Kaya Open-sources the rating system & updates

[Kaya.gs]

Hey Guys! Its been a while since i post , so i will give some updates after the announcement.

After reading many different suggestions for rating systems, different ideas and systems, after investigating the different theoretical models and example implementations, i concluded that the problem is rather large and it would be better to make a more informed decision on which system will fit Kaya the best.

Being open sourced means that anyone will be able to write their own rating system and add it to the public repository where anyone can check it out. The point of this is to have not only different systems, but also different implementations of each, and compare their accuracy with the same data set, and compare their performance.

For the first version of the server i will undoubtedly make some Elo-based system, which is easy to implement and with few users, it wont matter much which system is finally used. After that, we will compare the different solutions we/other people make , benchmarking them and testing their accuracy, and pick that system. Naturally that could gradually change over time, with tweaks and improvements, which will be very visible by the community.

The project is hosted here: https://github.com/conanbatt/OpenKaya

To contribute, you have to fork the project ,and once you get a version you would like to be commited, you make a Pull Request. I will look into it and if its good to go, merge it to the official repository.

The link above contains some documentation and a basic structure to deploy plug&play rating systems. There is a very simple example right now, which anyone can run locally on their computer.

Yoyoma made a WHR implementation in python, and id love to assist him into translating it into Ruby.

Along the rating system, there is also a score folder, which is for the Javascript algorithm to count territory given a board. A friend wanted to help us out and make that algorithm, so that is also public. Most likely, score estimation will have to be done from scratch in javascript too, so that algorithmic solution will probably be open source too.

Interested mathematicians-programmers are welcome to help around, your system might be the final system used by the server . If any developer needs aids understanding current code, github or anything else that blocks him on contributing in this case, you can contact me and i can help you around.


------------------------------------------------------------------------------------------------

All that said and done, id like to share some other news on Kaya from the past 2 weeks of development.

* We have set up a private blog only for supporters, where we are writing the latest news and advancements. IT currently has a snapshot of the actual *running* Kaya server, with chat and board included.

* We have now many new features complete on the server:
- there are go videos you can watch while chatting.
- A count-down timer for server events so you always know what is the next big thing.
- Chat commands like sharing links, sorting go comments, boldifying comments with your nickname, etc.
- Admin commands like announcements and booting
- A sandbox mode (similar to "Demonstrations" on KGS) where you can change the size of the board on the fly, even the url (you can name it as u want!)
- The basis of the karma system (you can +1 users!)
- and soon we are going to deploy the server on Amazon. Probably by the end of the week , founders will be able to log in and check out the current dev server.

Also in the past 2 weeks, the Quebec Go Association decided to get the Kaya.gs T-shirts. We added a new board just for them on the site.
Counting them, we have surpassed 100 supporters and over 4400 dollars!

Development is going full speed and on schedule. We have 15 "working days" to get it running remotely, and be able to play and more. We are currently missing the following things to get to Alpha:

- Deployment on Amazon (I estimate 5 days, only me working on that)
- Styling of the site. We have to contact the designer that did the prototype and come up with a cool whole page look&feel (I estimate 10 days with back&forth with the designer, mostly Polly working on that.)
- Scoring (? Mistery.)
- Rating system (1 or 2 days to make a simple Elo system and integrate it to the server safely and with tests)
- Private chat (optional, but it could end up being a piece of cake. Might be a 2 day thing.)

We are not far at all!

[uPWarrior]

Isn't an end-game scoring function enough?
I just had a look at score_test.js and you seem to be testing some pretty dubious situations such as test 1 or 2.

By end-game scoring I mean a function which proposes something in the "mark dead groups" phase. Scoring as it is requires perfect play; even pros have a certain degree of uncertainty in complicated fights. I think it should be made clearer what we request from this specific function, what is off-limits and what is not.

[mw42]

Have you plans to drop the .gs and Go Server from your name to avoid potential confusion with KGS?

Also,

We are not far at all!

Means you are not close to launching. Is this your intended meaning? The exclamation mark makes me think you might mean "we are not far away at all" which would mean you are close to launching.

EDIT: Need to find new words that mean "mean."

[xed_over]

Have you plans to drop the .gs and Go Server from your name to avoid potential confusion with KGS?

hasn't this already been discussed to death in the other thread?

the only confusion is from those who are looking for it.

[Kaya.gs]

Isn't an end-game scoring function enough?
I just had a look at score_test.js and you seem to be testing some pretty dubious situations such as test 1 or 2.

By end-game scoring I mean a function which proposes something in the "mark dead groups" phase. Scoring as it is requires perfect play; even pros have a certain degree of uncertainty in complicated fights. I think it should be made clearer what we request from this specific function, what is off-limits and what is not.

As the test shows, groups are already marked as dead, that decision has to be made by the players, and not a computer (regardless of how good computers can be at doing that).

Test cases 1 and 2 are not dubious at all, they just look edgy. The same scoring function that gives off results of typical games, should also give that result in those edge cases too (which is whole board neutral space and whole board territory of one color).

Besides counting sorrounded spaces, which is a rather straight forward algorithm, it has to count seki properly which is not so obvious.

[kivi]

Hi,
About the rating system: you say you will benchmark/compare different rating implementations. How will you do that? I had a look to the repository, couldn't see anything related to that. I don't mean to run a rating system with data, but to compare results of multiple rating systems. You mention "Our decision will be based mainly on the balance of Accuracy and Performance." I guess performance is computation time. But how do you define accuracy?

Another related thing, is the initial ratings. E.g., on KGS when you insert a new player there is no initial rating. But it is possible to add a player only once you have established players in the pool. Even when you have the pool already, and just adding one player there is the problem that same initial rating may not "feel" the same. If you add me to KGS and Tygem, both with initial rated as 3 dan, Tygem may have more work to do to stabilize me, as it has weaker ranks (3 dan KGS is about 5 dan Tygem).

[Kaya.gs]

Hi,
About the rating system: you say you will benchmark/compare different rating implementations. How will you do that? I had a look to the repository, couldn't see anything related to that. I don't mean to run a rating system with data, but to compare results of multiple rating systems. You mention "Our decision will be based mainly on the balance of Accuracy and Performance." I guess performance is computation time. But how do you define accuracy?

Another related thing, is the initial ratings. E.g., on KGS when you insert a new player there is no initial rating. But it is possible to add a player only once you have established players in the pool. Even when you have the pool already, and just adding one player there is the problem that same initial rating may not "feel" the same. If you add me to KGS and Tygem, both with initial rated as 3 dan, Tygem may have more work to do to stabilize me, as it has weaker ranks (3 dan KGS is about 5 dan Tygem).

Right now the current code benchmarks how long it takes for a rating system to process a set of data. It gets automatically written in the output file (a simple matter of showing how long it took to run).

Accuracy is measured in the following manner: given a large set of real games, you process a first part (say 90%) and then after you achieve a rating on the current players, you attempt to guess future results. You compare that with the remaining 10% of games, and thats how you get accuracy.
Current code right now doesn't make that comparison. I will get on that once i get a real data-set. Yoyoma had a few k's games at hand.

I dont get if the initial rating comment is a question or an affirmation.

[kivi]

Guess-future-results as accuracy test makes sense, thanks.

Let me turn the second part into a question. As a player I am curious: When a new user registers to Kaya, will he start with unknown rank (as in KGS), or will he enter his self-approximated level (as in many other servers)?

Also while benchmarking; will the rating system (that is being tested) be aware of the ratings of the players at the start. In other words, does the test data include any initial rating information (may it be self-assessed or, e.g., KGS given)?

[Kaya.gs]

Guess-future-results as accuracy test makes sense, thanks.

Let me turn the second part into a question. As a player I am curious: When a new user registers to Kaya, will he start with unknown rank (as in KGS), or will he enter his self-approximated level (as in many other servers)?

Also while benchmarking; will the rating system (that is being tested) be aware of the ratings of the players at the start. In other words, does the test data include any initial rating information (may it be self-assessed or, e.g., KGS given)?

That question is highly tied to the rating system being picked, so the answer is "depends on the rating system".

For the second part, it depends exclusively on the data-set.

Kivi, are you considering writing code? we can take the details to a private channel.

[SpongeBob]

Thanks for the update, it sounds very promising!

[yoyoma]

I am learning Ruby now, will convert my WHR implementation. Maybe throw in a Glicko implementation as well.

[uPWarrior]

Isn't an end-game scoring function enough?
I just had a look at score_test.js and you seem to be testing some pretty dubious situations such as test 1 or 2.

By end-game scoring I mean a function which proposes something in the "mark dead groups" phase. Scoring as it is requires perfect play; even pros have a certain degree of uncertainty in complicated fights. I think it should be made clearer what we request from this specific function, what is off-limits and what is not.

As the test shows, groups are already marked as dead, that decision has to be made by the players, and not a computer (regardless of how good computers can be at doing that).

Test cases 1 and 2 are not dubious at all, they just look edgy. The same scoring function that gives off results of typical games, should also give that result in those edge cases too (which is whole board neutral space and whole board territory of one color).

Besides counting sorrounded spaces, which is a rather straight forward algorithm, it has to count seki properly which is not so obvious.

You are right, I didn't look at the test cases carefully enough.
I'll implement it if I find free time. This is not a promise tough.

[Kaya.gs]

I am learning Ruby now, will convert my WHR implementation. Maybe throw in a Glicko implementation as well.

Thats awesome! if you need any help dont hesistate to ask. Whats missing from the code right now is the accuracy test for comparison.
Id love to do that after i get and parse a reasonable dataset. What games were you using for your WHR implementation?

the only thing that might confuse you from the code is the ->() notation, which is one of many ways to pass functions. Functions are not really first class citizens in ruby as in python.

You are right, I didn't look at the test cases carefully enough.
I'll implement it if I find free time. This is not a promise tough.

. Emiliano should have a first version this week actually, to do overall scoring. What he probably wont do is recognize those edge seki cases, and details with different rulesets.
Today i visited GoShrine and i found that its scoring system mis-counts the seki position. Polly couldnt find the algorithm they use to count in a few minutes, i didnt take a close look at it either.

[yoyoma]

I am learning Ruby now, will convert my WHR implementation. Maybe throw in a Glicko implementation as well.

Thats awesome! if you need any help dont hesistate to ask. Whats missing from the code right now is the accuracy test for comparison.
Id love to do that after i get and parse a reasonable dataset. What games were you using for your WHR implementation?

the only thing that might confuse you from the code is the ->() notation, which is one of many ways to pass functions. Functions are not really first class citizens in ruby as in python.

I have Glicko working. It is rough, for example I'm not sure how I should be doing my own local constants. Also I need the player class to have some more attributes, such as "RD" the rating deviation, and also a DateTime of the last game a player was in. I just added them to the system.rb file, maybe I should have added them from my own glicko.rb file.

Also I am having some trouble with git. I'm running Windows, and it's having trouble with the file ".rvmrc.09.25.2011-00:43:51". I would bet it's a compatibility issue with the filename being strange... Maybe you could just get rid of it if it's not necessary to help me workaround that? It looks like some sort of unneeded backup.

[xed_over]

Also I am having some trouble with git. I'm running Windows, and it's having trouble with the file ".rvmrc.09.25.2011-00:43:51". I would bet it's a compatibility issue with the filename being strange...

yeah, Windows doesn't allow ':' in filenames.