Agreed, but that being said, this is why I play on WBaduk or Tygem now. With my winning and losing streaks, a reasonable rating system should have had me falling down a rank and jumping up a rank. Winning 11 games in a row over a month, and losing 11 games in a month does not suggest that I have the appropriate rank. It suggests that I was playing at 5d during one month, and then at 3d at another month.

Tygem's system is elegant, I wish KGS's were more like that. KGS's system never rewards you or punishes you. Tygem tells you exactly what you have to do to jump up a rank, 2 ranks, or if you're in danger of falling down 1 or 2 ranks.

For every 20 games, it looks at your percentage and then decides whether to promote or demote.

KGS seems to be designed with the belief that somehow the last 180 games should be consistent with a single rating, which makes no sense to me.

_________________
Tactics yes, Tact no...

Unfortunately these two desires are in conflict. Human perception of what is reasonable with respect to how they should be ranked is not typically reliable and consistent (or at least, it's not reliable and consistent to the end of giving a good game. Many people will reliably consider themselves over-ranked or under-ranked). If your desire is to play games where you have a 50% chance of winning (i.e. properly handicapped), it is better to use a system you may not find intuitive. You can provide the fluidity users desire, however you must sacrifice some of the consistency and reliability of the rating system.

Remi Coulom actually looked at even-handicapped KGS games as sample data for evaluating rating systems. Over a period of two years the KGS rating system correctly predicted the game outcome more often than an Elo or Glicko system. A simple "Win X then promote, lose X then demote" would fare even worse in this respect (there was an algorithm that fared better than KGS, however it would be even more involved mathematically, and even less intuitive for the participant).

Of course all this falls outside the initial [2d?] issue, which an easily be resolved by playing a single rated game. None of the people affected by the [2d?] bug are actually involved in any ratings calculations (as they have no rated games in the previous 180 days)

I'm confused, if they were even-handicapped, how would any rating system fare better than 50%?

That tells me that KGS trusts its range inside of a rank, but it does not tell me that the players within that rank are less than a stone apart...

In fact, if anything, that tells me that there is more than a stone difference inside a single KGS rank, since otherwise, the results would be extremely uncertain.

_________________
Tactics yes, Tact no...

Some anecdotal evidence regarding Tygem...

I tend to prefer playing on Tygem, partly because the games tend to be more entertaining, partly because I'm not as invested in my rank, partly because of the more fluid rating system. HOWEVER, I must acknowledge, in line with recent comments from mef, that the quality of my opponents if very divergent. I'm rated 2d on Tygem (and 1k on KGS), but I often feel I am playing against players 5 stones weaker or stronger than me. I tend to dominate or be dominated. In contrast, on KGS, my games tend to be close, and I very rarely feel that I'm playing someone inappropriate.

Ratings have much finer precision than a single stone. A strong 1D (borderline promotion) vs. a weak 1D (borderline demotion) have essentially a full stone difference in strength between them, even though they play even (so the expected winrate is closer to 75-80% for the stronger player). As such, you would expect to be able to predict better than 50% of the games. Also worth noting -- the systems were trained with 5 years worth of KGS game data prior to the 2 year period analyzed -- so if a rating system did a better job of ranking these players, it would be able to pick out the games that were mismatches (because one player was ranked incorrectly, and the handicap isn't properly set by its own rating) and improve it's predictions even more so.


Edit/Addition: Also worth noting in this case to address your concern, the correct prediction rates were on the order of 54-57% , so it is still close to 50/50...it's just that some systems will correctly predict thousands more games/yr than others

I agree with shapenaji, the only bad thing about the KGS rating system is:
When I started to play on KGS winning 1 game was enough to go up a rank or more, this is considered normal until the rating system adjusted my "real" rank (without the ?).

But the more games I play the harder it becomes to change my rank.
In this month I reached 4k with a score of 18 wins and 12 losses (as 5k) with a winning streak of 11 wins (after the 7th win I was promoted), this sill seems normal because I played 20 games as 5k and won 60% of them.
But as 4k it is a bit confusing, my 4k game count is 9 wins and 2 losses that is about 80% after 11 games, after looking at my rating graph I need about 4 more wins to get promoted - if I can miraculously win the next 4 games without losing (~85%). 1 loss more means 1 win more, and here is the problem - not counting to losses I need now 11 wins to get promoted while as 5k it only took me 6 wins. I am willing to play and win/lose until I reach 20 games but I don't think this helps.
If it is going to be like this, then it would make more sense to use the tygem system, or to set a new rank every 20 games and if the win % is between 45% and 55% you can keep the rank. Something like that make more sense. Also another bad thing about the KGS rating system is when you played about 30 games and then don't play for about 4 weeks your rating changes by 3-5 stones! This is something I experienced when I started to play go.

Also there was a certain KGS 3d (don't remeber his name) who was inulted by other 3d's to be a sandbagger, he by the way played about 1000 games in the last months and he already won about 20 games, no losses but in his rating graph it moved only 1/4 stones up. Remember he has a 20 games 100% win rate as 3d but he need to win 60 more games to get promoted! He could make a new account but te same thing will happen again.

Well I can understand if KGS wants to keep it's rating system but it should only take the last 50 or 100 games into the calculations and not 6 months, which could be 1000 games! But that is ust my personal opinion, I do hope it is getting some adjustments.

See Mef, I see a problem with the idea that over time you will be able to lock on to a person's rank. I think that's only true if a person's growth is stymied. There are a lot of people who have hit a barrier. Statistically speaking, probably 90%+ of games played are played while at a barrier of some kind or other. (Simply because you spend a good deal more time at them, that's why they're barriers)

KGS may have an extremely accurate rating system by tracking a user over a long period of time. But, to me, that's like saying "Well, it's rained the last 3 months here in Oregon, therefore, by induction, it will be raining in July"

That system will be:

A) Frustrating, since it will not adapt quickly when a person breaks past that barrier
B) Capable of impeding growth, since if a person DOES start to do better, and gets no new challenges, they can fall back into their old barriers. In this way, it ends up enforcing its own ranks.

There is also serious evidence of a problem when players realize that these are issues, and do what has now become the norm, when you get stronger, you make a new account.

No one wants to lift the load of 180 days of previous games, especially when they're advancing. Why should they need 20 or so games just to get to the next rank? Why should they quit playing on KGS for a while just so that they have less inertia on their account? Why enforce this when people bypass it anyway?

_________________
Tactics yes, Tact no...

The funny thing is that I have the opposite problem. I play few enough games online that I can move my KGS rank quickly, but there's no way I can move my Tygem rank much at all.

Bear in mind that there is a lot of wishful thinking going on. If you have a group of people who all mistakenly believe that they're stronger than they are and create new accounts, a substantial fraction of them will end up with a higher rank, at least for a good long while.

But all this has me wondering if it's possible to create a system that is somehow intermediate between counting all games from the past 180 days and one that only counts the most recent n games?

Imagine that over the last 180 days, I have played either 20 or 1000 games on KGS. In the first case, the system has no idea how much time I'm spending on go. In the later, it knows that I'm spending lots of time. So it should think the probability that my strength is going up is higher, without looking at the results of the games. Could it therefore count the more recent games more heavily in the later case? Has this ever been tested?

I know that 179 day old games already count less--what I'm proposing is that for two players, one of whom is playing more on the server, their older games count even less.

_________________
Occupy Babel!

I believe (however I am not certain) that WMS has stated that there is a halflife of 45 games.

This sounds like making the simple comlicated, why should any rating system care how much time I spend playing on Go? I could play one game each month but still study a lot, thus spending a lot of time on go.
It doesn't really give a reason for increasing strenght. But to be fair: good results like a winning streak does also not really prove of increasing strenght.

After a winning sreak, always comes a losing streak (sooner or later). But if I lose 10 games straight, then I want to get demoted! The current rating system on KGS gives you the option to simply "lock" your rank by playing many games. People who want to keep track of their growth through KGS will get stuck at some point. Shapenaji called it "the barrier".

I encouterd a few 2k's that played worse then most 5k's on KGS, I had handicap of 2-4 stones but still they felt weaker. These people play 10+ games per day, even if they get stronger or weaker their rank will nit change at all.

Sometimes I'm amazed to see someone being stuck on 6k for a year while I just past it in a few months without studying at all, the reason is simple: I play not as much as them! Their rank is simply "frozen".


I think I read somewhere that the last 6 months are used to calculate the rating, because if you do not play for 6 months you will lose your rank.

45 days, not 45 games, but otherwise yes, that is the case.

One difficulty is that game weights must be symmetric in the KGS rank system. That is, a win that pushes you up in rank must push your opponent down equally strongly. When I experimented with asymmetric weights I got weird little groups of people who would spiral crazily up or down. So if the game fades fast for you, it must also fade fast for your opponent; if you play a lot and they don't, then what do we do? If we choose an intermediate half life, then you would get really broken stuff, where some of your games would fade away faster than others for reasons that have nothing to do with what they say about your strength.

I was happy to see Remi's results. Even though the games were even, and thus within ±1 stone, this is by the KGS rank system. If the rank system had been badly broken, then more accurate systems would have come up with different ranks for players and predict winners more accurately. Elo and glicko are very well respected rank systems, and KGS outperforming them tells me that my rank system is as accurate as it needs to be.

This also indicates to me that people who complain about getting stuck are seeing patterns that are not statistically valid. In basketball there is a strong belief amoung a lot of people that a player gets "hot hands" where they are shooting baskets exceptionally accurately. A statistician examining this phenomenon and found it to be invalid - players who hit a series of baskets were no more likely to hit the next than normal, it's just that human brains always look for patterns in sequences of events, and sometimes end up finding them where they don't actually exist. Sometimes KGS players do genuinely improve faster than the rank system can accommodate, but more often when people win or lose a lot and are disappointed by how little their rank changes, they are seeing patterns that are statistically outweighed by other data.

On the other hand, maybe accuracy isn't so important in a rank system. People like to see their rank go up and down. Maybe building an intentionally inaccurate rank system that responds too quickly to streaks would lead people to enjoy their games more even though it made their rank less meaningful. Not something I have time to experiment with now, but it is an idea.

In my progress to 4d KGS, I've had ~4 accounts. Each got stuck somewhere, I started winning, and decided that progression by winning 67%+ of my games was going to take forever. I started a new account, and after ~10-15 games, I was back to 50%.

Yes, there are people who mistakenly believe they are stronger, but the fact is that there is a lot of inertia, and go is just not nearly as fun if you are not getting a challenge.

There's a lot of wishful thinking on Tygem/WBaduk too. Folks set their rank too high. But, just as it is easier to gain ranks, it is also easier to lose ranks. So people eventually gravitate to the right rank anyhow. There's just a bit more noise.



I don't know about this, this doesn't factor in the considerable amount of time a person may be spending studying or playing over the board, and would seem to unfairly punish those forms of training.

I will say that, as it stands now, if you play a lot of games, those will weigh you down for a long time. There's a 45 day half-life (Thanks speedy for that stat btw), BUT the distribution of games is not uniform.

It would seem to me that every day should be treated as a single data point, NOT every game.

Games are just measurements of how a person is playing at a particular time, if they play 20 games in a day or 5 games in a day, those are both measuring the same parameter. So, you should store a performance rating and a confidence interval for each day. (Those might even be interesting stats to publish for the stat-hounds out there)

Those performance ratings should be combined (and older ones given less weight) to give you a players total rating.

That would make more sense to me.

_________________
Tactics yes, Tact no...

And following that idea up,

Take the performance ratings, if the performance ratings spike (whether up or down) magnify their impact.

Basically, if a person is playing at or near their weighted average, consider that noise. If there is a spike, the signal-to-noise ratio should go way up. That should tell you that something has happened. You should be able to weight based on that parameter.

_________________
Tactics yes, Tact no...

I'm not sure what this even means. "Performance ratings"? Of the rank system or the players? And signal-to-noise is a communications term, I'm not sure what it means with respect to ratings or game plays.

Performance ratings of the players, A LA what they do for chess GM's during tournaments. They calculate a performance rating for that player based on their results against the other players during that tournament.

if a person rated 6d has the following results for a given time period (you could calculate the performance ratings on a per-day basis or a per-week basis, whatever)

7 wins-6 losses Performance Rating: 6.8d
6 wins-8 losses Performance Rating: 6.1d
5 wins-4 losses Performance Rating: 6.5d
7 wins-4 losses Performance Rating: 7.3d
8 wins-3 losses Performance Rating: 7.5d



Now, the signal-to-noise stuff comes out of looking at the pattern of the performance ratings

suppose I have 10 weeks of performance ratings that are well fit by a normal: mean = 6.5 sd = 0.4

Now suddenly the next data point changes the standard deviation dramatically. That spike in the standard deviation is a signal that something has changed.

This is different from, say, a sociological experiment. In this case, recent unusual results are more important than the average.

_________________
Tactics yes, Tact no...

I think you are searching to find an algorithm that will fit what you see as a pattern. As I said in my earlier message, I trust statistical calculations more than humans.

KGS can underrate quickly improving players. I have always stated this. Nick, I would not be surprised if you were one of the few based on what I know about your playing strength and how quickly you got there. But very few players improve quickly enough for that to bite them, and even fewer truly become weaker that quickly. If 99% of the players are well suited by the current algorithm, then I would not call it more accurate to make a change to better fit the few if it causes false changes in the rest; and calling a few wins or losses a trend and adjusting because of it will definitely cause false rank changes in most players.

@wms: Now that is something I understand!
But one question remain: is it true that it takes longer to rank up if you play more games, because I had the feeling that you deny that fact?
I know it is true that you rank up fast if you don't play many games.

Let's say you have played 400 games at 4d and won half of them (randomly spaced out). You then win 20 in a row. Shapenaji, do you want this to make you rank up to 5d? And if you lost 20 in a row go down to 3d? I think that would be wrong as all those old games let us know this spike was not significant and you are still probably a 4d who just had a nice winning streak. If, on the other hand, you had 20 games at 4d and had won half of them, and then won 20 in a row, your rank should go up as they are more significant. According to my understanding of the KGS rating system, this is how it works (I'm sure wms can correct me if I'm wrong).

I would agree though that KGS ranks can have rather a lot of inertia if you play loads of games (and there's drift), but I went from 30k all the way to 3d in 2 years on my single Uberdude account without feeling the rating system holding me back. I was typically playing a few games a day and had a high win rate (up to 85%, man I miss those days when improving was so easy ). In fact I did make new accounts for short periods, but that wasn't because of rating inertia but because my bad connection made me a chronic escaper.