It is currently Mon Mar 18, 2024 10:51 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 143 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 8  Next
Author Message
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #41 Posted: Sun Jan 26, 2020 2:23 pm 
Lives in sente
User avatar

Posts: 914
Liked others: 391
Was liked: 161
Rank: German 2 dan
Just because your system spews out numbers to four significant digits doesn't mean that these numbers have the meaning you ascribe to them.

Specifically, it doesn't even mean that these numbers are internally consistent, not even in the long run (which we don't have, due to lack of data volume). It also doesn't consistently mean that if the number goes up/down, the strength goes up/down.

_________________
A good system naturally covers all corner cases without further effort.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #42 Posted: Sun Jan 26, 2020 2:32 pm 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
dfan wrote:
I have similar opinions about the AGA system, largely based on running the WHR rating algorithm on American Yunguseng Dojang games (there are over 5000 serious even games by now) and seeing how the resulting ratings compare with AGA ones.

There are a few problems that I believe I observe about the rating system. One is that the emphasis on handicap games means that it's hard for the system to be incredibly surprised at any particular result. In a chess tournament, you might be (the equivalent of) 10k and beat a couple of 5ks, and rocket up. If those are 5-stone handicap games, the fact that you won a couple of coin-flips isn't so impressive. Add to this a large amount of inertia in the rating system (and apparently the fact that it imposes unreasonable demands on winrate) and you get a relatively stagnant rating pool.

There is thankfully a pressure relief valve in the form of a rule that if you enter a tournament at least three ranks up from your established rating, and win at least one game, your rating resets entirely. The bad news is that this seems to be by far the most efficient way to get even a modest increase in skill recognized. If you are rated 4k and think you are 3k, I believe that the best way to achieve it is not by regular play but by entering tournaments at 1k until you win one game. That doesn't feel like a healthy system.

I didn't know the AGA rating system works like that, but it doesn't suprise me that such measures are needed to overcome what I would call flaws of the system. The EGF system has different measures, but it's basically a similar fix for a similar problem.
But these arbitrary measures are really needed to keep a reasonable alignment between the rating system and reality. Abolishing measures like these would break the rating systems.

I suppose some go players are aware of this, but improving the rating systems is hard. And by that I don't mean the technical part. Go players are humans and humans tend to prefer a status quo.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #43 Posted: Sun Jan 26, 2020 2:59 pm 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Bill Spight wrote:
It goes against the grain, I suppose, but I have long believed that ranking in terms of handicap stones, despite the fact that skill in giving or taking handicap stones is variable, it better than ELO ratings or score differences in differentiating skill levels. The main reason for my belief is that, over a large range of skill levels, handicap stones have a roughly linear relationship. E. g., if player A is three ranks stronger than player B, that means that a handicap of 3 stones with White giving komi normally gives each player around a 50-50 chance of winning the game. If player B is four ranks stronger than player C, then if A gives C a handicap of 7 stones, White giving komi, that will also normally give each player around a 50-50 chance of winning. Elo ratings, OTOH, will tell us that A will crush C in an even game. :lol:

I think go ranks have no meaning without handicap. Handicap defines ranks. If go had no handicap system and all games would be even games, we would just use an Elo rating system.

But the go rating systems all try to align winrates to handicaps. Tournament games in Europe tend to be even games, so you don't really need ranks for that, but it's nicer to have opponents of your own level in even games. Go players tend to have ranks in their clubs bases on handicap, so you want to use those club ranks in tournaments to put players in evenly skilled McMahon groups. It's also reasonable to update ratings based on tournament game results.

This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run. I do think the go rating systems are reasonably effective in this respect, but there is clearly room for improvement.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #44 Posted: Sun Jan 26, 2020 3:39 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
gennan wrote:
I think go ranks have no meaning without handicap. Handicap defines ranks. If go had no handicap system and all games would be even games, we would just use an Elo rating system.

But the go rating systems all try to align winrates to handicaps.
{snip}

This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run. I do think the go rating systems are reasonably effective in this respect, but there is clearly room for improvement.


Back in the 1970s I devised a rating system for the New Mexico Go Association. The AGA had already devised an Elo system, but I took the handicap basis seriously and did not do so. OC, there was no theory at the time about how to base a rating system on handicaps (Is there one now? ;)), so I drove by the seat of my pants. :) I divided ranks into two, so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi, etc. Instead of complicating the calculation of ratings, I had each game count the same number of points, and increased the point range of each successive rank by 5%, going upwards. Since most of our players were in their 20s I added inflation points to the winners' scores. After three years, based upon the results of visiting players and of our players playing elsewhere, I promoted everybody by ½ rank and added more inflation points. Two years later I had no reason to change the system.

OC, such a system faces the problem of players who make very rapid progress at the SDK level and above. Other systems do, as well, but the very short ranges of ranks in the DDK range do not present much of an obstacle. :) Human intervention may be required. I might have done so for Janice Kim, but after her summer as a go student in Korea, she was cautioned not to play with amateurs. So that problem solved itself. ;)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #45 Posted: Sun Jan 26, 2020 6:36 pm 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
gennan wrote:
This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run.

Bill Spight wrote:
so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.

OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #46 Posted: Sun Jan 26, 2020 7:29 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
jann wrote:
gennan wrote:
This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run.

Bill Spight wrote:
so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.


Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

Quote:
OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.


Why would the handicap not change during the run in the alternate universe, as well?

IOW, what is the problem?

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #47 Posted: Sun Jan 26, 2020 11:55 pm 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
I just realised that I misread something on https://wismuth.com/elo/calculator.html#elo_diff=250. The 50 Elo in the table at the bottom of the page was for IGS (PandaNet), not KGS. The winrates/Elo values for KGS are similar to EGF around 3k and 5d according to that table. Sorry about that.


Last edited by gennan on Mon Jan 27, 2020 1:08 am, edited 2 times in total.
Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #48 Posted: Mon Jan 27, 2020 12:00 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
jann wrote:
gennan wrote:
This puts a responsibility on go rating systems to convert well between ratings and ranks and to keep this conversion consistent in the long run.

Bill Spight wrote:
so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi

There is another problem with "handicap" in this case (faux H1). Suppose a player plays 100 games like this and loses all by 3-4 pts (no komi). This would correspond to winning all by 3-4 pts (with komi). I doubt there is a completely correct handling of such results.

OC a typical system calls this 0-100 and drops his ratings. But in another universe he decides to reject "handi" games and play even (same opponents, same move sequences). There these (the same performance) are called 100-0 and huge rating gain.


I think results like that are highly unlikely. There is allways statistical variation / ramdomness in playing quality. That can be expressed as a standard deviation of ranks. I estimate the standard deviation around shodan is roughly 1 rank. So a strong shodan playing a 100 game match against a weak shodan on josen will almost never end 100-0.

A simplified model for this would be to have a coin flipping match where the coin is fair or slightly biased, so that the heads/tails odds are 50/50 (match on josen = fair handicap) or 56/44 (match on even = odds favouring the strong shodan). It's possible to toss heads 100 times in a row with such coins, but it's highly unlikely. With the fair coin (50/50 odds), it will happen about one time in a million matches of 100 games. Playing 100,000,000 games of go would take about a 100,000 years when they play 3 games a day and take a day off on Sundays, so both players will probably have died from old age before it happens. With the biased coin (56/44 odds), it will happen more often, but still I think it's not going to happen within their lifetimes.

These are all back-of-the-envelope calculations (in fact, I didn't even use that) and go matches may not be exactly the same as a coin flipping matches, but it should be a decent first approximation for some ballpark estimations.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #49 Posted: Mon Jan 27, 2020 1:19 am 
Gosei
User avatar

Posts: 1747
Liked others: 176
Was liked: 488
Bill Spight wrote:
I divided ranks into two, so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi, etc. Instead of complicating the calculation of ratings, I had each game count the same number of points, and increased the point range of each successive rank by 5%, going upwards. Since most of our players were in their 20s I added inflation points to the winners' scores.


My guess is that any simple ranking system like that will work if most of the games are played with the proper handicap. But the question is how many points you should give to the winner and to loser in the case of an even game between players of different ranks.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #50 Posted: Mon Jan 27, 2020 1:55 am 
Gosei

Posts: 1492
Liked others: 111
Was liked: 314
I never really understood why the weight of handicap games wasn't reduced. Knocking off 10% of the normal contribution per handicap stone shouldn't be too radical an approach to acknowledging the additional uncertainty introduced by handicap go. KGS blocks anything greater than 6 stones :)

_________________
North Lecale

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #51 Posted: Mon Jan 27, 2020 2:00 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
jlt wrote:
Bill Spight wrote:
I divided ranks into two, so that a strong shodan would take White against a weak shodan, receiving ½ pt. komi, etc. Instead of complicating the calculation of ratings, I had each game count the same number of points, and increased the point range of each successive rank by 5%, going upwards. Since most of our players were in their 20s I added inflation points to the winners' scores.


My guess is that any simple ranking system like that will work if most of the games are played with the proper handicap. But the question is how many points you should give to the winner and to loser in the case of an even game between players of different ranks.


Actually, I did work that out for mismatches up to 3 ranks, after which it became ridiculous, based upon different plausible distributions. But nobody played any rating games with rating mismatches. ;) OC, that could be a problem if one player is known to be stronger than his or her rank. Why play a rated game against that player if, say, you could win only 5 pts. or lose 200 pts.?

Edit: I didn't mind, myself. For two reasons. 1) I didn't mind. 2) Because of the the greater point range for the upper dan levels, it was easy to absorb a larger ratings loss than usual without affecting the rank. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #52 Posted: Mon Jan 27, 2020 4:20 am 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
gennan wrote:
I think results like that are highly unlikely.

Sure, that is an overexaggerated example, but the underlying problem seems valid to me.

gennan wrote:
A simplified model for this would be to have a coin flipping match where the coin is fair or slightly biased, so that the heads/tails odds are 50/50 (match on josen = fair handicap) or 56/44 (match on even = odds favouring the strong shodan). It's possible to toss heads 100 times in a row with such coins

IMO the odds are not this low. Any board result in the -1 - -6 range would suffice.

Bill Spight wrote:
Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

This is just an extreme example, but the delay of promotion can be explained by the system updating only once a day, for example.

Quote:
IOW, what is the problem?

The problem is that the player's new rating (9k or 9d) will depend solely on whether he accepted "handi" or played even. And not on his strength or skill (games, opponents and move sequences being identical).

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #53 Posted: Mon Jan 27, 2020 4:31 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
jann wrote:
Bill Spight wrote:
Why do you think that the result you would get (100 losses) would be incorrect? (OC, the failure to promote or demote the players with such a streak is not correct. You seem to be assuming that.)

This is just an extreme example, but the delay of promotion can be explained by the system updating only once a day, for example.


One reason for making the calculation of the new rating so simple was that the players could do it themselves, after each game. :)

Quote:
Quote:
IOW, what is the problem?

The problem is that the player's new rating (9k or 9d) will depend solely on whether he accepted "handi" or played even. And not on his strength or skill (games, opponents and move sequences being identical).


As I mentioned above, I worked out tables for mismatches, but the players preferred to play rating games with the prescribed handicaps. Especially since it was always possible for them to know their ratings instantaneously. :)

There was no tradition of playing games with the wrong handicaps.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #54 Posted: Mon Jan 27, 2020 7:17 am 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
I didn't mean the example specific for that system, but as a general problem in interpreting the results of "h1" games.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #55 Posted: Mon Jan 27, 2020 7:46 am 
Gosei

Posts: 1589
Liked others: 886
Was liked: 527
Rank: AGA 3k Fox 3d
GD Posts: 61
KGS: dfan
Javaness2 wrote:
I never really understood why the weight of handicap games wasn't reduced. Knocking off 10% of the normal contribution per handicap stone shouldn't be too radical an approach to acknowledging the additional uncertainty introduced by handicap go. KGS blocks anything greater than 6 stones :)

The AGA system does treat larger handicap games as being less informative.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #56 Posted: Mon Jan 27, 2020 7:51 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
jann wrote:
I didn't mean the example specific for that system, but as a general problem in interpreting the results of "h1" games.


Excuse me, you've got me confused. First, what is the problem except for not updating often enough? Which is a problem for any system that does not do so. (The New Mexico system updated after every game. Something you did not know.) Second, what do handicaps have to do with anything? You can get the same thing with even games.

You have two players who are close in strength who play a long series of games in which the weaker player just barely loses every time, and the rating system does not update their ratings until afterwards. As a result, their ratings are widely apart and do not reflect their true difference in skill.

Hmmmm. Now suppose that their ratings are actually updated after every game. With even games their ratings still diverge quite a bit. With a handicap system, the handicap changes, which will change who wins. The weaker player will get a larger handicap and start winning. Then their ratings will start to return to their previous values. At some point they will be back to their original handicap, and their ratings will start to diverge again. And so on, and so on. The handicaps could flip back and forth a number of times during the run, but their ratings will not diverge widely, as will happen with only even games.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #57 Posted: Mon Jan 27, 2020 8:32 am 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
I think we talk about different things.

With higher handicaps the games themselves differ, so there is no real problem in blindly accepting the win/loss result as THE result, and update the ratings for that.

But with "H1" against the same opponents, all the games move by move can be completely the same whether the games were played as no-komi games or as even games (OC the player may play differently but not necessarily so, especially not in the ama dan range).

So if board results in the range of -1 and -6 are somehow over-represented (like the extreme example above), the assigned rating will mostly depend on something that has absolutely nothing to do with the player's strength (whether he accepted "h1" or they played "even"). This is not a simple math/rating problem but a more general one (and OC if you adjust after every game the problem is less significant).

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #58 Posted: Mon Jan 27, 2020 9:55 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
jann wrote:
This is not a simple math/rating problem but a more general one (and OC if you adjust after every game the problem is less significant).


OK, let me generalize. :) Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest, results with no variability will defeat the method.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


This post by Bill Spight was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #59 Posted: Mon Jan 27, 2020 10:16 am 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
Bill Spight wrote:
Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest

The percentage of correct answers is an exact, factual data (just like the percentage of various board scores). The percentage of wins (given those board scores) depends on an arbitrary parameter "komi".

Top
 Profile  
 
Offline
 Post subject: Re: KGS Ranking adjustment?
Post #60 Posted: Mon Jan 27, 2020 10:31 am 
Gosei
User avatar

Posts: 1747
Liked others: 176
Was liked: 488
@jann: normally, if a strong 1 dan plays against a weak 1 dan, then his winrate with komi 0.5 will be just a bit lower than his winrate with komi 7.5, so for the rating system to be fair, he should be awarded a little more points for a victory with komi 0.5 than for a victory with komi 7.5. How much is "a little more points" is not easy to determine, this has to be calculated using experimental data (see the link in the first post by gennan).

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 143 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 8  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group