Life In 19x19http://www.lifein19x19.com/ KGS Ranking adjustment?http://www.lifein19x19.com/viewtopic.php?f=24&t=15681 Page 4 of 7

 Author: Bill Spight [ Mon Jan 27, 2020 11:02 am ] Post subject: Re: KGS Ranking adjustment? jann wrote:Bill Spight wrote:Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contestThe percentage of correct answers is an exact, factual data (just like the percentage of various board scores). The percentage of wins (given those board scores) depends on an arbitrary parameter "komi".Well, komi is not arbitrary.

 Author: Bill Spight [ Mon Jan 27, 2020 11:09 am ] Post subject: Re: KGS Ranking adjustment? jlt wrote:@jann: normally, if a strong 1 dan plays against a weak 1 dan, then his winrate with komi 0.5 will be just a bit lower than his winrate with komi 7.5, so for the rating system to be fair, he should be awarded a little more points for a victory with komi 0.5 than for a victory with komi 7.5. How much is "a little more points" is not easy to determine, this has to be calculated using experimental data (see the link in the first post by gennan).In the NM system, the average difference between strong shodan and weak shodan is defined by a winrate of 50% with ½ pt. komi. We could even get more fine-grained and distinguish between levels according to exact komi. The problem with that, as I see it, is that skill at go is a vector; it is only average skill that is reducible to a number. Trying for too much precision is self-defeating.

 Author: jann [ Mon Jan 27, 2020 11:27 am ] Post subject: Re: KGS Ranking adjustment? jlt wrote:@jann: normally, if a strong 1 dan plays against a weak 1 dan, then his winrate with komi 0.5 will be just a bit lower than his winrate with komi 7.5, so for the rating system to be fair, he should be awarded a little more points for a victory with komi 0.5 than for a victory with komi 7.5Sure, but as I wrote above, this strongly depends on the assumption that result range -1 to -6 is not over- or under-represented. This assumption can be wrong (especially if the sample count is low), and these samples carry less info than others (can be seen as both wins or losses).And in any case this means the rating will carry more random variance, since normally only the players' performance varies, while for "h1"/even the rating will vary and depend on an external factor outside of the player's board performance.

 Author: Bill Spight [ Mon Jan 27, 2020 11:39 am ] Post subject: Re: KGS Ranking adjustment? jann wrote:And in any case this means the rating will carry more random variance, since normally only the players' performance varies, while for "h1"/even the rating will vary and depend on an external factor outside of the player's board performance. Such as whether the player takes Black or White at chess?

 Author: gennan [ Mon Jan 27, 2020 11:45 am ] Post subject: Re: KGS Ranking adjustment? jann wrote:Sure, but as I wrote above, this strongly depends on the assumption that result range -1 to -6 is not over- or under-represented. This assumption can be wrong (especially if the sample count is low), and these samples carry less info than others (can be seen as both wins or losses).And in any case this means the rating will carry more random variance, since normally only the players' performance varies, while for "h1"/even the rating will vary and depend on an external factor outside of the player's board performance.What kind of mechanism would cause a sharp probability drop between 6 points win on the board and 7 point win on the board? Without any special mechanism, the probability would drop smoothly. Is this some specially rigged version of go, like a game on a 2x3 board with area scoring and mirror plays forbidden?

 Author: jlt [ Mon Jan 27, 2020 12:04 pm ] Post subject: Re: KGS Ranking adjustment? We could imagine a player A, rated 1.75 dan, and a player B, rated 1.45 dan, so that on average, if A takes white, then A's score is (say) -3 points + komi on average, but we don't know anything about the standard deviation. If A's playing style is such that the standard deviation is very small, then A will lose too many games with komi 0 compared to what A's and B's Elo rating would predict. On the other hand, A also sometimes takes black with komi 0 against player C, rated 2.05 dan, and A wins too many games against C, so that the two effects hopefully cancel out.And if they don't, then A will be demoted to 1.49 dan at some point, and will win against B again, so it's not really a big deal.

 Author: jann [ Mon Jan 27, 2020 12:13 pm ] Post subject: Re: KGS Ranking adjustment? gennan wrote:What kind of mechanism would cause a sharp probability drop between 6 points win on the board and 7 point win on the board? Without any special mechanism, the probability would drop smoothly.Sure, in the long run (and as I wrote if the ratings are adjusted frequently there is no real problem).But suppose that you only allowed to observe 10 games of a new player, and need to make the best guess of his strength. With 10 even games I'm confident that I can make a decent guess regardless of the outcomes (even if he happens to lose or win all).But with 10 of such half-stone "H1" games, if most of them happen to be in the -1 - -6 range (not too hard to imagine), I cannot even guess if he is stronger or weaker than his opponents.

 Author: gennan [ Mon Jan 27, 2020 12:24 pm ] Post subject: Re: KGS Ranking adjustment? jlt wrote:We could imagine a player A, rated 1.75 dan, and a player B, rated 1.45 dan, so that on average, if A takes white, then A's score is (say) -3 points + komi on average, but we don't know anything about the standard deviation. If A's playing style is such that the standard deviation is very small, then A will lose too many games with komi 0 compared to what A's and B's Elo rating would predict. On the other hand, A also sometimes takes black with komi 0 against player C, rated 2.05 dan, and A wins too many games against C, so that the two effects hopefully cancel out.And if they don't, then A will be demoted to 1.49 dan at some point, and will win against B again, so it's not really a big deal.I estimate an average human 1d player would lose about 150 points over a whole game (compared to perfect play). You are hypothesizing what would happen when the probability distribution of his total error has a very sharp peak, so that the standard deviation is as small as 1 point. My question is: Why? I'm pretty sure that such players don't exist in real life (unless you build an AI that has exactly this behaviour), so I fail to see the point of this scenario. IMO it has little to do with real world statistics.

 Author: jlt [ Mon Jan 27, 2020 12:25 pm ] Post subject: Re: KGS Ranking adjustment? @gennan: I agree with you, I was trying to interpret jann's messages.

 Author: gennan [ Mon Jan 27, 2020 12:28 pm ] Post subject: Re: KGS Ranking adjustment? jlt wrote:@gennan: I agree with you, I was trying to interpret jann's messages.Oh, I'm sorry. I missed that it was you instead of jann.

 Author: gennan [ Mon Jan 27, 2020 12:44 pm ] Post subject: Re: KGS Ranking adjustment? jann wrote:gennan wrote:What kind of mechanism would cause a sharp probability drop between 6 points win on the board and 7 point win on the board? Without any special mechanism, the probability would drop smoothly.Sure, in the long run (and as I wrote if the ratings are adjusted frequently there is no real problem).But suppose that you only allowed to observe 10 games of a new player, and need to make the best guess of his strength. With 10 even games I'm confident that I can make a decent guess regardless of the outcomes (even if he happens to lose or win all).But with 10 of such half-stone "H1" games, if most of them happen to be in the -1 - -6 range (not too hard to imagine), I cannot even guess if he is stronger or weaker than his opponents.Well yes, 10 games is not a lot of data if you only look at the win/loss ratio. And when the winrate is close to 100% or 0%, the information content of the data is even lower. You could extract more data by analyzing the quality of play in those 10 games. Then you have much more data (something like 1200 moves to analyze). I suppose an AI could do that for you. For example, you could extract an error probability distribution of his moves using KataGo and compare it with error probability distributions of typical players of known ranks.

 Author: jann [ Mon Jan 27, 2020 12:53 pm ] Post subject: Re: KGS Ranking adjustment? gennan wrote:Well yes, 10 games is not a lot of data if you only look at the win/loss ratio. And when the winrate is close to 100% or 0%, the information content of the data is even lower.My point was the relative value: 10 even games are a decent amount of data even for 0% or 100%, but 10 "H1" games can be much less informative.

 Author: Javaness2 [ Mon Jan 27, 2020 1:05 pm ] Post subject: Re: KGS Ranking adjustment? In the EGF system, komi can be 1 point for an even game. That is entirely valid.

 Author: Bill Spight [ Mon Jan 27, 2020 1:30 pm ] Post subject: Re: KGS Ranking adjustment? jann wrote:But suppose that you only allowed to observe 10 games of a new player, and need to make the best guess of his strength. With 10 even games I'm confident that I can make a decent guess regardless of the outcomes (even if he happens to lose or win all).But with 10 of such half-stone "H1" games, if most of them happen to be in the -1 - -6 range (not too hard to imagine), I cannot even guess if he is stronger or weaker than his opponents.And your point is?