It is currently Wed May 12, 2021 8:35 am

 All times are UTC - 8 hours [ DST ]

 Page 4 of 7 [ 140 posts ] Go to page Previous  1, 2, 3, 4, 5, 6, 7  Next
 Print view Previous topic | Next topic
Author Message
 Post subject: Re: KGS Ranking adjustment? #61 Posted: Mon Jan 27, 2020 11:02 am
 Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
jann wrote:
Bill Spight wrote:
Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest

The percentage of correct answers is an exact, factual data (just like the percentage of various board scores). The percentage of wins (given those board scores) depends on an arbitrary parameter "komi".

Well, komi is not arbitrary.

_________________
At some point, doesn't thinking have to go on?

Visualize whirled peas.

Everything with love. Stay safe.

Top

 Post subject: Re: KGS Ranking adjustment? #62 Posted: Mon Jan 27, 2020 11:09 am
 Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
jlt wrote:
@jann: normally, if a strong 1 dan plays against a weak 1 dan, then his winrate with komi 0.5 will be just a bit lower than his winrate with komi 7.5, so for the rating system to be fair, he should be awarded a little more points for a victory with komi 0.5 than for a victory with komi 7.5. How much is "a little more points" is not easy to determine, this has to be calculated using experimental data (see the link in the first post by gennan).

In the NM system, the average difference between strong shodan and weak shodan is defined by a winrate of 50% with ½ pt. komi. We could even get more fine-grained and distinguish between levels according to exact komi. The problem with that, as I see it, is that skill at go is a vector; it is only average skill that is reducible to a number. Trying for too much precision is self-defeating.

_________________
At some point, doesn't thinking have to go on?

Visualize whirled peas.

Everything with love. Stay safe.

Top

 Post subject: Re: KGS Ranking adjustment? #63 Posted: Mon Jan 27, 2020 11:27 am
 Lives with ko

Posts: 167
Liked others: 0
Was liked: 25
jlt wrote:
@jann: normally, if a strong 1 dan plays against a weak 1 dan, then his winrate with komi 0.5 will be just a bit lower than his winrate with komi 7.5, so for the rating system to be fair, he should be awarded a little more points for a victory with komi 0.5 than for a victory with komi 7.5

Sure, but as I wrote above, this strongly depends on the assumption that result range -1 to -6 is not over- or under-represented. This assumption can be wrong (especially if the sample count is low), and these samples carry less info than others (can be seen as both wins or losses).

And in any case this means the rating will carry more random variance, since normally only the players' performance varies, while for "h1"/even the rating will vary and depend on an external factor outside of the player's board performance.

Top

 Post subject: Re: KGS Ranking adjustment? #64 Posted: Mon Jan 27, 2020 11:39 am
 Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
jann wrote:
And in any case this means the rating will carry more random variance, since normally only the players' performance varies, while for "h1"/even the rating will vary and depend on an external factor outside of the player's board performance.

Such as whether the player takes Black or White at chess?

_________________
At some point, doesn't thinking have to go on?

Visualize whirled peas.

Everything with love. Stay safe.

Top

 Post subject: Re: KGS Ranking adjustment? #65 Posted: Mon Jan 27, 2020 11:45 am
 Lives in gote

Posts: 410
Location: Netherlands
Liked others: 210
Was liked: 120
Rank: EGF 3d
Universal go server handle: gennan
jann wrote:
Sure, but as I wrote above, this strongly depends on the assumption that result range -1 to -6 is not over- or under-represented. This assumption can be wrong (especially if the sample count is low), and these samples carry less info than others (can be seen as both wins or losses).

And in any case this means the rating will carry more random variance, since normally only the players' performance varies, while for "h1"/even the rating will vary and depend on an external factor outside of the player's board performance.

What kind of mechanism would cause a sharp probability drop between 6 points win on the board and 7 point win on the board? Without any special mechanism, the probability would drop smoothly. Is this some specially rigged version of go, like a game on a 2x3 board with area scoring and mirror plays forbidden?

Top

 Post subject: Re: KGS Ranking adjustment? #66 Posted: Mon Jan 27, 2020 12:04 pm
 Lives in sente

Posts: 1237
Liked others: 98
Was liked: 403
We could imagine a player A, rated 1.75 dan, and a player B, rated 1.45 dan, so that on average, if A takes white, then A's score is (say) -3 points + komi on average, but we don't know anything about the standard deviation. If A's playing style is such that the standard deviation is very small, then A will lose too many games with komi 0 compared to what A's and B's Elo rating would predict. On the other hand, A also sometimes takes black with komi 0 against player C, rated 2.05 dan, and A wins too many games against C, so that the two effects hopefully cancel out.

And if they don't, then A will be demoted to 1.49 dan at some point, and will win against B again, so it's not really a big deal.

Top

 Post subject: Re: KGS Ranking adjustment? #67 Posted: Mon Jan 27, 2020 12:13 pm
 Lives with ko

Posts: 167
Liked others: 0
Was liked: 25
gennan wrote:
What kind of mechanism would cause a sharp probability drop between 6 points win on the board and 7 point win on the board? Without any special mechanism, the probability would drop smoothly.

Sure, in the long run (and as I wrote if the ratings are adjusted frequently there is no real problem).

But suppose that you only allowed to observe 10 games of a new player, and need to make the best guess of his strength. With 10 even games I'm confident that I can make a decent guess regardless of the outcomes (even if he happens to lose or win all).

But with 10 of such half-stone "H1" games, if most of them happen to be in the -1 - -6 range (not too hard to imagine), I cannot even guess if he is stronger or weaker than his opponents.

Top

 Post subject: Re: KGS Ranking adjustment? #68 Posted: Mon Jan 27, 2020 12:24 pm
 Lives in gote

Posts: 410
Location: Netherlands
Liked others: 210
Was liked: 120
Rank: EGF 3d
Universal go server handle: gennan
jlt wrote:
We could imagine a player A, rated 1.75 dan, and a player B, rated 1.45 dan, so that on average, if A takes white, then A's score is (say) -3 points + komi on average, but we don't know anything about the standard deviation. If A's playing style is such that the standard deviation is very small, then A will lose too many games with komi 0 compared to what A's and B's Elo rating would predict. On the other hand, A also sometimes takes black with komi 0 against player C, rated 2.05 dan, and A wins too many games against C, so that the two effects hopefully cancel out.

And if they don't, then A will be demoted to 1.49 dan at some point, and will win against B again, so it's not really a big deal.

I estimate an average human 1d player would lose about 150 points over a whole game (compared to perfect play). You are hypothesizing what would happen when the probability distribution of his total error has a very sharp peak, so that the standard deviation is as small as 1 point. My question is: Why?
I'm pretty sure that such players don't exist in real life (unless you build an AI that has exactly this behaviour), so I fail to see the point of this scenario. IMO it has little to do with real world statistics.

Last edited by gennan on Mon Jan 27, 2020 12:25 pm, edited 1 time in total.
Top

 Post subject: Re: KGS Ranking adjustment? #69 Posted: Mon Jan 27, 2020 12:25 pm
 Lives in sente

Posts: 1237
Liked others: 98
Was liked: 403
@gennan: I agree with you, I was trying to interpret jann's messages.

Top

 Post subject: Re: KGS Ranking adjustment? #70 Posted: Mon Jan 27, 2020 12:28 pm
 Lives in gote

Posts: 410
Location: Netherlands
Liked others: 210
Was liked: 120
Rank: EGF 3d
Universal go server handle: gennan
jlt wrote:
@gennan: I agree with you, I was trying to interpret jann's messages.

Oh, I'm sorry. I missed that it was you instead of jann.

Top

 Post subject: Re: KGS Ranking adjustment? #71 Posted: Mon Jan 27, 2020 12:44 pm
 Lives in gote

Posts: 410
Location: Netherlands
Liked others: 210
Was liked: 120
Rank: EGF 3d
Universal go server handle: gennan
jann wrote:
gennan wrote:
What kind of mechanism would cause a sharp probability drop between 6 points win on the board and 7 point win on the board? Without any special mechanism, the probability would drop smoothly.

Sure, in the long run (and as I wrote if the ratings are adjusted frequently there is no real problem).

But suppose that you only allowed to observe 10 games of a new player, and need to make the best guess of his strength. With 10 even games I'm confident that I can make a decent guess regardless of the outcomes (even if he happens to lose or win all).

But with 10 of such half-stone "H1" games, if most of them happen to be in the -1 - -6 range (not too hard to imagine), I cannot even guess if he is stronger or weaker than his opponents.

Well yes, 10 games is not a lot of data if you only look at the win/loss ratio. And when the winrate is close to 100% or 0%, the information content of the data is even lower.

You could extract more data by analyzing the quality of play in those 10 games. Then you have much more data (something like 1200 moves to analyze). I suppose an AI could do that for you. For example, you could extract an error probability distribution of his moves using KataGo and compare it with error probability distributions of typical players of known ranks.

Top

 Post subject: Re: KGS Ranking adjustment? #72 Posted: Mon Jan 27, 2020 12:53 pm
 Lives with ko

Posts: 167
Liked others: 0
Was liked: 25
gennan wrote:
Well yes, 10 games is not a lot of data if you only look at the win/loss ratio. And when the winrate is close to 100% or 0%, the information content of the data is even lower.

My point was the relative value: 10 even games are a decent amount of data even for 0% or 100%, but 10 "H1" games can be much less informative.

Top

 Post subject: Re: KGS Ranking adjustment? #73 Posted: Mon Jan 27, 2020 1:05 pm
 Gosei

Posts: 1361
Liked others: 110
Was liked: 283
In the EGF system, komi can be 1 point for an even game. That is entirely valid.

_________________

Top

 Post subject: Re: KGS Ranking adjustment? #74 Posted: Mon Jan 27, 2020 1:30 pm
 Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
jann wrote:
But suppose that you only allowed to observe 10 games of a new player, and need to make the best guess of his strength. With 10 even games I'm confident that I can make a decent guess regardless of the outcomes (even if he happens to lose or win all).

But with 10 of such half-stone "H1" games, if most of them happen to be in the -1 - -6 range (not too hard to imagine), I cannot even guess if he is stronger or weaker than his opponents.

_________________
At some point, doesn't thinking have to go on?

Visualize whirled peas.

Everything with love. Stay safe.

Top

 Post subject: Re: KGS Ranking adjustment? #75 Posted: Mon Jan 27, 2020 2:15 pm
 Lives in gote

Posts: 410
Location: Netherlands
Liked others: 210
Was liked: 120
Rank: EGF 3d
Universal go server handle: gennan

Handicaps provide fairly large increments of advantage/disadvantage. He worries that with small sample sizes and/or players with very small standard deviations, statistical anomalies could hide in those large increments.

My suggestion to handle this issue: If you worry about this, you could use komi in addition to handicap stones. Then you can make the handicap increments as fine-grained as half-point increments.

Use this as match conditions: If one player loses a game, you change the komi by 0.5 points in his advantage. If the komi goes below 0 or goes up to 14 points, you add or remove a handicap stone. If it's a jigo, you keep the same handicap.

Over the course of a match, the handicap + komi should gravitate to some specific value which would give a pretty good indication of the rank gap between these players.

Top

 Post subject: Re: KGS Ranking adjustment? #76 Posted: Mon Jan 27, 2020 2:48 pm
 Lives with ko

Posts: 167
Liked others: 0
Was liked: 25
If it's still unclear what I pointed at I doubt I could make it clearer. The above example of what could you tell after 10 komi / 10 no komi games looks like my best attempt summarizing it.

I see no problem with actual handi (H2+) games because in practice those are almost always played without komi. I also don't see a problem if there are a huge amount of games with frequently changing handicap / updated rating.

But without extra stones - and with the everyday server practice of offering a free choice (ranked differently OC, but that only helps in the long run) between the auto / nominal handicap or an even game with normal komi - there is a peculiar trait of those "h1" games: about 1 out of 3 games have no clear winner or loser (the recorded winner depends on the players' mentioned choice, which is arbitrary).

Top

 Post subject: Re: KGS Ranking adjustment? #77 Posted: Mon Jan 27, 2020 4:53 pm
 Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
jann wrote:
Bill Spight wrote:
Given a method of evaluation that has a probabilistic semantics, such as the percentage of correct answers on a test, or percentage of wins in a contest

The percentage of correct answers is an exact, factual data (just like the percentage of various board scores). The percentage of wins (given those board scores) depends on an arbitrary parameter "komi".

Here is what I meant. Any percentage other than 0% or 100% indicates variability in the correctness of answers to questions or of wins and losses.

_________________
At some point, doesn't thinking have to go on?

Visualize whirled peas.

Everything with love. Stay safe.

Top

 Post subject: Re: KGS Ranking adjustment? #78 Posted: Mon Jan 27, 2020 5:04 pm
 Honinbo

Posts: 10722
Liked others: 3612
Was liked: 3351
jann wrote:
If it's still unclear what I pointed at I doubt I could make it clearer. The above example of what could you tell after 10 komi / 10 no komi games looks like my best attempt summarizing it.

I see no problem with actual handi (H2+) games because in practice those are almost always played without komi. I also don't see a problem if there are a huge amount of games with frequently changing handicap / updated rating.

Well, it seems that you somehow think that komi is arbitrary.

Aside from that, you seem to be arguing something similar to the following:

There are two chess players who are close in strength. In ten games in which one player always took Black and the other player always took White. All ten games were draws. Who can tell which player is stronger? There is something wrong with having one player take Black and the other take White.

(The first player has the advantage in both chess and go. Go without komi is what you call H1. There is no komi in chess.)

I do not mean to parody you, but chess without odds seems to me to be analogous to what you are calling H1 without komi.

_________________
At some point, doesn't thinking have to go on?

Visualize whirled peas.

Everything with love. Stay safe.

Top

 Post subject: Re: KGS Ranking adjustment? #79 Posted: Mon Jan 27, 2020 6:01 pm
 Lives with ko

Posts: 167
Liked others: 0
Was liked: 25
Bill Spight wrote:
Well, it seems that you somehow think that komi is arbitrary.

For half stone rank diffs it is the players' arbitrary choice, both 0.5 and whole komi is very common. Lack of this choice is also why the chess analogy seem to fail.

Quote:
There are two chess players who are close in strength. In ten games in which one player always took Black and the other player always took White. All ten games were draws. Who can tell which player is stronger?

For 10 draws there is only one interpretation: B performed slightly above W. OC this is nowhere decisive (just 10 games afterall) but more than nothing. B being stronger is a bit more likely than the opposite.

This is not the same as 10 no-komi games all B+3. In that case there are two interpretations: W performed better (winning all with komi, instead of the expected 6 or 5) or worse (losing all without komi, instead of the expected 5 or 4). Harder to say who performed better (W OC, but only if you allowed to look at the scores, not just win%).

 This post by jann was liked by: Bill Spight
Top

 Post subject: Re: KGS Ranking adjustment? #80 Posted: Mon Jan 27, 2020 8:51 pm
 Lives in gote

Posts: 430
Liked others: 173
Was liked: 224
Rank: Australian 2 dan
GD Posts: 200
jann wrote:
This is not the same as 10 no-komi games all B+3. In that case there are two interpretations: W performed better (winning all with komi, instead of the expected 6 or 5) or worse (losing all without komi, instead of the expected 5 or 4). Harder to say who performed better (W OC, but only if you allowed to look at the scores, not just win%).

In terms of calculating ratings, there is no problem here. The only problem is using the ambiguous word "better". For this scenario, black performed above expectations, therefore black's rating needs to be adjusted upwards. Whether or not white is a stronger player than black is impossible to say from this data. It's possible that black is several stones stronger than white but chose to play slack moves (or safe moves) when ahead.

----

But there's a more fundamental issue here. We're getting hung up over a very implausible scenario. These two hypothetical players who can play a ten-game series and get exactly the same score (plus or minus one point) every time:
• First, Larry and Moe here are also playing games against other people, and those games also feed into the rating calculation, right? (Or if they're not playing anyone else, then does their rating calculation actually matter at all?)
• But even more important, this scenario is a one-in-a-thousand freak event, not business as usual. Humans just aren't that consistent. Even strong AI isn't that consistent yet. Look at the LZ promotion matches or the CGOS game results. Look at the many "golden era" challenge matches played without komi. Below pro level, if you can play me without komi ten times in a row and win every single time, then you're strong enough that you should be giving me at least three stones. (Or else this was a freakish run of games, and our next ten-game series will look very different). And if you're winning by exactly two or three points every time, you're choosing to play in a non-greedy style and you're very good at counting.
Of all the ways in which a rating system could go wrong, this is one of the last things we need to worry about.

 This post by xela was liked by 2 people: Bill Spight, gennan
Top

 Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending
 Page 4 of 7 [ 140 posts ] Go to page Previous  1, 2, 3, 4, 5, 6, 7  Next

 All times are UTC - 8 hours [ DST ]

Who is online

Users browsing this forum: No registered users and 1 guest

 You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot post attachments in this forum

Search for:
 Jump to:  Select a forum ------------------ Life In 19x19.com General Topics    Introductions and Guidelines    Off Topic    Announcements    General Go Chat    Beginners    Amateurs    Professionals       Lee Sedol vs Gu Li    Go Rules    Forum/Site Suggestions and Bugs    Creative writing    Tournaments       Ride share to tournaments Improve Your Game    Game Analysis    Study Group    Teachers/Club Leaders       Teacher advertisements    Study Journals L19²GO (Malkovich)    1-on-1 Malkovich games    Big Brother Malkovich games    Rengo Games    Other versions of turn-based games Go Gear    Go Books    Go Book Reviews    Computer Go    Gobans and other equipment    Trading Post    New Products/Upgrades/Sales Go Club Forums    Go Club Discussions       Honinbo Go League    American Go Association Forum       Go Congress 2011 volunteers       AGA volunteers ( non-congress)    Australian Go Association    European Go Federation Forum    Singapore Weiqi Association    KGS    ASR League    IGS    OGS    Tygem    WBaduk    Turn Based Servers    Insei League Events    Kaya.gs       King of the Hill