AI komi

[moha]

It is known that correct komi depends on the players' levels, with weaker players needing less komi for a fair game (since B makes less use of his advantage).

I didn't know that (do you have a source to satisfy my curiosity?). Do you also mean that a full handicap stone is worth about 13 points at 7d level, but perhaps only 6 or so at 30k level?

I don't have a definite source I'm afraid, though this came up a few times in the past (both here and elsewhere). Handicap stones are a good question, I'm not sure myself. I think the komi change is a bit smaller than your example (even for random players B has some advantage), and handicap stones after the first one are not completely linear. But for the first stone I suppose the komi*2 logic should hold.

Rank differences are measured by handicap stones. So when one would try to create a rating system separated by point gaps, the lower ranks would be significantly closer together than the higher ranks (somewhat similar as the true Elo gaps between lower ranks being much smaller than the true Elo gaps between higher ranks).

Yes, but since Elo (winrate) gaps are also/mostly affected by the different variance in players' performance at various levels, this would remain even if you would calibrate a system for 1 rank = 14 points (instead of 1 extra stone) for example.

[Bill Spight]

It is known that correct komi depends on the players' levels, with weaker players needing less komi for a fair game (since B makes less use of his advantage).

I didn't know that (do you have a source to satisfy my curiosity?). Do you also mean that a full handicap stone is worth about 13 points at 7d level, but perhaps only 6 or so at 30k level?

Here is an argument that may make sense, and it does not depend upon any particular model of the game. Suppose that all we know is that the first gote move gains 14 pts., that the next gote move gains at most 14 pts. and the last gote move gains nothing. Then the final score we expect to fall between 14 pts. and 0. (In real life we know better, OC. ) So our estimate of the final score that minimizes our maximum error is 7 pts. That, then, becomes our first guess for komi. (In real life we have a better estimate of komi than we have of how much the first gote gains, but bear with me.)

Now suppose that the players are so bad that they pick gainful plays randomly, but not so bad that they actually lose points. We estimate that the average play gains 7 pts. We do not know whether there will be an even number of plays or an odd number of plays. So half the time we estimate that the result with be 7 pts. for the first player (Black), and half the time the result will be 0. That gives us an estimate for the final score of 3½ pts., which is our guess for komi. This is one example that illustrates how komi should be smaller for bad players than for good players.

[ez4u]

...

Here is an argument that may make sense, and it does not depend upon any particular model of the game. Suppose that all we know is that the first gote move gains 14 pts., that the next gote move gains at most 14 pts. and the last gote move gains nothing. Then the final score we expect to fall between 14 pts. and 0. (In real life we know better, OC. ) So our estimate of the final score that minimizes our maximum error is 7 pts. That, then, becomes our first guess for komi. (In real life we have a better estimate of komi than we have of how much the first gote gains, but bear with me.)
...

I don't understand this at all. If in real life we know better, what does this strawman do other than create an artificial world where want you want to say, is true?

[Bill Spight]

...

Here is an argument that may make sense, and it does not depend upon any particular model of the game. Suppose that all we know is that the first gote move gains 14 pts., that the next gote move gains at most 14 pts. and the last gote move gains nothing. Then the final score we expect to fall between 14 pts. and 0. (In real life we know better, OC. ) So our estimate of the final score that minimizes our maximum error is 7 pts. That, then, becomes our first guess for komi. (In real life we have a better estimate of komi than we have of how much the first gote gains, but bear with me.)
...

I don't understand this at all. If in real life we know better, what does this strawman do other than create an artificial world where want you want to say, is true?

Well, it's not what I want to say. It's simple because it does not offer a model of errors, which may or may not be true. When I first heard of the idea that komi should be smaller for weaker players, back in the 1990s, I did not believe it. So I played around with errors, to see what would happen, and the idea makes a lot of sense. We could test it statistically, with kyu level data. I do not have that data.

Perhaps you would like to offer a reasonable model of go errors.

[jlt]

FWIW I looked at the last 94 lines of the 2018 EGC weekend tournament. Lomi was 6.5. Most of these players are DDK. I removed handicap games.

B-: 94 occurrences
W+: 80 occurrences
W-: 96 occurrences
B+: 80 occurrences

B- is not equal to W+ because some of these games were played against other players, but we can estimate that about 200 games were played. White won 49.7% of the time: ((W+)+(B-))/((W+)+(B-)+(W-)+(B+))=0.497 approximately.

For the last 86 lines of the 2019 EGC weekend tournament, White won 52% of the time.

For the last 100 lines of the 2019 EYGC-U12 tournament (komi 6.5, players mostly 13k or weaker, about 300 games), White won 49% of the time.

So I don't see any evidence that komi 6.5 is too large for weak players.

[ez4u]

My model of Go errors: Sh** Happens!

My "model" ("image" might be better) of the game:

• - Go is played by two opponents. Player Black plays first and then players White and Black alternate play. (append a copy of your favorite rules for the rest of the details)
  - There is an advantage to playing first. (We have a few centuries of experience that support this claim)
  - There are game-specific skills that allow player Black to better take advantage of playing first and equally allow player White to frustrate player Black's attempts to take advantage of playing first.

From this I expect that we do not see a lower komi as appropriate between less-skilled players. Rather what we expect is a greater variance in game results between less-skilled players.

In addition, it seems to me that all of our understanding of "skill" is performance based. What does it mean that komi would be lower for less-skilled players if we use komi/handicaps to define skill levels?

[ez4u]

Fortunately we may be able to test the question directly. We now have in our possession a formidable supply of games played between equal opponents across a range of strengths. I mean the archive of LZ's self-play games!

I downloaded the files all_1M.sgf - all_18M.sgf (18 files in all) from the archive at https://leela.online-go.com/zero/. These appear to be updated weekly or so and were last updated on 2019-12-17. Each file but the latest contains 1 million self-play games. AFAIK, all were played with Chinese rules and komi = 7.5 points. The earliest (all_1M.sgf) start with zero knowledge other than the rules and LZ bootstraps itself from random play to super-human levels as documented in currently 17,714,189 games.

The last file, all_18M.sgf only contains 714,190 games and, for some reason, all_7M.sgf only contains 999,999. By "contains" I mean that I wrote a short program to read each file searching for and counting the occurrences of "RE[", "RE[B", and "RE[W". These three strings should give us: total games with a result in the file, total wins for Black, and total wins for White. This seemed to work since the sum of the latter two always added up to the former and the former always equaled exactly 1,000,000 except for all_18 (expected) and all_7 (a surprise).

The winning rates for Black and for White are shown in the graph below. Notice that in the first 1,000,000 games (starting from random play) Black won more than 2/3 of the games. This quickly changed and already in the second million games White wins slightly more than Black. This trend of more wins for White increases as the game count (and LZ's strength) increases.

This seems to indicate that the more skilled the players (in this case the player-singular, since two instances of the same net are always playing each other) the less komi is necessary to compensate White for Black's having the first play. There may be other explanations for what we see here. Feel free to put forward your ideas. As always YMMV!

Winning rates for LZ self-play games.jpg (58.89 KiB) Viewed 9117 times

And for those who like numbers...

Game counts and winning rates by file.jpg (152.29 KiB) Viewed 9116 times

[xela]

I was sceptical of Bill's theory right up until five minutes ago. I've heard the argument before and thought it was flawed because it ignores the decreasing temperature towards the endgame.

So here we go. Data from https://github.com/gto76/online-go-games . A wide range of ranks but mostly DDK. 730935 SGF files, but some of them are problems, or text with error messages not a game record. Filter down to 662584 files that look like real games on a square board. Subset to 19x19 games with no handicap and 6.5 komi which ended with the score being counted: now down to 50988 games.

Turns out that the median score is W+9.5 (i.e. white 3 points ahead on the board), mean W+14.32.

Summary data attached for anyone else who wants to investigate. I've left in the weird komis for interest (e.g. the time someone tried -90 points komi). Oh, and there were four games where white won by over 400 points. I haven't looked at those SGFs and I don't want to.

(And while I was posting, someone else was looking at data too and finding that bots behave differently from humans. How about that!)

[moha]

Interesting ideas and data, thanks! Just a quick comment on the LZ set: komi 7.5 is known to be too much, and almost all of that graphs only shows how W is able to more and more capitalize on that advantage as it gets stronger. Perfect players win 100% W with 7.5 komi, this is the convergence target. With 6.5 (5.5) you would see B pulling ahead similarly, this is a different effect.

The interesting part of that dataset would be what happens inside the first 1M games (but LZ may have learned too fast to have enough data at nearly random and DDK levels, and there were also huge bugs in it around then which may distort results).

Similar distorting factors may be present in human games as well. The supposed effect is 1-2 points only, while the results are more subject to things like playing styles. Large scale fights favor B in a different way than quiet territory collecting, for example.

[hyperpape]

I specifically saw moha's comment, and thought "I have an axe to grind, but I won't derail the thread."

Since others have done the derailing, let me dump two links to previous discussions: viewtopic.php?p=222071#p222071, viewtopic.php?p=219462#p219462.

I'd suggested using AI training games to measure komi for weak players, glad to see Dave had more actual initiative than I did!

[jlt]

@xela: your stats are biased because if I counted correctly,

The two players have an identical rating in 62% of the games.
White is at least 1 rank stronger in 24% of the games.
Black is at least 1 rang stronger in 14% of the games.

(And sorry for derailing the thread further.)

[bayu]

@xela: your stats are biased because if I counted correctly,

The two players have an identical rating in 62% of the games.
White is at least 1 rank stronger in 24% of the games.
Black is at least 1 rang stronger in 14% of the games.

(And sorry for derailing the thread further.)

When OGS was taken over by Nova, the new managers decided for the tournaments to give the stronger player always white by default. Maybe it was less a decision and more of a bug turned feature. I don't know whether it is corrected by now, but it used to be the case at least for a couple of years. So, all the game data will be skewed. It even led to the conclusion that a komi 6.5 was too big for 9x9 (no surprise when giving the stronger player always white) so that it was reduced to 5.5.

[Bill Spight]

First, it is not my theory. Back in the 1990s, when I first heard the idea on rec.games.go, I believed as ez4u did. It was only after a couple of years that I looked into the question.

Second, I am glad that people are looking at data. It saves me from constructing some simple positions that fit ez4u's criteria and then playing several variations to illustrate the point.

Third, and most important, on the question of komi the percentage of wins is not the key statistic. It is the median of the results, which, OC, need not be an integer, and can remain the same even as the percentage of wins based upon a different komi value changes.

Edit: I wrote the above before seeing that xela, indeed, found the median.

[Bill Spight]

Data from https://github.com/gto76/online-go-games . A wide range of ranks but mostly DDK. 730935 SGF files, but some of them are problems, or text with error messages not a game record. Filter down to 662584 files that look like real games on a square board. Subset to 19x19 games with no handicap and 6.5 komi which ended with the score being counted: now down to 50988 games.

Here is a nit to pick, however, aside from the bias favoring White, which jlt pointed out. You should include human resignations. Unlike bots, humans are not going to resign games that they lose by only 1 or 2 points. Certainly not the weak humans for whom a low komi may be appropriate. OC, they may resign won games, but hopefully not as a rule. Also, as the Japanese pro data from the 1970s showed, to my surprise, the median was impervious to a small change in the komi actually used in games. Both the 4.5 pt. komi data and the 5.5 komi data had medians of 6.5. pts. So you don't have to stick to 6.5 komi games, as long as you control for rank.

[moha]

Both the 4.5 pt. komi data and the 5.5 komi data had medians of 6.5. pts.

We do not know whether there will be an even number of plays or an odd number of plays.

This is the parity issue I thought about earlier. Could it be that 6.5 points for Japanese is actually a very good choice even theoretically, not just by human statistics (another explanation for the more balanced results with it than with 7.5 for Chinese)?

It's clear that correct/perfect komi is integer for Japanese as well, maybe the same 7. But for near-perfect players, there are (usually) four cases to consider. The players either play the same number of competitive moves or B plays one more, for a certain territory score, then either odd or even dame remains. From these Chinese collapses 2x2 to the same (hence middle results only +5, +7, +9, with 7 presumably having the highest mass). But Japanese results retain all potentials (+5, +6, +7, +8), and +6 and +7 may have the same mass. Iow if we not only consider the single best minimax line but few similarly strong lines (maybe only those B+7 in Chinese), then maybe half of them is B+6 and half is B+7 because of those parity cases, so the distribution is not centered on 7 but on 6.5 actually?