It is currently Thu Mar 28, 2024 7:35 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 39 posts ]  Go to page Previous  1, 2
Author Message
Offline
 Post subject: Re: AI komi
Post #21 Posted: Mon Dec 23, 2019 1:55 am 
Oza
User avatar

Posts: 2401
Location: Tokyo, Japan
Liked others: 2338
Was liked: 1332
Rank: Jp 6 dan
KGS: ez4u
My model of Go errors: Sh** Happens! :D

My "model" ("image" might be better) of the game:
    - Go is played by two opponents. Player Black plays first and then players White and Black alternate play. (append a copy of your favorite rules for the rest of the details)
    - There is an advantage to playing first. (We have a few centuries of experience that support this claim)
    - There are game-specific skills that allow player Black to better take advantage of playing first and equally allow player White to frustrate player Black's attempts to take advantage of playing first.

From this I expect that we do not see a lower komi as appropriate between less-skilled players. Rather what we expect is a greater variance in game results between less-skilled players.

In addition, it seems to me that all of our understanding of "skill" is performance based. What does it mean that komi would be lower for less-skilled players if we use komi/handicaps to define skill levels?

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21


This post by ez4u was liked by 2 people: Bill Spight, gennan
Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #22 Posted: Mon Dec 23, 2019 6:13 am 
Oza
User avatar

Posts: 2401
Location: Tokyo, Japan
Liked others: 2338
Was liked: 1332
Rank: Jp 6 dan
KGS: ez4u
Fortunately we may be able to test the question directly. We now have in our possession a formidable supply of games played between equal opponents across a range of strengths. I mean the archive of LZ's self-play games! :study:

I downloaded the files all_1M.sgf - all_18M.sgf (18 files in all) from the archive at https://leela.online-go.com/zero/. These appear to be updated weekly or so and were last updated on 2019-12-17. Each file but the latest contains 1 million self-play games. AFAIK, all were played with Chinese rules and komi = 7.5 points. The earliest (all_1M.sgf) start with zero knowledge other than the rules and LZ bootstraps itself from random play to super-human levels as documented in currently 17,714,189 games.

The last file, all_18M.sgf only contains 714,190 games and, for some reason, all_7M.sgf only contains 999,999. By "contains" I mean that I wrote a short program to read each file searching for and counting the occurrences of "RE[", "RE[B", and "RE[W". These three strings should give us: total games with a result in the file, total wins for Black, and total wins for White. This seemed to work since the sum of the latter two always added up to the former and the former always equaled exactly 1,000,000 except for all_18 (expected) and all_7 (a surprise).

The winning rates for Black and for White are shown in the graph below. Notice that in the first 1,000,000 games (starting from random play) Black won more than 2/3 of the games. This quickly changed and already in the second million games White wins slightly more than Black. This trend of more wins for White increases as the game count (and LZ's strength) increases.

This seems to indicate that the more skilled the players (in this case the player-singular, since two instances of the same net are always playing each other) the less komi is necessary to compensate White for Black's having the first play. There may be other explanations for what we see here. Feel free to put forward your ideas. As always YMMV! :salute:
Attachment:
Winning rates for LZ self-play games.jpg
Winning rates for LZ self-play games.jpg [ 58.89 KiB | Viewed 6714 times ]

And for those who like numbers...
Attachment:
Game counts and winning rates by file.jpg
Game counts and winning rates by file.jpg [ 152.29 KiB | Viewed 6713 times ]

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21


This post by ez4u was liked by 2 people: gennan, hyperpape
Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #23 Posted: Mon Dec 23, 2019 6:26 am 
Lives in gote

Posts: 580
Location: Adelaide, South Australia
Liked others: 207
Was liked: 264
Rank: Australian 2 dan
GD Posts: 200
I was sceptical of Bill's theory right up until five minutes ago. I've heard the argument before and thought it was flawed because it ignores the decreasing temperature towards the endgame.

So here we go. Data from https://github.com/gto76/online-go-games . A wide range of ranks but mostly DDK. 730935 SGF files, but some of them are problems, or text with error messages not a game record. Filter down to 662584 files that look like real games on a square board. Subset to 19x19 games with no handicap and 6.5 komi which ended with the score being counted: now down to 50988 games.

Turns out that the median score is W+9.5 (i.e. white 3 points ahead on the board), mean W+14.32.

Summary data attached for anyone else who wants to investigate. I've left in the weird komis for interest (e.g. the time someone tried -90 points komi). Oh, and there were four games where white won by over 400 points. I haven't looked at those SGFs and I don't want to.

(And while I was posting, someone else was looking at data too and finding that bots behave differently from humans. How about that!)


Attachments:
OGS_result_list.csv [1.85 MiB]
Downloaded 315 times
Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #24 Posted: Mon Dec 23, 2019 6:35 am 
Lives in gote

Posts: 311
Liked others: 0
Was liked: 45
Rank: 2d
Interesting ideas and data, thanks! Just a quick comment on the LZ set: komi 7.5 is known to be too much, and almost all of that graphs only shows how W is able to more and more capitalize on that advantage as it gets stronger. Perfect players win 100% W with 7.5 komi, this is the convergence target. With 6.5 (5.5) you would see B pulling ahead similarly, this is a different effect.

The interesting part of that dataset would be what happens inside the first 1M games (but LZ may have learned too fast to have enough data at nearly random and DDK levels, and there were also huge bugs in it around then which may distort results).

Similar distorting factors may be present in human games as well. The supposed effect is 1-2 points only, while the results are more subject to things like playing styles. Large scale fights favor B in a different way than quiet territory collecting, for example.


This post by moha was liked by: ez4u
Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #25 Posted: Mon Dec 23, 2019 7:37 am 
Tengen

Posts: 4380
Location: North Carolina
Liked others: 499
Was liked: 733
Rank: AGA 3k
GD Posts: 65
OGS: Hyperpape 4k
I specifically saw moha's comment, and thought "I have an axe to grind, but I won't derail the thread." :)

Since others have done the derailing, let me dump two links to previous discussions: https://lifein19x19.com/viewtopic.php?p=222071#p222071, https://lifein19x19.com/viewtopic.php?p=219462#p219462.

I'd suggested using AI training games to measure komi for weak players, glad to see Dave had more actual initiative than I did!

_________________
Occupy Babel!

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #26 Posted: Mon Dec 23, 2019 8:24 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
@xela: your stats are biased because if I counted correctly,

The two players have an identical rating in 62% of the games.
White is at least 1 rank stronger in 24% of the games.
Black is at least 1 rang stronger in 14% of the games.

(And sorry for derailing the thread further.)

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #27 Posted: Mon Dec 23, 2019 10:01 am 
Lives with ko

Posts: 163
Liked others: 19
Was liked: 32
jlt wrote:
@xela: your stats are biased because if I counted correctly,

The two players have an identical rating in 62% of the games.
White is at least 1 rank stronger in 24% of the games.
Black is at least 1 rang stronger in 14% of the games.

(And sorry for derailing the thread further.)


When OGS was taken over by Nova, the new managers decided for the tournaments to give the stronger player always white by default. Maybe it was less a decision and more of a bug turned feature. I don't know whether it is corrected by now, but it used to be the case at least for a couple of years. So, all the game data will be skewed. It even led to the conclusion that a komi 6.5 was too big for 9x9 (no surprise when giving the stronger player always white) so that it was reduced to 5.5.

_________________
If something sank it might be a treasure. And 2kyu advice is not necessarily Dan repertoire..

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #28 Posted: Mon Dec 23, 2019 10:17 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
First, it is not my theory. Back in the 1990s, when I first heard the idea on rec.games.go, I believed as ez4u did. It was only after a couple of years that I looked into the question.

Second, I am glad that people are looking at data. It saves me from constructing some simple positions that fit ez4u's criteria and then playing several variations to illustrate the point.

Third, and most important, on the question of komi the percentage of wins is not the key statistic. It is the median of the results, which, OC, need not be an integer, and can remain the same even as the percentage of wins based upon a different komi value changes.

Edit: I wrote the above before seeing that xela, indeed, found the median. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #29 Posted: Mon Dec 23, 2019 10:35 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
xela wrote:
Data from https://github.com/gto76/online-go-games . A wide range of ranks but mostly DDK. 730935 SGF files, but some of them are problems, or text with error messages not a game record. Filter down to 662584 files that look like real games on a square board. Subset to 19x19 games with no handicap and 6.5 komi which ended with the score being counted: now down to 50988 games.


Here is a nit to pick, however, aside from the bias favoring White, which jlt pointed out. You should include human resignations. Unlike bots, humans are not going to resign games that they lose by only 1 or 2 points. Certainly not the weak humans for whom a low komi may be appropriate. OC, they may resign won games, but hopefully not as a rule. ;) Also, as the Japanese pro data from the 1970s showed, to my surprise, the median was impervious to a small change in the komi actually used in games. Both the 4.5 pt. komi data and the 5.5 komi data had medians of 6.5. pts. :) So you don't have to stick to 6.5 komi games, as long as you control for rank. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #30 Posted: Mon Dec 23, 2019 12:07 pm 
Lives in gote

Posts: 311
Liked others: 0
Was liked: 45
Rank: 2d
Bill Spight wrote:
Both the 4.5 pt. komi data and the 5.5 komi data had medians of 6.5. pts. :)
Bill Spight wrote:
We do not know whether there will be an even number of plays or an odd number of plays.

This is the parity issue I thought about earlier. Could it be that 6.5 points for Japanese is actually a very good choice even theoretically, not just by human statistics (another explanation for the more balanced results with it than with 7.5 for Chinese)?

It's clear that correct/perfect komi is integer for Japanese as well, maybe the same 7. But for near-perfect players, there are (usually) four cases to consider. The players either play the same number of competitive moves or B plays one more, for a certain territory score, then either odd or even dame remains. From these Chinese collapses 2x2 to the same (hence middle results only +5, +7, +9, with 7 presumably having the highest mass). But Japanese results retain all potentials (+5, +6, +7, +8), and +6 and +7 may have the same mass. Iow if we not only consider the single best minimax line but few similarly strong lines (maybe only those B+7 in Chinese), then maybe half of them is B+6 and half is B+7 because of those parity cases, so the distribution is not centered on 7 but on 6.5 actually?


This post by moha was liked by 2 people: Bill Spight, ez4u
Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #31 Posted: Mon Dec 23, 2019 3:13 pm 
Lives in gote

Posts: 580
Location: Adelaide, South Australia
Liked others: 207
Was liked: 264
Rank: Australian 2 dan
GD Posts: 200
OK, that'll teach me to take shortcuts with late-night data analysis, hoping to knock it off before midnight! I'll see if I can fix it in 15 minutes before going to work. What could go wrong?

----

Adding back in the resignations, but removing games with unequal ranks, gives 23,222 games.
Attachment:
OGS_result_list2.csv [752.35 KiB]
Downloaded 281 times


All 23,222 games: median score W+2.5 (after adjusting for komi, which varies between -99 and +30, but the vast majority are sensible numbers)

Restricting to 6.5 komi: 21,965 games, median score W+3.5, i.e. black is 4 points ahead on the board. Still seems to support Bill's argument.

Excluding resignations, allowing any komi: 7,866 games, median score W+1.5, mean score W+0.45.

Excluding resignations, 6.5 komi only: 7,378 games, median score W+1.5, mean score W=0.77.

Sorry I don't have time to format it into a nice table, the day job beckons.

Overall, white resigned 7,609 games and black resigned 7,747.

----

Medians! Yes, Bill, I've been paying attention to you :-)

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #32 Posted: Mon Dec 23, 2019 3:20 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
xela wrote:
OK, that'll teach me to take shortcuts with late-night data analysis, hoping to knock it off before midnight!


I sympathize. :) How many times have I resolved not to post late at night, because of misstatements, omissions, and goofs! :oops:

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #33 Posted: Tue Dec 24, 2019 1:18 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
@xela: it would be interesting to calculate the median for several categories of players: dan, SDK, 10-19k, 20-50k. The reason is that about 75% of the games are between players 20k or weaker.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #34 Posted: Tue Dec 24, 2019 1:41 am 
Oza
User avatar

Posts: 2401
Location: Tokyo, Japan
Liked others: 2338
Was liked: 1332
Rank: Jp 6 dan
KGS: ez4u
moha wrote:
...Just a quick comment on the LZ set: ...
The interesting part of that dataset would be what happens inside the first 1M games (but LZ may have learned too fast to have enough data at nearly random and DDK levels, and there were also huge bugs in it around then which may distort results)...

here is a closer look at the first 2 million games. Not because I think it will explain anything, rather because it looks cool. :cool: Note that LZ continued on the 5x64 net until around 3 million games, at which point it moved up to 6x128.
Attachment:
Winning rates for LZ self-play games 1st 2M games.jpg
Winning rates for LZ self-play games 1st 2M games.jpg [ 78.35 KiB | Viewed 6570 times ]

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #35 Posted: Tue Dec 24, 2019 6:48 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
You should be aware that some of the Leela Zero's "beginner mistakes" were utterly unlike anything a human would mistakenly do, and in specific ways that really would mess with the result.

For example, there was an early time when black would accidentally pass, and then white would pass and win. I think there were also periods where white learned that passing was "good" early in the game, as well as any time white got ahead in the number of stones (such as by captures), because sometimes black could pass in response and lose. Once black caught on to that trick though, black would win a ton of games because white would pass away the advantage every time after getting an advantage, until eventually white caught on to that it shouldn't be passing. And it was only white that could learn this "bad" strategy because it was only white that started with a positive komi such that passing might possibly seem good in some cases.

So, stupid stuff like that. The above story might not quite be right, but things "like that" were occurring sometimes I think. Careful about using early LZ to generalize about komi.


This post by lightvector was liked by 3 people: Bill Spight, ez4u, hyperpape
Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #36 Posted: Tue Dec 24, 2019 7:00 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
lightvector wrote:
For example, there was an early time when black would accidentally pass, and then white would pass and win. I think there were also periods where white learned that passing was "good" early in the game, as well as any time white got ahead in the number of stones (such as by captures), because sometimes black could pass in response and lose. Once black caught on to that trick though, black would win a ton of games because white would pass away the advantage every time after getting an advantage, until eventually white caught on to that it shouldn't be passing. And it was only white that could learn this "bad" strategy because it was only white that started with a positive komi such that passing might possibly seem good in some cases.


Just curious. How were those games scored on the board (befoce applying komi)? By area at the time? IMHO, proximity scoring may be an improvement. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #37 Posted: Tue Dec 24, 2019 7:33 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Yeah. Tromp-Taylor rules, of course. LZ was meant to replicate AlphaGoZero reasonably closely, so any change to the rules for scoring would also have not been appropriate.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #38 Posted: Tue Dec 24, 2019 7:58 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
lightvector wrote:
Yeah. Tromp-Taylor rules, of course. LZ was meant to replicate AlphaGoZero reasonably closely, so any change to the rules for scoring would also have not been appropriate.


Right. Thanks. :)

Ing adopted proximity scoring in his last set of rules, although the English translations I have seen assume regular area scoring. The only case I know of where the two differ for games that end normally is one seki shape. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: AI komi
Post #39 Posted: Wed Dec 25, 2019 8:53 pm 
Lives in gote

Posts: 580
Location: Adelaide, South Australia
Liked others: 207
Was liked: 264
Rank: Australian 2 dan
GD Posts: 200
jlt wrote:
@xela: it would be interesting to calculate the median for several categories of players: dan, SDK, 10-19k, 20-50k. The reason is that about 75% of the games are between players 20k or weaker.


OK. It's interesting and baffling. In fact most of the games are players 30k or below, including one game between two players who'd achieved a 51k rank. I think this is very noisy data.

For each rank band, I've calculated four medians. In order: all games; games with 6.5 komi only; all games with resignations excluded; 6.5 komi with resignations excluded. Numbers in brackets are the number of games.






























rank all games 6.5 komi no resign 6.5 komi, no resign
30k and below W+14.5 (13562) W+18.5 (13030) W+1.5 (3891) W+1.5 (3690)
20-29k B+4.5 (3464) B+4.5 (3173) W+0.5 (1630) W+0.5 (1499)
10-19k W+0.5 (2718) W+0.5 (2546) W+1.5 (1246) W+1.5 (1167)
SDK B+2.5 (2926) B+3.5 (2783) W+2.5 (936) W+2.5 (895)
dan B+3 (511) B+3.5 (421) W+1.5 (147) W+1.5 (121)


(This tool is handy for generating tables, although L19 doesn't seem to recognise the [th] tags, need to manually change those to [td], and I've no idea why it's inserting a massive space before the table.)


This post by xela was liked by: jlt
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 39 posts ]  Go to page Previous  1, 2

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot], lovego and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group