It is currently Mon Mar 18, 2024 7:33 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 17 posts ] 
Author Message
Offline
 Post subject: AI opinions on "fairest" komi by board size
Post #1 Posted: Thu Aug 27, 2020 9:41 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
I did a study of KataGo's opinions about the "fairest" komi for each square board size using the final 40 block network, for every size that the 40-block network has had direct training experience on. These opinions should reflect pretty well KataGo's experiences during self-play about its empirical winning frequency on these board sizes for different values of komi on those board sizes, and although KataGo is only one bot, might be indicative of what "fair" komi would generally be among bots trained to a comparable level on these board sizes.

Firstly, from separate earlier studies, we know (probable) perfect komi for smaller sizes:
Code:
SZ  Japanese  Chinese     
3      8         9
4      0         2
5     24        25
6      3         4


This post by lightvector was liked by 3 people: Elom0, Knotwilg, lebigot
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #2 Posted: Thu Aug 27, 2020 9:42 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Next my interpretation of the raw data. Most "fair" komi, in KataGo's opinion, assuming quality of play at KataGo's self-play-training level, plus some notes about that komi.

Keep in mind the fact that in Chinese rules, absent seki, odd komi are the ones that matter to cross on odd boards, while even komi are the ones that matter to cross on even boards. So, on odd sized boards, 5.5, 6, 6.5 are all the same for example, while 6.5, 7, 7.5 are all different. On even boards, it's the other way around. So there's this weird oscillatory behavior for Chinese rules on board size parity.

Code:
SZ  Japanese  Chinese     
7      8         9       
KataGo is pretty sure about both values, to such a degree that although formal proof is vastly beyond reach, these are probably decent guesses for the perfect komi.

8      9-*       10-*     
KataGo thinks these values are fair, and is *very* confident that it is not higher. But as for being lower... white has high-variance tactics that KataGo thinks probably don't work, but if they do, could push the perfect komi much lower, even by multiple points. The incredible asymmetry between higher komi confidence vs lower komi confidence is fascinating.

9      6-        7--     
These are the most fair, but KataGo thinks white is at least slightly easier to play in each case, especially in Chinese.

10     6-        6++     
Japanese: White is slightly easier to play.
Chinese: Black is easier to play.

11     6-     5.5,6,6.5+ 
Japanese: White is slightly easier to play.
Chinese: No seki is expected, unclear if perfect komi is 5 or 7, so fairest komi in practice is in between, but more likely to be 7, so black is slightly easier to play. (7 is less fair in practice because even if the most likely perfect komi, it's so much harder for black to play than 5.5,6,6.5 are for white to play).

12     6      6.5,7,7.5--
Japanese: Pretty much fair in practice.
Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is easier for white to play.

13     6+        7-       
Japanese: Black is slightly easier to play.
Chinese: White is slightly easier to play.

14     6+     6.5,7,7.5- 
Japanese: Black is slightly easier to play.
Chinese: Unclear if perfect komi is 6 or 8, enough that fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play.

15    6.5-        7-     
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: White is slightly easier to play.

16    6.5-     6.5,7,7.5-
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between, but more likely to be 6, so in between is slightly easier for white to play.

17    6.5-        7-     
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: White is slightly easier to play.

18    6.5-     6.5,7,7.5
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5, but more likely 6, so white is slightly easier to play.
Chinese: Unclear if perfect komi is 6 or 8, so fairest komi in practice is in between.

19    6.5         7       
Japanese: Unclear if perfect komi is 6 or 7, enough that fairest komi in practice is 6.5.
Chinese: Pretty much fair in practice.


This post by lightvector was liked by 3 people: Elom0, icarusmustburn, lebigot
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #3 Posted: Thu Aug 27, 2020 9:48 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Lastly, the raw data. These are tables of the observed winrates for KataGo from the opening position for different board sizes under different rules. This is the full range of board sizes that KataGo has had actual training on - size 7 to size 19. These were done with the final g170 40 block net with 100ks of playouts on the smaller boards, diminishing to 10ks of playouts on the large boards since those took longer, and using somewhat wider-than-normal search settings.

On the large boards the results appear quite stable even if searching longer, since the search is mostly just smoothly averaging over the large variety of permutations of the opening what the neural net says based on its self-play experience. On the smaller boards, tactical complications start happening much faster, and the values never stop fluctuating as the tactics are explored ever deeper, I often just chose an arbitrary point to cut it off, different runs might get different results.

Also of course, technically these are "KataGo Japanese" and "KataGo Chinese" rules, which might differ from the actual Chinese and Japanese rules in sufficiently pathological positions.

Japanese rules winrates:
Code:
SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J                                      99  50   3
8J              95  94  93  91  89  89  90  85  84  50   2
9J          89  84  80  70  48  26  17  13
10J     87  82  77  70  60  47  32  22  16
11J             76  68  59  48  37  28  22
12J                 67  59  51  42  35  28
13J                 67  59  52  45  38  31
14J                 65  59  52  46  40  34
15J                 65  60  54  48  42  37
16J                 63  58  53  48  43  38
17J                     57  52  48  44  39
18J                     56  53  49  45  41
19J                     56  52  49  45  42


Chinese rules winrates:
Code:
SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7C                                              99  50   2
8C                      95  92  89  88  90  79  79  80  85  50   1
9C              93  83  69  69  70  41  13  17  16  12   8
10C     93  85  76  75  77  60  31  34  32  18   7
11C             85  74  59  59  59  39  21
12C                     73  59  41  43  42  29  18
13C             83  72  59  59  60  45  31
14C                     71  60  45  46  46  34  23
15C             78  69  59  59  60  48  37
16C                     68  58  47  48  49  39  30
17C             74  65  56  57  57  48  39
18C                     65  56  48  48  49  41  33
19C             70  63  55  56  56  49  41


And the same two tables, interleaved and paired if you want to more easily compare Japanese and Chinese together for just a single komi:
Code:
SZ\KM  3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10. 10.5
7J                                      99  50   3
7C                                              99  50   2

8J              95  94  93  91  89  89  90  85  84  50   2
8C                      95  92  89  88  90  79  79  80  85  50   1

9J          89  84  80  70  48  26  17  13
9C              93  83  69  69  70  41  13  17  16  12   8

10J     87  82  77  70  60  47  32  22  16
10C     93  85  76  75  77  60  31  34  32  18   7

11J             76  68  59  48  37  28  22
11C             85  74  59  59  59  39  21

12J                 67  59  51  42  35  28
12C                     73  59  41  43  42  29  18

13J                 67  59  52  45  38  31
13C             83  72  59  59  60  45  31

14J                 65  59  52  46  40  34
14C                     71  60  45  46  46  34  23

15J                 65  60  54  48  42  37
15C             78  69  59  59  60  48  37

16J                 63  58  53  48  43  38
16C                     68  58  47  48  49  39  30

17J                     57  52  48  44  39
17C             74  65  56  57  57  48  39

18J                     56  53  49  45  41
18C                     65  56  48  48  49  41  33

19J                     56  52  49  45  42
19C             70  63  55  56  56  49  41


This post by lightvector was liked by 7 people: dfan, dhu163, Elom0, icarusmustburn, lebigot, marvin, Uberdude
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #4 Posted: Thu Aug 27, 2020 9:53 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
There are of course other things to observe in the raw data too.

Anyways, this was fun. Hope someone finds this interesting. :)

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #5 Posted: Fri Aug 28, 2020 3:49 am 
Oza
User avatar

Posts: 2404
Location: Ghent, Belgium
Liked others: 359
Was liked: 1019
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
Fascinating that fair komi hardly changes between board size 9 and 19, and that we've known this all along.


This post by Knotwilg was liked by: lebigot
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #6 Posted: Fri Aug 28, 2020 4:13 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
I find this very interesting!

I made charts for odd board sizes 9-19.

So for odd board sizes 9-19, the fairest komi is 6+ for Japanese rules and 7- for Chinese rules.

With Japanese rules, 6 komi looks like perfect komi.
When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then white has an advantage on smaller boards).

With Chinese rules, 7 komi looks like perfect komi.
When draws are unacceptable, 6.5 komi looks like the fairest alternative (but then black has an advantage, more so on smaller boards). With 7.5 komi, white has a fairly large advantage.

Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much.
Does anyone know if there are plans in China or the US to lower the komi to 6.5?

Has any AI research been done on area scoring with integer komi and a Button to break ties?


Attachments:
komi.png
komi.png [ 67.12 KiB | Viewed 13820 times ]

This post by gennan was liked by 2 people: Harleqin, lebigot
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #7 Posted: Fri Aug 28, 2020 7:42 am 
Lives in gote

Posts: 502
Liked others: 1
Was liked: 153
Rank: KGS 2k
GD Posts: 100
KGS: Tryss
Quote:
Evidence is mounting that 7.5 komi, as commonly used with Chinese rules (and AGA rules), is too much.
Does anyone know if there are plans in China or the US to lower the komi to 6.5?


With area scoring rules, you have 361 pts to share, so if there's no seki, the score difference on the board is always an odd integer (If black score on the board is B, and white score on the board is B, then W+B=361)

The only case where there's a difference between 5.5 and 6.5 komi is when there's an odd number of irreductible dame.

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #8 Posted: Fri Aug 28, 2020 9:11 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules.

But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi.

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #9 Posted: Fri Aug 28, 2020 11:40 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
gennan wrote:
Yes, the lower graph also shows that 5.5 and 6.5 komi have basically the same winrate with Chinese rules.

But black has 55% winrate with 5.5 or 6.5 komi, while white has 60% winrate with 7.5 komi. So 7.5 komi seems to be more unfair than 5.5 or 6.5 komi.


Well, why not a komi of 7 points? Even pros would probably have ties less than 5% of the time. Chess gets by with a fairly high rate of draws.

I still like Button Go, though. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


This post by Bill Spight was liked by: gennan
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #10 Posted: Sat Aug 29, 2020 12:26 am 
Oza
User avatar

Posts: 2401
Location: Tokyo, Japan
Liked others: 2337
Was liked: 1332
Rank: Jp 6 dan
KGS: ez4u
@lightvector

How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #11 Posted: Sat Aug 29, 2020 3:48 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Bill Spight wrote:
I still like Button Go, though. :)


KataGo supports area scoring + button Go as an evaluation method. I have not studied what its evaluations are or how they differ from those here (I suspect they would largely match the Japanese rules evaluations, but I haven't checked). I welcome anyone to do their own study. :)

ez4u wrote:
@lightvector

How much experience did katago have at the different komi levels/rule sets on different boards? I assume that the katago should be weaker at assessing the alternatives simply due to the difference in experience levels. It would be extremely interesting to see your thoughts in this regard.


Generally KataGo played easily 10x (order of magnitudewise) as many 19x19 games as any other board size. But also smaller boards are easier to learn - whereas if you only a have a few GPUs it takes weeks or months to reach pro level on 19x19 for example, I think it could be done on 9x9 in days. Also an enormous amount of knowledge generalizes between the board sizes, especially on the larger sizes. You can see this in the smoothness of the winrate estimates for the larger board sizes. If it were systematically the case that smaller board size play were abnormally "lower-quality" by a lot than the 19x19 play, you'd expect that empirically in self-play a given degree of komi-unfairness would not result in as pronounced an advantage for one side or the other, and then the neural net's output would reflect those statistics (fitting smooth-ish simple curves to statistics is very easy for neural nets), so you might expect a discontinuity in the sharpness of the winrates as you go from 19x19 to any smaller size. But to the contrary it looks quite regularly increasing in sharpness, as you'd expect if quality of play was fairly regular and the dominant effect was simply the higher variance of the large board due to the board being bigger / games lasting longer, given comparably high-quality play.

This makes intuitive sense to me. I'd think that 95+% of the "knowledge" between the board sizes is shared for board sizes 13x13 and larger. You learn the subtle ways that playing on 15x15 should adjust your overall judgment of territory vs influence and other fuzzy aspects of evaluation, and then modulo those shades of difference in judgment, almost everything you know about fighting and good shapes and what moves are effective in obtaining different kinds of trades carries over.

You can take the winrates as a fairly direct reflection of the self-play training statistics. For example on 9x9 with Japanese rules, KataGo must have experienced roughly fair results with 6 komi, and adjusting it merely to 5.5 komi gives black 70%/30%, more than a 2-1 advantage, whereas adjusting it to 6.5 gives white 74%/26%, almost a 3-1 advantage. While more extreme komis than this will have increasingly limited data, there should be quite many games at all of 5.5, 6, and 6.5, so presumably these are roughly the statistics from the games. Also self-play conditions have added noise and use only 200-1000 playouts - so basically you can say that slightly drunk ultra-blitz KataGo is roughly of a level on 9x9 where, at least against itself, a half-point advantage somewhere between doubles and triples your winning likelihood over the opponent. With the tens of thousands of playouts you'd get in match conditions, it should be yet much sharper.


This post by lightvector was liked by: gennan
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #12 Posted: Sat Aug 29, 2020 4:07 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
One thing I find interesting:

For 3,4,5,6,7 you see a pattern where the odd sizes have a larger advantage for black than the even sizes. Something to do with tengen being a uniquely effective move on small boards, whereas when the center is a 2x2 square, white can also split the board and you get a more balanced result? It sort of makes intuitive sense. But, size 8 breaks the pattern. It has a *larger* advantage for black than either 7 or 9.

Size 9 is the last size in which tengen is a good move - and in fact, contrary to what you hear circulated in some forums, it's not even necessarily the best move - size 9 is where non-tengen moves finally "catch up" and become just as good in quality, 4-4 and 4-5 openings are probably also draws under fair komi. After size 9, non central moves overtake tengen. Past size 9, it also becomes increasingly clear that the bots are no longer "nearly solving" the game, and the game becomes smoother and not dominated by one single highly swingy tactical fight. Both players have enough choices of how to play and maintain average move efficiency that aren't dictated by a high density of one-way-street tactics, that I'd conjecture that perfect komi remains near 6 or 7 perpetually after, near the practical fair komi. As opposed to being much more jittery like the small boards and it merely being beyond anyone's reach to find the needle-in-a-haystack line that makes you multiple points better than the current komis that appear to be fair.

So anyways, that would seems to leave board size 8 as an oddity, where the single-big-tactical-fight of the game results in black having a bigger advantage despite an even size, rather than a smaller advantage, and one of the last board sizes for a chance for this to happen before one might conjecture smooth averaging behavior to start to dominate.


This post by lightvector was liked by 3 people: ez4u, gennan, Maharani
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #13 Posted: Wed Nov 04, 2020 3:30 pm 
Judan

Posts: 6725
Location: Cambridge, UK
Liked others: 436
Was liked: 3719
Rank: UK 4 dan
KGS: Uberdude 4d
OGS: Uberdude 7d
Great stuff lightvector. I didn't realise KataGo now prefers white on Japanese 6.5, it used to prefer black. The 20b network I just tried in the BadukAI app still does. Any idea when this changed?

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #14 Posted: Thu Nov 05, 2020 11:08 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Uberdude - Not sure. But all the nets are here, feel free to check. :)

https://d3dndmfyhecmj0.cloudfront.net/g ... index.html

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #15 Posted: Mon Aug 14, 2023 4:11 am 
Lives in sente

Posts: 723
Liked others: 1018
Was liked: 30
Rank: BGA 3 kyu
KGS: Elom, Windnwater
OGS: Elom, Elom0
Online playing schedule: The OGS data looks pretty so I'll pause for now before I change it.
An update of this data on

Also, I am actually quite interested in KataGo's estimation for 6x6, 5x5, 4x4 and 3x3, perhaps even 2x2 as it would be interesting to see KataGo's thought process in getting to those likely correct komi

Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #16 Posted: Mon Aug 14, 2023 6:35 am 
Judan

Posts: 6082
Liked others: 0
Was liked: 786
2x2? To see if KataGo does worse than Bill Taylor's theorem (IIRC, Black's 1 point win under area scoring and superko)?


This post by RobertJasiek was liked by: Elom0
Top
 Profile  
 
Offline
 Post subject: Re: AI opinions on "fairest" komi by board size
Post #17 Posted: Mon Aug 14, 2023 12:44 pm 
Lives in sente

Posts: 723
Liked others: 1018
Was liked: 30
Rank: BGA 3 kyu
KGS: Elom, Windnwater
OGS: Elom, Elom0
Online playing schedule: The OGS data looks pretty so I'll pause for now before I change it.
RobertJasiek wrote:
2x2? To see if KataGo does worse than Bill Taylor's theorem (IIRC, Black's 1 point win under area scoring and superko)?


We can see KataGo's thought process for 2x2 on multiple rulesets!

Also, an idea is instead of trying multiple komis, simply train only on no komi games and increase the weight of a win according to winning score, so when a net wins by 3 points, the effect of it's weightings on creating the new version are 3 times what it would have been if it won by one point, for example. But it means the winrate for a particular komi is just derived from the proportion of games won by black by that margin or higher in these no komi games. The total number of games between area rulesets Chinese, AGA, Ing, New Zealand, and territory rulesets Korean, Japanese, etc would be even with the runs of rulesets within those also distributed equally, however, there will often be games played using one ruleset that would be legal in another, and in those cases the results of the winning margin should included in the data for those rulesets too, or better yet, whenever a move is played that is legal in some rulesets but not others, split the run into multiple branches that comply with different rulesets. I had the thought of making dual area-territory rules the standard in lentitear, so the final score is half the difference in number of stones on the board plus the difference in territory, and since fair komi for territory is 6.5 and fair komi for area is 7, fair komi in dual area-territory scoring with each weighted at a half would be 6.75!

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group