AI komi

[John Fairbairn]

I'm having trouble understanding the effect of komi in AI games. There seems to be no settled view either here or in what I read in the mainstream go press or books.

From what I can make out, it is now agreed that with the current 7.5 komi (Chinese rules) White has a significant advantage. My version of LZ starts with B-W win rates of 42.7% and 57.3%. You can quote books like Freakonomics all you like, but surely that's a LOT. But how much is a lot? We know anecdotally that pros consider 1 point a lot. I have just been reading something by Shibano Toramaru in which he is attempting to explain the disappearance of once popular fusekis such as sanrensei under AI influence. He makes the point that it is entirely down to win rates because White has found good counter-strategies, but that it could all change if EITHER Black eventually developed better counter-counter-strategies OR if komi was re-adjusted. Speaking of the adherence to win rates as the reason for the shift in pro attitudes, he says, "Actually, if we convert this to komi, it is maybe a difference of only about 1 point. It is therefore no surprise that there are pros who are saying 'Give me 2 points extra komi and I will play sanerensei or the Chinese style [again]."

I think statistics from human games also seem to suggest 6.6 komi (Japanese) is a tad too high, but 5.5. does favour Black.

So, in my gullible way, I assume that 57.3 - 42.7 represents about 1 point. Given that there seems to be a vague consensus that there is a margin of error of some 2 or 3 percentage points (but I'm unclear whether that means a rage of 2-3 pp or +/- 2-3 pp), in thumbnail terms that seems to mean 10 percentage points of win rate is tantamount to 1 point board count.

That's a lot for a pro. Shibano goes on to stress that this is all at pro level. He says that amateurs should ignore it and go on playing sanrensei and Chinese fuseki if that's what they like.

If that is all there was to it, I can wrap my head round it. But the reason I'm having trouble is that many comments here don't seem consonant with that.

One aspect is that different machines seem to produce quite different results.

The other, more troubling, aspect is that so many comments are couched in terms like "mistake" or "loses a lot" or "that move is no longer playable". Since the comments here are essentially from amateurs (who can't really appreciate a 1-point initial difference according to Shibano), is this a case of what we might call "swagger"? Or is it case of me completely misunderstanding the numbers (as usual)?

[RobertJasiek]

AI percentages are not linear, analyse move candidates incompletely and the relations to absolute points are guesswork. All we can say is that a current AI at a current turn uses percentages as relative empirical predictions. Hence, deriving komi from them is guesswork.

[xela]

First, the "winrates" are not probabilities. This has already been done to death on these forums. Whether 57% is a lot or a little depends on how the software is calibrated. You can't give intrinsic meaning to these numbers.

Second, we persist in using fractional komi. If the komi value ends in .5, then one player or another has a definite advantage. For weaker players, the advantage is small because the number of mistakes in an average game means that a couple of points more or less komi is lost in the noise. But the stronger the players, the bigger the advantage. With perfect play, you're not just looking at 51% or 57%, you're looking at white winning 100% of games (or perhaps we'll get a big surprise, see black winning 100% and learn that correct komi is bigger than we thought). Putting komi back to 6.5 or 5.5 isn't going to fix this problem, it just swings the advantage over to the black side maybe. If this bothers enough people, we'll eventually set komi to 7.0 and start running all tournaments as Swiss or McMahon format instead of knockout so that we can cope with a few draws. Jigo was OK in the 19th century, so why shouldn't draws be OK in the 21st century?

Third -- and this is going to offend some people, but we're on an internet forum here, what better place to start an argument? (*) -- we tend to put our pros on a pedestal, one part of Asian culture that the Western go community has well and truly embraced, and they don't exactly discourage that. I'm far from pro strength in go (to put it mildly), but I've been a pro in other areas of life, and I know a bit about developing a mystique. Those 1 or 2 point nuances in fuseki (and "move 2 lost the game"), are they real, measurable differences, or are they rhetoric to distract us from the frequent 3 point swings or 20 point swings in the middlegame fighting? I agree with the remark about amateur commentators and "swagger", but I suspect it's not only the amateurs sometimes.

----

(*) And I'm doing this close to midnight in my time zone! Nothing like a good rant to unwind before bedtime :-)

[YeGO]

With a non-integer komi, in principle, one side should be able to force a win (provided that we are talking about a ruleset that avoids "no results").

I think the observations from AI win rates suggests that the correct komi under area scoring should be 7. If that's the case, then with stronger AI, the white win rate for 7.5 komi should be even higher, and perfect players should win 100% of the games they play as white with 7.5 komi.

If the professional statistics for Japanese rules suggests that 6.5 komi might be too high, and 5.5 might be too low, then that seems to support the hypothesis that the correct komi for J rules should be 6.

Such a discrepancy between territory and area could arise if perfect play involves black making the last move to fill the last dame.

[John Fairbairn]

AI percentages are not linear, analyse move candidates incompletely and the relations to absolute points are guesswork.

It seems irrelevant that AI percentages are not linear if we are just talking about the initial position. Agreed the rest is a kind of guesswork, but surely of a Monte Carlo kind based on millions of data points, which has produced superhuman estimates - which seem good enough for most humans.

First, the "winrates" are not probabilities. This has already been done to death on these forums.

I own up to not really understanding this. The numbers are measuring something, presumably. If a 57% win rate for White means White has a greater likelihood of winning, that's a greater probability in layman's language, surely?

Whether 57% is a lot or a little depends on how the software is calibrated.

Understood, but the overwhelming majority of comments (here and in Japan, and as with Shibano) seem to be Leela-based, so "57% = a lot" seems fine unless Leela itself is being constantly and significantly re-calibrated - is it? If so, that seems to throw a lot of earlier comments here out of the window.

we tend to put our pros on a pedestal, one part of Asian culture that the Western go community has well and truly embraced, and they don't exactly discourage that.

You won't get any arguments from me on that. But I would point out that there are people here who go to extremes the other way, too, and throw professionals out of the window. I suspect most people take the "medical" view. If you're very sick and you can't await future research or wait until debates settle down, would you (in the present state of knowledge) go to a professionally qualified physician or to a computer scientist? (I have to say "suspect" because the current measles epidemic may be due to too many people reading he internet and not getting their kids vaccinated. But I'm in the jab camp.)

[luigi]

If the komi value ends in .5, then one player or another has a definite advantage. [T]he stronger the players, the bigger the advantage. [...] Putting komi back to 6.5 or 5.5 isn't going to fix this problem, it just swings the advantage over to the black side maybe. If this bothers enough people, we'll eventually set komi to 7.0 and start running all tournaments as Swiss or McMahon format instead of knockout so that we can cope with a few draws.

Even better would be to use komi 7.0 together with the button to avoid ties. Sure, with perfect play one side would still win 100% of the time, but, if komi 7.0 is a tie with perfect play, komi 7.0 plus button should achieve much better balance than 7.5 komi with any high-level play other than perfect play.

[moha]

It is known that correct komi depends on the players' levels, with weaker players needing less komi for a fair game (since B makes less use of his advantage). The higher the level, the less mistakes players make (the deviation of their error distribution gets smaller), and the higher the probability of ending the game exactly in a tie (with correct komi of 7). Komi 7.5 basically treats ties as white wins, which gets more and more significant as strength increase. Current Japanese games seem quite balanced on 6.5 komi up to pro levels (not surprising because of the above reasons). But this may change when strong bots start to support these rules, and find that it favors black. (Although there are some further questions and uncertainity here because of dame parity.)

Also note the complex relation between winrates - points - game phases - current lead. Giving up 10% in opening may mean a 1 point error, the smallest possible mistake. (Or it might even mean no mistake, just making the drawing sequence harder to find.) Giving up 10% later can mean something completely different, depending on those other factors.

[Knotwilg]

First, the "winrates" are not probabilities. This has already been done to death on these forums. Whether 57% is a lot or a little depends on how the software is calibrated. You can't give intrinsic meaning to these numbers.

Same as John. If the percentages don't mean anything, let's ignore them altogether.

Surely they do mean something, namely x/N where N = de number of playouts and x = the number of games won. Which is the probability to win a game, based on that sample of playouts.

Maybe beaten to death on this forum but still so that I didn't notice.

[Uberdude]

Surely they do mean something, namely x/N where N = de number of playouts and x = the number of games won. Which is the probability to win a game, based on that sample of playouts.

This is not correct. You are getting rollouts of old MCTS bots like CrazyStone and Zen before AlphaGo came along, mixed up with playouts of modern bots with a value network (or rather value-part of a combined policy/value network, AG Lee had separate ones). With those old bots a rollout did indeed play a load of psuedo-random moves to the end, score the finished game and count how many were won. With modern bots they don't do this, but instead a playout adds one more node to the tree of explored moves (with various factors deciding whether to go wider to new variations or deeper down already considered moves), and then use the value network part of the combined policy/value neural network to ask the network, "who is winning this (non-terminal) board position?". The winrate you see is some kind of average of those answers.

[lightvector]

They are probabilities, and have an intrinsic meaning, but a very particular one. In one sentence:

An estimated proportion of games that would be won by the recent history of training versions of that bot from a position in self-play training conditions, but averaging across all positions in the tree of an MCTS search.

Which means they are:

• Not the chance that pro players would win from the position (since pros will not play like the bot).
• Not the chance that amateur players would win from the position (since amateurs will not play like the bot).
• Not perfectly accurate (because it's the neural net's guesses, not some God-given truth).
• Not the chance that the bot would win the game against itself using <insert your hardware specs and time controls here>. (because your specs don't match self-play training conditions).
• Not the chance that the bot would win the game against itself even in self-play training conditions! (this one is subtle. It's because of the averaging over the MCTS search tree, rather than being specifically an estimate of the that board position in the game alone. This difference is actually significant! Also because it's a prediction about the recent history of training versions, not the current version itself).
• Not necessarily an "objective" guide to good moves or good positions, even when the neural net is CORRECT about its self-play training winning chances. (because an objectively worse move could lead to objectively better self-play chances if the results are easier to understand and follow-up on for the net - and a net may know and correctly predict this - and what's easy for a net to understand may differ surprisingly from a human's, such as in ladders or semeai or tricky rare tesuji).

But they are useful numbers still because even though are NOT any of these things that you actually care about, they may statistically correlate with these things you care about. Then, way to use them is to not treat them as absolute guides, but rather to interact with a specific bot repeatedly and with experience let your own brain fuzzily learn in what situations the correlation is weak and unuseful, or tight and highly useful.

(edit: better wording)

[Knotwilg]

The winrate you see is some kind of average of those answers.

OK, but even if it's an aggregate, it's still a bot's best guess of winning probability? Or at least a relative measure which you can't compare against other bots but against other moves?

[Bill Spight]

First, the "winrates" are not probabilities. This has already been done to death on these forums.

I own up to not really understanding this. The numbers are measuring something, presumably. If a 57% win rate for White means White has a greater likelihood of winning, that's a greater probability in layman's language, surely?

As I understand from discussions here from those in the know, winrates are actually estimates of the probability that the bot, playing itself from the current position, will win the game, assuming other conditions that are not spelled out. (Edit: lightvector explained winrates in a previous note while I was writing this. ) I also gather that these estimates are not checked by having the bot play itself from those positions. Instead, they are used to help the bot choose its next play, and are indirectly verified by how well the bot plays. The point is that the developers of bots are (with some possible exceptions) trying to develop the strongest bots that they can. For that purpose the accuracy of the difference in winrates for different plays is more important than the accuracy of the winrates themselves. For instance, if a bot overestimates the winrate for White at move one, that is of little importance for the bot's level of play, as long as the winrate estimate is accurate at the end of the game. All of this pertains to the next quote.

Whether 57% is a lot or a little depends on how the software is calibrated.

Understood, but the overwhelming majority of comments (here and in Japan, and as with Shibano) seem to be Leela-based, so "57% = a lot" seems fine unless Leela itself is being constantly and significantly re-calibrated - is it? If so, that seems to throw a lot of earlier comments here out of the window.

IIUC, Leela has been constantly recalibrated, but in small increments. And the accuracy of the initial winrate does not have high priority. Besides, why should it?

AFAICT, the people who really understand winrates have not made much of an effort to explain them to the consumers of winrates, amateurs and pros who use winrates to evaluate plays and positions. I suppose that people who sell their go playing software have not done a good job of explaining how to interpret winrates.

we tend to put our pros on a pedestal, one part of Asian culture that the Western go community has well and truly embraced, and they don't exactly discourage that.

You won't get any arguments from me on that. But I would point out that there are people here who go to extremes the other way, too, and throw professionals out of the window. I suspect most people take the "medical" view. If you're very sick and you can't await future research or wait until debates settle down, would you (in the present state of knowledge) go to a professionally qualified physician or to a computer scientist? (I have to say "suspect" because the current measles epidemic may be due to too many people reading he internet and not getting their kids vaccinated. But I'm in the jab camp.)

To take the measles outbreak first, I don't think that we can lay that at the feet of computer scientists, but at cranks, kooks, sensational journalists, and, since I live in the US, politicians who ignore or even oppose science. As for physicians vs. computer scientists, that's a false dichotomy. Computer scientists may write software that physicians and medical researchers use, but that's about as far as their relevance goes. But physicians these days do not simply rely upon their expertise and personal experience, but also rely upon science. Back in the 1980s it was demonstrated that computer programs made more accurate initial diagnoses than general practitioners. Today, a GP is likely to order diagnostic tests. There are also protocols in hospitals and emergency rooms that doctors and nurses follow that were developed with the help of computer scientists and statisticians. In the local medical center that I go to there are hand sanitizers at the outside entrance and to the entrance of every patient's room. Everybody uses them when they come in and go out of a patient's room. OC, the medical profession has known about the importance of clean hands for centuries (Edit: more than a century. Semmelweis was born in 1818.), but it took statistics to reveal and verify the importance of cleaning them so regularly and often within hospitals.

[gennan]

It is known that correct komi depends on the players' levels, with weaker players needing less komi for a fair game (since B makes less use of his advantage).

I didn't know that (do you have a source to satisfy my curiosity?). Do you also mean that a full handicap stone is worth about 13 points at 7d level, but perhaps only 6 or so at 30k level?

Rank differences are measured by handicap stones. So when one would try to create a rating system separated by point gaps, the lower ranks would be significantly closer together than the higher ranks (somewhat similar as the true Elo gaps between lower ranks being much smaller than the true Elo gaps between higher ranks).

[gennan]

Even better would be to use komi 7.0 together with the button to avoid ties. Sure, with perfect play one side would still win 100% of the time, but, if komi 7.0 is a tie with perfect play, komi 7.0 plus button should achieve much better balance than 7.5 komi with any high-level play other than perfect play.

Why do we need a tie breaker for a perfectly played game? I feel that a tie is a prefectly fine result for a perfect game.

Even for human go tournaments, I feel that organisers worry too much about ties. Even when go has integral komi, chess and draughts have much more ties than go will ever have, and they can still run tournaments. I even feel there is some elegancy in having some go games ending in a tie. In my club we play with integral komi, and I think there is always some strange sense of satisfaction when a game ends in a tie, almost as if both opponents win.

[Bill Spight]

The other, more troubling, aspect is that so many comments are couched in terms like "mistake" or "loses a lot" or "that move is no longer playable". Since the comments here are essentially from amateurs (who can't really appreciate a 1-point initial difference according to Shibano), is this a case of what we might call "swagger"? Or is it case of me completely misunderstanding the numbers (as usual)?

Such terms are, IMHO, the result of the fact that humans are in general better at words than numbers, and also of the fact that we are still guessing what the numbers mean for pro games and for our games. Let me illustrate from my use of the word, playable. Unless otherwise stated, if I say that a play in a given position and similar positions is no longer playable, I mean that, based upon the numbers available to me, I think that pros will stop making that play in those conditions within a decade. Let me illustrate more fully with sanrensei, which you mention in your note.

I have just been reading something by Shibano Toramaru in which he is attempting to explain the disappearance of once popular fusekis such as sanrensei under AI influence. He makes the point that it is entirely down to win rates because White has found good counter-strategies, but that it could all change if EITHER Black eventually developed better counter-counter-strategies OR if komi was re-adjusted.

These counterstrategies are, I suppose, evaluated in terms of winrates, so it ultimately comes down to winrates, as Shibano says. But it also comes down to the development of counterstrategies. That is, the pros were not simply satisified by the winrate estimates of sanrensei itself, but used that information to help develop countermeasures to sanrensei. That is a good example of how to make use of winrate estimates.

Using the Elf commentaries I have done a small amount of research on sanrensei. You may take this as my opinion, based upon anecdotal evidence. One thing about winrates which may not be appreciated is that they depend, not only on the move in question, but upon alternative moves. This is different from evaluations based upon static points, which may remain approximately the same, even when better moves are available elsewhere. The existence of a better move will lower the winrate estimate of the move in question, as a rule. Now down to cases.

First, delayed sanrensei is almost certainly playable. That is, if the other side of the board has been developed and converting nirensei into sanrensei looks playable, it almost certainly is. This is the result of the lack of a better alternative in the other side of the board.

Second, if there is an open corner, sanrensei is almost certainly not playable. The open corner should be occupied. Not that this opportunity will present itself with any likelihood, but when it does, take it.

Third, if the opponent has played on the 3-3 in each of the other two corners, sanrensei is almost certainly playable. Elf estimates it as losing only 1% to par, which is well within Elf's margin of error.

Fourth, if you have nirensei opposite nirensei, which is a common situation, sanrensei is, based upon its current winrate estimate, questionable. Elf thinks that it loses 4% to par, AlphaGoTeach thinks that it loses 2% to par. Leela Zero, I don't know. Both of those losses are close to what I think is the borderline between playable and unplayable (for pros, as I said). Since this position occurs so often, I suppose that the pros have developed counterstrategies for it, and have decided that sanrensei is unplayable in that position.

Fifth, if White has played 3-4, 3-5, or 5-4 in one or both of the other corners, approaching a corner is better than sanrensei, and sanrensei will lose more than 4% to it, in Elf's estimation. That's enough, currently, for me to classify sanrensei as a minor error. A loss of 7% would be enough for me to opine that sanrensei in that case is unplayable. It seem like the pros are ahead of me in that regard for sanrensei.