For a fixed neural network, (e.g. choose and fix a specific one of the Leela Zero networks for use in Lizzie), so long as the neural network is not too vastly stronger than me (e.g. Elf), I find I can readily develop a sense of what the neural network's winrate corresponds to over the course of a few game reviews with it of slow-paced games I played. I bet you can too if you try the same.

For slower-paced games where I have time to think as well as to count a few times over the course of the game, like everyone else, I have my own sense of "how much" I'm ahead. And for me, that "how much" feeling is definitely nonlinear in points. If we're headed into endgame and I've counted that I'm ahead by 5 points taking into account who has sente, against an equally-ranked opponent that feels to me like quite a solid buffer and hard to lose, whereas if it's still midgame (e.g. I'm ahead by 5 points in solid territory, and center access and influence and development potential all seem roughly equitable, but there are still invasions and fighting yet to happen), then it's really anybody's game still.

Over the course of doing several game reviews with a fixed neural net, I find I can pretty readily associate the neural network's winrates with my own sense of aheadness that I felt during the game. I'll find that when I felt so-and-so-much ahead the neural net will typically say numbers from 80%-90%, and when I felt so-and-so-much slighly-behind the neural network will typically be saying numbers from 30%-40%, etc. Sometimes it will say numbers that that are very different and outside of that range, because my own evaluation is just way off and I was misjudging something, because of course the bot is much stronger than me. But on average through repeated interaction it seems pretty easy to me to develop this intuitive correspondence. Then, when the bot reports a percentage change like 5%, I have a very intuitive "how much" that is - it's 1/6 of the the amount of "aheadness" that I typically associate with the bot saying 80% versus an even game 50%.

The trick is that you have to fix the neural network, and you have to be using it actively reviewing your own games so that your intuition calibrates to it. Different neural networks from different sources (like ELF vs Leela Zero) will have very different confidence scales, unsurprisingly typically the stronger neural nets will give values that are much more extreme for any given fixed "amount of advantage", because of course the stronger a player and opponent both are, the more a fixed amount of advantage is likely to result in the advantaged player winning.

So that does mean that when you see *other* people make posts saying this or that bot spit out this or that winrate, it's hard to interpret, because what the number means depends heavily on what neural network they're using, and it may easily not be one that you've used enough yourself to get a feel for. That's definitely unfortunate, and means communication about winrates on a forum like this often rightly feel adrift and ungrounded. But as for simply reviewing your own games, winrates from the current neural networks are already quite usable once you calibrate yourself to one particular network, just don't use one that's so much stronger than you that you can't do that (e.g. ELF network shooting up to saying 95% win the moment someone makes a small opening error).

Sorry I'm joining this conversation a little late...



Perhaps I am getting too hung up on a word, but to me, for Black to be "really ahead" suggests the game has been solved. We aren't estimating who is ahead - Black is really ahead. Yet if it has been solved, we would see a winrate of 1 or 0, as someone mentioned earlier.

Implicit in the winrate, therefore, is the idea that the game hasn't been solved and that maybe this question cannot be answered to everyone's complete satisfaction.



I think this may depend on where you are in the game. At some point (perhaps in yose) the winrate will spike close to 1 or collapse close to 0 when things become certain. Suppose White has a bad position (low winrate) at move 100. It plays its best after that, but cannot overcome its bad position and loses. I would imagine Black's winrate will rise after W makes the best moves she can, if for no other reason than we are getting to the end of the game and White's chances to turn the game around are disappearing.

Glad to have your input.



People use fuzzy language like ahead, behind, having the edge, having chances, etc., all the time. Furthermore, people may disagree in their assessments, even experts. Which means that people make mistakes. So player A may say, Black is ahead and then Player B may say, no, Black is really behind. Usually we do not have an objective and practical way to decide whether A or B is right, but in this case we may. Let suitably strong and matched bots play the game out from that position many times. If White usually wins, then Black was not really ahead. If you will, this is a very weak form of solving the game.



I agree that whether we want to call a play that loses 3% in winrate a mistake may differ in different parts of the game. In the recent past, MCTS bots' winrate differences in the endgame have definitely been peculiar. My impression is that the current best NN bots are better in that regard, but I don't really know.



IIUC, in theory that is supposed to happen only some of the time. E.g., if at move 200 Black has a winrate of 80% then 20% of the time Black's winrate should drop to 0 by the end of the game.

Edit: Both of these questions may also be addressed if we have error estimates. So if Black is estimated to be 55% ahead, with an average error of 10%, I would hardly say that Black was really ahead. But if the average error was 1% I would be willing to offer my opinion that Black is really ahead. And if White's play increased Black's winrate by 3% my confidence that it was an error would depend upon the error estimates of both winrates. And the error estimates should generally drop as the end of the game approaches.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

In the not too distant past, some MCTS bots evaluated the game in terms of points. As I understand, they were not as successful as those that evaluated the game in terms of winrates. In my considered opinion, evaluation by points requires the concept of temperature, as well. For instance, suppose that you are 2.5 points behind but it is your move. If the temperature is 7 you have good chances to win, but if it is 3 you do not.

I know of no reason that neural networks cannot make evaluations in terms of points and temperature, but winrates continue to be successful, so who is going to give such an approach a try?

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I don't think it is reasonable to expect a deviation term from the SAME SOURCE as the winrate (estimated probability of winning). If a bot would think it's chances are 55% with high potential error, he could adjust that towards 50%. It seems provable that for any given set of information, the winning probability (it's best estimate) always collapse to a single number.

If you are interested in how good these estimates are, their practical correlation to actual game outcomes in the long run, you could measure this nicely as I suggested earlier. But with such accuracy table available, you or the bot itself could again adjust the bot's winrates/guesses (depending on game phase etc. if you have such data), so it would again collapse to a single (corrected) net probability.

It would be different if the bot would evaluate to an estimate of score (some already do this btw), a normal-ish distribution with a deviation term. This deviation (and more extra data) would be meaningful for some decision making (like small sure win is better than high EV with high deviation). But even then you could calculate / collapse to a win probability in the end - a single "goodness" value is necessary for sorting your options to choose the best one.

I don't think so. That is, even with an estimated winrate, you can also have an error term. However, they are not giving an error term, so there we are.

I am suggesting other ways of getting error estimates.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I am also joining late in this discussion. I ran a lot of tests using various configuration of bots.

Test 1: top pro game. Analysed by Leela Zero network #157, but using the Ray machine (because it knows about ladders). Go review partner was used to run the analysis; GPU was a 1050 or 1080 Ti.



What this means that with 1600 playouts the black moves were valued 3.24% (on average) below the choice of the bot and that the standard deviation of these differences was 4.84.

So for this game the evaluation was unstable until about 50,000 playouts.

Edit 2018-08-12: ran the 1M playouts per move last night. Still no sign of further convergence.

Test 2: 5d amateurs game, same setup

Part of the problem is the mindset of trying to solve this with one AI. More things would be possible with multiple strengths exploring the same position. A position that is won for Ke Jie is not necessarily won for me. I don't care if a bot says it thinks it has a won position. I want it to simulate what I would do, or an opponent of my level would do.

Yes, winrates depend upon strength.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Could you give an example of such a dual evaluation where it is not possible to collapse it to a single probability of winning?

All it requires is for the "collapsed" probability to equal the estimated probability. The error term does not disappear.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I used to work on SQL Server, and when you run a query, you can get the query plan, ie., what steps the computer wants to take to run the query. There are many different ways to execute a query to get the same result, but the computer picks the plan having the best cost.

Seems reasonable, but what does it mean to have best cost?

Long ago, some guy did some benchmarking on different parts of query execution, and the cost could be correlated to runtime- probably from some computer in the 90s.

Nowadays, query "cost" has no meaning in itself - it only matters that the computer chooses the cheapest one.

I feel the same way about win percentages. We can use the relative values of candidate moves to see what the computer might select in that position, but drawing more than that seems to be a stretch to me.

_________________
be immersed

I think it does (for binary probabilities). For example, a 60% estimation will remain 60% only if the error/deviation is 0 (no new info), but will become 50% if the error term is very large ("this is the kind of situation where our estimation function is almost always wrong, does not correlate well to experiments, so in reality we know almost nothing") and so on. You didn't say where the error data comes from (external/internal), but in both cases it becames part of our given set of information (input for the revised estimation function).

Even if one tries to make a theoretical distinction between being sure to win half of the games from here, or having no clue about who is ahead (doubtful already, as far as win probability goes), once the relative value of these two is decided the bot gets the single scalar goodness / win confidence value for sorting. (All this is similar to binary distributions that cannot have a variance independent of ev.)

I think you are confusing a probability with an estimated probability. There's a whole literature on this, and our opinions are of little interest.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I have the impression that the term "winrate" is used for different things:

• WR_raw: the winrate estimated by the raw neural network. For a human this would correspond to picking the most intuitive move and try to guess at a glance the chances of winning.
• WR_n: the winrate estimated after n playouts, where n is a large number (given Jan van Rongen's tests above, n=50000 would be a good compromise between accuracy and computation time). For a human, this would correspond to estimating winrate after deep reading.
• WR_true: the limit when N tends to infinity of the proportion of won games when N test matches are run starting from the position.

The best way to estimate WR_true would be to run a large number N of test matches and calculate the proportion of won games (this is what AlphaZero did to create its teaching tool). Call this proportion p. The estimate of WR_true is p, and we can estimate the error by 2 sqrt(p(1-p)/N) (if we want a confidence interval of about 95%).

However we usually don't do that because it's too computationally expensive, so we use WR_raw or WR_n (or WR_m for a smaller number m, like m=1000) as estimates. The number WR_n takes more time to compute, but is probably a better estimate, than WR_raw.

So currently we consider that WR_n is a good estimator of WR_true, but we don't know how large the error |WR_n-WR_true| can be. It might be possible in the future to train a computer to give a good estimate the error, but for the moment we can't do that. Perhaps |WR_n-WR_raw| gives an idea of the magnitude of the error, but some tests would be needed to determine whether this is true.

I hope I'm not (similarity, not identity). What I say that even if you can make the theoretical distinction, for a bot's perspective there is no viable way to maintain or make use of an error term, since the best corrected guess of win probability will assimilate it, the bot can not act on it (even on exact, externally and posteriorly measured accuracy data). It can only be used outside of the process of finding the best move, for other purposes.

If you use "playouts" in the same sense bots do, then note that for MCTS search this latter, true value can only be 0 or 1 (convergent to minimax solution), very different to a win proportion in selfplays starting from the position.

This assumes sample independence, which is not really true here. Consider an early position with a few stones. Most of bot estimates will be around 50%, both raw net and long searches, and also millions of selfplays will result in similar values. Which is very different from the correct value of 0 or 1, the result of nearly-infinite search. If you do longer and longer searches to the extreme, you will see the winrate estimate somewhat stabilizing first, but start to change heavily later. (btw Alphago teach only did long searches, not selfplays from positions IIRC)

But we can at least do the posterior calculation of correlation of bot winrates to actual outcomes.

Hi,
Sorry, I haven't read everything yet, but, would it make more sens if we replace "Win rate" by "Confidence"?

So a 55% confidence means LeelaZero is confident she would win 55% of the games starting from that current board position, against an opponent of the same strength.

So now, if for the same position, ELF has a confidence of 68%, it does not really contradict with LeelaZero own confidence, it's just that ELF has more confidence for that game position than LeelaZero.

If human players were making different confidence statement for a same board position could be a matter of difference of level, or style. And 2 human players of same strength could reasonably have different confidence level for a similar position.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

I do not like the term win rate either, it is an estimated value for a move on a (0,1) scale. So the neural net when fed with a position gives for each (legal) move two values: the plausibility that it will be played and the estimated value. Both are fed into the MCTS to generate plausible sequences of playout. That means that many implausible sequences will never be considered within a compuattionally feasonable number of playouts.

The Win_raw_ is thus of limited interest, it's like playing with 1 playout.

Next you could wonder is the bot will converge to a single opinion when the number of playouts increases. In the pro game above it did not: the difference remained at 30-35% between the 100K and 400K playout runs.

So there is no concept of "the best move" from a bot; it is conditional to the number of playouts. Then hopefully the "win rate" converges? It does not. The average deviation between 100K and 400K playouts is still above 1% (5% for one particular move).

The bot doesn't use the error term. We do.

Edit: The fact that bots do not always choose the play with the best winrate shows that they have different ways of dealing with winrate uncertainty. If they needed error terms, they would calculate them.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

For now I consider the winrate: The bot "feels" this is better.

With the AIs available today you always have to play out the variations for a few moves to get a better picture and sometimes big changes in winrates occur after a few moves.