Was the ear-reddening move a divine move?

Bill Spight · **#21**

Tryss wrote:

And there's the possibility that the network is sensitive to some very specific small perturbations (this added stone is the perturbation).

This article (about neural network in image classification) show something very interesting :

Quote:

Abstract

Recent research has revealed that the output of Deep Neural Networks (DNN) can be easily altered by adding relatively small perturbations to the input vector. In this paper, we analyze an attack in an extremely limited scenario where only one pixel can be modified. For that we propose a novel method for generating one-pixel adversarial perturbations based on differential evolution. It requires less adversarial information and can fool more types of networks. The results show that 68.36% of the natural images in CIFAR-10 test dataset and 41.22% of the ImageNet (ILSVRC 2012) validation images can be perturbed to at least one target class by modifying just one pixel with 73.22% and 5.52% confidence on average. Thus, the proposed attack explores a different take on adversarial machine learning in an extreme limited scenario, showing that current DNNs are also vulnerable to such low dimension attacks.

https://arxiv.org/pdf/1710.08864.pdf

I think our neural net are less sensitive to this, but these kind of things can happens

Back in the 90s I came up with the idea of testing evaluation functions by comparing two positions which differ at only one point on the board, or by moving a stone to a neighboring empty point (based upon the concept in psychology of just noticeable differences). In go the evaluation function (or neural network) should be very sensitive to some such perturbations, insensitive to others.

Edit: OC, the paper concerns perturbations that are unnoticeable to humans.

Bill Spight · **#22**

John Fairbairn wrote:

That seems to say, passing is worth more to Black than playing certain moves on the board. Counter-intuitive?

Not to me. Some moves are bad, some moves are very bad.

And what's worse on the empty board than the 1-1?

John Fairbairn wrote:

Could someone usefully comment on what I regard as slightly unusual behaviour by Lizzie/LZ (I am using rounded figures for simplicity):

Given Lizzie's margin of error, rounding to the nearest integer is correct.

Kirby · **#23**

Maybe this was answered in a different thread, but practically speaking, what does the difference in win percentage mean, concretely?

That is to say, if black is winning with 65% win rate, I interpret that as a board where black can win. If the rate is 75%, how does it differ in a practical sense? Blaknis still going to win, right?

dfan · **#24**

Kirby wrote:

Maybe this was answered in a different thread, but practically speaking, what does the difference in win percentage mean, concretely?

That is to say, if black is winning with 65% win rate, I interpret that as a board where black can win. If the rate is 75%, how does it differ in a practical sense? Blaknis still going to win, right?

The big relevant discussion is here, but the one-sentence version is that basically if Leela Zero produces a win rate of 65% in some position, it means that Leela Zero thinks that Black would win 65% of the time if it played itself starting from that position. (If you don't like probabilities for bot play, you can turn this into statements about the money odds it would want in order to place a bet on Black or White.)

Of course you can argue about the precise semantics of pretty much every word in that sentence, but that's the intent.

Regarding your second question, it is true that to God every* position should have an evaluation of either 0% or 100%, but bots are not as strong as that.

Bill Spight · **#25**

Kirby wrote:

Maybe this was answered in a different thread, but practically speaking, what does the difference in win percentage mean, concretely?

That is to say, if black is winning with 65% win rate, I interpret that as a board where black can win. If the rate is 75%, how does it differ in a practical sense? Blaknis still going to win, right?

Nope.

As dfan points out, the winrate is an estimation of how often the bot will win playing against itself many times from the current position or with a certain play. There are unstated assumptions about the constraints on the play, such as time limits.

Let us say that the estimate of a 75% Black winrate is correct. Then, as a number of self-play games continue from the current position or with the given play, 75% of the time the winrate estimate should approach 100% (with fluctuations, OC), until finally Black wins, and 25% of the time the winrate estimate should approach 0%, until Black loses. Given that humans are weaker than bots, we should estimate winrates for humans that are closer to 50%, unless we know better.

yakcyll · **#26**

I think the key question is how are specific playouts different from one another? Is it just a matter of following different branches at each level with different proportions? Obviously Leela's isn't flawless and it doesn't produce binary results, so there has to be some level of randomness incorporated in the process.

lightvector · **#27**

Since again there seems to be confusion about this, relating mostly to oversimplifications of the relevant ideas, here's an attempt to describe succinctly and accurately without any of those usual easy-to-misinterpret oversimplifications.

65% win rate means that following what the bot considers to be likely good play by both sides over the next few moves¹, on average the resulting positions are ones that the neural net believes are "similarly good" to positions it's seen in the training data where the player-to-move in that data won about 65% of the time². The training data usually consists of a slightly old-and-weaker version of that bot playing itself many times using a certain fixed number of playouts³, and with much heavier randomization than normal⁴.

¹Of course, occasionally the bot's reading may entirely be overlooking a good move by one or both sides.
²But still limited by the neural net's ability to understand and compare those positions. Larger nets will on average have better understanding. But they can still massively blunder/misjudge from time to time.
³In particular, this means that the 65% is NOT an estimate of how likely this version would be to win with the potentially very different number of playouts that you are running it with.
⁴More randomization provides the neural net with richer and more varied training data to learn from, but also means that the bot in the training data is much more likely to blunder than normal, which of course also affects the win % just like the other things mentioned in (³).

Tryss · **#28**

And my feeling is that we can somehow describe the percentage of LZ (but not elf) in human terms with something like this :

45-55% : equal game
55-65% : slight advantage
65-75% : noticeable advantage
75-85% : huge advantage
over 85% : won game (LZ will resign under 10%)

Also, note that the winrate are not independant during the same game, so if you want to test the accuracy of LZ prediction, you need to fix a move number, then test the distribution

For exemple, if we want to know if LZ predictions are accurate for a category of players (for exemple pro players, or low dan on internet), an experiment could be like this :

Take 1000 games of these players that didn't resign before move 120, evaluate for each the position at move 80, group the games by 5% LZ predictions, and see if the results are close to the predictions.

After that, we could also verify the "internal consistency" : take two data points for each games, and see how the predictions correlate with the true result. Are the earlier estimate more reliable than the second?

I may try to do a python program like this and test it on something like 100 games of strong sdk with 5 predictions group (0-15, 15-40, 40-60, 60-85, 85-100). (but no garantee)

Kirby · **#29**

lightvector wrote:

Since again there seems to be confusion about this, relating mostly to oversimplifications of the relevant ideas, here's an attempt to describe succinctly and accurately without any of those usual easy-to-misinterpret oversimplifications.

65% win rate means that following what the bot considers to be likely good play by both sides over the next few moves¹, on average the resulting positions are ones that the neural net believes are "similarly good" to positions it's seen in the training data where the player-to-move in that data won about 65% of the time². The training data usually consists of a slightly old-and-weaker version of that bot playing itself many times using a certain fixed number of playouts³, and with much heavier randomization than normal⁴.

¹Of course, occasionally the bot's reading may entirely be overlooking a good move by one or both sides.
²But still limited by the neural net's ability to understand and compare those positions. Larger nets will on average have better understanding. But they can still massively blunder/misjudge from time to time.
³In particular, this means that the 65% is NOT an estimate of how likely this version would be to win with the potentially very different number of playouts that you are running it with.
⁴More randomization provides the neural net with richer and more varied training data to learn from, but also means that the bot in the training data is much more likely to blunder than normal, which of course also affects the win % just like the other things mentioned in (³).

Thanks for the succinct explanation - and to others who have similarly added to the discussion.

Given this explanation, could you elaborate on what is happening when the win rate is changing as the number of playouts increases? E.g. if I let LZ sit there, the percentages start to change. Obviously, the training data hasn't changed, so something about the playouts happening right now are affecting the win rate, right? Is it just that it's finding different board positions a few moves ahead that match up to newly found positions that are similar to different training data positions (thereby adjusting the probability)?

Thanks.

hyperpape · **#30**

Tryss wrote:

I think our neural net are less sensitive to this, but these kind of things can happens

I don't know if you meant go programs by "our neural nets", but in a very real sense, our human neural nets are similarly vulnerable, as a recent paper apparently demonstrated: https://spectrum.ieee.org/the-human-os/ ... ial-images.

lightvector · **#31**

Kirby - yes, pretty much. For example, if in its current favorite move it reads deeper and discovers that it actually blunders a group to die without compensation that the neural net initially didn't perceive as killable, then the evaluation of that move will go way down. (Mechanically, this is because after searching deeper now the search tree for that move contains many positions where the opponent has played the killing move making the dead shape more obvious, which now the neural net correctly recognizes as very bad for the player whose group died).

Basically its doing something very similar to what you might do with more time to think - reading deeper and reevaluating based on that reading.

Bill Spight · **#32**

Kirby wrote:

lightvector wrote:

Since again there seems to be confusion about this, relating mostly to oversimplifications of the relevant ideas, here's an attempt to describe succinctly and accurately without any of those usual easy-to-misinterpret oversimplifications.

65% win rate means that following what the bot considers to be likely good play by both sides over the next few moves¹, on average the resulting positions are ones that the neural net believes are "similarly good" to positions it's seen in the training data where the player-to-move in that data won about 65% of the time². The training data usually consists of a slightly old-and-weaker version of that bot playing itself many times using a certain fixed number of playouts³, and with much heavier randomization than normal⁴.

¹Of course, occasionally the bot's reading may entirely be overlooking a good move by one or both sides.
²But still limited by the neural net's ability to understand and compare those positions. Larger nets will on average have better understanding. But they can still massively blunder/misjudge from time to time.
³In particular, this means that the 65% is NOT an estimate of how likely [i]this version would be to win with the potentially very different number of playouts that you are running it with.[/i]
⁴More randomization provides the neural net with richer and more varied training data to learn from, but also means that the bot in the training data is much more likely to blunder than normal, which of course also affects the win % just like the other things mentioned in (³).

Thanks for the succinct explanation - and to others who have similarly added to the discussion.

Given this explanation, could you elaborate on what is happening when the win rate is changing as the number of playouts increases? E.g. if I let LZ sit there, the percentages start to change. Obviously, the training data hasn't changed, so something about the playouts happening right now are affecting the win rate, right? Is it just that it's finding different board positions a few moves ahead that match up to newly found positions that are similar to different training data positions (thereby adjusting the probability)?

Thanks.

Emphasis mine.

lightvector wrote:

Basically its doing something very similar to what you might do with more time to think - reading deeper and reevaluating based on that reading.

Elsewhere I discuss some differences between Elf's evaluations of the same game with settings of 100K playouts and 200K playouts. As a working hypothesis I assume that the winrates with the 200K setting are more accurate than those with the 100K setting. OC, both settings are far greater than those in the training data, else we would still be waiting for those games to finish.

By that token what is being estimated is not clear. Arguably the two different settings are estimating different things. Another unknown factor has to do with the fact that as the level of play increases, the estimated winrates approach 0 or 1, so we should expect the winrates with 200K to be more extreme than the winrates with 100K. How much more we -- at least, I -- don't know.

Kirby · **#33**

Nice - thanks for the explanations.

Tryss · **#34**

hyperpape wrote:

Tryss wrote:

I think our neural net are less sensitive to this, but these kind of things can happens

I don't know if you meant go programs by "our neural nets", but in a very real sense, our human neural nets are similarly vulnerable, as a recent paper apparently demonstrated: https://spectrum.ieee.org/the-human-os/ ... ial-images.

I was talking about go programs, but thanks for this very interesting article!

Mike Novack · **#35**

Back to the initial question (because the discussion may have drifted off)

We may be missing the point. The matter might not be "what is the BEST move possible" but "what is the move in practical across the board play between two humans that makes the outcome CLEAR?"

In other words, human A has analyzed the position in terms of basing saving the game on a particular resource and human B makes a move (the ear reddening move) that makes it clear that this resource does not exist. THAT would explain the ear reddening.

Deeper analysis, by either humans using more time than over the board allows or by our current very strong go programs might reveal even better moves in the sense that they assure victory irregardless of that resource. But that might not be immediately clear (no ear reddening)

Knotwilg · **#36**

Mike Novack wrote:

Back to the initial question (because the discussion may have drifted off)

We may be missing the point. The matter might not be "what is the BEST move possible" but "what is the move in practical across the board play between two humans that makes the outcome CLEAR?"

In other words, human A has analyzed the position in terms of basing saving the game on a particular resource and human B makes a move (the ear reddening move) that makes it clear that this resource does not exist. THAT would explain the ear reddening.

Deeper analysis, by either humans using more time than over the board allows or by our current very strong go programs might reveal even better moves in the sense that they assure victory irregardless of that resource. But that might not be immediately clear (no ear reddening)

In my case my toes flush read when caught by surprise. I keep my shoes on.

Kirby · **#37**

Mike Novack wrote:

Back to the initial question (because the discussion may have drifted off)

We may be missing the point. The matter might not be "what is the BEST move possible" but "what is the move in practical across the board play between two humans that makes the outcome CLEAR?"

In other words, human A has analyzed the position in terms of basing saving the game on a particular resource and human B makes a move (the ear reddening move) that makes it clear that this resource does not exist. THAT would explain the ear reddening.

Deeper analysis, by either humans using more time than over the board allows or by our current very strong go programs might reveal even better moves in the sense that they assure victory irregardless of that resource. But that might not be immediately clear (no ear reddening)

So what we need is not a neural network trained to win games, but rather, a neural network trained to demoralize humans!

EdLee · **#38**

Hi Kirby,

Quote:

rather, a neural network trained to demoralize humans!

Unfortunately, this is also past tense... for some pros.

Mike Novack · **#39**

Kirby wrote:

So what we need is not a neural network trained to win games, but rather, a neural network trained to demoralize humans!

Well ........ first of all, what we have currently are neural nets trained to "find the best next move" and "analyze chances for winning for each side". That is NOT quite the same thing as being "trained to win games". If behind, the objectively best move, but one which keeps things clear and simple might not be as good as a move not quite as good in objective terms but which made the game very complicated with lots of tempting ways for the opponent to go wrong. I ahead, the reverse might be true. In terms of the objective of winning the game.

We MAY be able to train the nets to do this (there may be difference between "simple" and "complex" in terms of the number of alternatives close in evaluation)

I don't think we have been training these neural nets to examine a sequence of moves made by the opponent to predict "what is the opponent aiming at?" << and thus, figuring out how to block that >> That appears to me to be a very different problem.

Kirby · **#40**

Mike Novack wrote:

Kirby wrote:

Well ........ first of all, what we have currently are neural nets trained to "find the best next move" and "analyze chances for winning for each side". That is NOT quite the same thing as being "trained to win games". If behind, the objectively best move, but one which keeps things clear and simple might not be as good as a move not quite as good in objective terms but which made the game very complicated with lots of tempting ways for the opponent to go wrong. I ahead, the reverse might be true. In terms of the objective of winning the game.

If the win probabilities are based on positions similar to the training data, as I understand, the win percentage would represent the probability that the bot (or maybe an earlier version of it) would win against itself. Making things complicated or simple shouldn't have an impact, unless it results in a difference in calculated win rate for the bot.

Playing against a human might be different. The bot wasn't trained to win against humans, so the move that gives the best win rate for bot vs. bot might be a different move than the move that has the best odds of beating a (given) human.

Was the ear-reddening move a divine move?

Who is online