Bots really strong when receiving handicap?

[SpongeBob]

When looking at bot-against-human games where the bot receives handicap stones according to the rank difference, it seems that the human player struggles hard and does not reach the 50% winning percentage. At least this is the impression I get when I look at some KGS games.

When I think about it, I come up with this explanation: Bots are playing much more 'consistently' than humans, basically meaning they do not 'blunder' that often. When I play handicap with white against a human player, I would say that black could normally win these games if he pulls himself together and does not mess up someplace. These blunders from the black player occur less often when a bot is playing black.

[ez4u]

Check the series of games between bigbadwolf 8d and zen19d 6d on July 11th. One game on three stones and six on two stones. Zen won the 3-stoner but bbw was 4-2 on two stones. Just a little counter example to you impression.

[Mike Novack]

When looking at bot-against-human games where the bot receives handicap stones according to the rank difference, it seems that the human player struggles hard and does not reach the 50% winning percentage. At least this is the impression I get when I look at some KGS games.

Some games? Informal estimate?

Doing statistics (correectly) isn't as easy as it might at first seem. If you have a background in it we could presume you knew how to take a proper sample from which you could have drawn a conclusion (that not 50%). But in that case you would have likely told us the "confidence level" to be attached to the conclusion (the confidence that your conclusion meant something, not pure chance).

The point is that if you flip a fair coin 10 times the result will be something other than 5-5 mpre likely than not. Would you conclude from a result of 4-6 or 3-7 that the coin was weighted?

[speedchase]

Check the series of games between bigbadwolf 8d and zen19d 6d on July 11th. One game on three stones and six on two stones. Zen won the 3-stoner but bbw was 4-2 on two stones. Just a little counter example to you impression.

I think this is more evidence of the theory that bots lose their strength when their weakness is studied.

[ez4u]

Is it known that bbw has studied the weaknesses of zen (or bots in general)?

[topazg]

Is it known that bbw has studied the weaknesses of zen (or bots in general)?

I think the implication is that simply noticing habits of the program in the earlier games increased BBW's performance in the later ones.

[ez4u]

Is it known that bbw has studied the weaknesses of zen (or bots in general)?

I think the implication is that simply noticing habits of the program in the earlier games increased BBW's performance in the later ones.

Hmmm.... You think the sequence 0 (3-stones) followed by 1,0,1,0,1,1 (2-stones) really shows that?

[Mike Novack]

Unfortunately (and in spite of my earlier comment) probably does believe that.

What ez4u is saying (and I most certainly agree) is that if you see a sample of just 6 splitting 4-2 that does not justify any conclusion let alone the one implied.

OK --- for six trials flipping a fair coin the chances are less than 1/3 for you to get 3 heads and three tails. Just under 1/2 for you to get either 4-2 or 2-4 (combined. It's ~1/5 might split as badly as 5-1 or 1-5 (combined) and ~1/30 might be as bad as all heads or all tails (combined). That's all just pure chance, not bias of the coin.

And while the MCTS have some particular behaviors not quite proper to describe as weakness. More a matter of "style of play", loose, leaving positons unfinished with potential all over the board. Some humans play this way too. Now if you have not before encountered this style of play or encountered it only very rarely there may be a learning curve so you might improve after some experience. But that's less a matter of learning the program's weakness as eliminating one that you had (misjudging how to play against this style).

[ez4u]

...

What ez4u is saying (and I most certainly agree) is that...

"That is as well said as if I had said it myself."
- Jonathan Swift. 1667-1745
Better actually

[emeraldemon]

Most modern Go engines try to find moves that maximizes the win percentage of playouts. They make no distinction between B+0.5 and B+10.5, only between B+ and W+ . While most humans understand that winning by more isn't the same as being more likely to win, I think we may not be quite as good at adjusting out play to be safe when we're ahead and dangerous when we're behind. In handicap games this translates to Bots being good at not taking risks against stronger players.

[hyperpape]

It's not obvious, though, since at the beginning of the game, the bot is very far ahead, and so the engine may play very slack moves if it is just trying to maximize win probability. Thus, there's a lot of research into the question of whether you can use dynamic komi (i.e. pretending to give your opponent a large komi at the beginning of the game, and slowly reducing that as the game progresses) to get better results.

I suspect your effect is real, Emeraldemon, but which effect predominates is unclear.

[speedchase]

Hmmm.... You think the sequence 0 (3-stones) followed by 1,0,1,0,1,1 (2-stones) really shows that?

Actually I misread the first post. sorry

[ez4u]

Most modern Go engines try to find moves that maximizes the win percentage of playouts. They make no distinction between B+0.5 and B+10.5, only between B+ and W+ . While most humans understand that winning by more isn't the same as being more likely to win, I think we may not be quite as good at adjusting out play to be safe when we're ahead and dangerous when we're behind. In handicap games this translates to Bots being good at not taking risks against stronger players.

I would think this would translate to bots giving away too much of their advantage early, when their lead is the greatest. In other words they are too willing to compromise on safe moves as long as their playouts show they are still ahead even if only by a minimal amount.

[speedchase]

I would think this would translate to bots giving away too much of their advantage early, when their lead is the greatest. In other words they are too willing to compromise on safe moves as long as their playouts show they are still ahead even if only by a minimal amount.

I think they use adjusted komi to compensate

[Mike Novack]

[
I would think this would translate to bots giving away too much of their advantage early, when their lead is the greatest. In other words they are too willing to compromise on safe moves as long as their playouts show they are still ahead even if only by a minimal amount.

You aren't looking at the consequences of what you have just said.

If it is true that there are always safe moves that retain a small but positive advantage that eventually results in a won game. Not unlike a match sailboat race (race between two boats) where the one in the lead maneuvers not to increase its lead but to always remain between the boat behind and the next mark.

Remember, the MCTS programs are not seeing how much the lead has increased or decreased form the next move but playing games out to the end and seeing who eventually won. Remember that it's up to the player behind to start taking chances (and the player ahead responding to that).