Zen beats pro with 2 stones handicap, another at EGC?

[Uberdude]

Putting DeepMind's helicopter airlift onto the summit of the Go AI mountain aside, Zen and CrazyStone and the other bots are making good progress. Last month Zen beat Takemiya Yoko 6p (son of Takemiya Masaki) with 2 stones handicap (game below), a first for a commercially available program. In a few days (27th, 4pm UTC) it will play Cho Hyeyeon 9p (who was the 2nd strongest woman in the world for most of the 2000s behind Rui Naiwei, and may be known to some for her English-language blog and creative life and death books) at the EGC in Russia. She is ranked #270 in the world (3149) on goratings.org to Takemiya Yoko's #766 (2818) so should pose a stiffer challenge. For reference Fan Hui 2p, who AlphaGo beat even in October 2015 is ranked #539 (2984) though I think that may be a bit of an overestimate given most of his wins are against Europeans who were probably seeded with too high ratings; Hajin Lee 3p (Youtuber Haylee) who beat Crazy Stone in a rather embarrassing even game last month is #574 (2960).

[pookpooi]

I'm confused because http://www.computer-go.info/h-c/index.html report that yoko won, not zen.

[macelee]

This isn't the first time. A few weeks ago there was an event in China in which Li Yuanqun (#566 goratings) beat Zen with 2 handicap stones (although the program led for most of the time and only lost by playing a silly yose move). Then a strong amateur 6-dan also managed to beat Zen with h2. Two weaker amateurs (about 5d) scored 1-1 in even games against Zen. In a final game a pro tried a h3 game and lost.

[Uberdude]

I'm confused because http://www.computer-go.info/h-c/index.html report that yoko won, not zen.

That too briefly puzzled me but then I checked the SGF which both says Zen won and counting manually confirms it, so concluded it was a mistake.

[Uberdude]

This isn't the first time. A few weeks ago there was an event in China ...

How many is a few? Takemiya Yoko's loss was 7th June. But you say Zen didn't beat a pro with 2h, just almost beat until a silly. You say Zen beat a pro on 3h but that already happened a few months back when it beat Kobayashi Koichi (see http://lifein19x19.com/forum/viewtopic.php?f=18&t=12906). Also when we get reports of "playing Zen" one has to be careful what version/strength: I recall illluck commenting that when Luo Xihe was bragging about beating it on 9 stones it was set up to play far below its strongest. But it doesn't surprise me a strong Chinese 6d amateur can do better than a mid-ranking Japanese pro .

[macelee]

I cannot find the precise dates but they were held about 3 weeks ago. They were using the latest commercially available version, using a server type of machine (which is stronger than average desktop/laptop PC), and giving the AI 60 - 120 seconds each move.

In a separate event held on 14th July Zhang Yiming 3p also played a h2 game and won easily. This time the AI was ran on a very powerful machine (a recent XEON 32-core CPU with 3 minutes per move).

From these games I can see that the yose level on the AI is very weak. Let me put it this way: if the AI could not build a very large lead after mid-game stage, it would have no chance against players around my level.

[lightvector]

From these games I can see that the yose level on the AI is very weak. Let me put it this way: if the AI could not build a very large lead after mid-game stage, it would have no chance against players around my level.

Possible, but are you sure you're not being misled by the tendency of MCTS to give up points when ahead? Because normally MCTS only cares whether or not it wins the game, rather than how much it wins by. If I recall, even AlphaGo made many obvious endgame "mistakes" when it was ahead in the cases where those points were not actually relevant to the final outcome of the game.

[macelee]

I don't believe that I am misled. I played about 30 games by myself. It looks to me that these AI programs don't choose yose moves based on their values. They do give up points quite easily when they are leading. They also don't know what to do when they are behind which is a big problem. In several games I played with Crazy Stone, when I managed to build up some comfortable (few points) leads, the program simply entered a crazy mode, playing insane moves, such as dead in gote, or ignoring by sente moves etc. and ended up losing 100+ points.

[Krama]

2 stones against some mediocre pro?

Meaning it still takes 4 stones against top chinese or korean pros.

Maybe 3 but that is probably not a big of an improvement since last year.

[hyperpape]

It's still not clear whether or not you get the point. There are three questions:

1) Does the AI give up enough points in yose to lose games it was winning?
2) Does the AI make bad moves and lose games it had a chance to come back in?
3) Does the AI play dumb moves in yose that don't change the outcome?

Only 1 & 2 affect the outcome of the game. 3 is unpleasant and obnoxious, but doesn't reflect the program's "strength", only how enjoyable its games are.

I can't figure out which you're describing from your post. It sounds like (2) might be happening, but I'd like you to be more definite.

Edit: My apologies. I see you said that the yose move made the program lose. I was looking at your later response and didn't see that it was clear before.

[Mike Novack]

I don't believe that I am misled. I played about 30 games by myself. It looks to me that these AI programs don't choose yose moves based on their values. They do give up points quite easily when they are leading. They also don't know what to do when they are behind which is a big problem. In several games I played with Crazy Stone, when I managed to build up some comfortable (few points) leads, the program simply entered a crazy mode, playing insane moves, such as dead in gote, or ignoring by sente moves etc. and ended up losing 100+ points.

You are a much stronger player than we are, but like Hyperpape says, its isn't at all clear you understand what the program is doing << we might be stronger than you in analyzing the behavior of a computer program >>

The fact that you say things like "these AI programs don't choose yose moves based on their values" is sort of an indicator of that. No, they aren't "thinking like a human". They aren't counting points at all. They are choosing the move based on "more likely to result in winning the game". You are making the assumption that selecting the move that maintains or increases the lead in points would do that and perhaps as a human have to think that way. Seems so obvious to you. Especially as you have no access to a thought process that could return "probability of winning the game".

In other words, if in a game where before entering small yose the program was leading by about 10 points and you saw it in small yose squandering 9 1/2 of those points you are deciding that if it had been only 9 points ahead it would have lost. Wrong understanding of what the program is doing.

Take a closer look at the "wrong moves" made by the program. Did they change the outcome?

[Bill Spight]

From these games I can see that the yose level on the AI is very weak. Let me put it this way: if the AI could not build a very large lead after mid-game stage, it would have no chance against players around my level.

Possible, but are you sure you're not being misled by the tendency of MCTS to give up points when ahead? Because normally MCTS only cares whether or not it wins the game, rather than how much it wins by. If I recall, even AlphaGo made many obvious endgame "mistakes" when it was ahead in the cases where those points were not actually relevant to the final outcome of the game.

MCTS "cares" about the "probability" of winning, but that probability is based upon more or less random playouts. It has not been demonstrated that that approach produces fewer endgame losses than making the largest play -- even though that simple strategy is flawed.

[Uberdude]

Take a closer look at the "wrong moves" made by the program. Did they change the outcome?

A few weeks ago there was an event in China in which Li Yuanqun (#566 goratings) beat Zen with 2 handicap stones (although the program led for most of the time and only lost by playing a silly yose move).

[Uberdude]

2 stones against some mediocre pro?

Meaning it still takes 4 stones against top chinese or korean pros.

Maybe 3 but that is probably not a big of an improvement since last year.

Indeed, the headline of Zen beats pro on 2 stones for the first time isn't so good when it's a weak pro. I'm not sure which is more impressive, beating Kobayashi Koichi 9p (3141 goratings) on 3h or Takemiya Yoko 6p (2818) on 2h. That's why I'm looking forward to the EGC match with Cho Hyeyeon (3149) as winning that on 2 stones would pretty clearly be better than the win against Kobayashi. Hopefully it will be broadcast.

[lightvector]

From these games I can see that the yose level on the AI is very weak. Let me put it this way: if the AI could not build a very large lead after mid-game stage, it would have no chance against players around my level.

Possible, but are you sure you're not being misled by the tendency of MCTS to give up points when ahead? Because normally MCTS only cares whether or not it wins the game, rather than how much it wins by. If I recall, even AlphaGo made many obvious endgame "mistakes" when it was ahead in the cases where those points were not actually relevant to the final outcome of the game.

MCTS "cares" about the "probability" of winning, but that probability is based upon more or less random playouts. It has not been demonstrated that that approach produces fewer endgame losses than making the largest play -- even though that simple strategy is flawed.

Indeed!**

Having followed and understood most of the research on computer Go and having observed how MCTS behaves in practice for many years now, here is my best knowledge of the situation:

- Go programs do fairly poorly if they end up behind in even or low-handicap games because the wins they perceive are from mistakes in the playouts that are likely to be significantly below the level of play of the opponent. The distribution of mistakes a human will make falls off far more sharply than that of a playout as the mistake gets simpler and more extreme. So if a program "sees" no way to win except to try to elicit a clear mistake from the opponent, then since it has no good model of what mistakes are likely and perceives many extraordinarily unlikely mistakes to be merely somewhat unlikely, it will try to elicit those extraordinarily unlikely mistakes and do things that are easy to refute. And the more extreme a mistake it sees as needed, the more trivial and easy to refute. Modeling and exploiting the distribution of mistakes an opponent is likely to make in reality is a very hard problem and is, as far as I know, unsolved.

- Go programs do fairly well if they end up ahead. I have only rarely seen a strong program give up enough against an equal opponent to actually lose a game that it was winning at the onset of endgame. Solidifying your critical territory (the portion you need to keep to win) to the point that even a blitzing-misclicking-mid-kyu-player** would successfully defend it against all break-ins and would not be capable of flubbing the remaining unfinished borders is actually not a terrible way to secure victory. It works fine if you can afford the moves to do so, even if it looks weird and involves giving up obvious other territory (such as the portion in excess of what you need to win). Unlike humans, bots don't have the issue of miscounting what they're expecting to get, so close margins aren't as risky for them. The cases where this backfires are when the program is terribly mistaken to begin with about what it expects to get. This happens with messy semeai or sekis, or with unusual tesuji that a bot entirely overlooks. But on the whole, strong Go programs on appear to be on average better at winning won games than human players of equal overall skill, even if the moves they use to do so look stupid.

- It's easy to underestimate the strength of a Go program if you try to judge it based on its moves. In both of the above cases, the play looks stupid and far below what one would expect from a human of comparable level. But the vast majority of the stupid looking moves are ones that the program will almost never make if that move is the difference between being ahead and being behind, and it will only rarely ever fall behind as a result of such moves. If you are attempting to judge the strength of a Go program by its moves, mostly you should focus on the quality of moves it makes in positions where you would neither be confident that you would win or that you would lose even after being given a little time to count and judge the position.


**One minor quibble: "random playout" is misleading because it connotes that the playout is uniformly random or otherwise completely ignorant. In the strongest Go programs, closer to the truth would be "played out loosely resembling a mid-kyu-player blitzing moves as fast as possible, with some chance to misclick". That is, vastly better than uniform random would be, and cognizant of liberties and basic shapes and tactics. But also likely to mess up a bunch and vastly weaker than the overall program.