--New Facebook Go AI - Can You Tell Who? A Turing Test--

[Starstorm3]

So I'm sure most of you have seen the article(s) on Google and Facebook's new competition to create a pro-level Go AI. But you may not know that Facebook has been testing their program on KGS for a few weeks now. It combines neural nets with Monte Carlo, which has produced some fascinating results. Bur when the article noted that it "felt human" in its moves, I knew I had to test it out (plus to see its strength). As anyone here can attest to, pure Monte Carlo AIs don't play human-like at all. So I just wanted to give my thoughts on it, and conduct a little Go experiment to boot, to anyone who cares. (For reference point, I'm KGS 4/5d myself, and I played it in an even game, random color)

Well, it managed to achieve 3d on KGS pretty quickly, so they seem to be off to a solid start. I have to say that I agree with the article's conclusions on that point entirely - its moves were extremely human, except for some extra-desperate endgame moves that humans would scoff at as insulting. It also doesn't understand the concept of "you can't win two 1 point kos at the end". But its general ko-playing skills are much improved from pure Monte Carlo I think.

As for its playstyle, it was at times conservative I thought. On the local level, it probably played nearly as well as myself. It avoided making as many stupid moves as I did, but it also played too conservatively or didn't take sente when it should have. But in general, it defended where it was supposed to and tried to attack my weak stones. However, globally, as one might expect, it was a bit weaker. I made some (very) foolish moves to be sure, but I won the game.

Finally, the experiment: as a Turing test, I've included the first 150 moves of a game I played against the Facebook AI AND the first 150 moves of a random game I played against a human player.

WHO IS THE AI?




Have fun, Go world!!

[hyperpape]

I went with Random Player 2. I think random player4 is out because of the broken shape early on in the lower right. Beyond that, I don't have a strong feeling, but player2 seems "restrained" and natural in the way that I associate with the bot, having watched several games.

I would give that less than 50% chance of being correct, though.

[DrStraw]

I would assume a bot knows joseki so Randomplayer4 is eliminated based on move 15.

I eliminated RandomPlayer 1 based on move 10.

Hard to say between the other two but I am inclined to think that that maybe RandomPlayer 3 is the bot.

[Jhyn]

I think RandomPlayer2 feels too pacific and japanese pro-like to be found on KGS, but I didn't give it much thought.

[MinjaeKim]

By the way this is a playable sequence for both as far as I know. Black plays 'a' to keep white not alive while taking both sides, and 'b' is to take sente.

$$c next 'a' or 'b'
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . 2 . . . |
$$ | . . . . . . . . . . . 9 . O . . 0 b . |
$$ | . . . , . . . . . , . 3 8 4 1 , X a . |
$$ | . . . . . . . . . . . . 6 5 . . . . . |
$$ | . . . . . . . . . . . . . 7 . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------

Click Here To Show Diagram Code

[go]$$c next 'a' or 'b'
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . 2 . . . |
$$ | . . . . . . . . . . . 9 . O . . 0 b . |
$$ | . . . , . . . . . , . 3 8 4 1 , X a . |
$$ | . . . . . . . . . . . . 6 5 . . . . . |
$$ | . . . . . . . . . . . . . 7 . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

[Uberdude]

Interesting post, my vote is for the bot being:

Player 2, with white being the OP in both games. 2nd choice would be player 4. Apart from the somewhat passive shape based play, one of the biggest clues for me was the failure to atari at j15 for move 39, I think few humans could resist that (and I think here it is a good atari).

I also found DrStraw's reasons for dismissing the players interesting, and I think they actually reveal a weakness of dogmatic human thinking based on learning josekis 40(?) years ago. Firstly, as MinjaeKim said, not cutting is another joseki, aiming to play on both sides and pressure the group as a whole. I presume DrStraw expects this joseki to happen which is a joseki popular in old Japanese games back when he was forming his go knowledge, but nowadays is even locally considered good for white (Hwang Inseong cited in josekipedia; black has an inefficient shape and is open on one side, plus white has the aji of the cutting stone). There are other options after cutting, for example extending rather than playing atari after the ko lock shape and I recall Go Seigen talking about this in one of his books or commentaries, perhaps DrStraw expected this? But on the board in question the old joseki is even worse for black as his k4 stone ends up too close, and leaves a weakness at h3. If k4 were at j4 maybe it would be ok for black, but personally I would probably still take white. And then there is his logic that "player didn't play joseki" + "bot should know joseki" -> "player isn't bot". So the "not joseki" claim is dubious, how about "bot should play joseki"? For monte carlo bots I would think they play less joseki than a human. With these new neural net pattern matching ones they may actually play lots of joseki, if those joseki appeared in pro games they are trained on. There are actually quite a lot of joseki sequences I know that if you do a pattern search for you find surprisingly few and sometimes no hits. Maybe they aren't such a joseki as we think (or maybe the database is deficient). Would a NN + MCTS bot play lots of joseki? I don't know.

Then there his dismissal of player1 based on move 10. Does this mean he thinks 10 is a bad move and a bot wouldn't do that, or 10 is a good move but too creative for a bot (but MCTS are pretty creative)? To me 10 is a normal light move, again perhaps more modern than the Go on which DrStraw was raised which would see a local reply.

Btw OP, are you Hourousha from old OGS?

[Bill Spight]

I also found DrStraw's reasons for dismissing the players interesting, and I think they actually reveal a weakness of dogmatic human thinking based on learning josekis 40(?) years ago.

Fortunately, I didn't learn any joseki, so I avoided that problem.

Firstly, as MinjaeKim said, not cutting is another joseki, aiming to play on both sides and pressure the group as a whole. I presume DrStraw expects this joseki to happen which is a joseki popular in old Japanese games back when he was forming his go knowledge, but nowadays is even locally considered good for white (Hwang Inseong cited in josekipedia; black has an inefficient shape and is open on one side, plus white has the aji of the cutting stone).

That's my feeling, but recently I actually ran across a similar dumpling in an ancient game between top players, along with the resultant ponnuki. It seems like it has been considered acceptable for some time.

For monte carlo bots I would think they play less joseki than a human.

Moi aussi.

With these new neural net pattern matching ones they may actually play lots of joseki, if those joseki appeared in pro games they are trained on. There are actually quite a lot of joseki sequences I know that if you do a pattern search for you find surprisingly few and sometimes no hits. Maybe they aren't such a joseki as we think (or maybe the database is deficient).

Joseki is a term that has been around so long that it has acquired more than one meaning. I am something of a purist, and think that a sequence that has not appeared frequently in pro play is not joseki, even if it is equitable.

Then there his dismissal of player1 based on move 10. Does this mean he thinks 10 is a bad move and a bot wouldn't do that, or 10 is a good move but too creative for a bot (but MCTS are pretty creative)? To me 10 is a normal light move, again perhaps more modern than the Go on which DrStraw was raised which would see a local reply.

To me, is a low dan move, being neither here nor there. is placed to reduce whatever Black tries on the top side; if White wants to develop on the top, then the approach to the top right corner is more dynamic. Anyway, is at the bot's playing level, but I doubt seriously if it comes close to matching anything that it has been trained on.

[Sennahoj]

To me, is a low dan move, being neither here nor there.

Haylee's latest video shows Rui Naiwei playing a move very similar to (to my eyes)

https://www.youtube.com/watch?v=_oNpTDm1ITg

[Uberdude]

To me, is a low dan move, being neither here nor there.

Haylee's latest video shows Rui Naiwei playing a move very similar to (to my eyes)

https://www.youtube.com/watch?v=_oNpTDm1ITg

That is the most frequently played move (7 hits) in that exact whole board position (20 hits) in the pro game db on ps.waltheri.net. Played by such esteemed professionals as Kobayashi Koichi, Cho Hun-hyeon and Cho Han-seung. I think it could be played be a 3d bot or a 3d human, so doesn't tell me anything. I doubt many 5k humans would play it though, that sort of light play is rarer there (certainly I think I only started playing such moves later).

[skydyr]

I'm inclined to think it's player 2 as well. The play seems normal but passive, and there are weird life and death/basic shape issues.

[Bill Spight]

To me, is a low dan move, being neither here nor there.

Haylee's latest video shows Rui Naiwei playing a move very similar to (to my eyes)

https://www.youtube.com/watch?v=_oNpTDm1ITg

Not to mine.

Here is Rui's move.

$$B Moves 1 to 10
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . 5 . 6 . . . . . 7 . . . . . |
$$ | . . . 2 . . . . . , . . . . . , 1 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . 0 . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . 8 . . . |
$$ | . . . . . . . . . . . . . . . . 9 . . |
$$ | . . . 4 . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . 3 . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------

Click Here To Show Diagram Code

[go]$$B Moves 1 to 10
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . 5 . 6 . . . . . 7 . . . . . |
$$ | . . . 2 . . . . . , . . . . . , 1 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . 0 . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . 8 . . . |
$$ | . . . . . . . . . . . . . . . . 9 . . |
$$ | . . . 4 . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . 3 . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

is well placed with regard to the , enclosure, and White has no approach such as the move I suggested as more dynamic.

[Bill Spight]

To me, is a low dan move, being neither here nor there.

Haylee's latest video shows Rui Naiwei playing a move very similar to (to my eyes)

https://www.youtube.com/watch?v=_oNpTDm1ITg

That is the most frequently played move (7 hits) in that exact whole board position (20 hits) in the pro game db on ps.waltheri.net. Played by such esteemed professionals as Kobayashi Koichi, Cho Hun-hyeon and Cho Han-seung. I think it could be played be a 3d bot or a 3d human, so doesn't tell me anything. I doubt many 5k humans would play it though, that sort of light play is rarer there (certainly I think I only started playing such moves later).

$$Wcm8 Moves 8 to 12
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . 2 . . . . 3 c . 5 b . . . . . |
$$ | . . X , 4 1 . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . a . . . . . , . . . . . , . . . |
$$ | . . . B . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . . . O . . . |
$$ | . . . . . O . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------

Click Here To Show Diagram Code

[go]$$Wcm8 Moves 8 to 12
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . 2 . . . . 3 c . 5 b . . . . . |
$$ | . . X , 4 1 . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . a . . . . . , . . . . . , . . . |
$$ | . . . B . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . . . O . . . |
$$ | . . . . . O . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

Point well taken.

is also the popular choice when is at "a". However, in both cases it has the lowest winning percentage of all the choices on the top side, 57% in this case and 50% when is at "a". Furthermore, after the winning percentage drops to 50% here and 40% in the other case. The approach at "b" has a winning percentage of 67% here, 75% in the other case. In both cases "c" is the winningest move so far. (FWIW. With so few instances, trying to find the best play is impossible. However, it is interesting that is the worst on the top side in both cases so far. )