AI misses move of the year

John Fairbairn · #1

White to play in the position below.

In real life, White was Pak Cheong-hwan. Black was Kang Tong-yun and the setting was the Tygem-backed Korean National Squad League. This is a 20-man event designed to train players to resist China. It has a first prize of only about $23,000, but perhaps more important is the coaching given. Time limits are 1 hour each.

The coaching staff chose the move in the privy chamber below as the "Best Move of 2020". It's rather easy - at least I saw it instantly. But it scored with the judges because (a) Pak had planned it some time back and (b) AI overlooked it. They didn't say which AI, or what AI proposed instead.

Cassandra · #2

John Fairbairn wrote:

They didn't say which AI, or what AI proposed instead.

KataGo does not have any problems with finding the tesuji for White in this position.

John Fairbairn · #3

Quote:

KataGo does not have any problems with finding the tesuji for White in this position.

I expected this sort of comment - otherwise I'd be claiming to be better than AI myself! It wasn't stated, but given the conjunction with the point about Pak seeing it many moves ahead, I'm assuming that what was meant was AI didn't notice it at the same early point that Pak planned it. But I suppose the counter to that is: how do they really know?

I still wonder whether connect-and-die (oi-otoshi) could hold problems for certain AI programs, though. Lightvector?

Cassandra · #4

John Fairbairn wrote:

It wasn't stated, but given the conjunction with the point about Pak seeing it many moves ahead, I'm assuming that what was meant was AI didn't notice it at the same early point that Pak planned it. But I suppose the counter to that is: how do they really know?

I still wonder whether connect-and-die (oi-otoshi) could hold problems for certain AI programs, though. Lightvector?

It would help answering some of your questions, if you were able to supply a SGF of the game's sequence until the position in question.
Provided that your initial source made that sequence available.

John Fairbairn · #5

Quote:

It would help answering some of your questions, if you were able to supply a SGF of the game's sequence until the position in question.
Provided that your initial source made that sequence available.

No, sorry, I haven't got the full game. If anybody has access to the February issue of Baduk it might be there.

Bill Spight · #6

John Fairbairn wrote:

I still wonder whether connect-and-die (oi-otoshi) could hold problems for certain AI programs, though. Lightvector?

For go, it is well known that strong humans, even amateurs, are better at certain types of positions than most modern AI. In particular, they are positions where depth first reading pays off, since humans are best at that, rather than breadth first or best first reading. Neural networks are very good at playing the averages, based upon the positions they are trained upon. While sacrifices to take away a dame are not uncommon, such sacrifices to win a semeai with several dame are uncommon in the self-play games upon which nearly all of today's top bots are trained. They do not encounter them often enough in their training to learn them. My guess is that if, say, Zen, were able to search 20k nodes at the depth where it reached the given position, it would find the sacrifice. Katago is trained differently, which enables it to do better.

bernds · #7

John Fairbairn wrote:

Quote:

It would help answering some of your questions, if you were able to supply a SGF of the game's sequence until the position in question.
Provided that your initial source made that sequence available.

No, sorry, I haven't got the full game. If anybody has access to the February issue of Baduk it might be there.

I've seen the game before, but I can't remember where it was. Googling shows a video:

https://www.youtube.com/watch?v=uY2aCYb9sDM

and an article with a link to a downloadable SGF:

https://boardtobitsgo.wordpress.com/2020/10/05/amazing-tesuji-in-pro-game/

Cassandra · #8

John Fairbairn wrote:

It wasn't stated, but given the conjunction with the point about Pak seeing it many moves ahead, I'm assuming that what was meant was AI didn't notice it at the same early point that Pak planned it. But I suppose the counter to that is: how do they really know?

There are some aspects to consider, which I would like to mention here, based on my experience with KataGo @ Igo Hatsuyôron 120:

If "the AI didn't notice" White's tesuji, this move cannot be part of the look-ahead sequence(s) that are shown / given in the interface programme that has the AI run behind.
=> However, it is not mandatory at all that the moves given in the look-ahead are really choosen when their (earlier suspected) time has come.

If "Pak planned" White's tesuji at an "early point" of the game, the succeeding sequence must be a ONE-WAY street of OPTIMAL play by both sides.
=> If the requirements are met in the game, the AI could only be blamed for not PLAYING that tesuji at due time. Not for anything that did not happen before that moment.
=> Otherwise, the AI must not be blamed.

AI programmes are not designed to find "beautiful" (human understanding) moves. Not even the "stongest" ones (human understanding).
=> AI is ONLY trained to win the game.
For several weeks now, I have been struggling with analysing a position in Igo Hatsuyôron 120, where KataGo's PLAY is very dependent on the (reverse) komi applied.
To keep a long story short:
Letting KataGo continue the game from that position (playing for both sides), there is a wide range of komi (1 to about -10), wherein the games have the general tendency to end (very) close.

dfan · #9

It can be hard to say when AIs notice moves, partially because they are also evaluating the most promising-looking moves for both sides, so they don't spend lots of time thinking about how to refute opponent moves once they're sure the move doesn't work.

In this game, White extends at J10 because he knows that the cut at J8 doesn't work, and so does KataGo. As soon as J10 is played it detects that J8 is bad for Black, and presumably it wouldn't think so if it didn't know the concrete refutation.

It does have a hard time choosing between J10 and H9. It may be that its preference (at least with low playouts) for H9 is the kind of thing the commentators are talking about. (Of course, they were likely using a different AI, or at the the least a different network.)

jlt · **#10**

After 200k playouts, 15-block Katago (g170e-b15c192-s1305382144-d335919935) still can't find the game's move. I don't know if it is superhuman, but I believe it is at least strong pro level.

Attachment:

Capture.JPG [ 31.11 KiB | Viewed 6992 times ]

Cassandra · **#11**

Analysed with KataGo (40b, as of 2021-03-27), using additional parameter setting for ANALYSIS, as recommended by Lightvector:
analysisWideRootNoise=0.1,nnPolicyTemperature=1.1,cpuctExploration=1.5,cpuctExplorationLog=0.6

As experienced also with Igo Hatsuyôron 120, "simply" letting KataGo PLAY might not be sufficient.
I waited with noting the values of win rate (and playouts) until the favourite move had 50k playouts.

Please note that KataGo thinks that B 87 was a mistake, largely affecting White's win rate (and continuation) at W 88.
And yes, KataGo does NOT think that the tesuji line is the surest way to victory ...

Pio2001 · **#12**

On my smartphone, BadukAI + Katago only sees the move after 8900 visits, but he thinks that it is a terrible move ! 16.9% chances of winning vs 73.9% for J6.

But if I play it, it instantly sees that white is winning.

Taking back the move, he again thinks that it is terrible : the calculations for a given move are not added to the previous tree search.

lightvector · **#13**

I was going to post here after getting back from work today but looks like Cassandra beat me to it. But I guess I can still make it human-explainable.

Older networks for KataGo have trouble seeing this move, yes. Newer katago networks should find it okay, especially if you are using the settings recommended for analysis. These settings will probably be made default for analysis in the next release.

And yes, at least in Kata's opinion, the tesuji is a good way to refute the immediate previous black move, but if black plays a better move, white actually should not play the tesuji. Unlike the extreme case of igoh120, there aren't any super-sharp tactics after this is resolved, so score and winrate should behave a bit more smoothly and robustly for judging these relative lines if you play them fully out.

So firstly, the game move:

Click Here To Show Diagram Code: [go]$$Wc Black's mistake, now tesuji becomes very good. $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X 2 . . . . . . . | $$ | . O O X X . . . X O 1 X . O . . X O . | $$ | . X O X . O . . . . . . X O . . X O . | $$ | . O X O . X 3 X X O . X O X . X . . . | $$ | . O X O . . X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . . X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

And KataGo gives this as a continuation:

Click Here To Show Diagram Code: [go]$$Wc Black's mistake, continuation $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . 9 . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . 8 . . . . . | $$ | . . O O X X . O X O X X . . . . . . . | $$ | . O O X X . . . X O O X . W . . X O . | $$ | . X O X . O 4 . 5 7 6 . X W . . X O . | $$ | . O X O . X C X X O . X O X . X . . . | $$ | . O X O . . X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . . X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

is urgent to stop white's two stones from jumping out and giving black two weak groups, and white takes sente.

Instead, black should play this way:

Click Here To Show Diagram Code: [go]$$Wc Black's better move $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X . 2 . . . . . . | $$ | . O O X X . . . X O 1 X . O . . X O . | $$ | . X O X . O . . . . . . X O . . X O . | $$ | . O X O . X . X X O . X O X . X . . . | $$ | . O X O . . X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . . X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

If white tries the tesuji now:

Click Here To Show Diagram Code: [go]$$Wc Black's better move, white unideal tesuji $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . 9 . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X . X 8 . . . . . | $$ | . O O X X . . . X O O X . W . . X O . | $$ | . X O X . O 4 . 5 7 6 . X W . . X O . | $$ | . O X O . X 3 X X O . X O X . X . . . | $$ | . O X O . . X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . . X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Same result as before, but black captures the two white stones with *much* thicker shape. The improvement from this capture over the looser capture is not small, KataGo puts it at 17% winrate and 6 points!

White is still winning this game, but relatively speaking, this result is good enough for black that actually KataGo prefers *not* playing the tesuji for white. Instead, white should just push through and make life locally:

Click Here To Show Diagram Code: [go]$$Wc Black's better move, white's best response. $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X . X . . . . . . | $$ | . O O X X . 6 4 X O O X . O . . X O . | $$ | . X O X . O . 5 3 . . . X O . . X O . | $$ | . O X O . X 7 X X O . X O X . X . . . | $$ | . O X O . . X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . . X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Click Here To Show Diagram Code: [go]$$Bc Black's better move, white's best response, continued $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X . X a . . . . . | $$ | . O O X X . X X X O O X . O . . X O . | $$ | . X O X . O 1 O O . 3 . X O . . X O . | $$ | . O X O . X O 2 . O . X O X . X . . . | $$ | . O X O . 5 X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X 7 6 4 X 9 X 8 . 0 X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

And now white is alive. If you find it a bit surprising that this is better for white than the tesuji line, I think it's instructive to add a black stone at "a" (NOT the correct next move for black, but instructive) and compare.

Click Here To Show Diagram Code: [go]$$Wc Tesuji result (white to play next) $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X . X X . . . . . | $$ | . O O X X . . . X O O X . O . . X O . | $$ | . X O X . O X . O O X . X O . . X O . | $$ | . O X O . X . X X O . X O X . X . . . | $$ | . O X O . . X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . . X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Click Here To Show Diagram Code: [go]$$Wc Local life result with black adding one more stone (white to play next) $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . . . . . . | $$ | . . O O X X . O X O X . X X . . . . . | $$ | . O O X X . X X X O O X . O . . X O . | $$ | . X O X . O X O O . X . X O . . X O . | $$ | . O X O . X O O . O . X O X . X . . . | $$ | . O X O . X X O O O O X O X . X O . . | $$ | . O X O . . X O X . O O O O X X O . . | $$ | . X X X X O O X X X O . O X . . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

I think the key points that make the local life result slightly better are:

* White is NOT alive in the tesuji result. White's group, despite poking out into the center, has no eyes anywhere, it only gets a half-eye as a result of the two dead black stones! (Black can falsify it in gote at H9). Black is very thick on both sides, so attacking white later is a real possibility.
* In the local life result, white is now safely alive. White's two stones at J9 are not connected, but their value was not high to begin with, since in all variations they are only cutting groups that are strong or alive.
* White has gained a lot of endgame value at the bottom due to the sacrificed stones at G2.

Lastly, if the local life result is so great, why can't white play it also when black plays the solid connection?

Because there is a risky sequence where black can go for the kill:

Click Here To Show Diagram Code: [go]$$Bc Risky sequence depending on which black circled stone is there $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . O X . . . | $$ | . . . . . . . . . . . . . . . O . . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . 7 . . . . . . . . . . . . | $$ | . . . , O . . . O , . . . . . , . . . | $$ | . . . X O . X . O X X . . . a . . . . | $$ | . . O O X X . O X O X B B b . . . . . | $$ | . O O X X . X X X O O X . O . . X O . | $$ | . X O X . O . O O 6 1 . X O . . X O . | $$ | . O X O 4 X O . . O . X O X . X . . . | $$ | . O X O . 2 X O O O O X O X . X O . . | $$ | . O X O 5 3 X O X . O O O O X X O . . | $$ | . X X X . . . X . X . . 8 X 9 . O . . | $$ | . . X . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Black barely escapes with :b7:

, and that group is still fragile. Next white runs out around a or b. White's inside group has a lot of liberties, and black has three weak outside groups to manage. With the hanging connection, black's shape is worse for liberties, and with three weak black groups and bad black shape on the middle group, KataGo is relatively confident that white can kill something or capture some piece to live. With the solid connection, KataGo isn't sure.

AI misses move of the year

Who is online