LZ misses a corner tesuji

Knotwilg · #1

Click Here To Show Diagram Code: [go]$$W Moves 190 to 191 $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O . | $$ | X X X X . X O X X . O . O O O O X X b | $$ | X O O O X X O O O , . O X X X X . . . | $$ | X X O O O O X . O X . O X . . . a 1 . | $$ | O O X O . . X . O O X X X . O . X 2 . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

White has made a placement. How to follow-up?

Leela Zero prefers one of A/B by 15% but she's wrong about it. She misses a tesuji, even after 21k playouts
Five moves down the sequence, she suddenly finds the tesuji and the game is reversed.
KataGo makes a similar mistake but I haven't let it play out as many games.

So, which is best? And which mistake does LZ/KG make - as me and my opponent in the game.

gennan · #2

I'm reading this for a:

Click Here To Show Diagram Code: [go]$$W a $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O . | $$ | X X X X . X O X X . O . O O O O X X 5 | $$ | X O O O X X O O O , . O X X X X 4 . 3 | $$ | X X O O O O X . O X . O X . . 2 1 O . | $$ | O O X O . . X . O O X X X . O 6 X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

I suppose that move 3 may be missed, but I would be quite surprised if LZ and KG both miss such a move.

When I compare that with b

Click Here To Show Diagram Code: [go]$$W b $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O . | $$ | X X X X . X O X X . O . O O O O X X 1 | $$ | X O O O X X O O O , . O X X X X 4 . 3 | $$ | X X O O O O X . O X . O X . c . 2 O . | $$ | O O X O . . X . O O X X X . O . X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

Now black has an option to gain about 2 points with c later.
The game does seem pretty close, so this might explain why b loses 15% for white.

jlt · #3

mhlepore · #4

@jlt:

SoDesuNe · #5

I just checked with KataGo but due to the close game it very much depends on the komi value.

With 6.5 komi, White is ahead any way and KataGo even sacrificed the placement stone.

With 0.5 komi, Black wins any way according to KataGo and recommends neither 'a' or 'b' but the first line kosumi. Black lets White connect up.

Bill Spight · #6

My first thought.

Bill Spight · #7

jlt wrote:

Edit: Maybe not.

Knotwilg · #8

Haha, you guys make it even better. I hadn't noticed the ko.
jlt's move was what I spotted indeed. Kata/LZ didn't take it consider it until the Black descent.

It's an interesting position!

I'll add a full endgame analysis, where this position plays an even bigger role, as there are many more threats available. Go to 154

lightvector · #9

I'm curious what version of KataGo you were using, and how many playouts / what komi you tried. Similarly LZ.

Did the bots actually miss jlt's sequence, or did you interpret them evaluating most of jlt's sequence as good for white (except for the very last bit) as missing jlt's sequence? Because in fact most of that line *is* good for white. It's possible that they hadn't missed it at all, but evaluated it as good for white because they went on to read the ko as the counter to jlt's sequence and evaluated the ko as winning!

Click Here To Show Diagram Code: [go]$$Wc $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O . | $$ | X X X X . X O X X . O . O O O O X X b | $$ | X O O O X X O O O , . O X X X X . . c | $$ | X X O O O O X . O X . O X . . . a O . | $$ | O O X O . . X . O O X X X . O . X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

I think the summary of the situation is that "b" and "c" are about the same and both work simply and easily. "a" is one point better, but is very complicated because if black resists, it requires a messy ko fight.

I'm playing around with the latest public 40 block KataGo net, mid thousands to low tens of thousands of playouts clearing the cache a few times to see what it thinks on fresh retries ("clear_cache" in GTP console) on Japanese rules ("kata-set-rules japanese" in GTP console, Lizzie will not do this automatically for you) and I get the following results:

Japanese rules, 7.5 komi or higher: White is clearly winning (by bot standards) and can just play "b" or "c", which is good enough to win: 85%ish. Trading 1 point for simplicity. Clearing the cache and repeating seems to result in it picking b with decent reliability for me, although not 100% consistent.
Japanese rules, 6.5 komi: White is favored (by bot standards) with "b" or "c", but it's less clearly winning. 65%ish. Clearing the cache and repeating seems to result in it picking "b" or "c" most of the time. Here KataGo has a little bit of "blindspotness" by failing to play "a". Based on the variations it gives, I think sees jlt's line easily, and furthermore sees that white can counter with a ko, but initially doesn't see quite right way to start the ko and/or feels it's messy enough that it would rather go with the 65%ish moves. If you actually play "a" on the board then the winrate quickly goes up to 75% to 80%, since now it's found the right way to start the ko and is reading it deeply enough to see that white can fight it well, in which case missing "a" is clearly a mistake. But if "b" or "c" still win by 0.5, then at least maybe it's not a mistake that costs the game yet.
Japanese rules, 5.5 komi: Now white is actually losing if playing b or c, probably by half a point. So KataGo seems to consistently choose "a" now. Again, it's already seen past jlt's issue to seeing that white can counter with ko, and however messy the ko initially appears, it's better than playing "b" or "c" and heading into a losing endgame. And with several thousands of playouts, it thinks the ko is eventually winning for white by a bunch, so black must not descend and instead must fight the normal endgame, where maybe white has a bare lead, by 0.5. (60%).

The ko line looks like this. White goes for the extra point, and black resists like jlt showed.

Click Here To Show Diagram Code: [go]$$Wc First $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O 6 | $$ | X X X X . X O X X . O . O O O O X X 4 | $$ | X O O O X X O O O , . O X X X X 5 . 3 | $$ | X X O O O O X . O X . O X . . 2 1 O . | $$ | O O X O . . X . O O X X X . O . X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

Next white cuts!

Click Here To Show Diagram Code: [go]$$Wc Next $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . b O O X | $$ | X X X X . X O X X . O . O O O O X X X | $$ | X O O O X X O O O , . O X X X X O . O | $$ | X X O O O O X . O X c O X . . X O O e | $$ | O O X O . . X . O O X X X . O 1 X X . | $$ | . . . . . . . . . . O O O X O O O X d | $$ | . O O O . . . O . . . X O O O X X a . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

White has time to cut here because it gains white a liberty via the cut at "a". If black tries "b" for example, white will play "c" and kill the big black center group, winning 4 liberties to 3. Black can play "d" or "e" to make it a ko instead... but even though this isn't the ko white is "expecting", it's just as winning for white as the other expected one.

Click Here To Show Diagram Code: [go]$$Wc Next next $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . 5 4 3 | $$ | . . . . . X X O X . . O . . . a O O X | $$ | X X X X . X O X X . O . O O O O X X X | $$ | X O O O X X O O O , . O X X X X O . O | $$ | X X O O O O X . O X c O X . . X O O . | $$ | O O X O . . X . O O X X X . O 1 X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X 2 . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

So in response to the cut, black must protect right away. The result of being able to interpose this cut of course, is that now the ko is extremely heavy for black because the center group is at stake.

Then white initiates 3-4-5. At this point, if black immediately begins fighting the ko with "a", it's immediately losing, so KataGo starts suggesting moves all around the board to try to generate bigger threats, but KataGo isn't optimistic about it. (80% for white).

So instead we get this:

Click Here To Show Diagram Code: [go]$$Wc Black can't fight ko so must back down $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O . | $$ | X X X X . X O X X . O . O O O O X X . | $$ | X O O O X X O O O , . O X X X X 5 . 3 | $$ | X X O O O O X . O X . O X . . 2 1 O . | $$ | O O X O . . X . O O X X X . O 4 X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . | $$ | X . . X O O X . . . . . . X . . X O . | $$ | . X X O O X O X X O O O O O X X O . . | $$ | X X O X X X . . . X . X . . X O O . . | $$ | . O O O X . . . . X . X O O X X O . . | $$ | . . O , X . . . . , O O X O O O . . . | $$ | . O . O X . . . . X X O X X . X O O . | $$ | . . . O O X X X . . . X . . . X X O . | $$ | . . . . O O O X . . . . . . . . . . . | $$ ---------------------------------------[/go]

And this leads into a very, very close endgame (at Japanese 5.5 komi) where white has the barest advantage and is hoping to win by 0.5.

Knotwilg · **#10**

Thanks lightvector. Since I hadn't spotted the ko at the time of publishing the problem, I may obviously have misinterpreted LZ's or KG's analyses.
What struck me and remains unexplained, is that (my version of) Lizzie, including both engines, recommends the move that gains one point, probably missing out on the turn at the 1-2 point. Then when played out, it discovers the turn and changes the winning rate drastically.

Perhaps the newer versions of the engines make a perfect analysis of the situation.

Bill Spight · **#11**

Knotwilg wrote:

Thanks lightvector. Since I hadn't spotted the ko at the time of publishing the problem, I may obviously have misinterpreted LZ's or KG's analyses.
What struck me and remains unexplained, is that (my version of) Lizzie, including both engines, recommends the move that gains one point, probably missing out on the turn at the 1-2 point. Then when played out, it discovers the turn and changes the winning rate drastically.

Perhaps the newer versions of the engines make a perfect analysis of the situation.

I think we still have to keep in mind the question of what we mean by a perfect, or near perfect, analysis. Winrate estimates assume errors. Errors at a high level of play, but errors nonetheless. For humans, the question is psychological. Which play is more likely to induce an error by my opponent? With apologies to Arthur C. Clarke, we are not quite ready to speak of the psychology of computer programs.

The changing winrate estimates further down in the game tree suggest a horizon effect. I don't know when, if ever, we will be rid of those.

Currently, I think, humans and bots have different strengths and weaknesses. The best analysis, I think, comes from combining the two. IOW, from the human interacting with the program. As, I believe, lightvector has suggested elsewhere.

RobertJasiek · **#12**

Knotwilg wrote:

Perhaps the newer versions of the engines make a perfect analysis of the situation.

No. AI needs a major redesign (such as always using dynamic komi as a start of better analysis, then assess different move candidates if they are related to different winners for adjacent komi) before they are reliable (let alone "perfect") sources for the endgame. AI only optimises winning percentages and ignores endgame counts.

jlt · **#13**

KataGo does not only take into account winning percentages but also the number of points. KataGo may not play perfect endgame, but may be superhuman (i.e. makes fewer errors than humans)?

lightvector · **#14**

Knotwilg wrote:

Thanks lightvector. Since I hadn't spotted the ko at the time of publishing the problem, I may obviously have misinterpreted LZ's or KG's analyses.
What struck me and remains unexplained, is that (my version of) Lizzie, including both engines, recommends the move that gains one point, probably missing out on the turn at the 1-2 point. Then when played out, it discovers the turn and changes the winning rate drastically.

Perhaps the newer versions of the engines make a perfect analysis of the situation.

Are you using LZ272 or something near that and/or KataGo's latest nets (such as g170e b20 s438M)? Also, my winrate estimates to fluctuate a lot in the first few thousand playouts as reading it all out happens, so if you are, maybe it's not enough playouts? Different runs will also vary a little.

For example, here's the evaluation with the latest 20 block net instead of the 40 block net, on 5.5 komi (which low enough to force KataGo to want the ko, since everything else loses):

Attachment:

20ba.png [ 157.54 KiB | Viewed 8388 times ]

And if you hover over it, it's definitely beginning reading out black's resistance at this point, even at these few playouts:

Attachment:

20bb.png [ 162.01 KiB | Viewed 8388 times ]

lightvector · **#15**

And here's the same for LZ272: also easily discovered black's resistance and well on to reading out the ko:

Attachment:

lz272.png [ 162.19 KiB | Viewed 8387 times ]

LZ can only use 7.5 komi Chinese and doesn't care about score, but ironically, it seems on average more inclined than KataGo to play the score maximizing move even at 7.5 komi. Probably just random differences in policy and things like that. Both nets agree that even at 7.5 komi, if you read out the ko line deeply enough, it's better, it's just a question of pushing through the mess to read it deeply enough when the simple moves look better at first.

But although sometimes LZ272 settles on the ko line, in many runs it often also behaves like KataGo and prefers giving up 1 point since it doesn't read the ko line deeply enough. For example in this run it's discovered the ko line, but still prefers the hane. Here, it's a little less confident in the hane than KataGo would be (77.3% for white vs ~85%) which makes some sense because usually KataGo is a bit stronger than LZ at the endgame, but 77% is still consistent with it being enough to win "comfortably" if the rest of the endgame is mistake-free.

Attachment:

lz272b.png [ 174.65 KiB | Viewed 8387 times ]

Knotwilg · **#16**

Thanks lightvector. Pardon my ignorance but I'm using Lizzie 0.7 and Katago 1.2

I think the matter has been settled conclusively by yourself, others and the AI (recent versions with sufficient playouts).

I'll focus on the full endgame analysis now, using the knowledge provided here about the (ko) variations.

Bill Spight · **#17**

jlt wrote:

KataGo does not only take into account winning percentages but also the number of points. KataGo may not play perfect endgame, but may be superhuman (i.e. makes fewer errors than humans)?

Oh, I think there is no question that KataGo and LZ play at a superhuman level. But not in every position.

I was reminded if this in a chess endgame lecture by Yasser Seirawan. ( https://www.youtube.com/watch?v=YI5CQL_ ... 3&frags=wn ) Chess engines have played at superhuman levels for many years. Seirawan showed a position that he had been sure at first sight was a win for Black. His engine, however, evaluated the position as only slightly favorable for Black. In his lecture, to illustrate Black's advantage Seirawan played as Black against the engine as White. After not many moves the engine reevaluated the position as equivalent to Black being a queen ahead or so.

I got the impression that Seirawan enjoyed being better at the endgame than the engine.

Skill at both chess and go is multidimensional. In both games humans are still better in some ways than the programs.

jlt · **#18**

Bill Spight wrote:

But not in every position.

Of course I agree that some positions are more adapted to the human mind than to the engine. I just believe that KataGo makes fewer errors on average, even during the endgame phase.

In the variation showed by lightvector

Click Here To Show Diagram Code: [go]$$Wc $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X X O X . . O . . . . O O . | $$ | X X X X . X O X X . O . O O O O X X . | $$ | X O O O X X O O O , . O X X X X 5 . 3 | $$ | X X O O O O X . O X . O X . . 2 1 O . | $$ | O O X O . . X . O O X X X . O 4 X X . | $$ | . . . . . . . . . . O O O X O O O X . | $$ | . O O O . . . O . . . X O O O X X . . | $$ | . X O X O . . O . . . X X O X O X . . | $$ | . O . X O . O X X , O X O X X , X . . | $$ | . X X X . O X . . . . X . . . . X O . |[/go]

Black loses two stones but gets sente. Black could have saved them in gote by playing :b4:

at

. I wonder how a human reasoning could have come to the same conclusion that it's better to sacrifice the stones for sente.

RobertJasiek · **#19**

jlt wrote:

Bill Spight wrote:

I just believe that KataGo makes fewer errors on average, even during the endgame phase.

Endgame phase: do you think so even when KataGo is clearly ahead?

jlt · **#20**

RobertJasiek wrote:

jlt wrote:

I just believe that KataGo makes fewer errors on average, even during the endgame phase.

Endgame phase: do you think so even when KataGo is clearly ahead?

Unlike other bots, KataGo doesn't play stupid moves when it is clearly ahead, and I haven't heard of stronger players complaining about such behavior, so my initial guess is "yes"... But you are an endgame expert, so maybe you could try by yourself to feed KataGo a few endgame positions and check if it makes mistakes?

LZ misses a corner tesuji

Who is online