My new hobby is to use LZ to analyse problems from books. Mostly of the whole board, shape, haengma type rather than l&d. It is quite easy to find problems where LZ has the opposite opinion from the book answers.

I'll show here a few from 'Making Good Shape' by van Zeijst and Bozulich. (Incidentally, I always found the book difficult, and took it to mean I have a poor brain for shape. At some point I tried to memorise all the answers from the book, but many of them would just not stick to my brain.)

The book problems are not whole board problems, so I tried to put a minimal amount of reasonably placed stones on the board that makes LZ interested in the problem area. We can discuss of course how much the rest of the board matters.

So far I looked at the first 5 problems in the book. On the first 3 LZ agreed, but on problems 4 and 5 it already disagreed.

Problem 4:


Here the original problem is for black to move for stones in the top right corner of the board. Book answer is the circle, and LZ thinks it is -4.5%. LZ answer depends on the rest of the board but I could not make it agree with the book move on any board. My intuition was the angle move that has 58% in the above image, so LZ somewhat agrees.

Continuation if B plays the book move:


Book continuation for W is the white circle, inducing B nobi (incidentally, LZ would then jump, not nobi).
LZ as W continues with the kosumi. Now B could theoretically cut but that would create bad aji in the corner it seems.

Then problem 5, again B to move, with top right stones shown in the problem:


Here book answer is the kosumi that LZ finds slightly inferior. LZ wants to descend, which the book calls a 'failure', because white descent next threatens the cut. Looking at the continuation, LZ does not worry about the cut but would e.g. sacrifice the 2 stones.

Instead LZ worries about this peep:


Of course W could peep after the earlier descent too, but according to LZ it would be less helpful then.

I will maybe post more cases in this or another thread, maybe from other books too.

[Edit: Tried to improve some wording, and added the last sentence]

Problem 4 is already known to be dubious, Mingjiu Jiang 7p disagreed with the book answer (didn't suggest LZ's attach but the 56.5% kosumi from the corner), see validity section here: https://senseis.xmp.net/?MakingGoodShape. But yes, a nice exercise, I've been doing some similar things myself.

Continuing the exercise I find it more and more fascinating. I decided to go through the first 9 problems of the book, which are on the first problem page. Now that I did it, I'll put the analysis in the next posts, and then give some summary afterwards.

So on problems 1-2 LZ agrees with the book. Problems 4-5 I already went through.
I actually claimed that LZ agrees on problems 1-3 with the book. I now need to confess I actually had only checked problems 1-2, and thought problem 3 is so obvious I can skip it. But fortunately I decided to check it with LZ anyway, because the result was very interesting. I promise from now on I will have actually checked all problems I mention.


So, problem 3:


I added the black stone in the lower left corner so that there is a consistent number of B and W stones and the win rates are reasonable.

Here the problem description asks to connect the black stones. The book answer is the well known double keima tesuji. But how is it possible that LZ likes the hane as good or better?

The book calls the hane a failure, because white can cut first at 3-3 and then separate the black stones, ending like this:


But notice the win rate for black has now increased! It seems that the issue is sente, i.e. white can only cut in gote. Also the white wall has the bad defect. It seems this is why LZ does not worry about the cut much at all. The issue is not only because the board is so empty. Even after adding many moves, LZ does not find the cut necessarily very interesting.

Actually after the initial black hane white can also try to cut by wedging at 5-3. Whether that works, depends on a ladder. That also seems to affect how LZ evaluates the book 'failure'.

In conclusion, I would not say that the book is wrong, because it asks to connect the stones. Still, if the book 'failure' is on a real board often better than the correct answer, I think the problem is not very well constructed. If the goal is to teach the double keima connection, that could be done in a better context.

On problem 6, LZ seemed to agree with the book unambiguously.

Then problem 7:


Again, the circled intersection is the book move. On this board, LZ clearly prefers the kosumi-tsuke, which the book calls a failure.

Here is the book continuation for the 'correct' move:


Here is the book continuation for the 'failure':


According to the book, now B "stones are not as thick, and White has more room to maneuver". But e.g. on this board LZ sees things the other way round. Notice that the 'failure' has the lower win rate for W, i.e. is better for B in LZ evaluation.

You can notice that the board is quite empty. Maybe one issue is who gets to play in the empty corner. After playing around I found that on some boards with B stones in upper left corner LZ likes the book answer. It still seems to me it likes the failure more often.

I see that the book answer has value, because it shows a way for black to wrap around the W stones (in gote). However calling the other option a failure seems to be just wrong.

The reckoning begins...

This is going to be ugly.

There is a BIBA video where Blackie plays BensonDarr and loses. At one point Diana Kosegi, who is sitting beside him commenting, says:

Shape is not important. Sente is important.

I have to find that point in the video. It's awesome.

Problem 8 was really interesting:



I had to add quite many stones elsewhere to make the problem area 'hot' enough.

The problem asked to separate the white stones, so the sequence started by the hane and then inside cut is an obvious way to do it.

LZ consistently wanted to play the tsuke instead, as in the image above.

Look at the variations here:



Variation 1 shows what LZ wants to do, building a higher and stronger wall and cutting a stone on the center side.

Variation 2 shows the book answer with one LZ continuation. B cuts some stones, but at the end it seems B shape is not so great and W stones look light to me.

In this case I again would not say that the book answer is wrong, but it seems that Leela Zero found a more interesting way to play. So if the point is to teach the hane-inside-cut sequence, the problem could be better constructed. And if the reader is left wondering if there are more interesting moves, he or she could be right...

[Edit: corrected some black/white mistakes]

Then problem 9:


Here the problem asks where B should play to "preserve his shape".

It was again surprisingly difficult to make LZ interested to play in the problem area as B at all. As white it wanted to hane, but then black can live on the side and white cannot wrap black solidly without losing sente. When B wanted to play, it wanted to jump rather than nobi, which is the book answer. The image does not show the win rate for the nobi, but it was considered a mistake of a couple of percentage points.

The book tells us that the hane is a mistake, because white will cut and black cannot capture the cutting stone. LZ agrees with that, but suggests a better move than the book.

Hi Calvin,

My summary on the Leela Zero analysis of the 9 first problems of 'Making good shape':

3/9 problems: LZ agrees clearly. (Problems 1, 2, 6)
3/9 problems: LZ disagrees clearly, finding the ‘failure’ better on most boards. (Problems 4, 5, 7)
2/9 problems: LZ agrees that the 'failure' is wrong, but finds a better local move than book solution. (Problems 8, 9)
1/9 problem: LZ thinks doing what the problem asks is often just wrong. (Problem 3)

Of course it was just a few problems. I now regret that I did not sample problems randomly, but started from the beginning. If I find time, I'll look at a few random problems more.

Thanks, this was fascinating. I did a similar experiment with some problems from Test Your Go Strength (and those are whole-board problems, which is more fair to Leela Zero, though maybe the komi is off by 1 or 2), and found some interesting results, although I doubt I will find the energy to write them up as nicely as you did here.

I really appreciate this thread. Much food for thought.

As a composer of problems I am familiar with the following phenomenon. You have a certain idea in mind when you compose a problem and, after some effort, you succeed in making up a problem to illustrate or embody that idea. Later you realize (hopefully) that there is some aspect of your problem that you completely ignored, but is important. This happens a lot, at least to me. The thing is, even if your solution is still superior to other plays, you really ought to address the other aspect, or redo the problem. Anyway, I sympathize with composers to whom this happens.

Problem 3 is not really a composition by the authors, even if they did not copy it from somewhere else. It is a standard position that illustrates the tesuji to connect underneath. If the problem is presented as such: How does Black connect? Which was how I learned it, that's fine. The hane does not connect, so it is a failure. If the composer adds that connecting may not be best, that's good. If not, I don't particularly see that as a problem. OTOH, I learned that tesuji as an SDK, and I have never considered not connecting, but playing the hane instead and taking sente. So that's a blind spot for me. I blame you, Sakata!

So I sympathize with the composers, but as the saying goes, it's not what you don't know, it's what you know that ain't so. This is a very important study. Many thanks, zermelo!

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

So, I checked 5 more problems. Now I chose them randomly with a software RNG.
Problems were 107, 164, 36, 211, and 186. Of these, on 36 and 186 LZ agreed perfectly (though I'd say 36 is more of a tsumego than shape problem).

Problem 107 is pretty interesting:


I admit I thought the answer must be obvious. But no! LZ thinks that the 'failure' is at least 5 percentage points better than the book solution.

See variations here:


Variation 1 shows how LZ would continue the book solution.

Variation 2 shows how LZ would continue the 'failure' variation, which it judges better for white.

Again, the problem is supposed to illustrate how white can stay connected (though that goal is not mentioned in the problem setting). It just fails to consider that getting cut can actually give a better result.

Then problem 164:



Here LZ thinks it's move is 3.5% better than the book solution (the 2nd line hane). It agrees that the book failure (descent to 2nd line) is much worse. If white plays the hane, LZ does not block but plays the keima on the center side. However, even if B responds as in the book, LZ thinks it is worse for white than starting with the 5th line tsuke above.

Last problem, 211:



The book asks for the vital point of white's shape, and suggests the angle play. LZ is not impressed, but wants to play one move further from white stones. I checked quite many board configurations, and LZ never seemed to like the 'vital point' best. Sometimes it wanted to block from the center side. The 'failure' given by the book is the 4th line move that LZ does not like either.

See some variations below. The win rate differences are not very big:


So a summary on these 5 problems:
2/5: LZ agrees with the book
1/5: LZ has the opposite opinion of success and failure
2/5: LZ agrees on 'failures', but sees a better local move than the book solution

Hi Bill,

"It's not what you don't know, it's what you know that ain't so."
I know this quote originally came from Mark Twain

Very interesting stuff zermelo. About the double knight connection of problem 3, I think having empty corners still available is too distorting in increasing the value of sente, though the lesson that it it ok not to connect but let the opponent cut in gote is indeed very valuable. So I think more care needs to be taken in creating a realistic whole board setting for the problem. One idea is to pro pattern search and stick the whole pro game in. I find only one such game when you include the details of the white wall (Cho Sonjin 6p vs Asano Hideaki 6p in the Ryusei from 1990). In this position LZ #157 started off liking the hane (with the double knight connect 1% behind) but then decided tenuki was better (if black descends at top right just answer). If hane and black cut and then descend in gote black loses 20%, this makes sense as white already has a fairly strong group on the right side and the peep of cutting point is nice (at least atari for ko instead of descend for black). After hane it expects black to wedge and then there is the question of the ladder and that seems to confuse LZ somewhat: when black wedges the expected sequence is wedge, descend, split, clamp, descend, extend/peep outside, connect, live 2-2. But when you play out the sequence LZ realizes extend/peep can be cut instead of black doesn't have ladder (he can squeeze though). After even more playouts after the initial hane and wedge LZ would atari from underneath and fight the ko.


Or here is an old OGS game of mine in which this double knight connection didn't happen but it was a possible sequence I had read and would have been happy to play (it's a kind of san-ren-sei invasion joseki), let's see what LZ thinks:

Nope, doesn't like it, hane is 9% better! If black does cut then the ko happens, white has some threats lower left but it says fine to ignore black f2 to win ko, but I don't like that wall having to run. What does white gain by winning the ko and spending 1 extra local move vs knight connection: a thicker shape and a few more points but I find it hard to prefer that when the cost is ignoring a black ko threat.

Now here is my attempt to create a vaguely realistic position in which the book answer is correct, without first checking it in LZ:

I was successful! I think this is because I gave black those 2 stones on the right side so that it is important and large for white to connect to the r14 stone, and the p13 stone takes care of the ladder with the clamp sequence after hane and wedge. Playing around with the position, just p13 and no p10 is enough to make it strongly prefer knight connection, but with just p10 it prefers hane (realizes ladder good for W). With p10, no p13, and black near tengen to make the ladder it then prefers the knight connection. It actually seems pretty quick to realize the importance of the ladder, for example taking off quite a few stones and then plopping a random ladder-making stone at g7 (vs l4 say) is enough to make it switch from hane to knight connect within 1000 playouts.

For problem 7, it's hard to find a realistic position as that looks like the far approach to 5-3 joseki, but then black made an unusual block on 2nd line instead of tenuki or extend or jump, and then white tenukid (could be reasonable to a big place if black's block was stupid and slow) rather than defend corner shape or pincer. Here's a recent pro game with a pincer, LZ says block was slow (prefers extend), prefers defend corner with kosumi to black's pincer, and agrees with the attach rather than cut across knight move (would capture not connect when white ataris). The pincer obviously makes quite a difference here, for example with the book sequence the peep at p6 is annoying.

Or this Dosaku game if you make Dosaku play lower left shimari instead of kosumi defence (which LZ agrees with) then LZ wants to play the same attach instead of sacrifice. The peep (o5 here) after the gote wall seems to be a key minus of the book sequence.

Edit: about problem 107, I wonder what LZ thinks of p6 same shape at move 50 in this game from daal: viewtopic.php?p=227310#p227310

If we start with an empty board and say: "Black to play," LeelaZero offers just two moves (4-4 and 4-3). But many top pros will happily play 3-3. This month alone (August 2018) at least 15 pros have played this. And other AI bots have played it recently, too (e.g. Golaxy beat Ke Jie with it).

We believe there is a margin of error for all bot predictions (early in the game, at least). How substantial is matter of debate, but the fact that it is mentioned so often suggests many people regard it as too important to overlook.

In the case of the problems here, we get distortions because of the lack of stones elsewhere (not to mention that the question in the original is not necessarily "what is the best move" but "how does Black...."). We appear to have no sense of scale of this distortion.

Even with a full-board problem we can get distortions. Komi may have been different from 7.5. The answer may have been provided by a ghost-writing amateur, etc.

Then we have possible distortions due to style. We don't even properly understand the style of bots yet.

And of course bots are human - they are written by humans who make programming errors. They fall off ladders, etc.

I accept that over a large number of games, bots will beat humans almost every time. But is that because every move is better than a human's or just because fallible humans make the last big mistake through a lapse of concentration?

All of those points will have already been raised or considered by others. But for me what they amount to is just a huge question mark over any bot judgement. I have absolutely no feel for which factor counts more than others or by how much. In short, I know there is a question mark but have no idea what it means. I am like the caveman terrified by a lightning bolt.

There are several people here, however, who work with numbers or statistics and who presumably do have some feel for what's going on, what we can trust, what we can laugh off. What do they make of it all?

To answer John, yesterday I did a separate exerice as Zermelo with some other 6d teaching material on whole boards (may publish later) and decided to continue, playing the "best move" each time. What you then see is still a wild fluctuation of the win rate, as LZ discovers new lines down the road which it had previously pruned. At one point the winning probability flipped sides while it had been at 70% on another.

Conventional wisdom averages out. AI averages out but differently. The former seems to care more about local efficiency, the latter about global sente. Clearly AI values sente a lot but in some cases only discovers down the road what sente delivers, which is no surprise. Conventional play is more honte-ish because it is less optimistic/certain of what it can do with the initiative.

I don't think we're at a point yet where AI can convincingly decide on textbook moves. But surely they lead to unbiased and very interesting arguments. Like Bill says, we want to reread a problem as "IF you want to connect ... but here's a way of playing that focuses on sente instead of connecting"

Wow! What setting did you use? The more visits, the less chance of overpruning, I suppose. Or for analysis, there's the LZ version modified to do less pruning. https://github.com/AncalagonX/leela-zer ... v0.16-next

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I've thought about what useful this exercise actually teaches, and it maybe comes down to this: If a stronger player tells that some position is better than another, but you cannot with reasonable effort understand why, then just ignore that opinion for the time being. There's a good chance that he/she/it is wrong, and in addition, if you don't understand the reason, you probably cannot learn anything generalisable about the case anyway.

I don't know if this is something controversial or something that most people agree on already. At least for me it is a new way of thinking. I thought that a way to improve my positional evaluation is to use the judgement of stronger players as 'training data'. Now I feel I have used a significant portion of my study time just to learn some noise instead of the signal.

But of course you need to learn from _something_. I'm quite sure that we humans cannot learn zero-style (though for a devil's advocate, has this actually been proven...?). For instance from this 'Making good shape' book we can train our 'policy network' by taking the book solutions as training examples (just don't take the 'failures' as negative examples). We can also learn how to succeed in some local goals like cutting, connecting, or capturing some stones (but only when we actually understand the specific tactics clearly).

Obviously this does not come only from this little study. I think similar conclusion can be made in general from bot evaluation of joseki and pro opinions.