Click Here To Show Diagram Code

[go]$$W Taisha variation
$$ ------------------
$$ | . . . . . . . . .
$$ | . . . . X c . . .
$$ | . . . O X O b . .
$$ | . . X X O X a . .
$$ | . . X O W . . . .
$$ | . . . . . . . . .
$$ | . . . . . . . . .
$$ | . . . . . . . . .[/go]

This is a position resulting from a variation to the taisha joseki, where White has connected on the outside, which assumes the ladder at a works. On the Sensei's Library's page the 3 options for White's next move discussed are a, b, c. "a" is the ladder, "b" is sente and prepares to cast a net, "c" is a forcing move before playing the ladder.

The SL contributors have one of these as the best move, allegedly backed up by conventional wisdom, one as inferior and one as a classical mistake. When reviewing this with Lizzie however the evaluation is not the same.

What do you think? What is conventional wisdom? What does the AI say? Why?

Probably conventional wisdom says that 'c' is worse than 'a'. It's a forcing move that doesn't gain many points, and kind of aji-keshi. 'b' is a nice net, but seems worse to me...

If I'm correct that conventional wisdom says that 'c' is worse than 'a', then my guess is that AI recommends it :-p

_________________
be immersed

Click Here To Show Diagram Code

[go]$$W Taisha variation
$$ ------------------
$$ | . . . . . . . . .
$$ | . . . . X c . . .
$$ | . . . O X O b . .
$$ | . . X X O X a . .
$$ | . . X O W . . . .
$$ | . . d . . . . . .
$$ | . . . . . . . . .
$$ | . . . . . . . . .[/go]

I don't think that there is any conventional wisdom, which is why the taisha joseki has so many variations. However, Waltheri shows the ladder as the overwhelming choice, with d second. c does not show up in Waltheri, probably because it is a beginner's move.

I suspect that the bots generally prefer d. Trick question.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

_________________
be immersed

_________________
be immersed

I'll check again for 'd'.

Here's a first reply for the early birds.



edit: seeing Kirby's reply, my setup was colors reversed and two black stones at the bottom corners. Apparently the global setup matters.

As Knotwilg says, the global setup matters.

My preliminary investigation.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Bizarre. Repeating the exercise shows a different result. Katago, like conventional wisdom, wants to play the ladder.

Click Here To Show Diagram Code

[go]$$B Ladder
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . 8 . . . . . . . . . . . . |
$$ | . . . O O . 6 7 . . . . . . . . . . . |
$$ | . . X O X . 5 . . , . . . . . O . . . |
$$ | . O O X X . . . . . . . . . . . . . . |
$$ | . 2 X O 3 . . . . . . . . . . . 9 . . |
$$ | . 4 . 1 . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

Kata gives Black 42% and losing by >2 points. Black continues locally with and to get some more resilience for his group before taking sente with
The overall inferior position was already a fact after connecting on the outside in the taisha variation.

Click Here To Show Diagram Code

[go]$$B Forcing
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O O . . . . . . . . . . . . . . |
$$ | . 2 X O X . . . . , . . . . . O . . . |
$$ | . O O X X . . . . . . . . . . . . . . |
$$ | . 1 X O 5 . . . . . . . . . . . . . . |
$$ | . . . 3 . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . 6 . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . 4 . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

Forcing first and then play the ladder allows for a ladder breaker, which is conventional wisdom. The double approach outweighs the strong position obtained in the upper left. Black trails by 32% and >6 points. So the forcing move is a "4 point loss".

Well, it's been a good exercise, even if conventional wisdom was finally confirmed by AI. I can't reconstruct how I came to a different insight in the first place.

Interesting, I ran into this exact issue when testing ladder positions in LZ. I thought I'd start with your first diagram and see if the bot chose 'a' or 'b', but it kept wanting to play 'c' first! So I had to put that move on the board before I could begin asking LZ whether or not it liked the ladder.

I wasn't sure whether to start a new thread for this post, but (a) it was this thread that prompted me to post, and (b) it all seems to come under what I regard as the true message behind the thread title: inexplicable behaviour by bots. There's a lot of that about for us mere mortals, of course, but two areas seem to have been mentioned often that make us wonder whether the bot behaviour is not just inexplicable but may be plain wrong. Ladders is one. Life & death is the other.

I was looking at the problem below (White to play). This is from the Xianji Wuku (Arsenal of Immortals' Devices). What had sparked my interest was that this had the same name, Shooting Sparrows with Gold Pellets, as a faintly similar problem in the Xuanxuan Qijing. There are examples where the XW takes an XXQJ problem not as a straight crib (unlike the much cribbing Guanzipu) but as the basis for a slightly altered problem with a new twist. This turned out not so much to have a new twist but was in a much more practical configuration than the XXQJ version, so a definite improvement.




I could have left it there, but this thread was live, so I idly decided to check out the solution on Leela. It's actually quite easy for humans once you spot the pretty obvious caterpillar connection, and I wasn't really trying to confirm the solution as much as to see whether the whole area was big enough for Leela to even play there, instead of an another corner.

At first I had a bit of a problem to make Leela play in this corner - this is a known issue, of course, hence my interest. But I managed that with not too much effort by adding about half a dozen stones somewhere, but was then taken aback both by Leela's choice and by the list of candidate moves.

The correct answer (according to the ancients) is in the game record. But Leela chose A. In this corner it also ranked B (the "right" answer) highly. Initially, I thought it was demonstrating a flaw in the original solution. But the variations shown for both A and B show Black ending up alive.

The original problem has White killing Black. Someone here may confirm Leela was right and the ancients (and I) were wrong, but then something else quite different but still inexplicable arises. Why (if Leela thinks Black is alive) is Leela playing in this corner at all? All its moves are aji keshi.

I can more or less understand why a bot may miss a L&D solution - they are not trained on L&D and don't actually read the way we do - but I do think this should be a trivial problem for a bot. But I do have trouble in getting my head round why it plays ajikeshi.

Someone may come along and say katago does find it and gefurtel finds it but with a different line, etc. etc.

But what it all adds up to, it seems to me, and not just with ladders/nets and L&D but with very much else, is that you just can't trust the bots. They are fickle and mysterious. We can all agree they are stronger than humans, but is that just because they make fewer mistakes, or mistakes with fewer bad consequences, and not because their actual moves are actually correct even most of the time?

And when we think we are learning from bots, are we learning mistakes? I do notice that the fuseki moves that pros use after studying with bots have changed quite radically ov er the past couple of years, and the bots have themselves change their "opinions."

Sans doute. The very idea of winrates is predicated on mistakes by the bots.



This is why, IMO, the concept of margin of error for winrate estimates is important. Our preliminary results for the concordance of different bots is on the order of 80%. So if we want a single answer to what to play, I don't think we can get it for most positions. What I do think we can get is a good idea where not to play. It would help to know where the threshold is, though.



I think if we could find a good way to restrict or focus the search of bots, with maybe some refinement of conditions, they could beat humans at life and death, semeai, and so on, normally finding the correct moves. I still hope that bots can be trained to play difference games on separate boards and reliably compare two different moves. But I don't think I can get to that this year. And that is a more modest objective than finding the correct play. You have to make the correct play one of the moves to be compared.

Many thanks for this example. Not too long ago I ran across an example where a bot recommended a tenuki instead of saving a sufficiently large group that saving would gain around 14 pts., comparable in size to the opening move. In addition, the possible kill was of the kind that Baduk Doctor had recently shown as a kyu level problem, adding a stone to take away a half eye. Even with global reading, it seems unlikely that a top bot would miss the kill. So either the other play was quite hot, which I seriously doubt, or the bot somehow misread the local L&D, which is possible, or the bot reckoned that is was behind and was playing for a mistake, which I think is most likely. OC, a dan level human, and many kyu players, would save the group, except perhaps in a handicap game, because we would expect our human opponent to see the kill.

Given the way that bots are trained, it is not easy to get them to make technically correct plays, or to be sure that they have done so.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

If we are learning the wrong moves by studying the bots, then pros are also learning the wrong moves because they are studying the bots. Of course they are better at it than we are but that doesn't mean the activity in itself is wrong for us to do.

In these latest two inquiries, it turns out that
1) the AI confirmed the conventional wisdom to play the ladder, not the forcing move. I was wrong in my first investigation and stood corrected by Bill.
2) the AI suggested a different move than the move by Feng Lun, didn'teven take the professional move into account, because the diagram was wrong, as found by Bill.

I'm not thinking of AI as an absolute source of truth. I'm using it as a double check. Conventional wisdom / pro commentary / pro move says A, what do bots think? What can I learn from that? If they are confirmed, it becomes more reliable advice, if they aren't we should be more suspicious. It's not about adopting new gospel and throwing out old gospel, that's not how I approach it.

For a long time we have been saying that 3-3 invasions should not be played as early as what we then saw AI do. Today, pros have adopted these early 3-3 invasions and all resulting joseki. That's the reverse double check. I still dislike early 3-3 invasions and avoid them, but there's more esthetics and romanticism to that than rationalized improvement of play.

I don't think that is happening here. There were people who took that approach when AI go first hit the streets, but it's calmed down. In my view that's in VERY large part due to Bill Spight, who tirelessly reminds us about margins of error, variability an other factors - all opaque to me but I trust him.

However, I do wonder if there's nevertheless a tendency, possibly subconscious, to overrate how good bots are.

I say this partly on the grounds that in the past there was a similar tendency to overrate how good pros are. You can see this even among pros!

Yet when I did the Go Seigen match series (starting with Kamakura), which was based on synthesising very many pro commentaries, I was struck by how often there were major disagreements between top pros. In some cases this extended to "brilliant" move according to one pro, "awful move" according to another.

I can't put a figure on any of this, but Bill's estimate (in another thread) of "only" an 80% consensus among bots as to the best move feels about right to me on the basis of my experience with hundreds of pro commentaries. Some people will say 80% is a lot. I'm one of those who say the missing 20% is a lot. Would you accept laser eye surgery if told the success rate is about 80%. I use that as an example only because I know a lady who took similar odds on surgery and is now bitterly regretting it.

Maybe the rest of us should try even harder to put in the caveats and qualifications that Bill keeps putting in - while never denying the bots are stronger than us overall, of course, and while continuing worthwhile investigations like this thread.

This reminds me of a classical problem in my (and many's) professional area. I'm working for a digital mapping company and although the work has evolved from manual to automated, the basic principle is still that we take sources and compile them into a representation of the world that fits customers' needs / end user use cases. The left side problem is to know how close we are to reality (the right side is to understand what users will do with the representation). For example: do we have all the restaurants in France correctly mapped (right side: which ones do we really need)? Which are we missing? Which ones in our map our out of business? Etc ...

To solve this question, we compare sources, often a sample for scoping. If source A & B confirm the presence, that's better than A confirming it and B denying it. Or is it? After all, A & B themselves are representations of reality. Maybe A relies on us, or we rely on A, and both of us are wrong. So we have to understand mutual dependency of two sources and put that into the equation. And the freshness of the source.

Today you will see more "confirmation bias" when comparing pro moves with AI. Older games will probably show a bigger deviation hence will contain more cases where the pro was right and the AI turns out to be wrong. Maybe AI should go back to study the classics one day ...

I'm not sure I agree with the framing of this example. If you told me an airplane had a 99% chance of not crashing, I would not get on that plane. But if you told me 99% of bots say this is the best move, I am going to believe them.

But more important to me is the relative evaluation percentages. When 80% of the AIs prefer Move A to Move B, do they think Move B is fatally bad, or is it just a tenth of a percentage point worse? Learning to avoid huge mistakes is where AI shines, in my opinion.

Something that I have been looking at lately is the revision of the bot's evaluation of a human's play that was off the bot's radar or got relatively few rollouts. In reviewing pro games I was surprised how often the bot, given a fairer comparison, gave a higher winrate to the pro's choice over its own. The difference was typically less than 2%, but sometimes substantial. We are still talking about fewer than 10% of the plays, but it's not a rare occurrence.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Well, I'd probably say the same, but at the same time I'd say the framing here is suspect: apples and pears. I wouldn't get on the aeroplane because crashing would kill me. But I wouldn't care about the percentages on best move because it's just a game and has no effect on me. The bot 99% part also waves a red flag for me - the result of reading books like Freakonomics. I can't say whether it's suspect, but I wouldn't be surprised if a guy like Levitt shot it down.



I agree this is valuable (especially with Bill's various cautions), and I think it's how pros are mainly trying to use it. But, just in passing, I think a lot of pro plays that are being counted as pros having learnt something from AI may stem from a quite different impulse: experimentation. When Shin Fuseki became all the rage, very many pros tried it out simply to get a handle on it. We know this because many of them contributed essays, letters and whatnot to go magazines explaining what they were doing and (eventually) explaining why they did or did not agree with Shin Fuseki principles (I think the fairest simple summary, which might apply today with AI, is "Too hard"). That was an era of pros working hard to get in touch with fans. We are not seeing the same level of openness now with AI, in an age when fans are more or less taken for granted by pros. But there are some strong indications of experimenting (internet accounts among other things) and I'd be rather surprised if experimentation didn't actually account for most of the examples we see.

10 examples seemed a poor sample size to me and I have a tool for this.
I analysed 91 positions at 30k playouts with elf v2 (which is terrible at ladders):



Re: trust - I don't have very much confidence at all that bots or pros are giving us 'correct' moves for more ambiguous opening positions but I'm not particularly concerned. I think in go we all spend a lot of time learning things that aren't quite correct but if we're lucky they are correct enough to take us a little bit further.
I agree that bots are limited but can show us ideas to explore and develop our own individual understandings of. They're also great for spotting blunders!

I would like to borrow the position below (i.e. just before Knotwilg's analysis) to show that "conventional wisdom" and "AI" don't necessarily belong together (or maybe even on the same planet).

I happened to notice along the way to the original position that katago (in my case the 20-block net) does not necessarily want to cut at "a" below. Serious attention is given to the hane at "b" instead. Faithful readers may remember that in my case deeper analysis requires turning things on and then going somewhere else and doing something else for a while. So I did and I did. The question is, what was the resolution that I found when I returned?

Click Here To Show Diagram Code

[go]$$Bc What is the next move?
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O O b . . . . . . . . . . . . . |
$$ | . . X O X . . . . , . . . . . O . . . |
$$ | . . O X X . . . . . . . . . . . . . . |
$$ | . . a O . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21