(quoted from another thread)


Setting aside the phenomenon of bots ignoring moves that are almost as good, and focusing just on the fact that if you want a bot to analyze a specific move much more accurately you can play that move on to the board - I'm wondering why this isn't already intuitively obvious to most people. It has seemed completely obvious to me ever since MCTS bots existed - a bot isn't analyzing enough a move that you want to know about? Well play it on the board and now 100% of the analyzing will be for that move.

This isn't special to Go or MCTS even either. It's also what I see tons of people do all the time intuitively with chess bots, online and offline. Chess bot suggests move A puts you up a pawn's worth. What about move B instead? Well, play it on the board and have the chess bot tell you now whether you're still up a pawn or whether actually now you're even or behind.

Maybe subtle differences in Go UI and Chess UI are what result in this? Or maybe just the Go community is still just really inexperienced with what the Chess community has known by now for decades?

This has surprised me a bit as well. I think part of it is that Go players are still learning to use and interpret these sorts of tools, but another aspect is that a fair amount of current Go AI analysis is not performed interactively in practice; you feed your game to GoReviewPartner or OGS or what have you and get back a report. So it's not as natural to a lot of people (yet) to use these tools to explore interactively rather than perform a static analysis. I don't think this was ever such an issue with chess engines, although comments such as "Kasparov's 23rd move was not considered by any engine, and they don't see how good it is until three moves later!" have always been popular.

Is it a common blind-spot to not realize you can check that way?

I thought John was not pointing out the technique so much as the consequence. Though the bot didn't consider our move, that doesn't imply that it's not a good move--sometimes it's just that the bot has a very different style.

_________________
Occupy Babel!

Probably lightvector was pointing to the phrasing: "too common - troubling even". If it's easy to explore other options that might be good, what is there to be troubled about?

_________________
be immersed

Precisely.

It is glib to say the technique is easy and obvious. It is debatable to say it is obvious, because people unfamiliar with how these programs work (like me) don't know what to trust. There's been enough argument among "experts" about the meaning and/or reliability win rates on this site, for heaven's sake.

It may be easy (when you know how) but that doesn't mean people do it. It's also easy to take your litter home with you - how many do it?

My strong impression is that this "easy" thing is not done very often here (which is why Bill's efforts stand out to me) and the result is that comments typically of the type "the human played here but the bot played there" - which I find "too common" - leave a residual sense that the human was a duffer, even if he was a pro. That is what I find "troubling."

My own sense of what is going on is that bots beat humans mainly because they are much more consistently accurate. They do not get distracted or tired. They do not make rash moves just because the human has gone to the toilet. For the most part I believe the pros do know what they are talking about. as more of these "easy" checks might show. And of course they can talk about everything far better than the bots.

You were curious about Showa's move. You tried it out to see what your bot thought, and got an answer. Seems like you figured it out.



Easiness depends on how you are using Lizzie/LZ. If you have a pro game loaded in Lizzie with pondering mode on, as you go through the game, you'll see LZ's analysis of the moves you're exploring in the game tree. From this perspective, it's pretty easy to get a bot's analysis of what the pro played. In addition, you'll also see a handful of alternative moves that LZ may have considered - but it won't show you all possible moves. Nonetheless, the move the pro played in the game you reviewed is there already, because that's the move the pro played.

The example you gave was slightly different in that you wanted to explore a move that wasn't loaded or played. LZ had some possible suggestions, but it's not exhaustive. But then when you played the move, you got the analysis, as expected.

So if you are interested in seeing what LZ thinks of a particular move, I would recommend either loading that move from an SGF or playing that move. LZ won't explore every possible move, especially if it's not one that's part of the game you're reviewing...

Good luck!

_________________
be immersed

I think this IS specific to MCTS. Average users may not be aware of the differences between chess-style and go-style analysis - they assume the bot analyzes the position, and that's all there to it.

To an extent this is true: the bot should find a move whose "real" winrate is not far from the objectively best move's one (its search would return garbage otherwise). But this only means unconsidered moves are unlikely to be significantly better (and most of them likely significantly worse).

BTW this is partially the UI's fault as well. LZ for example allows some custom shaping of its search tree, the UI should actively extend analysis to all branches it sees in the SGF game tree, even if the bot would not consider the game move by itself. This will likely be done as UIs mature.

One advantage of playing the move if it one of the bot's lesser choices is that playing the move will give you more playouts, and therefore a more reliable winrate estimate. In one case I saw the winrate estimate gain 20%, making it the new first choice.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Even when a move is not the AI's choice, the move is fine if the winrate isn't affected much. So when analyzing with an AI, I mostly look at moves where the winrate clearly drops. And even if the winrate drops quite a bit, crosschecking with KataGo may reveal it only loses 1 point. Also in those cases, I wouldn't consider it a reason to deeply investigate alternatives. It's more like an inaccuracy than a clear mistake (at least for my level of mid-dan).

Why isn't playing a move you want a bot to analyze obvious?

Analyzing with a bot is a skill obviously

Sorry for wording things a bit bluntly - yes I agree very much with everything you mentioned! The thing that I find striking is that is that at least among the Chess players I know, very few if any of them would suggest "the human played here but the bot played there" would imply that the human move was bad, without further analysis. And among strong players I've seen a lot of practical knowledge about the ways bots are and aren't useful for human analysis.

And it's not as if the evaluations given by (pre-AZ) Chess programs are much better than winrates. Ostensibly they're scaled to be units of material (i.e. pawn = 1), but in practice you quickly notice that they often differ significantly from it due to all sorts of positional factors or tactics, sometimes with no obvious or fixed correspondence to the material in a given position. So in practice they too are an opaque value with no useful intrinsic meaning other than "higher is better on some ill-defined scale". But experienced players generally already know that very small differences are not necessarily trustworthy, and have an understanding for what differences actually do mean in practice. And I highly doubt that it's due to most of them having read a statistical error analysis written by Chess bot devs of the flavor that that Bill has wished from Go bot devs.

These observations might not be entirely representative as I haven't actually delved *that* much into Chess. But to the degree they are at least somewhat accurate, I suppose what's going on is that there hasn't actually been a generation of strong Go players who have grown up with the bots from the start. And I guess that there's also not that much overlap between the players of the two games, so all the confusions in the Go community are unravelling only painfully slowly, despite the fact that seemingly right next door in computer Chess thousands upon thousands of people already have the answers.

This reality is only a few years away.
Like the generation who never saw a world without global connections at their fingertips.

I doubt it, too. They are relying on experience. And I'm gaining experience, too, feling my way along as I try to understand the Elf commentaries. That's why I have revised my estimated margin of error for Elf (when not too far from 50%) from 5% to 4%. But chess players are ahead of us by a decade or two in gaining that experience. And it seems to me that win rate estimates are better than centipawn estimates in that they are checkable. It's just that nobody has bothered to check. If I were in academia I might do the research.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I admit I have been guilty of suggesting the human move was inferior because the bot didn't consider it, although the qualification was largely implicit and in the eye of the beholder. As I'm gaining more experience with LZ (my current analysis tool) I have learnt to be cautious, not always to trust the bot and to investigate these unseen moves by feeding them. Although this is a small community, my misguided efforts should not have a large impact.

The title of this thread is a bit quora-trolling-alike. "Why is X" while X is debatable. For those who have installed an analysis tool, it is obvious to play a move you want the bot to analyze. I'm usually a late adopter of technology (and once I have a brand, I stay loyal). I'm a natural conservative, not an early adopter. I think if you are smart enough to play go, you are smart enough to install a bot for analysis purposes. It's just a matter of following instructions.

Now that we're here, what frustrates me when analyzing with LZ is, in order of priority:

1. That White starts out with 55%, suggesting komi is too large, but we don't seem to think that part of her assessment is true.
2. Ladders. They are not rare and can play a big role in a local evaluation. Then you see the ladder forming and facepalm ...
3. When you play out a suggested sequence with winning percentage x% on the first move, on any move (i) in that sequence the winning percentage x_i can vary by a lot.

We don't? I think it's true, and I thought most people did too. So long as it's understood in the correct context, i.e. LZ thinks that in games between itself white has a slight advantage with the 7.5 komi. That such an advantage exists in games of a superhuman player doesn't mean a weaker player can make significant use of that advantage.

P.S Title is obvious to me too
P.P.S I think Bill really should install Lizzie.

People should also have this in perspective, which is much easier with the new version 0.7 of Lizzie. Start with an empty board. Switch engines for LZ to katago (under engines at the top of the window). Katago likes White with a winrate of about 55.3%. Now go into the Game menu at the top, click on Set gameinfo and set the komi to 6.5 instead of 7.5. Now katago likes Black with a winrate about 54.3%. So we are talking about nearly a 10% difference due to one point of komi.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

I think there's three things going on here:

• UI design
• The "width" of go compared with chess
• The nature of Monte Carlo tree search

To my mind, go is a much more inherently visual game than chess. The idea that a diagram with numbered moves can show not just a board position, but also the history of a game -- you just can't do that in chess. Chess players are moving pieces around, not adding them, so it's much more natural to think in coordinates or in text.

Another factor is that the chess board is so much smaller, and involves six different types of pieces in an asymmetric starting position (no need to think about mirrored or rotated positions), so coordinates each have their own individual character. H7 is that square where white wants to sacrifice a bishop. C5 is where black tries to undermine white's pawn centre in hypermodern openings. And so on. You don't have the cognitive overhead of: "N12, now is that a knights move away from the stone I played 20 moves earlier, or is it a large knight's move?"

This feeds into the UI design. When I used to analyse chess positions using SCID, you have a window with the board position, then a separate text window with moves, evaluations and variations. All text. Drawing things on the board doesn't help. But with Lizzie, you can light up the board to show how the engine is "thinking". The immediacy of it is great, but it can also be misleading. A "missed" move sticks out in a way it wouldn't in a text list.

Width: in chess there are rarely more than 20 legal moves in a position, and often only two or three moves that are worth considering in the sense that they don't lose the game obviously and rapidly. If a move isn't near the top of the engine's list then it's generally not worth considering. Exceptions are rare and pretty obvious (e.g. where one player has such a large advantage that nearly any move will be a winning move). In go ... well, I don't need to spell it out in this forum!

And tree searching. Chess engines will evaluate every legal move (although they'll explore some more deeply than others), so nothing is completely overlooked (although once in a while an evaluation will be in error). MCTS guided by a neural net is just weird. The fact that it's possible to ignore very good moves and still play at a superhuman level is hard to understand, but we have plenty of empirical evidence that it's possible! With modern computing power, there should be no problem rewiring an engine to give, say, 10 playouts to every legal move before going into the usual search strategy. You'd lose a bit of strength, especially in fast games, but it might improve things as an analysis tool?

Obviously true, but there are already indications...

I was intrigued by the following fuseki, played in an exhibition game at a festival for the Nihon Ki-in to thank fans for their support. It was between two new pros. Both are just 13 and so in virtually their entire go careers Zen and later AI bots have been available. From their styles of play we can safely assume they have been using them.



I can only talk in terms of impressions, but FWIW it is normal now for even the more established players to adopt the AI style for the first few moves. They then revert to human type but throw in a few AI-type shoulder hits later on, if they remember. These young players, in contrast, seem to continue the AI style throughout the entire long fuseki.

As to what is characteristic about that style - and again I stress I am giving only my own impressions - I think it is that there is a strong emphasis on the seriai (running battle) style but with the modern difference that the fighting is at more at a distance. Whereas seriai fights in go used to be galleons pounding each other at close quarters, clashing, and being boarded with hand-to-hand fighting ensuing, now a seriai fight is battleships taking pot-shots at each other a couple of miles apart.

A few other changes result from that. For example, extraneous things like reconnaissance (probes) becomes more important. But a more fascinating one, to me, is that the concept of good shape seems also to be changing. Prophylactic shapes are becoming more flexible (in crude terms: wider). We might even say shapes are becoming more adventurous. If this right, I assume it is all part of the drive to achieve maximum efficiency and overconcentration.

Obviously I'm too weak to say the players in the above game have achieved naximum efficiency, but I think most of us here are strong enough to say there is extraordinarily little overconcentration (a couple of examples of potential heaviness that cancel each other out, perhaps?).

There is, incidentally a very early potential ladder in what is considered an AI joseki. Does that not confuse the bots?

Well, AlphaGoTeach gives White only about a 3% advantage, and the Elf commentaries give White about a 1% advantage. Both use a huge number of playouts.



It's the first player, Black, who has the possible advantage as the skill level increases. So if anything, as bots get better we should see the estimated advantage for White decrease with the 7½ pt. komi. But, for human players, anyway, statistics for over 40 years have indicated that the modal result on the board in an even game has been 7 pts.



You've come to the right place. Dealing with ladders is right up lightvector's alley.



The number of playouts is a factor there. That's why the Elf commentaries do not continue such sequences when the number of playouts drops to 1500. Even with a bot as strong as Elf, the Elf team did not trust its winrate estimates with so few playouts.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Really (rushes to LZ's download page) ?!

and

Really (now not so sure anymore what to think of bot based analysis) ?!