I played this game with KataGo and as you can see I didn't do very well

This really wasn't unexpected. Especially since I did something different. I decided I'd only play next to my previous stones or one space jumps (sometimes I'd jump over the white stones). Why did I do this strange thing - you may ask. Well, I had played some games with KataGo were I first setup some impossible situation and either tried to live a group or kill a group. Yet, if you look at points loss in these games it is not that remarkable and this is despite myself not even trying to play to win the normal game of Go.

Now, what is a big mistake? When discussing games and variations these days it is likely that someone will quote the computer and its points loss or percentages. Such discussions appear to always be underpinned by the point of view, one that I regard as something of a fallacy, that larger points loss means it's a worse mistake. A counterpoint to this view is that there are mistakes that the player already knows how to avoid and then there are mistakes that the player never saw coming. When we put it this way, mistakes don't appear to have much to do with the how big the points loss is.

In the first case, above, it doesn't matter how big the mistake is the player already knows to avoid it. Not that, avoiding making mistakes of this sort doesn't require lot of practice. In the second case, it is not clear from the context what the remedy is; maybe there is something new that needs to be learned. However, what if every time you are in danger of making mistakes you could see it coming and defuse the problem? If you can do this and not play worse moves than you would make if the dangerous situations were allowed, then you would be much, much, tougher player. Right?

It is in this context that I offer the following game where I only played one space jumps and next to my own stones as a study. How bad can bad moves really be before it becomes ridiculous? Maybe next time you make a bad move you can think if you could simply have played some one space jumps

Maybe someone cares to guess which is the worst move in terms of points loss and how bad it was? Or suggest better one spaces

So your advice is "don't play moves which you already know are bad"?

The idea of reviewing games using a quantitative filter is to figure out if the "bigger" mistakes lead to any particular understanding. You can also ignore the size of the mistakes and try figuring that out for all moves. You can also disregard any evaluation by AI and try figuring it all out for yourself, like in the old days.

When reviewing my own games, I take my own thought process as a guideline and AI as the control mechanism. Then I take the big mistakes as per IA, to understand the unknown unknowns. Likewise for pro grames, I take pro comment and AI evaluation. You can disagree with and mock the approach but I don't see which meaningful alternative you are suggesting.

Still taking the challenge:
5, 9 11, 17, 31, 33, 49

No. I just believe that the biggest hurdle for most people is to consistently apply what they have already learned. You can't really expect to perform at a high level when the performance requires something you have only just began to learn, something you never had time to internalize.

How about an example of a fictious 10 kyu who can consistently play about 50 good enough moves in a game. They can be said to know how to play some good moves, however they lack the ability to consistently play more of those good enough moves. The potential for improving is huge, I'm calling him Fico now, Fico analyzes every one of his games with the computer and he read on the Internet that he should only worry about moves that have very large points loss because everyone, even pros, make many moves with small points loss all the time. This rings true for Fico, he knows he isn't close to their level anyway and the Internet is trustworthy source for everything. It is not like Fico could play better moves than the pros, now can he? And when Fico analyzes his games on his oversized GPU it clearly indicates everyone except top pros are making small errors all the time but Fico himself is consistently playing 10 or so moves that lose 10+ points; that is the difference, isn't it? Fico thinks he has confirmed what he read to be true!

Well, maybe Fico isn't right though. What if Fico would only play one space jumps? He wouldn't be on the brink of SDK glory if he played that way, he would hardly win a game - ever! You can see why in my game with KataGo. All the same, the number of large 10+ errors would go down, it would go to ZERO.

My advice to Fico would be to learn to identify natural moves and to practice, and when he makes mistakes to focus on the first and last critical mistakes and see if he can find some solution which he is confident he could also implement in a game. But this wasn't really about teaching Fico.


You seem to take an overly hostile approach here. It is all well that you wish to use "a quantitative filter [...] to figure out if the "bigger" mistakes lead to any particular understanding", hopefully it is also fine that I have concluded that the magnitude of the mistake isn't all that interesting when you studying. Since we hold such apparently opposite views shouldn't it be more interesting to converse about the topic instead of rejecting them out of hand.

Did you consider if there were big (points loss) mistakes in the game with KataGo?

I think kvasir is on to something. There was (from memory) a Kansai Ki-in pro who recommended a way of playing in which every move was to be a one-space jump (ikken tobi) or diagonal step (kosumi) - alternately. Alternately means in terms of the local position, not in terms of time. Obviously, if you had to connect or the like somewhere, you did that, but the idea was that with your alternate ikken tobi/kosumi shape, you were likely to be making efficient shapes.

This, of course, was well before AI and it was not entirely novel even then. There is as per a well-known proverb to play that way locally. This pro, however, recommended it for an overall way to play. As I recall it, the next stage was to go on and add small knight's moves (keima) to your armoury. All the while, you were supposed to be studying what effect the various shapes had. You were storing in your intuition the DNA strands that make up life & death in go. A knight's move has a big weakness, which you can study by comparing that shape to the more secure ikken tobis and kosumis.

This approach is not about winning games directly: it is about studying - creating a go brain for all seasons, not a lawnmower that doesn't work unless the weather's right. You are studying in vitro, not in vivo, so that you can control and/or predict outcomes.

In Japan, this approach was taken a step further when the Nihon Ki-in published an "all about" series, such as "All about the hane" and "All about the sagari". Again, the idea was to absorb the nuances of each shape and its associated suji.

I will revert to my current favourite analogy of ballet (Because pros there are dedicated to constant study and a level of discipline that resembles that of go pros). Every ballet pro, every day, spends 60-90 minutes on barre work, all levels of dancers in the class doing exactly the same exercises. These can start with what you might regard as the ikken tobi of the ballet world - the tend (the pointed foot extended along the ground to the side, then the front, then the back. Then you might do a plié (bending the knees to go down on your hunkers). And so it goes on, the most basic moves practised repeatedly for best part of an hour (every day, remember). Only then do they do the keimas of the go world - jetés and leaps in the centre of the room. But even then they all do the same thing. It's only in later rehearsals that they practice the tricky stuff - the tsumegos, josekis and fusekis of the ballet world.

In short, both go players and ballet dancers who follow this sort of regime are concentrating on their technique (suji in go). As a result they make very, very few gross mistakes. In real, live action, The dancers react in ways go pros would recognise. They do make mistakes, but they are rarely apparent to the audience (e.g. doing three pirouettes instead of four). Many mistakes or omissions are because of fatigue, not lack of knowledge or technique, or in order to play safe.

I got the impression kvasir was likewise recommending studying technique and burning it into one's intuition. I would suggest that AI is not necessary to do this, and may even be a distraction unless you can train yourself to regard it as nothing more than a strong opponent, as opposed to an oracle.

Always on the assumption that I've not entirely missed the point of kvasir's experiment, I believe pro practice (there are similar things in the Chinese and Korean literatures) confirms the tenor of his thoughts.

For those who may be tempted to try this approach themselves, a word of warning. This is not about studying good shape in the form of pretty patterns with names, like sake bottles and tables). It is not about katachi (static shape) but about suji, which is the technique of creating or adapting shapes dynamically. My estimate of the ratio of mentions of suji and katachi in the Japanese literature is at least 9:1. But how many times do you hear westerners talk of suji relative to katachi? 1:99999?

Probably, but you don't need AI review, or pro commentaries or go books to apply what you already know. You need focus, concentration, clarity of mind, managing time and emotions.



We're both on the Internet. Fico might believe me or you, it doesn't make the internet more or less trustworthy.



Where would he get the idea?



Good advice.




Ah, the old "you're not only wrong but also stubborn" trick. And we're only 1 post in the debate.

You are playing a deliberately bad game, which fails a review approach I'm using for games where I deliberately play well. And it's not the only thing I do when reviewing, far from it actually. The fact there are other approaches doesn't invalidate this one.

Where you try to invalidate it, you are applying a logical fallacy. The fact that known bad moves don't necessarily result in big point differences is not logically opposed to the statement that big point differences can point to major mistakes. Moreover it's not a practical way for learning with AI. What other criterium could you use when using AI for discovery? Or should we simply ignore AI altogether then? Fico is confused.



Yes. DId you consider my answer?

That reminds me of a French go teacher, of whom I saw a video in which he was reviewing his pupil's games, backed with Katago as it's the norm nowadays. However, sometimes he was saying things like "for the AI this is a big mistake but for me it's fine, this move makes a lot of sense" (as we can often hear here and there, often based on the assumption that "IA is thinking in terms of perfect play, the move you made is enough for we poor humans") and also, more unexpectingly, things like "AI says this is a good move, but I disagree ; it is a mistake and you should learn how to avoid it".
In this specific situation, this move can maybe be followed by a 14-moves sacrifice sequence that may work correctly (and one that the player obviously did not spot), but locally the player missed a tesuji that was right and safe, and worthy to remember for later games, in which similar situations can occur.

I liked this attitude.

I too am under the impression that, focusing only on "big points-loss mistakes according to AI", we tend to be obsessed by some very complicated and extremly specific game-maker moves that can end a particular fight, and to forgot about plain lines of play. Thinking only about the twisting point itslef, and not the whole process leading to such a twisting point to exist — a proper "suji", if I understand correctly what John Fairbairn try to teach to us.

You can benefit from all those things but only if you really do know what to apply. A typical kyu player might know a knight's move is a good defence against an approach move. But he knows it only because AI, pros or books have told him so. You can also benefit from instruction (or practice) in seeing good moves that you do "know" hidden away in a complex position. You need to understand the nuances, which includes cases when an apparently good move is actually a bad move (and vice versa).

This is not limited to dan players or tactics. I "know" what tewari is and can usually explain the merits of moves on that basis. But I recently looked at a commentary by Chen Zude on a game by Huang Longshi. Chen pointed out that Huang played in a certain way because a tewari analysis proved the opponent had effectively lost a move. This was essentially a question of strategic evaluation. I understood the explanation instantly, but would never have thought of applying my knowledge of tewari to that particular position. And of course only a human could put me on the right track.

I have these "Ah, so" moments in almost every commentary I read. Focus, concentration, clarity of mind, managing time and emotions play no part in these deliberations. I have failed simply because my knowledge is too stiff, not flexible enough to adapt, and limited by the relatively few previous contexts I have seen where this knowledge is relevant. Accordingly, a lack of hubris needs to be added to those attributes. I don't really know what I know. On the other hand, I know enough as a dan player to know that I am still very weak.

Those reviews and commentaries could reinforce the lessons that you are learning.



I think the moves in the game give a different perspective on bad moves. Many moves in normal games are already worse in terms of evaluation than mistakes made when someone is not really playing Go. I can put it bluntly: when trying to play Go you shouldn't be happy with moves that are no better than if someone is not playing Go.

This deliberately bad game is in some ways better than many peoples normal games. For example, if the criteria is the largest points loss then it is better than most of my games for one thing. Of course most of the bad moves are early in this game, which is usually not what happens. I suspect that if I started playing in the same way in a middle game position the the errors would also be smaller than one would maybe expect.

As to how Fico can review games with AI. How about considering which good moves were available and not pay much attention to the indicated magnitude of the error? When Fico finds a mistake, maybe he should asses it in terms of which moves were available and if he thinks he could have played those moves. Thinking in terms of AI evaluation might not be needed, especially for players at Fico's level. However, he could reflect on if he would have been better off playing some half baked one space jump or really anything that is clearly not a full move (but this might be too advanced for Fico), that could help him identify moments when he is losing the plot in some way.



You were still editing your first message when I was replying. There was only the first line when I started writing the reply. I saw the rest later but didn't connect what the numbers where. I'll post the evaluations in the next post.

The summary statistic for the score change for the black moves on 10k playouts are:

Minumum: -4.1
25% Percentile: -2.0
Median: -1.0
75% Percentile: 0.1
Maximum: 0.7
Mean: -1.1

Note that this is as displayed in KaTrain which sometimes shows the best move as positive change in score.



Ranked based on score change these moves are number:
4, 9, 17 (shared), 7, 21 (shared), 3, 17 (shared)

Your summary statistics are as follows:

Minumum: -3.7
25% Percentile: -3.1
Median: -2.6
75% Percentile: -1.7
Maximum: -1.5
Mean: -2.5

Not so bad suggestions.


Here are the score change and rank for each move if anyone is interested in that.

This is brilliant!

So I decided to try it. Voila! I beat KataGo at 10k playouts with 5 stones. I'm not sure if I took liberties with how to alternate ikken tobi and kosumi, I assumed I meant to pick which ever I liked.

The only problem is that some will suspect I was playing GnuGo and not KataGo. However, white really is KataGo. It's the superb robustness of the ikken tobi and kosumi that left white with very few options to challenge black. As it turns out I was able to keep the lead through the endgame, the endgame turned out to be easier for me than usually.

I wonder. Is a move that tries to achieve something but fails badly, eg. because of a misread, or because it was the start of an "all-or-nothing" abandon sequence, really worse than a random tobi that doesn't even try to do anything meaningful ?
I think I would be happyer with the former.

Usually if you are certain that your move works you can just play it. You should trust yourself. However, if it turns out that you misread often then you are likely doing your game some disservice, unless you can calibrate your sense for when something works.

Many all-or-nothing moves are played when one side is losing anyway and they want to create a chance or just go out with a bang. It is strategic and it's only important if you can increases your practical chances this way or not.

The point is also that if some half baked move doesn't give up your chances to win the game then there should be many other better moves that might prefer. Maybe some adventurous move strikes you as fun, there is nothing wrong with that, but maybe the adventurous move is a mistake if it damages your chances more than a move you would not be happy to play. Conversely, if your move is better than every move that you wouldn't be happy with, then this move is probably fine.

What is stopping us from consistently finding equal or better moves than those half baked one spaces? Maybe we could learn to do exactly that.

Food for thought indeed.

I'd offer that the main factor stopping us is 'reading' ability, and inexperience of standard sequences.

I want to come back on the initial debate, without being hostile or defensive - ok maybe a little defensive. Let me state again how I think one should review their own games. In fact it can be observed: BadukDoctor is doing so in his videos.

1. right after the game, open it in an editor for review; NOT AI
2. go through the game, remembering your thought process, the choices you made, the things that you realized were changing the outcome for better or worse, while they were being played
3. open the game again, in an AI interface
4. go through it again, asking AI the questions or checking the ideas developed in (2)
5. where does AI find a major mistake (point of % difference) which didn't even occur to you

(5) is the point getting criticism here, as a heuristic for finding key learning points. I'm not doing (5) exclusively in my reviews and I don't recommend it as an exclusive, all encompassing device either. When reviewing pro games with commentary, I substitute (2) with the commentary. When reviewing pro games without commentary, there's not so much you can do besides (5). You can form your own opinion about the game but chances are low you can second guess the pro decisions. I would say such exercises are probably more entertaining than educational.

On another note, I reviewed a 10 kyu game, first without AI. The idea was to next review with AI to see if the major lessons I would draw from the game, towards the 10 kyu, would be possible to draw by the 10 kyu for themselves, using AI. After my own review, I'm skeptical indeed, because the play is all over the place. I'll still conclude the exercise to figure out if "big point differences" is a good heuristic for self review at the 10k level.

Unless by 'here' you mean L19 in general, I fail to see where this is happening. As I understood the OP, AI is not even central to the debate. Katago was just a convenient way to find a strong player for the experiment. It could just as easily have been a friendly pro you had locked up in your wardrobe.

I concede that the reference to "big mistakes" could be a red herring, but even there the tenor of the whole OP suggested to me that what was mean was "important mistakes" - nothing to do with numbers.

Women's football is fascinating analogy.

Imagine a penalty kick. A premier league player misses one and the game ends in a loss instead of a draw. That's a big as well an important mistake. The pro and his coaches (the equivalent of using AI in go) will spend a lot of time analysing ways to make sure that doesn't happen again. This analysis will include advanced things such as the goalkeeper's size, his preference for one side or the other, but also other arcana such as the state of the pitch, or the game, or the weather, and of course the temperaments of the various possible penalty takers.

But in women's football we used to see penalty kicks being missed simply because the penalty taker could not kick the ball hard enough or accurately enough. The coaches in women's football therefore concentrated on improving fitness and strength and intuition, and simply practised kicking in general. We have seen the results for ourselves - the women's game is now blessed great skill and excitement, at least at the highest levels.

Amateur dan players in go are still at the level of the early female penalty takers. Remember that Go Seigen famously said of a player "he's very weak - he's only a 4-dan pro." As proof of his words, he beat Yasunaga Hajinme 4-dan down to four stones. So, I for one, certainly think that amateur dans playing with AI, except as an experimental device, are being wrongly obsessed with bling and "Go faster" decals.

What they need, instead, is go's equivalent of kicking or heading a football reliably every time. One of the main words you come across in Japanese books by go pros who give advice to weak amateurs is to play "consistently." So (again on the assumption I co0rrectly understood kvasir's intent), your task is to find a way of playing in good style (i.e. with good suji or good flow) so that, while your moves may not be the best by pro standards, they are all reliable and consistent - and contain no big mistakes.

Only when you acheive that standard can you step up and offer to take the penalties in the World Cup.

As a concrete example, consider a typical large-handicap game. White approaches at c6 and Black defends at c3. That is a mistake but not a big mistake. It's good suji (a kosumi that sets up an couple of ikken tobis). It's just not the best (and much more difficult) suji of an ikken tobi or keima. Black's "par for the grade" move might not even elicit comment. But consider if Black instead of c3 plays the burasagari at c4. That's NOT good suji, outside of specific contexts. It would just elicit a forcing move on the other side at f3, and Black is already halfway to a heavy, overconcentrated group and White has a couple of flexible outside stones. Such a move as that burasagari would certainly elicit comment. Amateur teachers tend to say, "you should normally play e3 or e4 here." That is, "in this situation, do this or that". Pro teachers tend to instead to highlight the "bad suji." That is, "in ALL situations, play good suji as the basic minimum."

My interpretation is that kvasir's experiment bears this out. Nothing to do with AI directly.

OP says: "When discussing games and variations these days it is likely that someone will quote the computer and its points loss or percentages. Such discussions appear to always be underpinned by the point of view, one that I regard as something of a fallacy, that larger points loss means it's a worse mistake."



I didn't understand the OP that way. Because strong players won't usually comment in terms of point or percentage differences.

In the meantime, the experiment has led to an interesting tangent, so I'm willing to let go of what I perceived as the kernel.

Your comments confirm that there is a widespread view that the magnitude of the change in evaluation is of crucial importance. It is in the fifth step of your method that the computer evaluation appears to be taken at face value to identify key learning points. This is not to the exclusion of all other approaches and it doesn't appear to be the primary tool. However, when it comes to professional games without commentaries it is the primary tool, at least this what I understood.

This post wasn't directed at your method of reviewing games or anyone else's. And, of course, there are other widespread views that I also find suspicious (i.e. that every minuscule changes in evaluation is important, that crossing from evaluating that white is winning to black winning is very important,...). However, this experiment is more relevant to how changes in score evaluation are perceived than changes in other form of evaluation.

There is also, like John points out, a broader point. That is a comment about what is important when reviewing and studying from the human and personal perspective. Just as with the example of penalty kicks in women's football, it would similarly be difficult to make much progress in Go unless the player's weaknesses are identified correctly.

The only way to be objective would be to take your role as a player as the starting point. It is after all a game between two human players. In contrast, the computer's evaluation will only tell you what the likely result is if two similar programs play each other from some position. This is often very different from what happens if you were to play another human player. That is, the computer evaluation is not the most objective evaluation. To have a more objective outlook you need to add your own perspective of the game and that of your opponents.

Similarly, if you wish to find your own mistakes it is not objective to conclude that if it is a mistake for someone else then it is a mistake for yourself. Especially, if the someone else is a computer program, which "thinking" you can't even begin to imitate and it is inevitable that you will play poorly in comparison.

There was also the other point. Sometimes we try so hard to fail spectacularly. That is, any move would have been better than our move. One quality of good players is striking a balance between pushing to find good moves and avoiding bad moves. Or more precisely, this is a quality that good human players exhibit.

You understand correctly and I do stand by that criterium. One reason to treat "mistakes" by order of point difference is that larger differences are falling outside the uncertainty boundaries of the system. Another reason is they are likely to impact the game more. It's almost a circular reasoning trying to explain this. That's probably the reason why it's widespread too ...



Since these minute variations are close to the precision of the system, these are generally disregarded. I don't find those practices so widespread.



Of course. But AI evaluation is a starting point. And since we don't have Sai on our shoulders to tell us on a daily basis what we're doing wrong and why, AI is the best we have. Repeating: we shouldn't just stare at it. We should reason for ourselves and then verify. And with many repetitions of that try finding patterns. We can ask a forum like this for a shortcut too. At 1-2d, I won't get massive reply with high confidence. So I resort to self review with AI.




I would argue the opposite. Adding your own perspective makes it subjective.



I beg to differ. As a father, perhaps, and then still there are proven practices to raise a kid. As a Go player, hardly. I might enjoy the game more when killing a big group than by playing better endgame but that doesn't mean a 10 losing move which is a reckless attack is "good for me".

I dislike analogies with a passion but penalty kicks are proven to statistically score more when kicked high in the corners. If you know the goalkeeper, you may factor in personal differences. And there remains a fair deal of psychological warfare. When you decide to go for the middle because you thought you outsmarted the keeper, the stats will tell you the high corners were better options. When you go for a high corner and you fail, the lesson is to train harder on high corners, rather than going for the middle next time. There are exceptions. My compatriot Eden Hazard was extraordinarily good at waiting until the very last moment, observe the keeper and change direction in the very last fraction. We're talking top professionals now. Top pros are entitled to defy the odds.

At least you will learn more from the former: it was played with good faith but turned out to be a mistake with massive impact. Maybe the lesson was very specific, maybe there's a technical pattern in there, maybe the general lesson is to read very hard in a fight and bail out to a conservative move when uncertain.

Deliberately playing badly, there's no learning. The fact that deliberate bad moves may not result in big point differences is not disproving that big point differences are likely to point to major mistakes, either.

But I think the OP's main point is that proper learning can't be correlated to major point swings in AI evaluation. I strongly disagree with that point of view. If learning doesn't reduce the randomness of the game, to bring the swings closer around the equilibrium, you're learning the wrong things.

I think it's a terrible analogy, but an instructive one :-)

If you kick a ball and it doesn't go where you intended or as fast as you intended, then it's immediately obvious what happened. Likewise if you're running towards the ball but an opponent gets there first because they're faster, or you have to slow down because you're tired... I think amateur footballers are well aware of the gaps in their technique and fitness.

In go, if you make a bad shape, or a bad choice about direction of play, the consequences of the mistake can be tens of moves later. And it's dependent on the opponent knowing how to punish the mistakes: in amateur games you often won't see any consequence at all. Without a teacher beside you, you can't see clearly. "Play more consistently" is good advice, in the sense that if you do it, you'll get better results. But it's not useful advice unless it's accompanied by a lot more instruction on what typical small mistakes look like and how to avoid them.

When you sit down to review a game without a teacher in the room, but with access to an AI, what are you going to do? Look at the numbers, or not look at the numbers? The numbers may not be perfect, but they're better than getting no help at all. We've already had many conversations about how the numbers can sometimes be misleading, and how to interrogate the machine, compare alternative moves, get its evaluations on different lines of play, get more information out of it than a single number... You need to work a bit to filter out the signal from the noise, and you won't get it right every time. (You won't get accurate, clear and useful advice from a human teacher every time either.) But that's not a reason to give up. Don't throw the baby out with the bathwater!

Practically speaking, if the AI review says you made three mistakes of 10 points or more, and 30 mistakes of around half a point, where are you going to start? Some of the half-point mistakes might be highly significant psychologically, and thinking about them might help you on the road to more consistent play. But many of them will lead you down paths that are not going to help at all. And if you can't see clearly, you don't know which is which. You can spend a lot of time lost in the wilderness. If you look at the 10-point mistakes, you won't strike gold every time, but I think you've got a lot more chance of finding something useful.