Computer Specs for Go Review with AI

xela · Post by **xela** » Tue Feb 25, 2020 5:31 pm

Bill Spight wrote:You understand that the win rate estimates based upon fewer than 100 visits are basically meaningless.

I think "meaningless" is too strong a word. If you care about the difference between a 53% move and a 55% move in the fuseki, then 100 visits isn't enough. But for reviewing your games and spotting kyu level mistakes (and even dan players make a handful of kyu level mistakes per game: weeding those out is a good way to improve), or for suggesting "here's another move you might have looked at" or "here's the natural move in this shape", small numbers of visits will still get the job done. Remember that the newer networks can play at dan level even on just one playout.

Of course more is better, and getting a GPU is a good idea :-)

Bill Spight · Post by **Bill Spight** » Tue Feb 25, 2020 6:03 pm

xela wrote:
Bill Spight wrote:You understand that the win rate estimates based upon fewer than 100 visits are basically meaningless.
I think "meaningless" is too strong a word. If you care about the difference between a 53% move and a 55% move in the fuseki, then 100 visits isn't enough.

As you may have noticed, I regard a 2% difference as within the margin of error, except perhaps with millions of visits.

But for reviewing your games and spotting kyu level mistakes (and even dan players make a handful of kyu level mistakes per game: weeding those out is a good way to improve), or for suggesting "here's another move you might have looked at" or "here's the natural move in this shape", small numbers of visits will still get the job done.

Well, that's what I recommend doing. Take the low number of playouts as suggesting a move, but do not rely upon the winrate estimate; make the play and get a high number of visits.

dino1019 · Post by **dino1019** » Tue Feb 25, 2020 7:06 pm

(Again, how AI helps human review a game, in particular a kyu player, I re-posted my comments at reddit/cbaduk here, note that the 3.5 vps speed for 40b on my notebook computer is shown above with a screenshot)

I have a few points to make here:

I am not sure whether a best move suggested by AI is a best move for a kyu level player and win rate (or score) itself might not be as relevant as win rate delta (or score delta). Note that win rate delta is the difference between the best move (the move with the highest win rate) and the actual move. If the delta is huge (>20%) then this move is probably a big mistake, and in this case, this is the key lesson for the player to learn. That's the reason I think the filter for the win rate delta in AI sensei is useful. (but 0.5 to 1 euro for full review of a game is too expensive)

The use case of game review with AI for me is Ah-Q app that connects to my notebook computer (Windows 10, via SSH) with CPU-only Leela Zero. Many people say 100 visits per move is good enough for a kyu player (15, 20, or 40 blocks?), this is not the case. First of all, we know that Leela Zero has a problem with ko, seki, ladder, and I also see a stupid mistake about life and death, if the visits per move is not high enough. In the meantime, I see most full game review services are with 1000 or 1600 visits per move, so I think this is a good reference number.

The reason I am looking at a PC with GPU is because the speed is important not only for a full game review but also for a live analysis. The live analysis in the above use case sees an issue if the calculation is too slow (in my current settings, it's only about 3.5 to 5 visits per second for LZ264(40b)), and in the process the best move can change around, which will be even more confusing for a kyu player (if the AI stabilizes quickly on one best move. then at least I know what the AI thinks the best move is).

I heard a 1060 GPU (4G) has about 200 visits per second with LZ 40b, this seems to a bit slow if 1000 to 1600 per move is preferred, so maybe a higher spec. GPU is better.

(Note in this context, visits = playouts = nodes)

xela · Post by **xela** » Tue Feb 25, 2020 7:41 pm

200 visits/sec is actually quite nice: you get to see LZ's "first thought", then see whether or not it changes its mind on further investigation over the next few seconds. So you get some insight into whether the best move should be obvious or whether it requires reading. Besides, if you're reviewing interactively, you probably can't take in information much more quickly than 5 seconds per move anyway. No harm in aiming for a faster GPU, but I'd say it's nice to have not essential.

The ladder problem is only minor, and pretty much solved with the 40b networks. The policy net can "see" the ladder shape and any ladder-breakers without any playouts. (I'm in the middle of testing this and hope to write up some results soon.)

Can the app work with KataGo, or is it LZ-only?

It's already been said here, but I want to emphasise: even with CPU only, if you're getting 3 visits/sec then probably something is wrong with your setup. On an i5 it should be going 5-10x faster. I don't know if it's the config on the notebook itself, or something to do with communication between the notebook and the app. But you might want to look into that before spending money on hardware.

jlt · Post by **jlt** » Wed Feb 26, 2020 12:10 am

I've self-reviewed a number of kyu games using 15-block networks, making sure that each move of each variation gets at least 100 visits (occasionally I let it run longer), and also had some games reviewed by strong players. The human's and the AI's suggestions can be divided into the following categories:

A. The human and the AI agree that a move was a big mistake, and suggest the same better move.

B. The human and the AI agree that a move was a big mistake, but the move suggested by the human would be considered as a mistake by the AI. Either the human didn't see the better move, or he saw it but chose not to mention it because he thought it was too complicated.

C. The AI thinks a move is a small mistake (about -3%), so is hard to detect in a self-review, but the mistake can easily be avoided by using human reasoning, so is pointed out by the human.

D. The AI notices a big mistake, and the human doesn't.

E. The AI makes a mistake.

Getting a fast computer eliminates "E", which occurs less than 1% of the time, but won't improve "C", which occurs much more frequently. So I still think that as far as kyu games are concerned, for teaching purposes humans are better than computers, and that the improvement we get from upgrading a computer is marginal.

Actually I haven't seen AI mistakes recently. I did notice a small number of mistakes with earlier networks like LZ157 at low playouts like:

misreading a ladder;
in a fight, suggesting to cut and create a weak group that ends up being killed without enough compensation

but I didn't notice such mistakes with Katago (since Katago reads ladders and counts liberties). It doesn't mean Katago doesn't make life-and-death mistakes, but they occur rarely enough so that it's not a problem.

Bill Spight · Post by **Bill Spight** » Wed Feb 26, 2020 1:05 am

It is not uncommon, even with top level bots with thousands of visits per candidate move, for a human to come up with a move that is completely, or almost completely off the bot's radar, but which, when played, has a winrate estimate within 1-2% of that of the bot's top choice, and is certainly quite playable. Sometimes the bot will give it an even higher rating than the bot's top choice.

dino1019 · Post by **dino1019** » Wed Feb 26, 2020 7:56 am

I want to comment on the five scenarios:
A, B, D are about big mistakes, actually there can be as many as 20 mistakes (both sides) for -5% mistakes, and several -20% ones for kyu players per game. This is where a kyu players can and need to learn and improve. As for whether AI and human teacher agree on the best move, I am not sure this is too much an issue, at least we need to know (1) this is bad move (2) for what reason, even a human teach can suggest more than one best moves, as well as the AI teacher. The thing a human teacher can do better is to suggest a good enough but not too complex move, which an AI teacher cannot yet.
As for C, the small mistakes (-1% to -3%), don't you think this is mostly for higher level players? Isn't this the difference between kyu and dan/pro players?
Therefore, I don't see why AI is not helpful for kyu players.
I do have a human teacher who helps me review games right now, for big mistakes the human teacher and AI teacher tend to agree that it is a big mistake, as for best moves, as discussed above, it's ok to have more than one, but understanding the concept and develop the ability to tell is important. The limitation of a human teacher is that you have to record the session or you might forget what the teacher has said, while the AI teacher can tell you the story over and over again. On the other hand, if the human teacher can "generalize the way of playing under specific situations" it's a plus, but from time to time, I cannot see the generalization and next time I still might not able to tell the right direction to play, or cannot judge the situation correctly. Another thing a human teacher can do but an AI teacher cannot is to recommend actions of the student to take, such as doing some kind of go problems, studying some books or videos.
Another related observation, it seems I have to study about 100 relevant go books before can become a dan player, but I guess many people can improve much faster than this.

jlt · Post by **jlt** » Wed Feb 26, 2020 8:28 am

About "C", I don't have a concrete example at hand but let me make an artificial one.

Click Here To Show Diagram Code: [go]$$ DDK mistake $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . O X . . . . . . . . . . . . . | $$ | . . . O . X . . B , . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , . . . . . , . . . . . , . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

According to 15-block LeelaZero, the last move is a very small mistake, -1.5%. Yet, if a player makes this mistake, I think it is important to correct it because it means he doesn't know what is the ideal extension from a wall, isn't aware of overconcentration problems, etc.

You might think that 1.5% is small, but in a 250-move game, small mistakes add up. Go game would be simple if just correcting big mistakes was enough to improve. Sometimes it's true, especially at DDK level, but my belief is that we need to pay attention to every detail, be patient and accept that a new piece of knowledge will result in an improvement of a tiny fraction of a stone.

Bill Spight · Post by **Bill Spight** » Wed Feb 26, 2020 9:33 am

dino1019 wrote:As for C, the small mistakes (-1% to -3%), don't you think this is mostly for higher level players? Isn't this the difference between kyu and dan/pro players?

My well informed opinion, having studied winrate differences for a few years, is that, even with 100k rollouts per move by a top bot (which is spread over a number of different candidate moves, so that the top choice usually has several thousand visits), we can't say that a winrate difference of 3% or less between the winrate of a play and the winrate of the bot's top choice is a mistake. If the top choice usually has only a few hundred visits, then I think that we require an even greater winrate difference before we think that a move is a mistake.

Kirby · Post by **Kirby** » Wed Feb 26, 2020 9:44 am

One thing that might help reduce noise would be to analyze the same position using different bots.

Bill Spight · Post by **Bill Spight** » Wed Feb 26, 2020 9:50 am

dino1019 wrote:Another related observation, it seems I have to study about 100 relevant go books before can become a dan player, but I guess many people can improve much faster than this.

IMHO, even better than the 100 books to shodan heuristic is the 1,000 games to shodan heuristic. But there is a caveat. 1,000 quick games against weak players is unlikely to get you very far. It is important to review your own games, something you are obviously doing. It is also important to play against stronger players, so that you do not pick up bad habits from your opponents, and so that your opponents can punish many of your errors. The usual advice is to play against players three stones stronger than yourself.

Good luck!

Bill Spight · Post by **Bill Spight** » Wed Feb 26, 2020 10:15 am

jlt wrote:About "C", I don't have a concrete example at hand but let me make an artificial one.

$$ DDK mistake
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . O X . . . . . . . . . . . . . |
$$ | . . . O . X . . B , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . , . . . . . , . . . . . , . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . O . . . . . , . . . . . X . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------
Click Here To Show Diagram Code
[go]$$ DDK mistake $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . O X . . . . . . . . . . . . . | $$ | . . . O . X . . B , . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , . . . . . , . . . . . , . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]
According to 15-block LeelaZero, the last move is a very small mistake, -1.5%. Yet, if a player makes this mistake, I think it is important to correct it because it means he doesn't know what is the ideal extension from a wall, isn't aware of overconcentration problems, etc.

You might think that 1.5% is small, but in a 250-move game, small mistakes add up. Go game would be simple if just correcting big mistakes was enough to improve.

As you probably know, I agree in general. I also sort of agree about this example. I would agree more if the extension were on the 3d line.

But the reason I agree in this case is that humans figured out long ago that this extension is too close to the two stones. We know why it is a mistake.

But if all we know is that a top bot thinks that a move loses 1.5% to par, we don't even know that it is a mistake. We can guess why it might be a mistake, bet we don't know. In this particular case we have corroborating evidence from human thought and experience.

Now, my Last Open Corner heuristic, which strongly favors occupying the last open corner (there being exceptions, OC), is largely based upon 1-2% differences. Sometimes you get differences of 10% or more, more often you get differences between 5 and 10%, and even more often you get small differences. Without the corroboration of the large differences, I would not advance that heuristic.

Bill Spight · Post by **Bill Spight** » Wed Feb 26, 2020 10:59 am

What do we teach a DDK about this move?

Click Here To Show Diagram Code: [go]$$Bc DDK move $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . O X . . . . . . . . . . . . . | $$ | . . . O . X . . 9 a . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , . . . . . , . . . . . , . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

15 block LZ with some number of rollouts says that this loses 1.5% to par. As it turns out, there is a recent pro game with the plays up to this point, Tuo Jiaxi, 9 dan (W) vs. Lian Xiao, 9 dan, 2018-08-06, which is in the Elf commentaries. This extension did not show up in the commentaries, getting fewer than 1500 rollouts, so I don't know what winrate estimate it got. The extension to a got 5.2k rollouts, for a loss of 1.3%. (Elf gives larger differences than LZ, so maybe LZ gives it something like a loss of only 1%.) Should we tell the DDK why the extension to a is better? Of course. It is established go knowledge and generalizes to other situations.

But Lian Xiao played

in the next diagram, and Elf agrees, with 26.7k rollouts. I also agree.

Click Here To Show Diagram Code: [go]$$Bc Elf's top choice $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . O X . . . . . . . . . . . . . | $$ | . . . O . X . . . a . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . 9 . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . , . . . . . , . . . . . , . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . b . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . , . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Elf also likes b, with a "loss" of 0.1% and 11.9k rollouts. Should we teach the DDK those moves, as well? I say yes. Go is not for the faint of heart.

But is a playable? Well, Elf prefers the other plays, but not by enough to say that a is a mistake. And in another game vs. Tuo, 2018-05-14, Zhao Chenyu, 6 dan, played that very extension.

Gomoto · Post by **Gomoto** » Wed Feb 26, 2020 11:16 am

Ke Jie:

Click Here To Show Diagram Code: [go]$$B $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . O . | $$ | . . . . . . . . . . . . O . . . . O X | $$ | . . . . O X . . B . . . . O . O O X . | $$ | . . . O . X . . . , . . . . X X X . . | $$ | . . . . . . . . . . . . O . . . . X . | $$ | . . . O . . . . . . . . . . . . X . . | $$ | . . . . . . . . . . . . . O . . . X . | $$ | . . . . . . . . . . . . . . . O O O . | $$ | . . . . . . . . . . . . . . . . X . . | $$ | . . . , . . . . . , . . . . O O X . . | $$ | . . . . . . . . . . . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . X . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . X . . . | $$ | . . X O . . . . . , . . . . . , . . . | $$ | . . X O . . . X . . . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Gomoto · Post by **Gomoto** » Wed Feb 26, 2020 11:17 am

Cho Chikun:

Click Here To Show Diagram Code: [go]$$B $$ --------------------------------------- $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . X . . . . . . . . . . . . . | $$ | . . X X O X . . . . W . . O X . . . . | $$ | . . O O . . . . . , . . . O . X . . . | $$ | . . . . . X . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . X . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . O , . . . . . , . . . . . , . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ | . . . O . . . . . . . . . . . . X . . | $$ | . . . O X . O . . . . . . . . . . . . | $$ | . O . O . . . . . , . . . . . X . . . | $$ | . X X . . X . O . . . . . O . . . . . | $$ | . . . X . . . . . . . . . . . . . . . | $$ | . . . . . . . . . . . . . . . . . . . | $$ ---------------------------------------[/go]

Life In 19x19

Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI

Re: Computer Specs for Go Review with AI