Some Lizzie Ai questions

[JoeS1]

Hi I recently downloaded Lizzie and was using it to review some pro games to see how many mistakes pros make in a game. I had a few questions.

Does it matter what GPU you use? I'm using a 760 GTX, old card, so I was wondering if that affected the strength and ability to pick the best moves in the program.

What is policy exactly? I click the show policy option and it has some numbers and different colored moves, but the blue move in policy isn't always the recommended move from the program.

Is the first engine Leela Zero and how strong is it compared to Kata Go the second engine? Is it beatable by humans?

When you change komi from 7.5 the default to 6.5 the AI can compensate for that right?

[CDavis7M]

I haven't used Lizzie too much and I'm not an AI expert but maybe I can help.

Hi I recently downloaded Lizzie and was using it to review some pro games to see how many mistakes pros make in a game.

Interesting idea.

Does it matter what GPU you use? I'm using a 760 GTX, old card, so I was wondering if that affected the strength and ability to pick the best moves in the program.

It only affects the wait-time. Even a CPU does well enough if you are patient and just analyzing a few moves instead of an entire game.

What is policy exactly? I click the show policy option and it has some numbers and different colored moves, but the blue move in policy isn't always the recommended move from the program.

Policy refers to the neural network. It takes a board position as input, runs that through layers of a neural network, and outputs predictions of good moves to play. The policy neural network does not perform any "reading." It's sort of just pattern recognition. These recommendations/predictions are the starting points for reading variations.

Is the first engine Leela Zero and how strong is it compared to Kata Go the second engine? Is it beatable by humans?

They are similar enough in strength that it doesn't matter. It is beatable by humans and it is technically beatable by a monkey. Depends on luck and the handicap.

When you change komi from 7.5 the default to 6.5 the AI can compensate for that right?

First, I don't understand what the question is. Are you asking if changing an AI setting does not actually change the AI setting? Second, why would you change the komi in that way? Doesn't Leela default to Chinese rules/scoring with the standard 7.5 komi? I haven't messed with komi with Leela and I'm not sure if you can change it. I do set komi to 0 on KataGo sometimes for classic games. But KataGo isn't set to 7.5. The komi change does work and you can see by the analysis changing.

[jlt]

I don't have a GPU and don't use LeelaZero but in my opinion:

• What is relevant is the number of playouts. My guess is that with 100000 playouts at each move, both LeelaZero and KataGo are superhuman, and over 95% of their moves are equally good or better than top pro moves. Even my slow laptop without GPU can detect many mistakes by European pros with just 1000 playouts per move (however humans sometimes find better moves than the computer).
• Top policy moves are moves that are explored more often by the AI, but may not be the best moves (like first instinct is not always the move you choose once you have explored alternative moves).
• KataGo is stronger than LeelaZero.
• KataGo understands Chinese and Japanese rules, and understands different values of komi.

[thirdfogie]

I have used Lizzie and KataGo as suggested at the top of this thread.
The resulting SGF file is further processed to convert the output from KataGo
into human-readable comments and annotations. You still have to supply your own
insight, but it is sometimes possible to see the merit in KataGo's suggestions.

The following game shows what can be done. It was analysed with about
5000 or 6000 playouts - I forget which. It is Game 6 of the second Kisei
in 1978 and is notable for the fact that Kato (with black) opens with the High
Chinese and Shuko replies with the Low Chinese. I originally chose to look at
this game because it was not clear why Kato resigned on move 226. The
automatically-generated comments on moves 26 and 37 are examples that are not
a million miles from human understanding.

The LZ[] tags have been removed because they make the file very large and
because some SGF readers refuse to handle them.




Edited for clarity

The date in the posted file is April 2022, which represents the time I processed
it, not the date the game was played. I may remember to fix that in future, if
there is a future.

Further edited to remove a spurious comment that was added for test purposes.

[JoeS1]

When you change komi from 7.5 the default to 6.5 the AI can compensate for that right?

First, I don't understand what the question is. Are you asking if changing an AI setting does not actually change the AI setting? Second, why would you change the komi in that way? Doesn't Leela default to Chinese rules/scoring with the standard 7.5 komi? I haven't messed with komi with Leela and I'm not sure if you can change it. I do set komi to 0 on KataGo sometimes for classic games. But KataGo isn't set to 7.5. The komi change does work and you can see by the analysis changing.

Within the Lizzie program you can play against the AI and change Komi on the New Game option. I was asking whether it could process this, because I actually won a game by 0.5 points after setting Komi from default 7.5 to 6.5 and I thought the program was suppose to be stronger than a pro. So either my 760 was to blame or I confused the program, because when I reviewed it with KataGo I had like 0.3 percent chance of winning in the mid game. On White's turn several moves would all pop up with a 99.8% chance of winning and it started playing bad moves that allowed me to invade it's moyo. It was a bizarre game. I'd embed it but I don't know how to embed games on here.

[Cassandra]

I'd embed it but I don't know how to embed games on here.

Please refer to

viewtopic.php?f=5&t=833

for advice.

[lightvector]

Within the Lizzie program you can play against the AI and change Komi on the New Game option. I was asking whether it could process this, because I actually won a game by 0.5 points after setting Komi from default 7.5 to 6.5 and I thought the program was suppose to be stronger than a pro. So either my 760 was to blame or I confused the program, because when I reviewed it with KataGo I had like 0.3 percent chance of winning in the mid game. On White's turn several moves would all pop up with a 99.8% chance of winning and it started playing bad moves that allowed me to invade it's moyo. It was a bizarre game. I'd embed it but I don't know how to embed games on here.

Leela Zero only supports komi 7.5, and will play as if this is the komi regardless of what it actually is. It also doesn't optimize for score, so it may indeed play bad moves if it thinks they won't change the result with komi 7.5.

[JoeS1]

Well I keep getting the game data is invalid message anytime I try to embed it through Eidogo. Not sure what I did there as it loads on Drago and CGoban just fine. Even tried creating a New SGF file on CGoban and just merging the moves there, still invalid. No big deal I guess. Just thought it was interesting seeing the AI play the way it did. There appears to be a vast difference in moves between KataGo and LeelaZero. Reviewing the game with KataGo showed that KataGo was suggesting better moves. LeelaZero was throwing away points endgame, as was I, made 2 big mistakes. I just don't know why it played so poorly.

Within the Lizzie program you can play against the AI and change Komi on the New Game option. I was asking whether it could process this, because I actually won a game by 0.5 points after setting Komi from default 7.5 to 6.5 and I thought the program was suppose to be stronger than a pro. So either my 760 was to blame or I confused the program, because when I reviewed it with KataGo I had like 0.3 percent chance of winning in the mid game. On White's turn several moves would all pop up with a 99.8% chance of winning and it started playing bad moves that allowed me to invade it's moyo. It was a bizarre game. I'd embed it but I don't know how to embed games on here.

Leela Zero only supports komi 7.5, and will play as if this is the komi regardless of what it actually is. It also doesn't optimize for score, so it may indeed play bad moves if it thinks they won't change the result with komi 7.5.

Well that's interesting, but even if it's stuck to 7.5 komi, it would have lost by more points had I not made a couple huge endgame mistakes. Just now reviewing the game I corrected the mistakes and had LeelaZero play from there and it missed a squeeze that cost a lot of points while in the original game it caught the squeeze. Something isn't right.

[JoeS1]

I have a few more questions I remembered about the AI.

I noticed sometimes there is a move with a higher percentage, sometimes a whole percent that is in the red, while another move is in the blue, is there a reason for it not switching the colors when there's a higher percentage move?

Are the variations it gives only the best for the current color playing? I've seen it hang on a variation for a long time, tried to play through it but by the 4th move, the opponents move, the AI changed the move it recommended. The original move in the variation wasn't even considered by the AI and was considered a mistake I guess. Found this odd.

Is there anyway to force the AI to consider a move? Like hovering over a point hitting a button, then the AI reads what would happen if you played there?

[jlt]

The move (A) in blue is the one which has been explored most often. Another move (B) with higher percentage may be red because it has been explored less than A, so the calculated percentage has low confidence. But the engine will start exploring it more, and the color of B will change. If the percentages stay stable, then B will eventually turn blue. But if the percentage of B drops, then B may stay red or even disappear.

Are the variations it gives only the best for the current color playing? I've seen it hang on a variation for a long time, tried to play through it but by the 4th move, the opponents move, the AI changed the move it recommended. The original move in the variation wasn't even considered by the AI and was considered a mistake I guess. Found this odd.

Is there anyway to force the AI to consider a move? Like hovering over a point hitting a button, then the AI reads what would happen if you played there?

AIs are like humans, sometimes they are like "oops I didn't expect move 4, so in fact my move 1 was a mistake". Most of the time, after say 100000 playouts the AI will find a better move than that move 1, but in complicated situations humans may occasionally find better moves than the AI.

[JoeS1]

AIs are like humans, sometimes they are like "oops I didn't expect move 4, so in fact my move 1 was a mistake". Most of the time, after say 100000 playouts the AI will find a better move than that move 1, but in complicated situations humans may occasionally find better moves than the AI.

Is there a certain threshold of variations the AI needs to read before discovering these types of mistakes? I was assuming the 3 thousand or whatever was enough to determine the the best 4th move the opponent would play, but if you're talking about 100k, I realize my GPU might not be fast enough to process that many. I don't think it goes above 3k to maybe 5k variations before moving onto another move.

[thirdfogie]

Is there a certain threshold of variations the AI needs...

On my box, one can press 'a' on the keyboard and then tell Lizzie how many visits
to use for each move. The programs will then do as many visits as you tell them to.
If your GPU is relatively slow, the process will take relatively longer. I typically
analyse my own games with 5000 visits per move. It then takes 60 to 70 minutes to
analyse a full game: less if there is an early resignation. I have an NVIDA
GeForce GTX 1660, which supposedly has about 1/3 the performance of the current
top-of-the-range GPU that Robert Jasiek lusts after. My box runs Linux and your
mileage may vary, but the point is the only limit on the number of visits is how long
you are prepared to wait for a result.

[jlt]

Is there a certain threshold of variations the AI needs to read before discovering these types of mistakes? I was assuming the 3 thousand or whatever was enough to determine the the best 4th move the opponent would play, but if you're talking about 100k, I realize my GPU might not be fast enough to process that many. I don't think it goes above 3k to maybe 5k variations before moving onto another move.

On my version of Lizzie, I just click through the game and the AI explores variations from move n when the board position is on move n. So you can just stay a few seconds on easy positions, and let it ponder a few minutes on complicated ones. The AI with 3k-5k playouts per move is already way stronger than European pros but humans may occasionally find better moves in complicated fights. I don't know if 100k is a threshold, I mentioned this number because I've seen (rare) situations in which the AI discovered the human move after about 100k playouts, but I can't make many tests to check since my computer is too slow.

[lightvector]

There is no threshold, and there will never be a threshold until the point where the entire game is perfectly solved, which we are astronomically far away from (5x6 is the largest rigorously solved board, 7x7 is the largest board where we might be able to plausibly claim optimal play in all possible main lines without being able to prove it).

As you increase the number of playouts, mistakes will become very gradually more rare, but there is no magic number past where mistakes will stop happening. Mistakes will always happen. Including misjudging moves that some human players would correctly find - (in the most extreme case, for no other reason than that there are thousands of top-level human players that all think in different ways and will sometimes play different moves, some of which may happen to choose the better move).

[JoeS1]

Thanks for the replies. I was trying to find more info on Lizzie AI and came across KaTrain and AlphaGo Zero/Master games which I didn't know existed, and have been replaying those. Surprising to see it still makes mistakes as well, but they are typically giving up a point or two endgame or answering a KO threat in a way that adds another unnecessary KO threat. Those are the easy things even a kyu player could see. It almost seems like there should be a second pass where the program takes the top moves and analyzes which ones lead to the most points gained locally or something. This seems feasible endgame when there aren't many possibilities left.

When searching for a guide on Lizzie I found mention of KaTrain and have been messing with that. Lizzie doesn't seem to have an option to end the game and count the score. KaTrain does, but after downloading all the modules, I noticed it only seems to do a few hundred variations across a few moves total per minute. Is KaTrain using a more advanced version of KataGo? I'm on Lizzie version 0.7.4 which appears to be the latest version.

I'm using the Open CL, 1.11 windows 64 bit engine, because I think that's the only one for GPUs, but there are 6 models. There are 15 block, 20 block, 30 block, 40 block, latest model, strongest model. Does anyone know how these compare to one another? Is Lizzie using the 15 block? Most I could find was 40 block was better than 20 block, but I didn't notice much of a difference in calculations between 40 block and strongest version. Still seems incredibly slow.