It is currently Thu Mar 28, 2024 4:34 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 24 posts ]  Go to page 1, 2  Next
Author Message
Offline
 Post subject: Some Lizzie Ai questions
Post #1 Posted: Fri May 13, 2022 8:24 pm 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
Hi I recently downloaded Lizzie and was using it to review some pro games to see how many mistakes pros make in a game. I had a few questions.

Does it matter what GPU you use? I'm using a 760 GTX, old card, so I was wondering if that affected the strength and ability to pick the best moves in the program.

What is policy exactly? I click the show policy option and it has some numbers and different colored moves, but the blue move in policy isn't always the recommended move from the program.

Is the first engine Leela Zero and how strong is it compared to Kata Go the second engine? Is it beatable by humans?

When you change komi from 7.5 the default to 6.5 the AI can compensate for that right?

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #2 Posted: Fri May 13, 2022 11:46 pm 
Lives in sente
User avatar

Posts: 714
Liked others: 109
Was liked: 138
Rank: Shokyu
Universal go server handle: CDavis7M
I haven't used Lizzie too much and I'm not an AI expert but maybe I can help.

JoeS1 wrote:
Hi I recently downloaded Lizzie and was using it to review some pro games to see how many mistakes pros make in a game.
Interesting idea.

JoeS1 wrote:
Does it matter what GPU you use? I'm using a 760 GTX, old card, so I was wondering if that affected the strength and ability to pick the best moves in the program.
It only affects the wait-time. Even a CPU does well enough if you are patient and just analyzing a few moves instead of an entire game.

JoeS1 wrote:
What is policy exactly? I click the show policy option and it has some numbers and different colored moves, but the blue move in policy isn't always the recommended move from the program.
Policy refers to the neural network. It takes a board position as input, runs that through layers of a neural network, and outputs predictions of good moves to play. The policy neural network does not perform any "reading." It's sort of just pattern recognition. These recommendations/predictions are the starting points for reading variations.

JoeS1 wrote:
Is the first engine Leela Zero and how strong is it compared to Kata Go the second engine? Is it beatable by humans?
They are similar enough in strength that it doesn't matter. It is beatable by humans and it is technically beatable by a monkey. Depends on luck and the handicap.

JoeS1 wrote:
When you change komi from 7.5 the default to 6.5 the AI can compensate for that right?
First, I don't understand what the question is. Are you asking if changing an AI setting does not actually change the AI setting? Second, why would you change the komi in that way? Doesn't Leela default to Chinese rules/scoring with the standard 7.5 komi? I haven't messed with komi with Leela and I'm not sure if you can change it. I do set komi to 0 on KataGo sometimes for classic games. But KataGo isn't set to 7.5. The komi change does work and you can see by the analysis changing.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #3 Posted: Sat May 14, 2022 12:05 am 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
I don't have a GPU and don't use LeelaZero but in my opinion:
  • What is relevant is the number of playouts. My guess is that with 100000 playouts at each move, both LeelaZero and KataGo are superhuman, and over 95% of their moves are equally good or better than top pro moves. Even my slow laptop without GPU can detect many mistakes by European pros with just 1000 playouts per move (however humans sometimes find better moves than the computer).
  • Top policy moves are moves that are explored more often by the AI, but may not be the best moves (like first instinct is not always the move you choose once you have explored alternative moves).
  • KataGo is stronger than LeelaZero.
  • KataGo understands Chinese and Japanese rules, and understands different values of komi.


This post by jlt was liked by: thirdfogie
Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #4 Posted: Sat May 14, 2022 1:21 am 
Lives with ko

Posts: 128
Liked others: 148
Was liked: 29
Rank: British 3 kyu
KGS: thirdfogie
I have used Lizzie and KataGo as suggested at the top of this thread.
The resulting SGF file is further processed to convert the output from KataGo
into human-readable comments and annotations. You still have to supply your own
insight, but it is sometimes possible to see the merit in KataGo's suggestions.

The following game shows what can be done. It was analysed with about
5000 or 6000 playouts - I forget which. It is Game 6 of the second Kisei
in 1978 and is notable for the fact that Kato (with black) opens with the High
Chinese and Shuko replies with the Low Chinese. I originally chose to look at
this game because it was not clear why Kato resigned on move 226. The
automatically-generated comments on moves 26 and 37 are examples that are not
a million miles from human understanding.

The LZ[] tags have been removed because they make the file very large and
because some SGF readers refuse to handle them.




Edited for clarity

The date in the posted file is April 2022, which represents the time I processed
it, not the date the game was played. I may remember to fix that in future, if
there is a future.

Further edited to remove a spurious comment that was added for test purposes.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #5 Posted: Sat May 14, 2022 7:59 am 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
Quote:
JoeS1 wrote:
When you change komi from 7.5 the default to 6.5 the AI can compensate for that right?
First, I don't understand what the question is. Are you asking if changing an AI setting does not actually change the AI setting? Second, why would you change the komi in that way? Doesn't Leela default to Chinese rules/scoring with the standard 7.5 komi? I haven't messed with komi with Leela and I'm not sure if you can change it. I do set komi to 0 on KataGo sometimes for classic games. But KataGo isn't set to 7.5. The komi change does work and you can see by the analysis changing.

Within the Lizzie program you can play against the AI and change Komi on the New Game option. I was asking whether it could process this, because I actually won a game by 0.5 points after setting Komi from default 7.5 to 6.5 and I thought the program was suppose to be stronger than a pro. So either my 760 was to blame or I confused the program, because when I reviewed it with KataGo I had like 0.3 percent chance of winning in the mid game. On White's turn several moves would all pop up with a 99.8% chance of winning and it started playing bad moves that allowed me to invade it's moyo. It was a bizarre game. I'd embed it but I don't know how to embed games on here.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #6 Posted: Sat May 14, 2022 8:22 am 
Lives in sente
User avatar

Posts: 1308
Liked others: 14
Was liked: 153
Rank: German 1 Kyu
JoeS1 wrote:
Quote:
JoeS1 wrote:
I'd embed it but I don't know how to embed games on here.

Please refer to

https://lifein19x19.com/viewtopic.php?f=5&t=833

for advice.

_________________
The really most difficult Go problem ever: https://igohatsuyoron120.de/index.htm
Igo Hatsuyōron #120 (really solved by KataGo)

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #7 Posted: Sat May 14, 2022 11:55 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
JoeS1 wrote:
Within the Lizzie program you can play against the AI and change Komi on the New Game option. I was asking whether it could process this, because I actually won a game by 0.5 points after setting Komi from default 7.5 to 6.5 and I thought the program was suppose to be stronger than a pro. So either my 760 was to blame or I confused the program, because when I reviewed it with KataGo I had like 0.3 percent chance of winning in the mid game. On White's turn several moves would all pop up with a 99.8% chance of winning and it started playing bad moves that allowed me to invade it's moyo. It was a bizarre game. I'd embed it but I don't know how to embed games on here.


Leela Zero only supports komi 7.5, and will play as if this is the komi regardless of what it actually is. It also doesn't optimize for score, so it may indeed play bad moves if it thinks they won't change the result with komi 7.5.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #8 Posted: Sat May 14, 2022 1:15 pm 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
Well I keep getting the game data is invalid message anytime I try to embed it through Eidogo. Not sure what I did there as it loads on Drago and CGoban just fine. Even tried creating a New SGF file on CGoban and just merging the moves there, still invalid. No big deal I guess. Just thought it was interesting seeing the AI play the way it did. There appears to be a vast difference in moves between KataGo and LeelaZero. Reviewing the game with KataGo showed that KataGo was suggesting better moves. LeelaZero was throwing away points endgame, as was I, made 2 big mistakes. I just don't know why it played so poorly.
lightvector wrote:
JoeS1 wrote:
Within the Lizzie program you can play against the AI and change Komi on the New Game option. I was asking whether it could process this, because I actually won a game by 0.5 points after setting Komi from default 7.5 to 6.5 and I thought the program was suppose to be stronger than a pro. So either my 760 was to blame or I confused the program, because when I reviewed it with KataGo I had like 0.3 percent chance of winning in the mid game. On White's turn several moves would all pop up with a 99.8% chance of winning and it started playing bad moves that allowed me to invade it's moyo. It was a bizarre game. I'd embed it but I don't know how to embed games on here.


Leela Zero only supports komi 7.5, and will play as if this is the komi regardless of what it actually is. It also doesn't optimize for score, so it may indeed play bad moves if it thinks they won't change the result with komi 7.5.
Well that's interesting, but even if it's stuck to 7.5 komi, it would have lost by more points had I not made a couple huge endgame mistakes. Just now reviewing the game I corrected the mistakes and had LeelaZero play from there and it missed a squeeze that cost a lot of points while in the original game it caught the squeeze. Something isn't right.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #9 Posted: Sat May 14, 2022 8:17 pm 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
I have a few more questions I remembered about the AI.

I noticed sometimes there is a move with a higher percentage, sometimes a whole percent that is in the red, while another move is in the blue, is there a reason for it not switching the colors when there's a higher percentage move?

Are the variations it gives only the best for the current color playing? I've seen it hang on a variation for a long time, tried to play through it but by the 4th move, the opponents move, the AI changed the move it recommended. The original move in the variation wasn't even considered by the AI and was considered a mistake I guess. Found this odd.

Is there anyway to force the AI to consider a move? Like hovering over a point hitting a button, then the AI reads what would happen if you played there?

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #10 Posted: Sat May 14, 2022 10:34 pm 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
The move (A) in blue is the one which has been explored most often. Another move (B) with higher percentage may be red because it has been explored less than A, so the calculated percentage has low confidence. But the engine will start exploring it more, and the color of B will change. If the percentages stay stable, then B will eventually turn blue. But if the percentage of B drops, then B may stay red or even disappear.

JoeS1 wrote:
Are the variations it gives only the best for the current color playing? I've seen it hang on a variation for a long time, tried to play through it but by the 4th move, the opponents move, the AI changed the move it recommended. The original move in the variation wasn't even considered by the AI and was considered a mistake I guess. Found this odd.

Is there anyway to force the AI to consider a move? Like hovering over a point hitting a button, then the AI reads what would happen if you played there?


AIs are like humans, sometimes they are like "oops I didn't expect move 4, so in fact my move 1 was a mistake". Most of the time, after say 100000 playouts the AI will find a better move than that move 1, but in complicated situations humans may occasionally find better moves than the AI.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #11 Posted: Sun May 15, 2022 3:03 pm 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
jlt wrote:
AIs are like humans, sometimes they are like "oops I didn't expect move 4, so in fact my move 1 was a mistake". Most of the time, after say 100000 playouts the AI will find a better move than that move 1, but in complicated situations humans may occasionally find better moves than the AI.
Is there a certain threshold of variations the AI needs to read before discovering these types of mistakes? I was assuming the 3 thousand or whatever was enough to determine the the best 4th move the opponent would play, but if you're talking about 100k, I realize my GPU might not be fast enough to process that many. I don't think it goes above 3k to maybe 5k variations before moving onto another move.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #12 Posted: Sun May 15, 2022 4:34 pm 
Lives with ko

Posts: 128
Liked others: 148
Was liked: 29
Rank: British 3 kyu
KGS: thirdfogie
Quote:
Is there a certain threshold of variations the AI needs...

On my box, one can press 'a' on the keyboard and then tell Lizzie how many visits
to use for each move. The programs will then do as many visits as you tell them to.
If your GPU is relatively slow, the process will take relatively longer. I typically
analyse my own games with 5000 visits per move. It then takes 60 to 70 minutes to
analyse a full game: less if there is an early resignation. I have an NVIDA
GeForce GTX 1660, which supposedly has about 1/3 the performance of the current
top-of-the-range GPU that Robert Jasiek lusts after. My box runs Linux and your
mileage may vary, but the point is the only limit on the number of visits is how long
you are prepared to wait for a result.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #13 Posted: Sun May 15, 2022 10:19 pm 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
JoeS1 wrote:
Is there a certain threshold of variations the AI needs to read before discovering these types of mistakes? I was assuming the 3 thousand or whatever was enough to determine the the best 4th move the opponent would play, but if you're talking about 100k, I realize my GPU might not be fast enough to process that many. I don't think it goes above 3k to maybe 5k variations before moving onto another move.


On my version of Lizzie, I just click through the game and the AI explores variations from move n when the board position is on move n. So you can just stay a few seconds on easy positions, and let it ponder a few minutes on complicated ones. The AI with 3k-5k playouts per move is already way stronger than European pros but humans may occasionally find better moves in complicated fights. I don't know if 100k is a threshold, I mentioned this number because I've seen (rare) situations in which the AI discovered the human move after about 100k playouts, but I can't make many tests to check since my computer is too slow.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #14 Posted: Mon May 16, 2022 6:11 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
There is no threshold, and there will never be a threshold until the point where the entire game is perfectly solved, which we are astronomically far away from (5x6 is the largest rigorously solved board, 7x7 is the largest board where we might be able to plausibly claim optimal play in all possible main lines without being able to prove it).

As you increase the number of playouts, mistakes will become very gradually more rare, but there is no magic number past where mistakes will stop happening. Mistakes will always happen. Including misjudging moves that some human players would correctly find - (in the most extreme case, for no other reason than that there are thousands of top-level human players that all think in different ways and will sometimes play different moves, some of which may happen to choose the better move).

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #15 Posted: Sat May 21, 2022 6:41 pm 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
Thanks for the replies. I was trying to find more info on Lizzie AI and came across KaTrain and AlphaGo Zero/Master games which I didn't know existed, and have been replaying those. Surprising to see it still makes mistakes as well, but they are typically giving up a point or two endgame or answering a KO threat in a way that adds another unnecessary KO threat. Those are the easy things even a kyu player could see. It almost seems like there should be a second pass where the program takes the top moves and analyzes which ones lead to the most points gained locally or something. This seems feasible endgame when there aren't many possibilities left.

When searching for a guide on Lizzie I found mention of KaTrain and have been messing with that. Lizzie doesn't seem to have an option to end the game and count the score. KaTrain does, but after downloading all the modules, I noticed it only seems to do a few hundred variations across a few moves total per minute. Is KaTrain using a more advanced version of KataGo? I'm on Lizzie version 0.7.4 which appears to be the latest version.

I'm using the Open CL, 1.11 windows 64 bit engine, because I think that's the only one for GPUs, but there are 6 models. There are 15 block, 20 block, 30 block, 40 block, latest model, strongest model. Does anyone know how these compare to one another? Is Lizzie using the 15 block? Most I could find was 40 block was better than 20 block, but I didn't notice much of a difference in calculations between 40 block and strongest version. Still seems incredibly slow.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #16 Posted: Sat May 21, 2022 11:19 pm 
Gosei
User avatar

Posts: 1753
Liked others: 177
Was liked: 491
15 blocks is better for slow computers and 40 for fast ones.I don't know when a computer is fast enough to use 40 blocks.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #17 Posted: Fri May 27, 2022 2:53 am 
Dies in gote

Posts: 31
Liked others: 3
Was liked: 5
Does anyone else know any reason why there'd be such a difference in calculation of variations between Lizzie and KaTrain and what I asked in previous post? I know this community is kind of small. Is there anyone who knows how these programs work that has gone in depth on AI? I'm having a hard time Googling and finding specific info on AI Go.

Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #18 Posted: Sat May 28, 2022 9:42 am 
Beginner

Posts: 1
Liked others: 0
Was liked: 1
JoeS1 wrote:
Does anyone else know any reason why there'd be such a difference in calculation of variations between Lizzie and KaTrain and what I asked in previous post? I know this community is kind of small. Is there anyone who knows how these programs work that has gone in depth on AI? I'm having a hard time Googling and finding specific info on AI Go.


First of all there's just inherent variance. I can run the same sgf through KaTrain two times under 25,000 playouts a move and the evaluations will differ.

Then there's all the differences caused by the different components. If you're using different neural networks through the same engine, that will cause differences. And if you're comparing different engines entirely, such as Lizzie and KataGo, then that will also be the cause of differences (on top of the fact that you're inherently using different neural networks and inherent variances between runs).


This post by PelleasGo was liked by: Akura
Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #19 Posted: Mon May 30, 2022 11:24 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
jlt wrote:
15 blocks is better for slow computers and 40 for fast ones.I don't know when a computer is fast enough to use 40 blocks.


Probably 40 blocks is better no matter what speed your computer is at this point (specifically, the latest nets from https://katagotraining.org). It's trained so much more that it will just dominate. Also the 15 blocks net predates some specific joseki training (mi yuting's flying dagger, some 3-4 pincer joseki) that are known AI blindspots for bots that don't have training for such positions, so it will systematically make well-known mistakes on some joseki that new nets will get right even with far less compute.


This post by lightvector was liked by: jlt
Top
 Profile  
 
Offline
 Post subject: Re: Some Lizzie Ai questions
Post #20 Posted: Mon May 30, 2022 12:23 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
JoeS1 wrote:
Thanks for the replies. I was trying to find more info on Lizzie AI and came across KaTrain and AlphaGo Zero/Master games which I didn't know existed, and have been replaying those. Surprising to see it still makes mistakes as well, but they are typically giving up a point or two endgame or answering a KO threat in a way that adds another unnecessary KO threat. Those are the easy things even a kyu player could see. It almost seems like there should be a second pass where the program takes the top moves and analyzes which ones lead to the most points gained locally or something. This seems feasible endgame when there aren't many possibilities left.


Heh. Keep in mind that AlphaZero/Master put literally zero value on winning with a larger score versus a smaller score. The're not trying to gain points locally, or trying to avoid giving up points, in the first place. They're only trying to win, and if they see that giving the opponent free points in some endgame situations wins just as much as not giving those points, then they're indifferent, a win is a win. You can see this clearly in one of the AlphaGo vs Ke Jie games, where AlphaGo won by 0.5 points, but was ahead by far, far more (10+ points, if I recall).

KataGo "fixes" this by putting a mild weight on maximizing points as well. Winning/losing is valued much more, so KataGo will still give up points for safety, roughly to the degree that a human pro sometimes yields and plays more peacefully when ahead instead of stretching for the maximum possible. But even KataGo will not care things that don't affect points (like the final ko threat count), and may due to noise or positional difficulty in the search sometimes waste threats (Sometimes bots will waste threats at very similar moments that a human would deliberately waste a threat for timesuji! Although the reasons are a bit different). And of course, bots can still make mistakes, as mentioned before.

JoeS1 wrote:
When searching for a guide on Lizzie I found mention of KaTrain and have been messing with that. Lizzie doesn't seem to have an option to end the game and count the score. KaTrain does, but after downloading all the modules, I noticed it only seems to do a few hundred variations across a few moves total per minute. Is KaTrain using a more advanced version of KataGo? I'm on Lizzie version 0.7.4 which appears to be the latest version.

I'm using the Open CL, 1.11 windows 64 bit engine, because I think that's the only one for GPUs, but there are 6 models. There are 15 block, 20 block, 30 block, 40 block, latest model, strongest model. Does anyone know how these compare to one another? Is Lizzie using the 15 block? Most I could find was 40 block was better than 20 block, but I didn't notice much of a difference in calculations between 40 block and strongest version. Still seems incredibly slow.


I believe Lizzie by default comes with a pretty old 15-block Leela Zero, and a very outdated 20-block Katago. Neither of these nets is the best choice for analysis, you should just use KaTrain's strongest model. Which you can upgrade Lizzie to use if you like, by manually downloading all those updated versions and networks and replacing the existing versions in your Lizzie download with those.

Even the older nets will often provide similar answers in analysis situations, simply because they are still strong and in a position where there is a clear best move, or there are clear "normal" continuations, all strong bots, and often high dan or pro humans, might well agree on them. But at the times they disagree, likely the strongest 40b net will be right more often than it's wrong compared to older nets, even despite being able to compute about 4x less.

If you have more "basic" questions, you might find better answers asking your questions in discord chat: https://discord.gg/bqkZAz3 - it's a pretty quiet discord, but some of the people hanging out there know a lot more about the details of different bots and related programs than those elsewhere.


This post by lightvector was liked by 2 people: Akura, johnsmith
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 24 posts ]  Go to page 1, 2  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group