It is currently Thu Mar 28, 2024 8:56 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 9 posts ] 
Author Message
Offline
 Post subject: Some Qestions about KataGo and Lizzie
Post #1 Posted: Tue Mar 29, 2022 11:24 pm 
Lives with ko

Posts: 128
Liked others: 148
Was liked: 29
Rank: British 3 kyu
KGS: thirdfogie
This post is the result of time spent looking in detail at the SGF files output
by Lizzie and KataGo.

1. Sometimes the percentage win rate goes up when the 'correct' move is played.
The two images show how the analysis changed in a no-komi game with 5000 playouts.
KataGo increases White's win rate from 29.3% to 31.2%.
There is presumably some benefit from playing the correct move, in that the
game is now one move nearer its end, which slightly reduces the fog of uncertainty.
However, a change of 1.9% feels excessive if that is the only factor at work.

(With more playouts, KataGo slightly prefers White Q5.)

2. As can be seen from the screen captures, Lizzie shows different win rates
in the main board display and the win-rate graph. Data in the SGF file
reflects the win-rate graph. Why are the two not updated in parallel?

3. Lizzie stores the data from KataGo using lengthy 'LZ' tags in the SGF file
This tag was not mentioned in the thread on extending SGF
(https://lifein19x19.com/viewtopic.php?f=10&t=16425). Has 'LZ' become a de facto
standard?

My preferred SGF editor for Linux is Quarry, which ignores LZ tags
when reading an SGF file and preserves them on writing. This is
apparently correct behaviour, though other editors behave differently.

4. Is Lizzie now abandoned? What are the cool kids using now with Linux?

For reference, I have Lizzie 0.7.4 (patched version) and KataGo 1.7.4.


Attachments:
File comment: After White played the correct move.
after1.png
after1.png [ 493.08 KiB | Viewed 3245 times ]
File comment: Before White played the correct move.
before1.png
before1.png [ 492.64 KiB | Viewed 3245 times ]
Top
 Profile  
 
Offline
 Post subject: Re: Some Qestions about KataGo and Lizzie
Post #2 Posted: Wed Mar 30, 2022 3:21 am 
Lives with ko

Posts: 259
Liked others: 46
Was liked: 116
Rank: 2d
thirdfogie wrote:
1. Sometimes the percentage win rate goes up when the 'correct' move is played.
The two images show how the analysis changed in a no-komi game with 5000 playouts.
KataGo increases White's win rate from 29.3% to 31.2%.
There is presumably some benefit from playing the correct move, in that the
game is now one move nearer its end, which slightly reduces the fog of uncertainty.
However, a change of 1.9% feels excessive if that is the only factor at work.
I don't know what happened in this specific example, but AIs will change their opinions over time when evaluating a position. It could be as simple as looking at the position with a different number of playouts.

Quote:
4. Is Lizzie now abandoned? What are the cool kids using now with Linux?

Don't know about cool kids, but I made this.


This post by bernds was liked by: thirdfogie
Top
 Profile  
 
Offline
 Post subject: Re: Some Qestions about KataGo and Lizzie
Post #3 Posted: Wed Mar 30, 2022 6:41 am 
Oza
User avatar

Posts: 2408
Location: Ghent, Belgium
Liked others: 359
Was liked: 1019
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
FWIW - the difference between the percentage/points shown on the blue move and the percentage/points shown on the next blue move, after playing the blue move, is what I take as an AI's precision. I've never seen it as high as 2 points though, rather in the area of 0.7-0.8, given sufficient playouts on each move.


This post by Knotwilg was liked by: thirdfogie
Top
 Profile  
 
Offline
 Post subject: Re: Some Questions about KataGo and Lizzie
Post #4 Posted: Wed Mar 30, 2022 7:13 am 
Lives with ko

Posts: 128
Liked others: 148
Was liked: 29
Rank: British 3 kyu
KGS: thirdfogie
Hi bernds, thanks for the reply. I will try more KataGo playouts to check whether doing so reduces
the gain from playing the correct move.

I tried installing q5Go, but failed. The ZIP file was unpacked into a suitable directory.
It turns out that the qmake package was missing, so...
Code:
apt-get install qt5-qmake

Next...
Code:
mkdir ~/g5go-target
mkdir build
cd build
qmake ../src/q5go.pro PREFIX=~/q5go-target

However, qmake then complains as follows and gives up...
Code:
Project ERROR: Unknown module(s) in QT: core gui widgets xml network multimedia svg sql

Since I don't know what a QT module is or how to get one, I gave up too.
The system runs Debian bullseye (V11.3) with XFCE.
Maybe my barebones approach is part of the problem.

Top
 Profile  
 
Offline
 Post subject: Re: Some Questions about KataGo and Lizzie
Post #5 Posted: Wed Mar 30, 2022 7:22 am 
Lives with ko

Posts: 259
Liked others: 46
Was liked: 116
Rank: 2d
thirdfogie wrote:
Code:
Project ERROR: Unknown module(s) in QT: core gui widgets xml network multimedia svg sql

Since I don't know what a QT module is or how to get one, I gave up too.
The system runs Debian bullseye (V11.3) with XFCE.
Maybe my barebones approach is part of the problem.

Yeah, you need all the Qt5 packages as well. Unfortunately these are a bit confusing on Debian based systems. I think you probably need at least
Code:
libqt5svg5-dev
qtmultimedia5-dev
qt5-qmake
qt5-default

There was some discussion here.


This post by bernds was liked by: thirdfogie
Top
 Profile  
 
Offline
 Post subject: Re: Some Qestions about KataGo and Lizzie
Post #6 Posted: Wed Mar 30, 2022 7:29 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Quote:
2. As can be seen from the screen captures, Lizzie shows different win rates
in the main board display and the win-rate graph. Data in the SGF file
reflects the win-rate graph. Why are the two not updated in parallel?


The winrate graph presumably is looking at the root value, which is an average across all the different moves' values weighted by their visit count (or something similar to that). This will generally make it slightly worse for the side to move than the winrate of the top move for that player, but it is more robust. The winrate of the top move for a player may itself be slightly too optimistic for that player on average (due to bias from taking the maximum of a large number of noisy values).

Also, you may be looking for an option like "winrate_always_black" or something like that in the config. I *think* Lizzie by default shows the winrate from the perspective of the side to move rather than only from black's perspective, which if that's the case and I'm not just misremembering, is a really poor default. It is entirely inconsistent with the winrate you'd want to graph, and it makes it far more difficult to understand how the value changes as you play a move because rather than simply observing whether it went up or down, you instead have to mentally subtract from 100% every time the side to move changes. It would be great if the Go world standardized on showing black's winrate and score only, the same way that Chess has mostly standardized on reporting bot evaluations from the perspective of the first player to move in Chess.


This post by lightvector was liked by: thirdfogie
Top
 Profile  
 
Offline
 Post subject: Re: Some Qestions about KataGo and Lizzie
Post #7 Posted: Thu Mar 31, 2022 1:55 pm 
Lives with ko

Posts: 128
Liked others: 148
Was liked: 29
Rank: British 3 kyu
KGS: thirdfogie
I have now managed to get q5Go running. The hidden section gives the installation
details, mainly for my own use if the work needs to be repeated on a new PC or
with a new version of Debian. The rest of this post is a mixture of questions about q5Go
and an attempt to compare it to Lizzie, pending a decision on whether to switch.

Download the program's ZIP file and expand it into a new directory called
q5go-master. Then do the following...
Code:
apt-get install libqt5svg5-dev qtmultimedia5-dev qt5-qmake

The qt5-default package does not exist in Debian, and was not needed.
Code:
mkdir ~/q5go-target
cd ~/q5go-master
mkdir build
cd build
qmake ../src/q5go.pro PREFIX=~/q5go-target
make
make install

The next step is getting analysis to run with KataGo. For simplicity, all the
relevant files were copied from the old Lizzie directory to the new one for q5Go.
Code:
cd ~/q5go-target/bin
cp -af ~/lizzie/target/{katago,kata1-b40c256-s10336005120-d2519775087.bin.gz,gtp_custom-kata1-b40c256.cfg} .

Then use Settings in q5Go to add KataGo as the engine. The parameters passed to KataGo are...
Code:
gtp -model kata1-b40c256-s10336005120-d2519775087.bin.gz -config gtp_custom-kata1-b40c256.cfg

I probably could not have worked this out if I had not already done it
with Lizzie. The README.md file says nothing about it.
I then wasted time wondering why nothing happens after loading an SGF file and
clicking on Analysis. The solution was to disable a puzzling bear trap. In
"Settings"/"Preferences", choose the "Computer Go" tab, then highlight "KataGo"
in the list and click on "Edit". In the resulting dialog box, enable "Use for
analysis..." That setting should default to "Yes". (Perhaps it does, and I
accidentally unset it while thrashing around bindly.)

At first glance, q5Go has a lot of functions and is therefore more complex than
Lizzie and harder to understand. I am also mourning Lizzie's keyboard
shortcuts. For example, Lizzie's space bar switches analysis on and off,
Ctrl-0/Ctrl-1 changes the analysis engine and 'a' starts an analysis by
prompting the user to specify the number of playouts. This is much more
convenient than working through the menus in q5Go, especially since I don't
understand them yet.

After running an analysis, I tried saving. (It was not clear what saving meant
in q5Go, which is why I wanted to try it.) q5Go said "Unable to save" without
saying why. The next day, following a reboot, saving always worked. I have no
idea what changed. The "Export" menu item also failed, but I have not retried it.

q5Go saves extra data using "QKGV" tags, which answers the question
whether "LV" is a de facto standard. The structure of the data in "QKGV"
tags seems very different but I haven't studied it.

I tried to find out whether one program analyses faster than the other. Both
programs were set to analyse the same game from the same point using KataGo
with the same model and settings, AFAICT. The results were confusing and
inconclusive.

In 2 minutes, Lizzie managed to get 38.8k playouts from KataGo. g5Go managed
11.4k visits. I don't know whether visits and playouts are the same things,
and it is possible that the report taken from q5Go is only for visits to the best
move. The GPU temperature was similar in both cases. The only definite
finding is that Lizzie uses more CPU time, distributed evenly over the
available four cores, whereas g5Go mainly uses one core.

Measurement is harder with g5Go because 1) I don't really know the right way to
stop the analysis - the GPU often continues to run after I think it should have
stopped - and 2) the analysis detail disappears from the screen when I do
(try to) stop it. It would be better to analyse a whole game using both programs,
but that would be time-consuming and boring. Also, the two programs use different
methods to control the analysis: Lizzie limits the number of visits but q5Go limits
the time per move.

Since the only stupid question is the one you did not ask, here goes.

Q1. The attached image shows a screenshot from an analysis run. The assessment
graph at the bottom left is too small to be useful even with g5Go using all the
1920x1080 screen. (The attachment was shrunk for posting.) The
screenshots from Lizzie at the head of this thread are of a useful size. Can
the graph in q5Go be made bigger? There are lots of examples of big graphs
in the help, but it is not clear how to access them.

Q2. Lizzie's assessment graph shows the percentage win-rate, the points lead
and the blunders on the same axes. This is often cluttered but it is useful.
Can g5Go show more than one thing at a time?

Q3. When Lizzie saves an SGF file, she saves all the analysis inside LZ tags.
One can then reopen the SGF file later and review the analysis without
repeating it. Is such an off-line review possible with q5Go?

Q4. When analysing in batch mode, where does the output show up? I couldn't
find anything. Maybe this problem was related to the "unable to save" problem.

Thanks for your patience.

Edited for clarity by thirdfogie, 31 March, 2022.


Attachments:
File comment: Screenshot from q5Go.
q5go1.png
q5go1.png [ 397.95 KiB | Viewed 3057 times ]
Top
 Profile  
 
Offline
 Post subject: Re: Some Qestions about KataGo and Lizzie
Post #8 Posted: Thu Mar 31, 2022 2:16 pm 
Lives with ko

Posts: 259
Liked others: 46
Was liked: 116
Rank: 2d
thirdfogie wrote:
In 2 minutes, Lizzie managed to get 38.8k playouts from KataGo. g5Go managed 11.4k visits.
If the KataGo executable and options are the same, performance should be identical. It's probably a reporting difference.

Quote:
Can the graph in q5Go be made bigger? There are lots of examples of big graphs
in the help, but it is not clear how to access them.
You can drag the separators, or even drag the entire graph to a different position in the window. You can then also save the layout. When opening a window it'll try to restore the default layout; if you then add more panes (for example by switching on the win-rate graph if it wasn't open already) you can press "L" to reload the layout for the new set of panes.
Quote:
Q2. Lizzie's assessment graph shows the percentage win-rate, the points lead
and the blunders on the same axes. This is often cluttered but it is useful.
Can g5Go show more than one thing at a time?
I don't quite know what Lizzie does, but it doesn't sound like something q5Go can do at this point. You can switch the graph between score and winrate mode.
Quote:
Q3. When Lizzie saves an SGF file, she saves all the analysis inside LZ tags.
One can then reopen the SGF file later and review the analysis without
repeating it. Is such an off-line review possible with q5Go?
Scores and winrates are saved. If you're doing live analysis you have to add variations yourself if you want to save them (shift-click or middle-click). You can also use batch analysis to make a file that has the all the top lines at every move.
Quote:
Q4. When analysing in batch mode, where does the output show up?
There's a button with a "new board window" icon below the job queue.

Top
 Profile  
 
Offline
 Post subject: Re: Some Qestions about KataGo and Lizzie
Post #9 Posted: Sat Apr 02, 2022 2:25 am 
Lives with ko

Posts: 128
Liked others: 148
Was liked: 29
Rank: British 3 kyu
KGS: thirdfogie
Hi bernds, thanks for all the explanations and help.

I have learnt more about the differences in CPU usage between Lizzie and q5Go
when running the same analysis. The attachment shows the greater CPU usage by
Lizzie (on the left) compared to q5Go (on the right). Running the Linux "top"
utility explained it all: both programs run KataGo at about 104% CPU load, but
Lizzie updates the screen much more: Xorg runs at 70% CPU with Lizzie but
only at 10% to 30% with q5Go.

Another difference is that Lizzie re-uses saved analyses more than q5Go. For
example, if I quit Lizzie then restart it, open the same file and start
analysis at the same move, Lizzie somehow remembers the previous analysis and
restarts from that point. Lizzie even does this if the SGF file was deleted
and replaced by a fresh copy from an archive somewhere. q5Go always restarts
its analysis from the beginning, at least in my hands. This difference
probably explains my initial impression that Lizzie was faster.

The aggressive cacheing by Lizzie bedevils any attempt at detailed speed comparison.
The main conclusion is that GPU performance is not the only factor in analysis
speed. Are you reading this, Robert Jasiek?

I will now go back to using Lizzie, and work on the original project,
which was to extract useful textual comments from SGF files created
by KataGo. I had hoped to find a ready-made tool, but it looks like
I will have to write something myself.


Attachments:
File comment: CPU usage by Lizzie (left) and q5Go (right)
krells.png
krells.png [ 56.46 KiB | Viewed 2962 times ]
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group