Upgrading Lizzie and KataGo with Debian 10.

[thirdfogie]

Not sure this is worth posting, but here goes anyway.

I have been putting off updating my installation of Lizzie and KataGo, because
the original installation (Lizzie 0.7.2, KataGo 1.3?) was lengthy, as recorded
here.

viewtopic.php?f=18&t=17187

The first step was to download the latest files.

Code: Select all

Lizzie.0.7.4.Mac-Linux.zip
katago-v1.8.0-opencl-linux-x64.zip
kata1-b40c256-s6485784576-d1573360039.bin.gz

The new neural network came from https://katagotraining.org/networks/
A "Network File" version ending in bin.gz is appropriate. The file chosen was
the "latest known strong" network.

Note the use of OpenCL, not CUDA. As before, the GPU is an NVIDIA GeForce GTX 1660.

For historical reasons, all the files are placed here.

Code: Select all

~/lizzie/target

The first step is to back up the old installation so it can be restored if the new versions did not work.

After making the backup, expand the new zip files. The new Lizzie files expanded into a Lizzie
sub-directory and needed to be moved up one level.

Move

Code: Select all

kata1-b40c256-s6485784576-d1573360039.bin.gz

to the target directory.

Make the katago file executable.

Code: Select all

chmod +x katago

KataGo needs libzip.so.5, which is not available. The solution is
the time-honoured kludge of attaching a symbolic link with the new name
to the old library and hoping that everything still works.

Code: Select all

cd /usr/lib/x86_64-linux-gnu/
ln -sf libzip.so.4 libzip.so.5
cd -

Then run the following on the new neural network. Note that the online instructions say to run
katago.exe, but the .exe extension is not usual under Linux.

Code: Select all

./katago genconfig -model kata1-b40c256-s6485784576-d1573360039.bin.gz -output gtp_custom-kata1-b40c256.cfg

In response to questions, select Japanese rules and GPU device 0. Otherwise,
accept the defaults, including no pondering.

The only change in the recommended configuration was that the number of threads changed from 12 to 8 for the newer neural network.

Various futile attempts were made to install the new neural network. Perhaps they did succeed in
installing the new KataGo configuration.

Code: Select all

mv default_gtp.cfg old_default_gtp.cfg
cp -af gtp_custom-kata1-b40c256.cfg default_gtp.cfg
mv gtp_example.cfg old_gtp_example.cfg
cp -af gtp_custom-kata1-b40c256.cfg gtp_example.cfg

Here is the correct way. In the engine-command-list section of config.txt,
replace the line for katago with the following.

Code: Select all

./katago gtp -model kata1-b40c256-s6485784576-d1573360039.bin.gz -config gtp_custom-kata1-b40c256.cfg

The launcher for Lizzie on the XFCE desktop did not need to change.

Lizzie started with an open GTP console, which is not wanted. But that did not
happen again after the first exit following manual closure of the console.
There is a setting in the GUI for this anyway.

The following functional changes have been seen. Warning: it is early days with the new versions.
1. KataGo now sometimes recommends different moves in complex situations, as expected.
2. At the end of a game with the old versions, the estimated score by KataGo was commonly out by 0.5 points. That error is now reduced to 0.1 points.
3. I use Lizzie and KataGo to analyse human games. With the old version, Lizzie would sometimes get confused and continue
analysing the previous move after it had been told to move on one play in the game record. The new move appeared on the screen, but was
ignored in the analysis. I have yet to see this happen with the new versions. (The work-around was to discard the history by reloading the SGF file.)

Testing has begun with Debian 11 (bullseye). It also uses libzip4 AFAICT, but I
expect greater trouble with finding and installing the correct GPU drivers
and the switch from Xwindows to Wayland. I'll post here if things do work with Debian 11.

Many thanks to featurecat, lightvector and their many collaborators.

Edited to correct the symbolic link code and the double description of the optimisation step.
And some textual infelicities.

[thirdfogie]

This post is about getting LeelaZero, Lizzie and KataGo running on Debian 11 (bullseye).
Its main value is to record what was done and why. If it helps anyone else, that is a
bonus.

Debian 11 was installed alongside Debian 10 on the same computer, so the old programs and
configurations were readily available. Because the old installation had recently been
updated, the first plan was to copy everything in ~/lizzie/target from Debian 10 to 11.
However, Debian now has native packages for LeelaZero (0.17-1) and Lizzie (0.7.4), though
none for KataGo. I tried the two native packages first. In the event, the native version
of Lizzie did not immediately work. The native version of LeelaZero did work and is a much
smaller binary. Unfortunately, the native version took much longer to initialise every
time Lizzie started. Instead of investigating these problems, I reverted to the original
copy-across plan, which is recorded below in gruesome detail.

Append " contrib non-free" to every line in /etc/apt/sources.lst, then update and install
additional packages as shown below.

Code: Select all

apt-get update && apt-get dist-upgrade
apt-get install nvidia-detect nvidia-driver nvidia-libopencl1 libzip-dev xwayland
apt-get install libnvidia-glcore libnvidia-eglcore libboost-program-options-dev libboost-system-dev

Since I was not planning to recompile Leela Zero, I could probably have installed libzip4
instead of libzip-dev, and similarly with the the other *-dev packages, but no harm was
done.

The disk partitions containing Debian 10 are mounted via a series of /old* directories so
it was easy to copy the old installation to the new one.

Code: Select all

mkdir ~/lizzie/target
cd ~/lizzie/target
sudo cp -af /oldhome/user/lizzie/target/* .
sudo cp -af /oldhome/user/Desktop/Lizzie.desktop ~/Desktop/Lizzie.desktop  # The program launcher for XFCE.

The most significant files are listed below.

Code: Select all

-rw-r--r-- 1 user user      7134 Mar  6 09:17 gtp_custom-kata1-b40c256.cfg
-rw-r--r-- 1 user user 173528857 Feb 26 08:39 kata1-b40c256-s6485784576-d1573360039.bin.gz
-rwxr-xr-x 1 user user 109948336 Jan 13 22:42 katago
-rwxr-xr-x 1 user user   1353600 Dec 12 15:19 leelaz
-rw-r--r-- 1 user user   6195749 Sep 29 02:36 lizzie.jar
-rw-r--r-- 1 user user  21540103 Sep 29 02:51 lznetwork.gz

Since leelaz was compiled from source in Debian 10, it should really be recompiled for
Debian 11, but I decided to try running the old binary. (Experience has shown that this
trick often works with Debian.) I only use Leela to check when KataGo gets stuck or
suggests something outlandish and no such problems have occurred since the previous
upgrade to KataGo and Lizzie, so Leela is now of less interest.

The next step was to run leelaz stand-alone as follows.

Code: Select all

./leelaz

As expected, leelaz complained about libzip5 not being present. The fix was the symbolic
link kludge used in debian 10. (Ubuntu Long-Term Support version "focal fossa" does have
libzip5, which may be why the need for it got baked in to my build of leelaz using Docker.
I avoided Docker in the Debian 11 installation.) The next attempts to run leelaz
stand-alone showed that additional library versions used by leelaz also needed the
symbolic link kludge to make it seem that older library versions were present. The full
set of symbolic link commands appears below.

Code: Select all

cd /usr/lib/x86_64-linux-gnu
ln -sf libzip.so.4 libzip.so.5
ln -sf libboost_program_options.so.1.74.0 libboost_program_options.so.1.67.0
ln -sf libboost_filesystem.so.1.74.0 libboost_filesystem.so.1.67.0
ln -sf libboost_system.so.1.74.0 libboost_system.so.1.67.0

The hidden section is a reminder of two different ways to identify which Debian package
can supply a missing library file. These methods supplement my failing memory and are
particularly useful when a package is replaced by one with a slightly different name.

Method 1 uses Google with the search string 'debian package search <name>', where
<name> is something like docker or libzip. This typically returns a lot of results
which can be explored using search in page or Ctrl-F. The results seem to be
alphabetically ordered, so the top entries may be dominated by versions for Android or
Atomic (whatever that is).

Method 2 interrogates the current release and architecture, so it excludes results for
Debian 8 or the Android architecture. Type the following Bash command line.

Code: Select all

apt search 'expression'
# The 'expression' argument can be a regular expression such as '^docker'.
# The output can be piped to a 'grep -Ev' clause to prune unhelpful matches.

After all that, KataGo and LeelaZero now run well under Lizzie. I also learnt two
useful keyboard short-cuts in Lizzie.

1. Ctrl-0 and Ctrl-1 switch between analysis with LeelaZero and KataGo respectively

2. Pressing "." switches off KataGo's "show expected territory" function, which is visually
confusing when the board begins to fill up with stones.

As before, many thanks to featurecat, lightvector, Gian-Carlo Pascutto and
their many collaborators.

*EDITED* on 28 June 2021 to remove an unwanted space after "cd ~" in a code block.