Life In 19x19
http://www.lifein19x19.com/

Compiling 1 Engine and Its Interface for Debian 10 (Buster)
http://www.lifein19x19.com/viewtopic.php?f=18&t=17187		 Page 1 of 1
Author:  thirdfogie [ Fri Jan 10, 2020 4:29 pm ]
Post subject:  Compiling 1 Engine and Its Interface for Debian 10 (Buster)
The title is a nod to Xela's excellent work here.
https://lifein19x19.com/viewtopic.php?f=18&t=16046&start=0

I previously had Lizzie and Leela Zero working without a GPU under Debian 9, but the atttempt to
replicate it six months ago under Debian 10 failed, in ways that are now lost in the mists of time.
I decided to try again at Christmas when my son would be able to help out, and to buy and install a
GPU to make things harder. This post is a detailed account of what happened, because...
1) I like war stories; and
2) The record may help with the inevitable repeat when Debian 11 becomes the next stable release.

It is likely that 80% of what we did was unnecessary, but I got a working setup in the end, I learnt
about Docker and I get this chance to muse about my limitations in public. Extensive use is made of
hidden sections to improve the flow.

To start with the conclusion: we bought an Nvidia GeForce GTX 1660. It manages 250 to 1000 visits
per second, depending on the game state. Weights file
877d874de6dc08110158ac3b321f131cb6f276c5bd07eaa4bef9fdf12c9f3f68.gz was used.
As an odd test of accuracy, Leela reproduced AlphaGo's mistake in answer to Lee Sedol's move 78
in game 4 of their match.

Why was the task so hard?
1. The GPU card was installed for use with an existing Linux installation. If the GPU had been
present at the time of installation, it is possible that more support would have been built in to
the O/S already.
2. The available online instructions for Lizzie and Leela under Linux were written by and for the
Ubuntu distribution, and things are a bit different with Debian. This actually helped us during
software installation because my son has a laptop with a small pre-installed Nvidia GPU running
Ubuntu. He quickly got Lizzie and Leela up and running in that environment, which we could then try
to replicate under Debian.
3. The task may be a bit harder under Debian 10 than it was under 9. The next hidden section
describes my usual approach to Debian upgrades and how it failed to help in this case.

My 70-year-old eyes, shaking hands and inflexible brain would have been unequal to the task
without help.

The hidden section describes my usual approach to Operating System upgrades.
The UK computer writer Jack Schofield offers the following maxim which I paraphrase.
Whenever I find myself putting data into a new program or system, I ask myself "How can I get it out
again?". Experience has indeed taught me that it may not be easy to decode, decompress or decrypt
data which was encoded, compressed or encrypted by an older version of the same operating system.
The upgrade from Debian 8 to 9 was particularly troublesome. All the problems could be fixed, but
it took time to track them down. My policy now is to install each new version using dual boot and
run both versions alternately for one or two months until it is clear that everything works before
switching over entirely. In this way, it is often possible to execute an old program under the new
operating system to recover the data.

An upgrade can also introduce other problems if a program has changed its behaviour in ways that I
consider unacceptable. In that case, I can sometimes keep the old version or build the program from
source and change the behaviour myself, but there will be no automatic security fixes afterwards.

The upcoming change from X windows to Wayland in Debian may cause other problems: I use the XFCE
desktop environment because it is relatively simple and does not get in the way. It is not yet
clear that XFCE will run under Wayland.

Another example is xpdf, which is my preferred viewer for .PDF files. (It is simple and does not
get in the way.) It now has no Debian maintainer. If Debian abandons xpdf, I may be able to use an
old binary for the rest of my life.

Despite all this caution and suspicion, the upgrade to Debian 10 caught me out. The old binaries
for Lizzie and Leela would not run under the new O/S, and a newly compiled and updated Lizzie never
managed to run the leelaz program. It was probably a simple configuration error, but not one I
could find.


There are various online instructions for building Leela zero and Lizzie for Linux. We decided to
compile from source. This worked easily under Ubuntu, so we decided to prototype the method for
Debian by installing a Docker instance of Debian 10 on the Ubuntu laptop and then to repeat the
process inside Docker. That worked, though it was not possible to run Lizzie inside Docker because
the Debian instance that we chose is a small one that does not have a Graphical User Interface.
Instead, the resulting binaries were copied out of Docker into the host Ubuntu system, where they
did run. We decided to repeat the process on the target PC with Debian 10 as the host and the same
Docker instance of Debain 10. That also worked, and the binaries from Docker also ran on the Debian 10
host. Our scripts appear below. This same method will be used when Lizzie, Leela or the Operating
System are next upgraded, because new stuff can be tried out inside Docker without messing up the
existing running system.

GPU Choice and Installation
We found the GTX 1660 online for about £180. Two weeks later, it was on sale for about £210 from
the same supplier. Apparently, the price of GPUs varies with the price of cryptocurrencies.

The GTX requires at least a 400W power supply. The existing power supply has a big label on its
side reading "450W", so that was not a problem.

We had concerns about cooling inside the PC's case. An "Air Duster" aerosol can removed a lot of
dust from the cooling fins attached to the CPU heatsink, which helped a lot. The Air Duster was
available from Poundland (a Dollar Store in US English) at the time. We also decided to replace one
fan in the wall of the PC case that had a slightly noisy bearing. This was not easy, because the
old fan took its power directly from the main power supply unit, and we had bought a new fan that
expects to plug into a different kind of socket on the motherboard. The solution was a "VanDeSaiL 4
Way Black Sleeved Braided Fan Power Cable 15 Inch for 12V", which was quite cheap to buy. It
connects to the Power Supply and provides several extra sockets like those on the motherboard. The
result is bit more clutter inside the case, but the main problem for me would have been knowing that
such cables even exist.

There was enough room for the GPU board: at least we researched that properly. However, it was
supplied without a self-tapping screw to secure it to the PC's case. That was a problem, because
the GPU is relatively heavy, and it needs to resist the force when the HDMI cable is plugged in to
it. We borrowed a screw from a smaller and lighter board temporarily.


After improving the PC's cooling and installing the GPU board, we closed the case and moved the HDMI
lead from the socket on the PC motherboard to the socket on the new GPU. Fortunately, the Linux
kernel already contained the reverse-engineered Nouveau driver for Nvidia devices, so we could
still see screen output after rebooting, even though the screen resolution was wrong and some
desktop elements were therefore off the screen.

The following commands added the non-free driver and a detection utility. Non-free sources were
already enabled in /etc/apt/sources.list. We ran everything as root. You might want to be more
careful.

Code:
apt-get install -y nvidia-detect nvidia-driver nvidia-libopencl1


We then rebooted and adjusted the display settings using the GUI's Display Settings editor. We
changed the default 1024x768 resolution to match the actual screen and then adjusted the dots per
inch (DPI) until text looked right. The resulting display looked like it did before the GPU was
added. We also disabled the default nouveau driver:
Code:
echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf


Two more detailed and painstaking sets of instructions for Debian 10 are available here:
https://linuxconfig.org/how-to-install-nvidia-driver-on-debian-10-buster-linux

We then installed the following packages to support Nvidia and OpenCl. (The system probably had
some of these already, but asking for them again causes no problems.)
Code:
apt-get install -y cmake g++ libboost-dev libboost-program-options-dev libboost-filesystem-dev
apt-get install -y opencl-headers ocl-icd-libopencl1 ocl-icd-opencl-dev zlib1g-dev
apt-get install -y nvidia-opencl-icd nvidia-cuda-mps
apt-get install -y qtbase5-dev maven

The following hidden section contains all the details about the Docker build.

For information on Docker, see https://docs.docker.com/v17.12/install/linux/docker-ce/debian/.

We needed at least 10G spare space in /var and a reasonable internet connection. It took about 30
minutes on the target PC, with a lot of downloads.

The following script prepares the Debian-10 host for Docker.

Code:
apt-get remove docker docker-engine docker.io

apt-get update

apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg2 \
    software-properties-common

curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
apt-key fingerprint 0EBFCD88

add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
apt-get update;apt-get dist-upgrade
apt-get install -y docker-ce
docker run hello-world


The key fingerprint should look like this.
Code:
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22


The main script is go.sh (below). It calls the following three scripts in turn. The three
sub-scripts were used to exploit cacheing in Docker during development and testing.

Script build-leela-and-lizzie-1.sh
Code:
#########################################################
# Set up dependencies
#########################################################

# To use an nvidia GPU with leela-zero we need to install proprietary nvidia drivers.
# To install these we must enable the contrib and non-free parts of the default apt repos.
sed -iE 's/main$/main contrib non-free/g' /etc/apt/sources.list &&

# Remove a warning in apt that CLI utilities are not installed. We don't need them.
export DEBIAN_FRONTEND=noninteractive &&

# Now update repos and install apt-utils, which is necessary for apt (dont know
# why but it moans without). Note that when installing apt-utils, debconf will
# emit a warning that apt-utils is not installed, but the internet says this is ok.
apt-get update &&
apt-get -y dist-upgrade &&
apt-get install -y apt-utils &&

# Install nvidia driver. nvidia-detect must be present before
# nvidia-driver can be installed.
apt-get install -y --install-suggests nvidia-detect &&
nvidia-detect &&
apt-get install -y nvidia-driver nvidia-cuda-toolkit &&

# Install a few packages to support the build process
apt-get -y install git


Script build-leela-and-lizzie-2.sh

Code:
#########################################################
# Build leelaz
#########################################################

# Remove a warning in apt that CLI utilities are not installed. We don't need them.
export DEBIAN_FRONTEND=noninteractive

# Create a temporary directory to build leela
mkdir /tmp/go-program-build &&
cd /tmp/go-program-build &&

# Clone the leela-go repository from github and follow the instructions given there to build it.
git clone https://github.com/leela-zero/leela-zero.git &&
cd leela-zero &&
apt-get install -y clinfo && clinfo &&

git submodule update --init --recursive &&
apt-get install -y cmake g++ libboost-dev libboost-program-options-dev libboost-filesystem-dev \
                   opencl-headers ocl-icd-libopencl1 ocl-icd-opencl-dev zlib1g-dev &&

# Now install some qt5 dependencies, which cmake requires
apt-get install -y qtbase5-dev &&$G

mkdir build && cd build &&

cmake .. &&
cmake --build .


Script build-leela-and-lizzie-3.sh

Code:
#########################################################
# Build Lizzie
#########################################################

# Remove a warning in apt that CLI utilities are not installed. We don't need them.
export DEBIAN_FRONTEND=noninteractive

cd /tmp/go-program-build &&

# Install maven (which pulls in a jre) for lizzie
apt-get install -y maven &&

# Now clone the lizzie repo from github and build it.
git clone https://github.com/featurecat/lizzie.git &&
cd lizzie &&
mvn package


When the three sub-scripts are in place, it is finally time to run the script go.sh, which is as follows.
Code:
# go.sh

# First remove all pre-existing outputs
rm -rf outputs/*

# Use docker to build leela-zero and lizzie in a debian buster environment.
# The command uses the commands located in ./Dockerfile and builds a container
# called leelaz-build-env which will contain leelaz and lizzie.  The dockerfile
# invokes the scripts called buildscripts/build-leela-and-lizzie{1,2,3} inside
# the new debian environment.
docker build -t leelaz-build-env .

# Now copy the binaries into ./outputs.
mkdir -p outputs
docker run -it --mount type=bind,source=$(pwd),target=/mnt leelaz-build-env sh copy-outputs.sh

If the above process succeeds, the remaining operations are performed on the host system. Copy
lizzie-0.72-shaded.tar and leelaz from ./output and the weights file from ~/Download to a common
directory. Rename the weights file as lznetwork.gz, then run this command in the common directory.
Code:
java -jar lizzie-0.7.2-shaded.jar &

GPU Initialisation took about 2 minutes, then lizzie and leela were ready.

To tidy up the docker installation, do something like this.
Code:
docker system prune -f --volumes
docker container prune
apt-get purge docker-ce-cli maven   # There are probably many more that could go.
apt-get autoremove
rm -rf /var/lib/docker

There is much more information about removing Docker images here.
https://linuxize.com/post/

Now I just have to relearn how to use Lizzie for analysis, and learn about Katago which is not yet installed.
Author:  xela [ Sat Jan 11, 2020 4:52 am ]
Post subject:  Re: Compiling 1 Engine and Its Interface for Debian 10 (Bust
Impressive work! If you've managed all this, then setting up KataGo should be a piece of cake :-)
Page 1 of 1 All times are UTC - 8 hours [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/