Compiling some engines for linux

[pnprog]

Interesting: I didn't get that error for Ray because I'd previously installed libatlas-base-dev as a dependency for Pachi. I wonder how you managed to compile Pachi without it?

For Ray, the installation process for Ubuntu has been added to the readme.m:

Code: Select all

sudo apt install git make gcc g++ libcaffe-cpu-dev libboost-all-dev libgflags-dev libgoogle-glog-dev libprotobuf-dev libopenblas-dev
git clone https://github.com/pasky/pachi.git
cd pachi
make

My guess is that libatlas-base-dev provides the cblas component for Ray, and some other components necessary for Pachi that can be also provided by libopenblas-dev. So libatlas-base-dev or libopenblas-dev are both working for Pachi, but only libatlas-base-dev works for Ray. I will take time to try AQ and Ray from fresh Ubuntu install to see what exactly are the minimal dependencies required.

Also, I've recently changed Ray to add a command line option "--weights", so that you can easily switch between different weight files. My changes were merged into zakki's github repository just a couple of days ago, so if you don't have the --weights option then you might want to download the latest version of the lz branch and recompile.

Oh that's an useful option (and I have it on my current copy of Ray). With that, several profiles for Ray can be added to GTP programs while having only one copy of Ray. Thanks

[xela]

I've now managed to compile a non-crashing version of AQ! The trick, at least for me, was to change a few lines of the source code, as detailed at https://github.com/ymgaq/AQ/issues/73

Compiling this one was a pain, but all of the pain comes from TensorFlow, not from AQ itself. (I'm told that TensorFlow is actually pretty easy to set up if you want to call it from Python. It's the mix of TensorFlow and C++ that gets complicated.) The README file tells you to follow these instructions, but that's more a collection of hints than detailed instructions, and it's missing a detail that was essential for me (see note 3 below).

So let's say you've created ~/tensorflow, cloned the github repository in there as per the the instructions (all 400+ megabytes of it), making sure to do it recursively (git clone --recursive https://github.com/tensorflow/tensorflow), and unpacked the AQ source code to ~/tensorflow/tensorflow/AQ-2.1.1. And you've managed to install the latest bazel (currently version 17), and NVidia's CUDA, CUDNN and NCCL as dependencies. Now you need to do something like this:

Code: Select all

cd ~/tensorflow
./configure # see note 1
cd tensorflow/AQ-2.1.1/src
bazel build --config=monolithic :AQ # see notes 2 and 3
mv prob/* . # the *.txt files need to be in the same directory as aq_config.txt
find ~/.cache/bazel -name AQ
mv the_long_pathname_that_contains_runfiles/AQ .

and then you should be able to run your executable ~/tensorflow/tensorflow/AQ-2.1.1/AQ Remember that you can change configuration options by editing the aq_config.txt file.

Note 1: the configure script will ask you a lot of annoying questions, including your CUDA version and where you installed the libraries. Where it's asking whether to support something you've never heard of, it seems that it's safe to say no, and most of the defaults worked OK.

Note 2: bazel will download a few hundred megabyes of dependencies without asking you, and will create about three gigabytes of files in a hidden cache directory. The download-build-and-leave-junk-behind process took about 20 minutes on my machine.

Note 3: I don't know what the "monolithic" bit actually does (the bazel documentation isn't very informative here), but without it I get a bunch of "symbol not defined" errors. I got the idea of including it from https://github.com/tensorflow/tensorflow/issues/18739

[pnprog]

I've now managed to compile a non-crashing version of AQ!

The whole procedure seems pretty scarry
I will see if I can git it a try, but with the poor internet connection here, I am not confident I can make it.
Plus I am getting very low on disk free space.

But anyway, thanks for the (successful) effort, and sharing the methodology with us

By the way, keep in mind that AQ is trained for 6.5pt komi (and Japanese rules). For instance, in a 7.5pt komi game, AQ's value network might tell white is losing by 0.5pt while it is winning by 0.5pt in fact, and because of that, AQ may very forfeit the game (or start playing strange moves like AlphaGo VS LeeSedol game four). As black, AQ might very well happily let the opponent win by 0.5pt (believing black is wining) by loosing 0.5pt in a yose where it could have been avoided. It may select wrong jose move due to Japanese rules as well.

I just say that in case you intend to have AQ enter your tournament.

[xela]

By the way, keep in mind that AQ is trained for 6.5pt komi (and Japanese rules).

Thanks for the heads up. It's strange that the default configuration has "japanese rule = off" and "komi = 7.5".

I just say that in case you intend to have AQ enter your tournament.

I don't think I can do anything about it. I'll just enter it in the ranking list anyway. and unfortunately it will be at a slight disadvantage. I don't imagine it would make a huge difference, probably less than 100 rating points?

[pnprog]

It's strange that the default configuration has "japanese rule = off" and "komi = 7.5".

Indeed it's strange.

I just say that in case you intend to have AQ enter your tournament.

I don't think I can do anything about it. I'll just enter it in the ranking list anyway. and unfortunately it will be at a slight disadvantage. I don't imagine it would make a huge difference, probably less than 100 rating points?

I cannot predict, but you can keep record of the number of games AQ looses by 0.5pt (normally AQ should resign before scoring). This is explained in the readme

[pnprog]

Phoenix Go

I gave up on this one. I tried following the instructions at https://github.com/Tencent/PhoenixGo The problem is that it uses bazel as its build system, which is new to me, so I'm not well qualified to troubleshoot the problems. Bazel will download a few hundred megabytes of stuff, then try to build TensorFlow configured according to the selected PhoenixGo options, spit out a bunch of compiler warnings, then fail on one of the TensorFlow components. I didn't want to keep retrying it because it's downloading an awful lot of stuff each time. I did find some old forum posts saying that TensorFlow won't work with bazel version 12 or 13, so you should use bazel 11. I tried that, as well as the newest bazel (version 17), with the same results each time.

This is just to mention that the instructions for compilation were added on the github project page. Easy enough for me to be able to compile and run it.

[xela]

This is just to mention that the instructions for compilation were added on the github project page. Easy enough for me to be able to compile and run it.

Glad to hear that it worked for you! The instructions don't seem to have changed since I tried in September, except that the current version of bazel is now 19 not 17. Maybe a bazel update has fixed the problem?

[pnprog]

This is just to mention that the instructions for compilation were added on the github project page. Easy enough for me to be able to compile and run it.

Glad to hear that it worked for you! The instructions don't seem to have changed since I tried in September, except that the current version of bazel is now 19 not 17. Maybe a bazel update has fixed the problem?

Here are the condensed instructions that are working for me: https://github.com/Tencent/PhoenixGo/blob/master/docs/FAQ.md/#b0-it-is-too-hard-to-install-bazel-or-start-bazel

It tells how to install bazel which is useful to install AQ (i am on it right now).

[pnprog]

I also went again through the compilation of Ray. One thing is different now, git needs the --recursive option to fetch some code for the src/lz/Eigen folder.

Code: Select all

sudo apt install libatlas-base-dev #there are other dependancy, but that one in particular was missing for me
git clone --recursive https://github.com/zakki/Ray.git --branch lz --single-branch .
make

The compilation will still fail, with errors such as:

Code: Select all

src/UctSearch.cpp:1694:41: error: no matching function for call to ‘atomic_fetch_add(std::atomic<long unsigned int>*, int&)’
     atomic_fetch_add(&depth_sum, c.depth);
...
                                         ^
src/UctSearch.cpp:1695:37: error: no matching function for call to ‘atomic_fetch_add(std::atomic<long unsigned int>*, int)’
     atomic_fetch_add(&depth_count, 1);
                                     ^
...
Makefile:24: recipe for target 'src/UctSearch.o' failed
make: *** [src/UctSearch.o] Error 1

I found out that replacing lines 219/220 in src/UctSearch.cpp as follow fixes the issue:
Before:

Code: Select all

static std::atomic<uint64_t> depth_count;
static std::atomic<uint64_t> depth_sum;

After:

Code: Select all

static std::atomic<double> depth_count;
static std::atomic<double> depth_sum;

This also work, but I have no idea what the difference mean:

Code: Select all

static std::atomic<double_t> depth_count;
static std::atomic<double_t> depth_sum;

It's used to calculate and display the average tree search depth, so nothing critical I guess. Anyway, I sent a PR to the project.

[pnprog]

It tells how to install bazel which is useful to install AQ (i am on it right now).

I must admit I could not make that one work. I can compile, but the binary crashes when issuing genmove b command
Maybe I should try with an older version of bazel (I have been using version 19)

[xela]

Not an engine as such, but just as useful: I've just installed the fantastic q5Go version 9. Getting the source code from GitHub was easy, and it should build and install exactly as per the instructions, except for one minor but annoying detail.

q5Go relies on Qt version 5.10, but the official Ubuntu repositories only have Q5 version 5.9, and there are conflicting and confusing instructions on how to upgrade. It turns out that you can install Qt 5.12 into its own folder without messing up your "official" Ubuntu installation.

• Download the Qt installer from http://download.qt.io/official_releases/qt/5.12/5.12.5/
• Run the installer (chmod a+x longfilename.run, then ./longfilename.run). When it asks you to create/sign in to an account, just skip that step. Create a new directory to install into: let's say you've put it into /opt/qt5.12
• Create a file called ~/.config/qtchooser/qt5.12.conf. The file should contain two lines with the bin and lib directories of your new installation -- e.g. they might be /opt/qt5.12/5.12.5/gcc_64/bin and /opt/qt5.12/5.12.5/gcc_64/lib
• Test: type the command

  Code: Select all

  qmake -qt=qt5.12 -v

  It should show you the version number that you installed (5.12.5 at the time of writing) and the (stuff)/lib directory name.
• When you're building q5Go, replace

  Code: Select all

  qmake

  with

  Code: Select all

  qmake -qt=qt5.12

  For other projects, if you use plain qmake without the -qt bit, it will fall back to using your standard Qt installation.

From here you should be able to follow the instructions on GitHub.