Pachi 11.x for Windows, 32/64-bit (14 june 2016)

[Babelardus]

Hi, Babelardus here

Some time ago I've started to play Go, after having played Chess since a young age. I've always been an avid user of chess software, both for analyzing games and casual playing against old/weak chess engines. I actually have an electronic DGT chess board for this.

Compared to the number of Chess engines, Go engines are few and far between. I've read good things about Pachi, but had little success in using the current Windows build under the Drago GUI. The 32-bit Windows version on the Pachi site (http://http://pachi.or.cz/) is crash prone due to memory problems, especially when using large opening books and pattern files. According to Pachi's readme, it should limit the memory usage to stay around 1.4GB (excluding patterns and opening books), but it doesn't do so. I have had this version crash and terminate unexpectedly multiple times.

Therefore, I've decided to compile Pachi 11.0 myself to see if this can be improved. I failed to compile it under MinGW/MSYS. I tried both the native version and Cygwin version, both 32-bit and 64-bit, and custom make-files, but I kept receiving loads of compiler errors. However, it did work perfectly fine under Cygwin's default compiler, after a small change in its make-file. The only drawback is that this version needs Cygwin's Linux-to-Windows compatibility layer, which makes the engine somewhat larger.

Attached to this post you'll find this Cygwin-build, in both 32-bit and 64-bit. I've tested both of them under Drago, using the largest patterns and opening books. The 32-bit version, using the default memory settings (which is, provide no setting), caps out at 1750 MB. The 64-bit version, using max_tree_size=8192, caps out at 7850 MB. Both versions have played several full games without problems.

Note: Drago adds the option "-d 0" by default, due to the settings in its config file. If you leave this option in place, Pachi will be unable to load the pattern files or opening books. I do not know if this is a bug, or by design. After adding Pachi to Drago, click the [...] button next to "Parameters" in the engine settings window, and replace the command line, completely removing "-d 0":

32-bit version: -f "C:\Programs\Go\Engines\Pachi_11_0\32-bit\books\book_pro_6d.dat" threads=1
64-bit version: -f "C:\Programs\Go\Engines\Pachi_11_0\64-bit\books\book_pro_6d.dat" threads=1,max_tree_size=8192

Change the following where needed:
-f option path: point to the location where you have put the opening books.
threads=1: My 4-core i7-6820HQ with HT (4 physical cores, 4 logical) hits 90% when running 6 threads. This setting depends on the number of cores your CPU has available.
max_tree_size=8192: I can spare 8GB for Pachi if I want. Reduce this number if you don't have that much memory. The 32-bit version cannot use more than 2GB RAM; by default (when nothing else is specified), it'll top out at 1750 MB when using the largest books and patterns.

It is highly recommended to download the pattern files here: http://pachi.or.cz/pat/

If you haven't got an archiver to extract them, you can use 7-zip: http://www.7-zip.org/

The pattern files need to be in the folder where pachi.exe is located. If you want to install both the 32-bit and 64-bit version, you'll have to have two copies of the files, or work with symbolic links.

Note on GoGUI: Pachi's scoring commands are not compatible with GoGui. It does not score the game after it ends. You can hit "Score" in the menu, but the score will be wrong, as dead groups are not declared dead. You can get the correct score by clicking each dead group. (The other way around is also true: Fuego's scoring commands or not compatible with Drago, as mentioned int the Fuego thread.)

I hope this contribution is of use to you

Versions:
Pachi 11.0 Retsugen: http://asmoda.net/files/pachi_11_0_win32_win64.zip
Pachi 11.99 (bèta), 14-06-2016: http://asmoda.net/files/pachi_11_99_201 ... _win64.zip

[sorin]

However, it did work perfectly fine under Cygwin's default compiler, after a small change in its make-file. The only drawback is that this version needs Cygwin's Linux-to-Windows compatibility layer, which makes the engine somewhat larger.

Thank you for doing this!
Can you please also share how you built it using Cygwin, what changes did you make in the Makefile?

[Hane]

Downloaded and tested, working fine with patterns and book,

btw, there is a DCNN version of Pachi, does anyone have a Windows build of that ?

[Jaafar]

This version works perfectly (at least the x64 version I tested). Thanks a lot for sharing. You should contact the developer, maybe he'll add it to his site.

I think the "-d 0" parameter in drago is by design. When I removed it with the older exe the engine crashed (same for -d 3 or -d 4).

I added "-t =100000" to the arguments to give it a strength between 1d and 2d KGS (closer to 2d according to the facebook paper). It should be even higher because pondering is enabled by default.

It takes about 40s to make a move on an old i5 with 3 threads, it should be much quicker on recent hardware.

[Babelardus]

However, it did work perfectly fine under Cygwin's default compiler, after a small change in its make-file. The only drawback is that this version needs Cygwin's Linux-to-Windows compatibility layer, which makes the engine somewhat larger.

Thank you for doing this!
Can you please also share how you built it using Cygwin, what changes did you make in the Makefile?

It's my pleasure.

I've just installed Cygwin twice, both the 32-bit and 64-bit versions. Just make sure of the following points:

- Download the Pachi tarball. Extract it somewhere. Make sure you run 7-zip in admin mode, as the tarball contains symbolic links which will fail to be created if you run 7-zip as a user.
- Install both in their own folder.
- When installing, install the default gcc (C) and g++ (C++) compilers. If you install Cygwin 32-bit, don't install the Cygwin64 cross compiler, and vice versa. Also don't install anything related to MinGW.
- Don't add their "bin" folders to your path, or the versions will get mixed up.
- When compiling, use the Cygwin Bash shell. (Cygwin Terminal, in the Start Menu.)
- Type these commands (without quotes):

cd /cygdrive/d/path/to/pachi/folder

(d is the drive letter.)

make pachi

(wait)

- After pachi.exe is built, put it in a folder on your desktop, together with cygwin1.dll and cyggcc_s-1.dll (32-bit version), or cygwin1.dll and cyggcc_s-seh-1.dll (64-bit version). These files are found in the \bin folder of the 32-bit and 64-bit Cygwin installations.

- In the terminal, go to your desktop:

cd /cygdrive/c/users/username/desktop/pachi_folder

(or go where your desktop is)

strip -s *.*

(this strips debug information from the files)

- Obviously, the 32-bit version of Cygwin generates a 32-bit program, and the 64-bit version creates a 64-bit one. (The cross compiler can create a 32-bit program under a 64-bit environment, and the other way around, but that's not what we want to do here.)

- You need to do this twice: once for the 32-bit version, once for the 64-bit version, if you want them both.

===

The changes to the makefile I made were unnecessary. I uncommented the "WIN=1" directive, and that caused the library "regex" to be missing. I removed the reference to "-lregex", and compiled anyway.

Later, I realised that Cygwin is actually a Linux/Unix environment, even though it runs on top of Windows. It basically generates a Linux-program in a Windows executable format. But, Windows is not Linux, so even though the program is a Windows executable, it still "talks to Linux" when running. That is why you need the additional cygwin1.dll and the cyggcc*dll: this is the part of the Linux-to-Windows conversion layer which Pachi requires. (Some programs only need Cygwin1.dll, others need boatloads more.)

Due to this, I recompiled, without uncommenting "WIN=1", and Pachi immediately compiled without errors.

I did uncomment "DOUBLE=1", so Pachi can handle huge playouts, like >=1 million. (This is explained in the makefile.) So, in the end, the makefile is perfect, and the only change is including high-precision by uncommenting "DOUBLE=1".

I have updated the archive in the first post to use this new executable that doesn't have the "remove -lregex" workaround. I doubt it makes a difference; when compiling under Cygwin, Pachi doesn't need it or the compiler would have generated an error. You can replace your pachi.exe if you wish, though.

[Babelardus]

Downloaded and tested, working fine with patterns and book,

btw, there is a DCNN version of Pachi, does anyone have a Windows build of that ?

Very good

I have tried to build Pachi with DCNN, but I gave up. The 11.0 Stable tarball doesn't have this yet, so I downloaded the very latest source from GitHub. Then it started....

- Pachi needs the neural network "Caffe".
- So I download that and try to compile it. Boost is missing. (DAMN, that is a huge project to compile...)
- Fortunately, Cygwin has Boost built in, so I install it, and then try to compile Caffe again.
- Error: gflags missing.
- I find gflags (Google Flags), build it, installed it, and try to compile Caffe again.
- Error: glog missing.
- I find glog (Google Log). Compile. Install. Try Caffe again.
- Error: cblas.h missing.
- I find cblas (some sort of math library) and try to compile it.
- Crap. It's written in Fortran... so I install GCC Fortran, compile CBLAS and install it. Try Caffe again.
- Error: OpenCV missing.....

Dependency hell.

Visual Studio is able to automatically resolve and compile Caffe dependencies, but the pure Windows-version is still experimental, and I do not know if I can use it in Cygwin.

Here I gave up. My time to spend on compiling a program is not unlimited, sorry.

This version works perfectly (at least the x64 version I tested). Thanks a lot for sharing. You should contact the developer, maybe he'll add it to his site.

I think the "-d 0" parameter in drago is by design. When I removed it with the older exe the engine crashed (same for -d 3 or -d 4).

I added "-t =100000" to the arguments to give it a strength between 1d and 2d KGS (closer to 2d according to the facebook paper). It should be even higher because pondering is enabled by default.

It takes about 40s to make a move on an old i5 with 3 threads, it should be much quicker on recent hardware.

I already contacted Petr Baudis through his GitHub e-mail address and sent the compiled version to him.

[avysk]

This is strange -- I've compiled 64-bit version too (wrote down my steps here: viewtopic.php?p=205172#p205172), but it loads books / patterns happily with '-d 0'. On the contrary, without "-d 0" it spits out some noise about found book positions, and Drago cannot recognize the move made, waiting for it forever.

Also I use tree_size 3072, because at 4096 it stops thinking, spitting out moves immediately (*after* it's out of the book). I guess there's some overflow somewhere... I didn't try larger sizes, though.

[Babelardus]

This is strange -- I've compiled 64-bit version too (wrote down my steps here: http://www.lifein19x19.com/forum/viewto ... 72#p205172), but it loads books / patterns happily with '-d 0'. On the contrary, without "-d 0" it spits out some noise about found book positions, and Drago cannot recognize the move made, waiting for it forever.

Also I use tree_size 3072, because at 4096 it stops thinking, spitting out moves immediately (*after* it's out of the book). I guess there's some overflow somewhere... I didn't try larger sizes, though.

Your version is different from mine. Your version is native Windows, mine is a Linux version on top of the Cygwin1.dll API emulator.

Maybe "-d 0" is only necessary for a Windows-native version.

While you *think* you have created a 64-bit version... did you really? Many toolchains can cross-compile, and this can become very confusing. You installed mingw-w64-x86_64-clang. I assume it's MinGW-w64, and x86_64-clang, so a 64-bit toolchain and a 64-bit compiler. It could be though, that the default compilation mode of CLANG is 32-bit. I don't know; this is just a guess.

When running MinGW, you can run the toolchain in 32-bit (MinGW), or 64-bit (MinGW-w64), which includes a compiler that is 32-bit, or 64-bit *itself*. When running GCC, the 32-bit compiler creates 32-bit programs, unless you pass it the -m64 option; when running the 64-bit version of GCC, it creates 64-bit versions unless you pass it -m32. This is under Cygwin though; MinGW's GC does not support this option. It could be that CLANG defaults to 32-bit programs until you tell it differently.

To check, install the 'file' command under MSYS2, if it is not already there. Then go to the Pachi folder, and run 'file pachi.exe'. If it says something like "i386", "i686", you've inadvertently compiled a 32-bit version.

It could be that CLANG automatically sets the /LargeAddressAware option in its linker if you create a 32-bit Windows-version. This option provides a 32-bit program to use up to 3.2 GB RAM instead of 2 GB.

If you (accidentally) compiled a 32-bit version with the /LargeAddressAware option set, it would perfectly explain why your version of Pachi crashes as you set it to use more memory.

You could try to remove CLANG and install GCC. CLANG is a relatively recent addition to MinGW; it might not yet work correctly in all cases.

=========================

Thanks for pointing out MSYS2. I didn't know about this, and the fact that it runs as a shell with package manager..

I can't get Pachi, or any other program for that matter, to compile with the current version of MinGW/MSYS as available on mingw.org. In this version, MSYS is a Windows-native, ultra-minimal version of Cygwin. I see MMSYS2 has evolved back toward the direction of Cygwin, with a package manager and such.

Maybe I'll try to recompile all engines under this version, using GCC, after I finish the website where I'm going to put all of this stuff. (I finally found something to create a website about... YAY!)

[Babelardus]

I've just tested Pachi's memory consumption. I set it to use a maximum of 24GB of the 32GB my system has available. It stopped increasing just short of 20GB, and took 15 minutes or so to fill up while pondering. Maybe the memory usage would rise to the total of 24GB if the game progresses further, but I haven't tested that. I just played 30 moves or so.

So, Pachi can use *at least* 20GB of RAM without problems, when compiled under Cygwin/GCC as 64-bit.

[avysk]

Yes, my version is a 64-bit one, I happen to know a thing or two about compilers and file command -- since year 1995 or so

I guess there's some incompatibility in some data type size between Windows and Cygwin; something in windows is probably limited in bit width, that's why tree_size of 4096 breaks the things for me. And I have only one low-end Windows laptop with 8Gb of memory; the rest of computers in home are Mac / Linux / FreeBSD, so I cannot test if it takes up to 24Gb or not.

Can you try your version with exact 4096 tree size -- does it work, or does it start to throw moves very fast without thinking?

(And yes, it does not crash for me with 4096 tree size -- it just stops taking any time to think, and plays moves very fast. Bad moves.)

[avysk]

I've tried your version, and it crashes for me immediately, here's .stackdump

Exception: STATUS_ILLEGAL_INSTRUCTION at rip=00100421EB5
rax=0000000000000000 rbx=00000001006059C0 rcx=0000000100451D9D
rdx=0000000000000002 rsi=000000010044C870 rdi=00000000FFFFBA70
r8 =0000000000000000 r9 =0000000000000002 r10=0000000100000000
r11=0000000100421EB5 r12=00000000FFFFCC69 r13=0000000000000000
r14=0000000000000000 r15=0000000000000005
rbp=0000000000000001 rsp=00000000FFFFB990
program=C:\Users\alexey\Applications\pachi_11_0_win32_win64\64-bit\pachi.exe, pid 8140, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B

Running under msys2 says:

alexey@DUVEL MINGW64 /c/Users/alexey/Applications/pachi_11_0_win32_win64/64-bit
$ ./pachi.exe
4 [main] pachi (6244) C:\Users\alexey\Applications\pachi_11_0_win32_win64\64-bit\pachi.exe: *** fatal error - cygheap base mismatch detected - 0x18033A400/0x18033E400.
This problem is probably due to using incompatible versions of the cygwin DLL.
Search for cygwin1.dll using the Windows Start->Find/Search facility
and delete all but the most recent version. The most recent version *should*
reside in x:\cygwin\bin, where 'x' is the drive on which you have
installed the cygwin distribution. Rebooting is also suggested if you
are unable to find another cygwin DLL.

(32-bit version crashes as well)

[Babelardus]

Yes, my version is a 64-bit one, I happen to know a thing or two about compilers and file command -- since year 1995 or so

OK, that's good Then from henceforth I'm not going to include extremely detailed explanations on how to use the toolchains and command line

I guess there's some incompatibility in some data type size between Windows and Cygwin; something in windows is probably limited in bit width, that's why tree_size of 4096 breaks the things for me.

I assume you mean MinGW/MSYS2? The Cygwin version of Pachi works fine.

For reasons reasons of compatibility with old software, Windows keeps the maximum memory size for a 32-bit process down to 2GB, except when it has been compiled and linked with the LargeAddressAware option. Then the process can use 3GB.

Can you try your version with exact 4096 tree size -- does it work, or does it start to throw moves very fast without thinking? (And yes, it does not crash for me with 4096 tree size -- it just stops taking any time to think, and plays moves very fast. Bad moves.)

I've just tested this and it works. I've played a few moves in Pachi's opening book, and then some out of it. It played normally. Then I let it ponder, until the memory consumption stopped rising. It always stops somewhat short of the full allocation; this time, with 4096MB allocated to max_tree_size, it stopped at 3896MB. Pachi keeps playing normally at an average of 12-13 seconds per move, which is the default if you don't set specific timings.

pachi_memory2.png (18.98 KiB) Viewed 20698 times

I would suggest to recompile Pachi using GCC, and test again. Maybe CLANG's Windows-port in MinGW is broken. In the past, I've had good results with GCC under under MinGW, but I had to stop using that as for some reason, 'make' often can't find the programs and commands needed. I'll look into MSYS2.

(It seems they reversed it: MSYS was previously part of MinGW, now it seems the other way around, with MinGW being part of MSYS2. It's now the same structure as used by Cygwin.)

[Babelardus]

I've tried your version, and it crashes for me immediately, here's .stackdump

Exception: STATUS_ILLEGAL_INSTRUCTION at rip=00100421EB5
rax=0000000000000000 rbx=00000001006059C0 rcx=0000000100451D9D
rdx=0000000000000002 rsi=000000010044C870 rdi=00000000FFFFBA70
r8 =0000000000000000 r9 =0000000000000002 r10=0000000100000000
r11=0000000100421EB5 r12=00000000FFFFCC69 r13=0000000000000000
r14=0000000000000000 r15=0000000000000005
rbp=0000000000000001 rsp=00000000FFFFB990
program=C:\Users\alexey\Applications\pachi_11_0_win32_win64\64-bit\pachi.exe, pid 8140, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B

Running under msys2 says:

alexey@DUVEL MINGW64 /c/Users/alexey/Applications/pachi_11_0_win32_win64/64-bit
$ ./pachi.exe
4 [main] pachi (6244) C:\Users\alexey\Applications\pachi_11_0_win32_win64\64-bit\pachi.exe: *** fatal error - cygheap base mismatch detected - 0x18033A400/0x18033E400.
This problem is probably due to using incompatible versions of the cygwin DLL.
Search for cygwin1.dll using the Windows Start->Find/Search facility
and delete all but the most recent version. The most recent version *should*
reside in x:\cygwin\bin, where 'x' is the drive on which you have
installed the cygwin distribution. Rebooting is also suggested if you
are unable to find another cygwin DLL.

(32-bit version crashes as well)

Do you have your own version of Cygwin installed and included in your path? If you are just starting Pachi, without the DLL's in the folder, it'll start searching for them in your path. If you have a different version of Cygwin installed than I have, it will crash. It is indeed complaining about an incompatible version of Cygwin. This probably has something to do with the configuration of your Windows environment and/or DLL/version mixups.

I always keep the toolchains out of my path. I couldn't even include them, because I won't install GCC 32-bit and 64-bit at the same time in one Cygwin installation. I can, but then I need to start to do weird things to get stuff to compile with the correct compiler. It's easier to install Cygwin twice, and just use a BATCH file to temporarily include the wanted one in the path. This way, I can also use different toolchains or compilers, and have MinGW/MSYS installed as well, without clashes between different compilers and versions.

The one downside is that, after you lift the binary from the compiler's workspace, you need to include all needed DLL's in the folder where you put the executable. You can't, for example, include both the 32-bit and 64-bit version in your path. If you include the 32-bit version first, the 64-bit program will probably crash, and the other way around.

Note: Pachi has been downloaded 31 times as of today, and I have received no problem reports as of yet. Fuego, especially version r2029, of which no official binary exists (let alone a 64-bit version) outside of mine, is just as popular though.

edit: when compiling Pachi using GCC under MinGW64/MSYS2, I get the same errors as when using the older MinGW64/MSYS from MinGW.org:

make[1]: Entering directory '/c/users/marcel/desktop/pachi-master/random'
[CC] random.c
<command-line>:0:8: error: expected identifier or '(' before numeric constant
<command-line>:0:8: error: expected declaration specifiers or '...' before numer ic constant
<command-line>:0:8: error: expected declaration specifiers or '...' before numer ic constant
<command-line>:0:8: error: expected declaration specifiers or '...' before numer ic constant
<command-line>:0:8: error: expected declaration specifiers or '...' before numer ic constant
<command-line>:0:8: error: expected specifier-qualifier-list before numeric cons tant
<command-line>:0:8: error: expected specifier-qualifier-list before numeric cons tant
<command-line>:0:8: error: expected specifier-qualifier-list before numeric cons tant
<command-line>:0:8: error: expected specifier-qualifier-list before numeric cons....

And so on... it seems the Windows-port of GCC is broken, at least where Pachi (and Fuego) is concerned.

[Jaafar]

Pachi works fine with x64 and 3gb memory.

The fist version of fuego works fine too (x64) but the latest version take way too long for my computer compared with the first version . More than 2 mn per move compared to about 15s in the older version. Both with 100k playouts and 2Gb of memory.

[Babelardus]

Pachi works fine with x64 and 3gb memory.

The fist version of fuego works fine too (x64) but the latest version take way too long for my computer compared with the first version . More than 2 mn per move compared to about 15s in the older version. Both with 100k playouts and 2Gb of memory.

I have just tested this with both Fuego 1.1 and 1.1_r2029.
Threads: 6
Memory: 8GB
Time and playout settings: defaults

Both versions play a move after an average of 8.5 seconds, but Fuego 1.1 runs through 16.500 games per second, while Fuego 1.1_r2029 only does around 600. (This is reported in the GTP-window.) Therefore the r2029 is 27.5 times slower.

The executable of 1.1_r2029 is much larger than the older 1.1 version. (They have both been stripped from all debugging code and symbols.)

1.1 32-bit: 2.40 MB <=> 1.1_r2029: 8.80 MB
1.1 64-bit: 1.26 MB <=> 1.1_r2029: 8.70 MB

I've read in one of the newsgroups that the developer was busy adding patterns to Fuego. They seem to be already in there, built into the executable. When comparing 1.1 and 1.1_r2029, the latter plays MUCH better when out of the opening book. It plays much more like Pachi, which also uses patterns, although Pachi's are larger.

When testing 1.1 against 1.1_r2029, using 3 threads and pondering, or 6 threads without pondering (and 6GB RAM for each), Fuego 1.1_r2029 is MUCH stronger than the older 1.1, and it plays much more logical... even though its number of games played per second is so much lower.

Therefore, I wouldn't try to base your opinion on the number of playouts, at least not in version r2029. It seems to base its style of play much more on pattern recognition than on brute playout power. Compared to the older 1.1, version 1.1_r2029 is a completely different program and can/should not be run with the same parameters. (Except for number of threads and memory allocation.)

What option have you used to set the engine to 100.00 programs specifically? I haven't found any, but I may have missed it. I've got too many parameters from different engines in my head now

(By the way, it would be better if you'd post Feugo-related posts in that topic )