It is currently Thu Mar 28, 2024 11:01 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 390 posts ]  Go to page Previous  1 ... 5, 6, 7, 8, 9, 10, 11 ... 20  Next
Author Message
Offline
 Post subject: Re: Engine Tournament
Post #141 Posted: Sat Jan 20, 2018 2:16 pm 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
q30 wrote:
Try to use time control equivalent to 2 min per move. In this case fluctuations will be much smaller and difference between pondering and not pondering will be measurable...


You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.


Last edited by as0770 on Sun Jan 21, 2018 12:29 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #142 Posted: Sat Jan 20, 2018 2:24 pm 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
New entry in League B is Dream Go, update in League C Mathilda 1.25

Leela vs. AQ
Code:
    1. AQ 2.0.3                     12/16
    2. Leela 0.11.0 Beta 11          4/16

Configuration:
2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

AQ: AQ
Code:
aq_config.txt:
-main time[sec] =7200
-time controll =true
-japanese rule =true

Leela: leela_gtp_opencl –gtp –threads 4 –noponder

twogtp: gogui-twogtp -auto -white "leela_gtp_opencl --gtp --threads 4 --noponder" -black "AQ" -games 4 -size 19 -time 120 -verbose -sgffile xxxx


League A:
Code:
    1. Leela 0.10.0                 22/24
    2. Rayon 4.6.0                  19/24
    3. Oakfoam 0.2.1 NG-06          18/24
    4. Hiratuka 10.37B (CPU)         9/24
    5. DarkForest v2 MCTS 1.0        7/24
    6  DarkGo 1.0                    5/24
    7. Pachi DCNN 11.99              4/24

Configuration:
2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 2500 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
Hiratuka: Hiratuka-19×19.exe -po 175000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Oakfoam: oakfoam -c nicego-cnn-06.gtp
Code:
nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

Pachi: pachidcnn -f pachibook.dat threads=4,max_tree_size=2048,pondering=0
Rayon: rayon --thread 4 --no-debug

twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 120 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto


League B:
Code:
    1. Ray 9.0.1                    31/36
    2. Pachi DCNN 11.99             30/36
    3. Dream Go 0.5.0               29/36
    4. Leela Zero 0.9 (2018.01.01)  21/36
    5. MoGo 4.86                    18/36
    6. deltaGo 1.0.0                18/36
    7. Fuego 1.1                    15/36
    8. Michi C-2 1.4.2               8/36
    9. Orego 7.08                    8/36
   10. GNU Go 3.8                    2/36

Configuration:
1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
Fuego: fuego.exe –config fuego.cfg
Code:
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
Leela Zero: leelaz --gtp --threads 1 --w /path/to/Leelaz_best-network_2018_01_01 --noponder
Michi C-2: michi gtp
Code:
ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachi -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
Ray: ray --time 3600 --thread 1 --no-debug
twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto


League C:
Code:
    1. GNU Go 3.8                   25/28
    2. Hara 0.9                     18/28
    3. Matilda 1.25                 16/28
    4. Indigo 2009                  16/28
    5. Dariush 3.1.5.7              15/28
    6. Aya 6.34                     13/28
    7. Fudo Go 3.0                   7/28
    8. JrefBot 081016-2022           2/28

Configuration:
1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

Aya: Aya.exe –mode gtp –level max
Dariush: DarGTP.exe –level 10
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
GnuGo: gnugo --mode gtp --level 10 --resign-allowed
Hara: hara
IndiGo: Indigo.exe -gtp
JrefBot: java -jar jrefgo.jar 10000
Matilda: matilda
Code:
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1


twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto


League D:
Code:
    1. JrefBot 081016-2022          16/18             
    2. Iomrascálaí 0.3.2            15/18
    3. Crazy Patterns 0008-13       13/18
    4. Marcos Go 1.0                13/18
    5. AmiGo 1.8                    13/18
    6. Beancounter 0.1               8/18
    7. Stop 0.9-005                  5/18
    8. GoTraxx 1.4.2                 3/18
    0. CopyBot 0.1                   2/18
   10. Brown 1.0                     2/18

Configuration:
1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2

Amigo: amigogtp
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
GoTraxx: GoTraxx.exe
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 10000
Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp

twogtp: TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time 60 -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto



Links:
Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/winnt/game/fh673259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/winnt/game/fh688349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: https://github.com/gcp/leela-zero
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: https://lifein19x19.com/forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
Orego: https://sites.google.com/a/lclark.edu/drake/research/orego
Pachi: http://pachi.or.cz/
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/

Best,
Alex


Last edited by as0770 on Mon Jan 22, 2018 10:41 am, edited 2 times in total.
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #143 Posted: Sun Jan 21, 2018 6:59 am 
Dies in gote

Posts: 21
Liked others: 0
Was liked: 17
Thanks for running these tournaments, impressive lineup !

Right now the leagues are based on strength more or less if i understand correctly.
Just an idea: how about a gpu league and a cpu league for the top programs ?
This is already the case mostly, what i mean is that for programs like Leela which can do both, it'd be interesting to see how the cpu version fares. Not sure how practical this would be. Maybe you don't want to have to unplug your graphic card to prevent it from using the gpu =)

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #144 Posted: Sun Jan 21, 2018 10:24 pm 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
lemonsqueez wrote:
Thanks for running these tournaments, impressive lineup !

Right now the leagues are based on strength more or less if i understand correctly.


You do :)

One or two engines play in the upper and lower League to have a virtual connection between the Leagues.

lemonsqueez wrote:
Just an idea: how about a gpu league and a cpu league for the top programs ?
This is already the case mostly, what i mean is that for programs like Leela which can do both, it'd be interesting to see how the cpu version fares. Not sure how practical this would be. Maybe you don't want to have to unplug your graphic card to prevent it from using the gpu =)


I want to have a comparison between GPU and CPU engines so I don't want to make different Leagues.

In fact I tried to run the GPU engines also in CPU mode, this works well with Leela. It would play in the same League as Leela GPU but I don't want to have one engine playing twice in one League. This would strain the results. But you can find Leela CPU in the history :)

Rayon CPU is basically Ray 9.0.1 afaik. AQ won't work as CPU engine here and for Oakfoam as CPU engine I have to adjust some parameters and there where problems running it. And btw it is very, very weak. For other engines I would have to change the system configuration, but I don't want to mess up my system like this. So after all I would like to run the engines in both modes but I would face too many problems...

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #145 Posted: Fri Jan 26, 2018 10:37 am 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
This time I downsized the Leagues to get space for new entries.

In League B Leela Zero is updated to v0.11 and the last 5x64 network (2018.01.17). After that they changed to a 6x128 network.

Also there is a new Engine in League E: SimpleGo 0.4.3


Leela vs. AQ
Code:
    1. AQ 2.0.3                        12/16
    2. Leela 0.11.0 Beta 11             4/16

League A:
Code:
    1. Leela 0.10.0                    22/24
    2. Rayon 4.6.0                     19/24
    3. Oakfoam 0.2.1 NG-06             18/24
    4. Hiratuka 10.37B (CPU)            9/24
    5. DarkForest v2 MCTS 1.0           7/24
    6  DarkGo 1.0                       5/24
    7. Pachi DCNN 11.99                 4/24

League B:
Code:
    1. Leela Zero 0.11 (2018.01.17)    15/20
    2. Pachi DCNN 11.99                13/20
    3. DarkGo 1.0                      12/20
    4. Dream Go 0.5.0                  11/20
    5. Ray 9.0.1                        7/20
    6. Mogo 4.86                        2/20

League C:
Code:
    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:
Code:
    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:
Code:
    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:
Code:
    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:
League A: 2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time xx -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League B-F: 1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time xx -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Amigo: amigogtp
AQ: AQ
Code:
aq_config.txt:
-main time[sec] =7200
-time controll =true
-japanese rule =true

Aya: Aya.exe –mode gtp –level max
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
Dariush: DarGTP.exe –level 10
DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 2500 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
Fuego: fuego.exe –config fuego.cfg
Code:
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
GoTraxx: GoTraxx.exe
Hara: hara
Hiratuka: Hiratuka-19×19.exe -po 175000
IndiGo: Indigo.exe -gtp
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 10000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 1 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder
Matilda: matilda
Code:
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1

Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Michi C-2: michi gtp
Code:
ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Oakfoam: oakfoam -c nicego-cnn-06.gtp
Code:
nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachidcnn -f pachibook.dat threads=4,max_tree_size=2048,pondering=0
Pachi: pachidcnn -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
Ray: ray --time 3600 --thread 1 --no-debug
Rayon: rayon --thread 4 --no-debug
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp

Links:
Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/winnt/game/fh673259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/winnt/game/fh688349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: https://github.com/gcp/leela-zero
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: https://lifein19x19.com/forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
Orego: https://sites.google.com/a/lclark.edu/drake/research/orego
Pachi: http://pachi.or.cz/
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/

Best,
Alex

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #146 Posted: Sat Jan 27, 2018 6:57 am 
Lives with ko

Posts: 135
Liked others: 1
Was liked: 1
Rank: 30 kyu
as0770 wrote:
You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.

It depends on game randomness, that changes with the time control...

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #147 Posted: Sat Jan 27, 2018 10:10 am 
Dies in gote

Posts: 34
Liked others: 2
Was liked: 5
Rank: 3k
Some strong bots have been updated:
AQ v2.1.1
Leela 0.11.0
Ray 4.32
Pachi 12.00

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #148 Posted: Sat Jan 27, 2018 11:01 am 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
q30 wrote:
as0770 wrote:
You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.

It depends on game randomness, that changes with the time control...


Then we have to rewrite basic mathematical principles.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #149 Posted: Sat Jan 27, 2018 11:04 am 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
Cyan wrote:
Some strong bots have been updated:
AQ v2.1.1
Leela 0.11.0
Ray 4.32
Pachi 12.00


Leela 0.11 is running and I'll wait for the official release of Pachi 12.

I'll take a look at the others, thank you. :)

Edit: Tried to compile Ray 4.32 without success. First I had to get some libs of cntk 2.1 although in the readme the cntk version is 2.3. Then I ran into the next error messages. The author is not much interested in making the installation easier, not even in keeping the readme up to date. That's no problem, but I'll leave Rn until it is easier to install.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #150 Posted: Mon Jan 29, 2018 1:10 am 
Lives with ko
User avatar

Posts: 284
Liked others: 94
Was liked: 153
Rank: OGS 7 kyu
as0770 wrote:
Edit: Tried to compile Ray 4.32 without success. First I had to get some libs of cntk 2.1 although in the readme the cntk version is 2.3. Then I ran into the next error messages. The author is not much interested in making the installation easier, not even in keeping the readme up to date. That's no problem, but I'll leave Rn until it is easier to install.
Yes, and sadly it's doing prejudice to this otherwise strong bot :(

It would be nice to have somebody who is well versed in unbuntu/apt/ppa/compilation/deb to implement a dedicated ppa for ubuntu and all usual go software. This ppa would include updated deb files for commonly used go program (Leela, Sabaki...). Those hard to compile/install programs like Ray would come in sort of containers or snap applications (with all dependencies included).

Not quite a come back of Hikarunix, but still a big improvement.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #151 Posted: Mon Jan 29, 2018 11:37 pm 
Beginner

Posts: 13
Liked others: 1
Was liked: 3
I usually use Windows, and Rn has no maintainer on Linux.
Pull requests is welcomed.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #152 Posted: Wed Jan 31, 2018 2:50 am 
Lives with ko
User avatar

Posts: 284
Liked others: 94
Was liked: 153
Rank: OGS 7 kyu
zakki wrote:
I usually use Windows, and Rn has no maintainer on Linux.
Pull requests is welcomed.
I am confident that at some point, somebody will show up and provide some help for linux, keep up the good work! :tmbup:

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #153 Posted: Fri Feb 02, 2018 12:04 am 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
Now in League A: Leela Zero 5773f44c (2018.01.26), it lost 5 games because of a ladder. Also Leela is updated to v0.11.0.

Leela vs. AQ
Code:
    1. AQ 2.0.3                        12/16
    2. Leela 0.11.0 Beta 11             4/16

League A:
Code:
    1. Leela 0.11.0                    18/20
    2. Rayon 4.6.0                     15/20
    3. Oakfoam 0.2.1 NG-06             12/20
    4. Leela Zero 0.11 5773f44c         7/20
    5. Hiratuka 10.37B (CPU)            6/20
    6. DarkForest v2 MCTS 1.0           2/20

League B:
Code:
    1. Leela Zero 0.11 c83e1b6e        15/20
    2. Pachi DCNN 11.99                13/20
    3. DarkGo 1.0                      12/20
    4. Dream Go 0.5.0                  11/20
    5. Ray 9.0.1                        7/20
    6. Mogo 4.86                        2/20

League C:
Code:
    1. MoGo 4.86                       18/20
    2. deltaGo 1.0.0                   14/20
    3. Fuego 1.1                       13/20
    4. Michi C-2 1.4.2                  8/20
    5. Orego 7.08                       5/20
    6. GNU Go 3.8                       2/20

League D:
Code:
    1. GNU Go 3.8                      25/28
    2. Hara 0.9                        18/28
    3. Matilda 1.25                    16/28
    4. Indigo 2009                     16/28
    5. Dariush 3.1.5.7                 15/28
    6. Aya 6.34                        13/28
    7. Fudo Go 3.0                      7/28
    8. JrefBot 081016-2022              2/28

League E:
Code:
    1. JrefBot 081016-2022             16/20             
    2. Iomrascálaí 0.3.2               12/20
    3. SimpleGo 0.4.3                  11/20
    4. Crazy Patterns 0008-13           7/20
    5. Marcos Go 1.0                    7/20
    6. AmiGo 1.8                        7/20

League F:
Code:
    1. AmiGo 1.8                       19/20
    2. Beancounter 0.1                 15/20
    3. Stop 0.9-005                    10/20
    4. GoTraxx 1.4.2                    7/20
    5. CopyBot 0.1                      6/20
    6. Brown 1.0                        3/20

Configuration:
League A: 2h/game, pondering off, 4 threads, 2GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time xx -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

League B-F: 1h/game, pondering off, 1 thread, 1GB on 4 x Intel® Core™ i5-4210H CPU @ 2.90GHz, 8 GiB Ram and GeForce 840M/PCIe/SSE2
TWOGTP=”gogui-twogtp -black \”$BLACK\” -white \”$WHITE\” -games 2 -size 19 -time xx -sgffile xxxx”
gogui -size 19 -program “$TWOGTP” -computer-both -auto

Amigo: amigogtp
AQ: AQ
Code:
aq_config.txt:
-main time[sec] =7200
-time controll =true
-japanese rule =true

Aya: Aya.exe –mode gtp –level max
Beancounter: beancounter
Brown: brown.exe
Copybot: python /path/to/__main__.py
CrazyPatterns: CrazyPatterns.exe
Dariush: DarGTP.exe –level 10
DarkForest: taskset -c 0 bash cnn_evaluator.sh 1 /data/local/go and taskset -c 0 th cnnPlayerMCTSV2.lua --num_gpu 1 --num_tree_thread 1 --rollout 2500 --win_rate_thres 0.1
DarkGo: darknet go engine cfg/go.test.cfg go.weights
deltaGo: deltaGo.exe
Dream Go: export NUM_ITER=1375 and dream_go
Fudo Go: taskset -c 0 fudo –boardsize=19 –komi=6.5
Fuego: fuego.exe –config fuego.cfg
Code:
fuego.cfg:
uct_param_search number_threads 1
uct_param_search lock_free 0
uct_max_memory 1024000000
uct_param_player reuse_subtree 1
uct_param_player ponder 0
uct_param_player early_pass 1

GnuGo: gnugo --mode gtp --level 10 --resign-allowed
GoTraxx: GoTraxx.exe
Hara: hara
Hiratuka: Hiratuka-19×19.exe -po 175000
IndiGo: Indigo.exe -gtp
Iomrascálaí: taskset -c 0,1 iomrascalai
JrefBot: java -jar jrefgo.jar 10000
Leela: leela_gtp_opencl –gtp –threads 4 –noponder
Leela Zero: leelaz --gtp --threads 4 --w /path/to/Leelaz_best-network_yyyy_mm_dd --noponder
Matilda: matilda
Code:
matilda.h:
#define BOARD_SIZ 19
#define DEFAULT_UCT_MEMORY 1000
#define DEFAULT_NUM_THREADS 1

Marcos Go: marcos_go --patterns /path/to/patterns.txt --cycles_mcts 10000 --threads_mcts 1
Michi C-2: michi gtp
Code:
ui.c:
init_large_patterns("patterns2.prob", "patterns2.spat"); // Michis pattern files renamed because they have the same name as Pachis files.

MoGo: mogo
Oakfoam: oakfoam -c nicego-cnn-06.gtp
Code:
nicego-cnn-06.gtp:
param playouts_per_move_max 40000
param thread_count 4

Orego: java -jar /path/to/orego-7.08.jar threads=1 grace
Pachi: pachidcnn -f pachibook.dat threads=4,max_tree_size=2048,pondering=0
Pachi: pachidcnn -f pachibook.dat threads=1,max_tree_size=1024,pondering=0
Ray: ray --time 3600 --thread 1 --no-debug
Rayon: rayon --thread 4 --no-debug
Simple Go: python /path/to/play_gtp.py --node_limit=100
Stop: /usr/bin/java -ea -jar /path/to/stop-09-005.jar --mode gtp

Links:
Amigo: https://sourceforge.net/projects/amigogtp/
AQ: https://github.com/ymgaq/AQ
Aya: http://www.yss-aya.com/
Brown: http://ricoh51.free.fr/go/engineeng.htm
Beancounter: Private
Copybot: https://github.com/sirtango/ICopyMoves
CrazyPatterns: https://www.remi-coulom.fr/Amsterdam2007/
Dariush: http://ricoh51.free.fr/go/engineeng.htm
DarkForest: https://github.com/facebookresearch/darkforestGo
DarkGo: https://pjreddie.com/darknet/darkgo-go-in-darknet/
deltaGo: http://home.q00.itscom.net/otsuki/delta.html
Dream Go: https://github.com/Chicoryn/dream-go
Fudo Go: http://www.geocities.jp/hideki_katoh/
Fuego: http://fuego.sourceforge.net/
GnuGo: https://www.gnu.org/software/gnugo/devel.html
GoTraxx http://gotraxx.codeplex.com/
Hara: https://github.com/antoniogarro/Hara
Hiratuka: Non GPU version (10.37B): http://www.vector.co.jp/download/file/winnt/game/fh673259.html / GPU version (10.38B): http://www.vector.co.jp/download/file/winnt/game/fh688349.html
IndiGo: http://www.math-info.univ-paris5.fr/~bouzy/INDIGO.html
Iomrascálaí: https://github.com/ujh/iomrascalai
JrefBot: http://ricoh51.free.fr/go/engineeng.htm
Leela: https://sjeng.org/leela.html
Leela Zero: https://github.com/gcp/leela-zero
Marcos Go: https://github.com/MarcosPividori/Go-player
Matilda: https://github.com/gonmf/matilda
Michi C-2 https://github.com/db3108/michi-c2
MoGo: https://lifein19x19.com/forum/viewtopic.php?p=211091#p211091
Oakfoam: https://bitbucket.org/dsmic/oakfoam
Orego: https://sites.google.com/a/lclark.edu/drake/research/orego
Pachi: http://pachi.or.cz/
Rayon: https://github.com/zakki/Ray
Ray: https://github.com/kobanium/Ray
Simple Go: https://sourceforge.net/projects/londerings/
Stop: https://www.vanheusden.com/stop/

Best,
Alex


This post by as0770 was liked by: pnprog
Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #154 Posted: Sat Feb 03, 2018 3:04 am 
Lives with ko

Posts: 135
Liked others: 1
Was liked: 1
Rank: 30 kyu
as0770 wrote:
q30 wrote:
as0770 wrote:
You have no idea what you are talking about. Standard deviation doesn't change with the timecontrol.

It depends on game randomness, that changes with the time control...


Then we have to rewrite basic mathematical principles.

So, it will be good, if we will rewrite Your representations about basic mathematical principles...
For beginning, Standard deviation is square root of give right translation to English Yourself, that can be determined by next:
https://wikimedia.org/api/rest_v1/media/math/render/svg/1d1610b913011b6744f23f47e0920974b7f78f58,
where pi in our case depends among others on time control...

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #155 Posted: Sat Feb 03, 2018 5:15 am 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
q30 wrote:
So, it will be good, if we will rewrite Your representations about basic mathematical principles...
For beginning, Standard deviation is square root of give right translation to English Yourself, that can be determined by next:
https://wikimedia.org/api/rest_v1/media/math/render/svg/1d1610b913011b6744f23f47e0920974b7f78f58,
where pi in our case depends among others on time control...


Nice you tried to understand my points. Of course the probability _can_ change _slightly_ with the time control. But the result in a 1h match and a 2h match will be more or less the same. What you claim is that the result of a 2h match will show the relative strength more accurate than a 1h match, and that is nonsense.

Two engines of equal strength will have a 50% chance for a 1-1, a 25%chance for a 0-2 and 25% chance for a 2-0. If you double the time control from 1h to 2h the over all winning probability will _maybe_ change to 51:49%. Experience in engine matches in chess is that you get basically the same results in 1min/game and 2h/game as long as there is no significant bug. The difference of 1h/game and 2h/game match is not measurable. There is no reason why it should be different in Go. Even if the probability changes to 55:45%, you would need hundreds of games to prove the difference in strength. What I do is a tournament with 20 or 30 games. If I run the tournament twice I can get completely different results. This won't change with 2h/games or pondering on (League A is 2h on 4 threads btw).

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #156 Posted: Sat Feb 03, 2018 6:59 am 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
as0770 wrote:
q30 wrote:
So, it will be good, if we will rewrite Your representations about basic mathematical principles...
For beginning, Standard deviation is square root of give right translation to English Yourself, that can be determined by next:
https://wikimedia.org/api/rest_v1/media/math/render/svg/1d1610b913011b6744f23f47e0920974b7f78f58,
where pi in our case depends among others on time control...


Nice you tried to understand my points. Of course the probability _can_ change _slightly_ with the time control. But the result in a 1h match and a 2h match will be more or less the same. What you claim is that the result of a 2h match will show the relative strength more accurate than a 1h match, and that is nonsense.

Two engines of equal strength will have a 50% chance for a 1-1, a 25%chance for a 0-2 and 25% chance for a 2-0. If you double the time control from 1h to 2h the over all winning probability will _maybe_ change to 51:49%. Experience in engine matches in chess is that you get basically the same results in 1min/game and 2h/game as long as there is no significant bug. The difference of 1h/game and 2h/game match is not measurable. There is no reason why it should be different in Go. Even if the probability changes to 55:45%, you would need hundreds of games to prove the difference in strength. What I do is a tournament with 20 or 30 games. If I run the tournament twice I can get completely different results. This won't change with 2h/games or pondering on (League A is 2h on 4 threads btw).


Although, it's best not to take this heuristic too seriously, because a nontrivial change is possible. I haven't read it that closely, but my skim of the following thread https://github.com/gcp/leela-zero/issues/667 suggested that that Leela Zero has sometimes got noticeably different results between very small numbers of playouts, like 5, and a larger number number of playouts, like 1600, where the relative strength difference and even sometimes the ordering of strength would change between the neural nets.

It's not actually not surprising at all to me that Leela Zero in some cases could have quite a large difference in strength between tiny numbers of playouts and large numbers of playouts, enough to change the ordering between nets. For example new candidate nets often appear to vary in strength on the order of multiple hundreds of Elos, so training is very noisy, and there's no reason to expect that the quality of the policy part of the neural net and the value part of the neural net always vary together in the same way. And thinking in those terms, it's pretty obvious that you're measuring something fairly different at 5 playouts vs at 1600 playouts. With very few playouts you rely on the policy net more heavily.

I agree that if you're only running 20 or 30 games, then of course none of this matters, the noise in 20 to 30 games still dwarfs this. :)

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #157 Posted: Sat Feb 03, 2018 9:31 am 
Lives with ko

Posts: 135
Liked others: 1
Was liked: 1
Rank: 30 kyu
Quote:
as0770 wrote:
Quote:
q30 wrote:
So, it will be good, if we will rewrite Your representations about basic mathematical principles...
For beginning, Standard deviation is square root of give right translation to English Yourself, that can be determined by next:
https://wikimedia.org/api/rest_v1/media ... 74b7f78f58,
where pi in our case depends among others on time control...



Nice you tried to understand my points. Of course the probability _can_ change _slightly_ with the time control. But the result in a 1h match and a 2h match will be more or less the same. What you claim is that the result of a 2h match will show the relative strength more accurate than a 1h match, and that is nonsense.

Two engines of equal strength will have a 50% chance for a 1-1, a 25%chance for a 0-2 and 25% chance for a 2-0. If you double the time control from 1h to 2h the over all winning probability will _maybe_ change to 51:49%. Experience in engine matches in chess is that you get basically the same results in 1min/game and 2h/game as long as there is no significant bug. The difference of 1h/game and 2h/game match is not measurable. There is no reason why it should be different in Go. Even if the probability changes to 55:45%, you would need hundreds of games to prove the difference in strength. What I do is a tournament with 20 or 30 games. If I run the tournament twice I can get completely different results. This won't change with 2h/games or pondering on (League A is 2h on 4 threads btw).

You are quite right, if there is the same engine sparring. But even if there will be 2 simple MC engines (which will in sparring demonstrate mentioned by You chances with time on move --> 0), it may be difference in strength (i.e. in chances) dependent on time control because of difference in best move choice algorithm (and especially more complex engines with more complex algorithms).
You can try to compare 2 engines (with close strength levels) results with time and thread control, that You have used for league B-F, and results of these engines sparring with 2' per move and 4 threads...

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #158 Posted: Sat Feb 03, 2018 1:07 pm 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
q30 wrote:
You are quite right, if there is the same engine sparring. But even if there will be 2 simple MC engines (which will in sparring demonstrate mentioned by You chances with time on move --> 0), it may be difference in strength (i.e. in chances) dependent on time control because of difference in best move choice algorithm (and especially more complex engines with more complex algorithms).
You can try to compare 2 engines (with close strength levels) results with time and thread control, that You have used for league B-F, and results of these engines sparring with 2' per move and 4 threads...

You don't get the point. The statistical fluctuation is way too high to meassure little differences in strength. I won't play hundreds of games to prove you wrong.
Once again: This are two matches with the same engines and the same conditions:
as0770 wrote:
Pachi vs. Hiratuka 8:8
Pachi vs. Hiratuka 2:14

This discussion doesn't make any sense. No more replies by me.

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #159 Posted: Sat Feb 03, 2018 10:40 pm 
Lives with ko
User avatar

Posts: 284
Liked others: 94
Was liked: 153
Rank: OGS 7 kyu
as0770 wrote:
Now in League A: Leela Zero 5773f44c (2018.01.26), it lost 5 games because of a ladder. Also Leela is updated to v0.11.0.

Thanks for running the tournament and sharing the result. It's nice also to have the list of internet links :salute:

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Top
 Profile  
 
Offline
 Post subject: Re: Engine Tournament
Post #160 Posted: Sun Feb 04, 2018 2:24 am 
Lives with ko

Posts: 180
Liked others: 15
Was liked: 23
Rank: Beginner
lightvector wrote:
Although, it's best not to take this heuristic too seriously, because a nontrivial change is possible. I haven't read it that closely, but my skim of the following thread https://github.com/gcp/leela-zero/issues/667 suggested that that Leela Zero has sometimes got noticeably different results between very small numbers of playouts, like 5, and a larger number number of playouts, like 1600, where the relative strength difference and even sometimes the ordering of strength would change between the neural nets.

It's not actually not surprising at all to me that Leela Zero in some cases could have quite a large difference in strength between tiny numbers of playouts and large numbers of playouts, enough to change the ordering between nets. For example new candidate nets often appear to vary in strength on the order of multiple hundreds of Elos, so training is very noisy, and there's no reason to expect that the quality of the policy part of the neural net and the value part of the neural net always vary together in the same way. And thinking in those terms, it's pretty obvious that you're measuring something fairly different at 5 playouts vs at 1600 playouts. With very few playouts you rely on the policy net more heavily.

I agree that if you're only running 20 or 30 games, then of course none of this matters, the noise in 20 to 30 games still dwarfs this. :)


Of course with 5 playouts there will be different results, but we are talking about 1h/game vs 2h/game what is 7000 vs. 14000 playouts on my system.

It is also funny to follow the history when I replace or remove some engines, look at Ray:
Code:
    1. Ray 9.0.1                    29/32
    2. Pachi DCNN 11.99             28/32
    3. Leela Zero 0.9 (2018.01.01)  19/32
    4. MoGo 4.86                    18/32
    5. deltaGo 1.0.0                17/32
    6. Fuego 1.1                    15/32
    7. Michi C-2 1.4.2               8/32
    8. Orego 7.08                    8/32
    9. GNU Go 3.8                    2/32

Code:
    1. Leela Zero 0.11 c83e1b6e        15/20
    2. Pachi DCNN 11.99                13/20
    3. DarkGo 1.0                      12/20
    4. Dream Go 0.5.0                  11/20
    5. Ray 9.0.1                        7/20
    6. Mogo 4.86                        2/20

And at DreamGo:
Code:
    1. DreamGo 0.5.0                   15/20
    2. DarkForest v2 MCTS 1.0          12/20
    3. Pachi DCNN 11.99                12/20
    4. DarkGo 1.0                      10/20
    5. Ray 9.0.1                        9/20
    6. Mogo 4.86                        2/20

It do not replay the whole tournament, I just remove the old engines and add the new ones.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 390 posts ]  Go to page Previous  1 ... 5, 6, 7, 8, 9, 10, 11 ... 20  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group