LZ's progression

nbc44 · **#141**

Last attachment.

P.S. to be continued

nbc44 · **#142**

Another test (l0 v15, validation.exe .... --gpu 0 --gpu 1 --noponder -t 12 -q -d -r 10 --timemanage off) #157 vs #178 :

a). by visits 8001:3201 (157-178.zip)

Code:

The first net is worse than the second
#157 v #178 ( 110 games)
          wins        black       white
#157   43 39.09%   21 38.18%   22 40.00%
#178   67 60.91%   34 61.82%   33 60.00%
                   55 50.00%   55 50.00%
110 games played.
Status: 1 LLR -3.00809 Lower Bound -2.94444 Upper Bound 2.94444

P.S. more better than last test :^)

nbc44 · **#143**

Yet another test (l0 v15, validation.exe .... --gpu 0 --gpu 1 --noponder -t 12 -q -d -r 5 --timemanage off) #157 vs #178 :

a). by visits 9601:3201 (157-178-5.zip)

Code:

The first net is worse than the second
#157 v #178 ( 241 games)
          wins        black       white
#157  112 46.47%   57 46.34%   55 46.61%
#178  129 53.53%   66 53.66%   63 53.39%
                  123 51.04%  118 48.96%

241 games played.
Status: 1 LLR -2.95061 Lower Bound -2.94444 Upper Bound 2.94444

nbc44 · **#144**

So-called "time parity" test - 10 min per game.

Code:

gogui-twogtp -black "C:\APPS\l0gpu\leelaz.exe --gtp --weights=C:\APPS\net\d351f06e.gz -t 12 --gpu 0 --gpu 1 --noponder" -white "C:\APPS\l0gpu\leelaz.exe --gtp --weights=C:\APPS\net\58d1222a.gz -t 12 --gpu 0 --gpu 1 --noponder" -games 100 -sgffile 157-178-10m -auto -time 10 -komi 7.5 -verbose

Part1.
#157 (black) vs #178 : +48-52=0

nbc44 · **#145**

Part2.
#157 (white) vs #178 : +60-40=0 (???)

Finally:

#157 vs #178 : +108-92=0

moha · **#146**

nbc44 wrote:

So-called "time parity" test - 10 min per game.

This says almost nothing without mentioning your gpu or average nps.

nbc44 · **#147**

moha wrote:

nbc44 wrote:

So-called "time parity" test - 10 min per game.

This says almost nothing without mentioning your gpu or average nps.

2 х 1080ti with Temp. limit 80 C and Power limit 90%. :salute:

First two moves (part 1 & 2):

moha · **#148**

Interesting, there seems quite a visit fluctuation between positions (even for the 1st move on empty board, and even outside tree reuse?). ~~If low-visit move bottlenecks happen randomly throughout the game, that may have an impact on 40b's results.~~

Edit: Never mind, I figured it out.

Btw, what is the actual visit fluctuation, for example between early and late positions? Does it remain within these 25-30%?

nbc44 · **#149**

moha wrote:

Btw, what is the actual visit fluctuation, for example between early and late positions? Does it remain within these 25-30%?

FYI, "big data" in action

. Format - "black~white".

1). by visits

nbc44 · **#150**

2). by node/s

moha · **#151**

Thanks. Looking at LZ's reported "visits" some huge swings are expected since that includes tree reuse. But it seems there actually are large fluctuations in n/s as well, which is not subject to tree reuse (it's taken as playouts/time IIRC).

Bottom line is, it's not surprising if results with fixed (asymmetrical) visits differ somewhat from fixed per-game time.

nbc44 · **#152**

Yet another so-called "time parity" test - 10 min per game.

Code:

--precision half -t 12 --gpu 0 --gpu 1 --noponder

Part1: #157 (black) vs #181 : +32-68=0

Part2: #157 (white) vs #181 : +35-65=0

Finally: #157 vs #181 : +67-133=0

Is it a miracle or just an ordinary bug of "precision half" option? A farewell to #157?

P.S. L0-next 29/09/2018

Vargo · **#153**

Another brick in the (same) wall
#181 v. #157 (visits=12801 for #157 and visits=3201 for 181)

#181 wins 7-3

(twogtp v1.4.10, noponder, average length : 237 moves, #157 takes 10% more time than #181)

Attachment:

157_181_157isW.zip [4.86 KiB]
Downloaded 537 times

Attachment:

157_181_157isB.zip [4.91 KiB]
Downloaded 542 times

nbc44 · **#154**

Quick answer:

Code:

C:\APPS\l0gpu\validation.exe -k 157-181-10 -b C:\APPS\l0gpu\leelaz -n C:\APPS\net\d351f06e.gz -o "-g -v 12801 --gpu 0 --gpu 1 --noponder -t 12 -q -d --timemanage off -w" -b C:\APPS\l0gpu\leelaz -n C:\APPS\net\68824bbc.gz -o "-g -v 3201 --gpu 0 --gpu 1 --noponder -t 12 -q -d --timemanage off -w"

Code:

6 wins, 4 losses
10 games played.

#157 wins 6 - 4 :clap:

edit.

Code:

gogui-twogtp -black "C:\APPS\l0gpu\leelaz.exe --gtp --weights=C:\APPS\net\d351f06e.gz -t 12 --gpu 0 --gpu 1 -v 12801 --noponder --timemanage off" -white "C:\APPS\l0gpu\leelaz.exe --gtp --weights=C:\APPS\net\68824bbc.gz -t 12 --gpu 0 --gpu 1 -v 3201 --noponder --timemanage off" -games 5 -sgffile 157-181-visits -auto -komi 7.5 -verbose
&
gogui-twogtp -black "C:\APPS\l0gpu\leelaz.exe --gtp --weights=C:\APPS\net\68824bbc.gz -t 12 --gpu 0 --gpu 1 -v 12801 --noponder --timemanage off" -white "C:\APPS\l0gpu\leelaz.exe --gtp --weights=C:\APPS\net\d351f06e.gz -t 12 --gpu 0 --gpu 1 -v 3201 --noponder --timemanage off" -games 5 -sgffile 157-181-visits -auto -komi 7.5 -verbose

#157 wins 8 - 2 :clap:

x 2

Any idea?

Vargo · **#155**

We must have the same kind of PC (I have 2x1080Ti, i9-7920X), and except for your -t 12 -q -d , I have the same commands.
On mine, a 10min (per side) game lasts in fact around 820 sec, as LZ doesn't use up all its time, and a 12801v3201 visits game lasts about the same time, but slightly favours #157, as it takes 10% more time than 181.

181 won 66% of your 200 games at 10 min.

181 won 7 of its 10 games
181 won 4 of its 10 games
181 won 2 of its 10 games
that's 43% of the 30 games, but with ~10% less time than 157

I'd trust your 200 games result. Probably 3x10 games is not enough, or ... maybe it's 181's way of asking for 10% more time ;-)

nbc44 · **#156**

Due to the fact that your processor is faster, I suggest repeating your last test (10 games) with my settings:

Code:

validation -k 157-181 -b leelaz -n #157net -o "-g -v 12801 --gpu 0 --gpu 1 --noponder -t 12 -q -d --timemanage off -w" -b leelaz -n #181net -o "-g -v 3201 --gpu 0 --gpu 1 --noponder -t 12 -q -d --timemanage off -w"

P.S. I have 2 x 1080ti too and Xeon E5-1650 v4, win10.
P.P.S. Benchmark:

Vargo · **#157**

I don't think that the CPU is important, we have the same GPUs, and so, very similar benchmarks

Most probably, 10 or 30 games is not enough. For example, in your two 10 game matches, 181 doubles its winning percentage (20% and 40% , it's a huge swing). That's why I have more faith in your 200 game match

splee99 · **#158**

If you have two GPU's, why don't you try assign GPU0 to 181 and GPU1 to 157? I know this would make the speed slower, but it will make a fair match because some data maybe cached in a GPU during the game.

nbc44 · **#159**

Last so-called "time parity" test - 10 min per side.

Code:

--precision single -t 12 --gpu 0 --gpu 1 --noponder

Part1: #157 (black) vs #181 : +31-69=0

Part2: #157 (white) vs #181 : +53-47=0

Finally: #157 vs #181 : +84-116=0

A farewell to #157!!! :clap:

P.S. L0-next 29/09/2018

Vargo · **#160**

New network :clap:

(183)

Two 20 game matches #183 v #157 at time parity :

Match 1 :
LZ 015, GPU : 1x1080, no pondering, 5 min. per side , corresponding to ~800 visits for #183 and to ~2800 visits for #157
#183 wins 11-9 (all games by resignation, about 240 moves per game, and 230" used per side and per game)

Match 2 :
GPU : 2x1080Ti, no pondering, 7 min. per side , corresponding to ~3400 visits for #183 and to ~12800 visits for #157
#183 wins 12-8 (all games by resignation, about 240 moves per game, and 320" used per side)

Finally, 183 is better than 157, even at low time settings (less than 2" per move with a 1080)

If someone wants the games...

LZ's progression

Who is online