Someone at reddit/cbaduk (here) asked about 161 vs 157 at time parity...

I've run a 20-games match between #157 and #161 at 5 min. per game per side, no ponder, komi 7.5 (twogtp V1.4.10, 1xGTX1080)

LZ_015#157 v. LZ_015#161 :
LZ#157 wins 17-3 (10 wins as W, 7 wins as B)

Ouch... ! 20 games is not enough to be really sigificant, but stilll...

If someone wants the games, I'll upload them.

Actually, if a program A is stronger than B ( the win probability of A against B is >= 0.5), then there is under 0.13% chance that the program A get 3 win or less in 20 games. So the result is statistically significant.

But is statistically significant really significant?

This may sound flip, but it is related to the replication crisis.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Another 20-games match at 10min/game (1x1080): #157 wins 16-4 (9 times as W, 7 times as B)

And a 10-games match at something like 15min/game #157 wins 6-4 (3 times as W, 3 times as B)
(in fact it's a 5min/game match, but with 2x1080Ti, which corresponds to ~12-16 min/game on a 1080)

Again, if someone wants the games or the reports, I'll upload them.

In the 50 games played in these matches, #161 won only 11, that's 22%, and #161 has a better Elo score than #157, mmmmm... How come ?

Anyway, if someone wants the reports or the games, I'll upload them.

How come, indeed?

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Test matches which are used to calculate the Elo score of LZ networks are not with time parity, but with the same number of visits. 20-block networks take more time per visit than 15-block networks.

The Elo scale is at visits parity, that's why I'm doubtful about it... I would prefer something measuring "real" strength, and with the same scale as for human players, but maybe it's hard or impossible to do.

Anyway, I find all this very interesting and I hope LZ will bounce back !

Accumulation of errors without calibration, either to older versions of self or an external benchmark. (On a related note, dead-reckoning / inertial guidance systems (as for Apollo or ICBMs) are amazing feats of engineering, I was recently reading about them.)


I think I've managed to get twogtp working and am running a 147 vs 157 match now at 3200 visits.

Update: currently #157 leads only 7-3 as white. #157 6-0 as black before my PC crashed (maybe I shouldn't run this match and LZ and Elf analysis concurrently!).

20-games match between LZ015#157 and LZ015#163 at time parity
#163 is 11985 Elo
#157 is 11806 Elo

5-min games, 1x1080, twogtp V1.4.10

#157 wins 16-4 (10 wins as W, 6 wins as B)

I understand the ranking matches are at visits parity, and all that, but there's still something weird with the Elo scale used. For example, the ranking of L-zero has #163 higher than #157 in "Dan scale" ...

Here, #163 wins only 20% of its games against #157... It's only 20 games, but I doubt that #163 is stronger than #157.

Could you try #165? It is noticeably more aggressive than before.

20-games match between LZ_0.15#165 and LZ_0.15#157 at visits parity.

twogtp V1.4.10 --noponder --visits=1601 -komi 7.5 for both.
If you look at the .dat reports, you'll see that #165 takes at least twice as long as #157 to think.

#165 wins 14-6 (9 times as W, 5 times as B, all games by resignation)




Ok. Tomorrow, I'll run a 20-games match at time parity (#165 v #157). It will be interesting to compare results at visits parity and at time parity.

Thanks for the report. I had run one game between #165 and #157 with time parity and #157 won. However the battle was so heavy that only bots can deal with it. Anyway I hope we can see some progress from #161 to #165 from your test tomorrow.

40-game match at time parity between #165 and #157
(40 games, because I made a mistake, the first 20 games were all with #157 as White, so I had to run 20 more)

5 min per game, komi 7.5, no pondering, 1xGTX1080, twogtp V1.4.10
5 min per game on my computer is similar to 1600 visits for #165 or 3200 visits for #157

#157 wins 27-13 (15 times as W, 12 times as B, all games by resignation)

13 wins for #165, that's 32.5 %, much better than #161, which scored only 15% in a 20-game match at 5 min against #157.

The 40 games :

There's a new 40block network out: http://zero.sjeng.org/networks/e2be4815 ... 1c836a3.gz. I wonder how would this would do vs #157 on 5 mins time parity? It got a very respectable 42% vs Elf v1 (presumably at visits parity) and 83% vs #162. So in a 5 mins game it'll probably get about 800 visits (Edit: is that true, it seems not approx. linear in blocks if 15 to 20 is a halving of visits!) to #157's 3200. Will its superior intuition and judgement win out, or will it make tactical blunders like ladders due to not enough playouts and lose?

20-game match between LZ0.15 with e2be48 (256x40) and LZ0.15 with #157 (192x15)

e2be48 at 1600 visits, and #157 at 6400 visits.

I think time parity is very roughly :
1x visits for 256x40 or
2x visits for 256x20 or
4x visits for 192x15

So, e2be48 at 1600 visits v. #157 at 6400 visits is at time parity.

#157 wins 11-9 (5 times as W, 6 times as B)
I just realized (better late than never !) that twogtp can give a nice and complete .html report, with min, max, standard deviation...

LIKE THIS

AND THIS

With 2xGTX1080Ti,
Average time used per game is around 3min10sec (see html for exact numbers)
Average length is around 220 moves

So the latest 40b has about the same strenght as #157 at time parity ? Nice !

I doubt this could be said in a general sense.


The point of these deep and strong nets is to direct the search, shape the search tree. But with 1600 visits you don't have much of a search tree to be shaped (just a few moves lookahead). In those cases the bot won't be really strong anyway (even AGZ was only high dan amateur without search), and the most important is getting more visits (which #157 does).

But in serious games these bots do a lot more visits per move on good gpus. I suggest you repeat the test with at least 10k visits vs 40k. I would be surprised if 40b won't crush #157.

You must be right, because I've tested the new 40b (e2be48) against #157 at time parity : only 5min on 1x1080.
For e2be48, it corresponds to only 500-600 visits(?)
#157 wins 14-6
6 wins as W here , and 8 wins as B

Maybe I'll try tomorrow a mini match at time parity, with 3200 or 6400 visits for e2be48.
e2be48 should be stronger, we'll see.

Well, I did a game (it took 2 hours) at large visits and didn't get the result I expected: 15 block #157 as white with 80k playouts beat 40 block 40b_157_360k e2be with 20k playouts which is very close to equal time (15b took 3114s, 40b 3201s). Looks like it was a half pointer but black did a load of inside territory nonsense instead of resigning, should I adjust some parameter?



2nd game 40 block won, same opening for first 27 moves, #157 diverged.

Here is a 40k vs 10k match I just did :


40d win as white against #157, (and it's close to time parity, with 2407s for B and 2559s for W)