LZ's progression

[Satorian]

So that was 5 mins per side per game, these test are 1 minute. 184 is a 40 block so (assuming there isn't an LZ 0.16 fp16 speedup between the two tests, and there may be) 800/5 = 160 so a lot short of the several thousand I like to give the 40 blocks to show their strength. If there is fp16 difference that'll help at most a factor of 2.

Interesting. Thank you! So even with the FP16 best case, we'd be at ~320 playouts. That's a far cry from what I would consider reasonable, especially given your experience and reports on playout effects.

I would prefer to have the engines simply play at 10s or 20s or 30s per move, in pure byo-yomi. No time management, no fancy distractions, and a move speed that is realistic either for online go or longer OTB games.

[Uberdude]

... though if I can get a decent few thousand playouts in 20 seconds per move on my 1060 GPU, Vargo may be able to get the same in 5 seconds on his top-end 1080 Ti.

[Vargo]

mini 10 game match
Elfv1 v. V16_000513 converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 10, time parity, twogtp 1.5.0

Result : 10-0 for Elf...
I used -alternate, so, Elf-White seems to win 10 times, but in fact, Elf won 5 times as W and 5 times as B.


Something wrong with the parameters ?
(time is several seconds per move)

mgelf.jpg (347.8 KiB) Viewed 12320 times

I'll try another match, with an other converted network.

[Mike Novack]

... though if I can get a decent few thousand playouts in 20 seconds per move on my 1060 GPU, Vargo may be able to get the same in 5 seconds on his top-end 1080 Ti.

THAT is more or less what I have been trying to say. I was NOT saying a particular amount of time but one realistic scaled to what we decide to consider "adequate hardware" << so less actual clock time if tests run on a more powerful machine >>

For example, we might decide "should be a portable machine" so power consumption and cooling requirements limit the gpu to the equivalent of a 1060. Our question would then be "what is the most powerful (go strength( program if run on a powerful, but portable, gaming machine at fast, but playable for humans time control?"

If the actual tests were to be run on a machine X times faster then we would scale the time to 1/x of that.

[And]

mini 10 game match
Elfv1 v. V16_000513 converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 10, time parity, twogtp 1.5.0

Result : 10-0 for Elf...

v15(20x256) is better:
https://cloudygo.com/v16-19x19/eval-graphs
https://cloudygo.com/all-eval-graphs?bucket=

[Vargo]

another mini 10 game match
Elfv1 v. V15-000990-cormorant converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 5, time parity, twogtp 1.5.0

Result : 6-4 for Elf

time is several seconds per move. 10 games is too few, I'll run more games tomorrow.

[Uberdude]

mini 10 game match
Elfv1 v. V16_000513 converted
LZ0.16, komi 7.5, no pondering, 2x1080Ti, -r 10, time parity, twogtp 1.5.0

Result : 10-0 for Elf...
I used -alternate, so, Elf-White seems to win 10 times, but in fact, Elf won 5 times as W and 5 times as B.

Something wrong with the parameters ?
(time is several seconds per move)

I expect what we are seeing here is the reduced playouts that 40-block v16_513 minigo gets in this short time parity are really hurting it against the 20-block elfv1. If you ran a test of fewer games but with enough time for minigo to get around 3k playouts per move (so I guess Elf gets about 6-8k, playouts speed isn't quite linear in number of blocks) then you'd find it not so lopsided. (By analogy with 40-block LZ's, I don't have much feel for minigo's behaviour yet)

[Uberdude]

So I manually played a game between elfv1 (black) and minigo v16.513 (white), giving them >30s a move (approx ~3k for minigo and 6k for elf), giving them both extra time on hard moves, watching for where they agreed and differed. Elf as black (which it tends to favour) thought it was winning a lot, but had a blind spot for white p15 so its o17 was a lot worse than it expected. I don't have the time to play it to counting, but they both agree the game is pretty good for white (Elfv1 says 75%, minigo 85%). Here's the sgfs with winrates from each's point of view. Also elf missed q2 and winrate plunged on that, minigo saw it quickly (but probably not quickly enough for Vargo's setup).





And here's their view of the winrate, minigo's is much more stable/consistent. Does this mean it can't detect tiny mistakes as well as Elf can, or is Elf just being erratic and missing moves which cause it to change evaluation? I tend towards the latter.

elf winrate.PNG (287.94 KiB) Viewed 12192 times

minigo winrate.PNG (287.96 KiB) Viewed 12192 times

[splee99]

I have tried minigov15-modeste_converted and minigov15-cormorant_converted. They look indeed stronger than ELFv1. The minigo style is really far from human style. I don't see apparent activities of acquiring solid territory, nor building walls in the center. It just plays rock solid, so the ELF's sharp attack is not effective. However, this also means it is bad at (giving) handicap games because minigo is not good at attacking opponents weakness, in particular, regarding ko fights and capture races.

[Vargo]

10 game match
Elfv1 v. MG_000513_revenge

LZ0.16, twogtp 1.5.0 , 2x1080Ti, 8000 visits for Elf, 4000 visits for 513 (corresponding roughly to 2.6 and 3s/move)
-r 5 , no pondering
result : Elf wins 8-2

513_elf.jpg (388.04 KiB) Viewed 14584 times

NB. I forgot to change the default sgf name, so, the games are labelled mg990elf_v4000-x.sgf, but the network used is revenge_000513, as you can see in the command line.

[Vargo]

The first picture of this post shows contrasted results for 990_cormorant v. Elfv1 (cormorant wins at 10s/move and at 120s, but loses at 60s/move)

Here are two more 10 game matches with settings comparable to his 10s/move.
gpu: 2x1080Ti, constant 5s/move (5.03s effectively used) , twogtp, -alternate, -r 5, no pondering
All games by resignation, no error, no duplicate game.
Average length is 247 moves for the first 10 game match, and 212 moves for the second match.

Result : Elf wins 12-8

If we add the results of the three 10 game matches, we get ... 15-15
My conclusion is that 990 cormorant and Elfv1 are about the same strength at 10s/move with one 1080Ti.

Stats :

1.jpg (294.46 KiB) Viewed 14427 times

2.jpg (274.02 KiB) Viewed 14427 times

I'll upload the games if someone wants them.

[And]

new network 15x192 60:40 over f438268e:
https://github.com/leela-zero/leela-zero/issues/2192

[And]

My conclusion is that 990 cormorant and Elfv1 are about the same strength at 10s/move with one 1080Ti.

"v15-19x19_models_000990-cormorant_converted_quantized.txt" is stronger than "000990-cormorant_converted.txt"
https://github.com/leela-zero/leela-zer ... -459413040

[Pippen]

Is there a possiblity to download the last - say 100.000 games - from LZ's self play as a sgf collection?

[Tryss]

See here :

https://leela.online-go.com/zero/