so basically leela zero has made 0 progress in the last 4mo

[hydrogenpi7]

so basically leela zero has made 0 progress in the last 4months since the last 15 block 157 net

on average hardware that still beats the latest 40block even on time parity

[Vargo]

According to THIS SITE, the network
#157 (4 months ago) is 10.6d , and #191 (latest) is 12.85d
More than 2 stones difference...

But I'm a bit like you, I doubt very much #157 is over 2 stones stronger at time parity.
Maybe LZ is simply plateauing , but I hope I'm wrong...

[Uberdude]

on average hardware that still beats the latest 40block even on time parity

Please provide evidence of this (and qualify average hardware and what time per move). 1 month ago you might have been correct, but the recent 40b networks now surpass #157 in game/analysis-realistic times on my GTX 1060 (in my playing around on Fox and reviewing experience, particularly with ladders, but I've not done a big match for 100+ games). It is not enough to just say time parity, as the extensive research documented at https://github.com/gcp/leela-zero/issues/1914 shows at very short time parity tests the 40 block networks are better than #157 and have been for a while, whilst at more normal (e.g. 20s a move on a 1060 GPU) #157 was stronger for quite a while but that's changing recently.

Vargo did a 20-game match of #185 vs #157 with equal time (5 min/game on a 1080) on 1st Nov and 185 won. viewtopic.php?p=238518#p238518. Or #181 beating #157 in a 200-game time parity match back on Oct 1st: viewtopic.php?p=237567#p237567.

[Javaness2]

so basically leela zero has made 0 progress in the last 4months since the last 15 block 157 net

on average hardware that still beats the latest 40block even on time parity

One more shitty claim for the road

[Umsturz]

The new 40x256 nets are all way stronger. For example the net #157 got demolished by an inofficial net called 1fdfb1c5 (trained by bjiyxo) as you can see here in the first line:

http://zero.sjeng.org/network-profiles/ ... 96aea95a0c

This inoffical net got a 337 : 63 (84.25%) result against #157. And the current 40x256 nets are stronger than this older 1fdfb1c5 net.

The problem is that it is hard to compare different sized nets on the same hardware. Smaller nets get more playouts on the same hardware as bigger networks, which is an advantage. But bigger networks have more potential to evaluate and play the better moves.

[Vargo]

Tomorrow, I'll run a #191 vs #157 match at time parity, with reasonably long time settings.

I think #191 is stronger, but probably not by 2+ stones. At this level, it's a huge difference.

[tchan001]

I would like to think that bigger networks require better hardware for optimized results. It's like trying to solve math with a basic calculator vs an average computer. Basic addition and subtraction would be restricted to the input speed of the operator, but as we move up the scale to higher levels of mathematics the calculator will lose out. So in terms of progress with the use of a basic calculator, it would seem that moving up to higher levels of mathematics does not produce better results.

I have read in a Chinese go forum that with the new Leela zero engine (0.16), it would take a 2x 1080ti setup to be able to enjoy the real speed of the new algorithms.

Quoting the Chinese go forum with the use of google translate

The amount of calculation is large, the demand for graphics cards surpasses almost all large games. If you want to play a big weight like 40B, the two 1080TI is the entry configuration.

[Vargo]

10 game match #191 v. #157

no pondering, -r 10, komi 7.5

15 min games (per side and game) with 2x1080Ti
benchmark 520 n/s (#191) and 1510 n/s (#157)
average length : 243 moves
average time B : 642"
average time W : 645"
That's around 5-6 s/move, amounting to a reasonable 20" per move (???) for one average GPU.

Result : #191 wins 8-2

Too few games, but large margin.

If someone wants the games, I'll upload them

[bernds]

10 game match #191 v. #157

Result : #191 wins 8-2

I was curious, so I was trying to run something similar yesterday. While I got a similar result, the games all looked identical up to move 40 or so. If the same thing happened in your run, I don't think we can say anything about relative strengths - only that #157 misevaluates one particular fuseki.

I've tried to make an opening book for twogtp, consisting of a number of files with eight moves each, so that the programs would start from those positions. I've left it overnight, and it's not done, but so far the results are far more even, with #157 (playing as Black in every game so far, which you'd expect to be a disadvantage) winning slightly more often.

Hardware is a GTX 1060, and the full command line:

Code: Select all

gogui-twogtp -black "/local/go/software/leela-zero/autogtp/leelaz -w  /local/go/software/leela-zero/autogtp/net157-15x192final-d351f06e.gz --noponder  -g " -white "/local/go/software/leela-zero/autogtp/leelaz -w  /local/go/software/leela-zero/autogtp/net191-53f805d1.gz --noponder  -g " -auto -games 60 -verbose -sgffile lz157-191d -time "1+1/8" -openings /local/go/openings/ -debugtocomment

[Vargo]

While I got a similar result, the games all looked identical up to move 40 or so.

Nice to see that you have similar results. There's no duplicate game in my 10 games.
Below is a picture of the 10 games at move 60, they don't look too identical.

xx.jpg (319.82 KiB) Viewed 22464 times

[dfan]

I have read in a Chinese go forum that with the new Leela zero engine (0.16), it would take a 2x 1080ti setup to be able to enjoy the real speed of the new algorithms.

I am not sure why they say that. Leela Zero is already strong amateur strength with just a few visits (which you can easily get with no GPU at all). Uberdude's setup seems to be performing quite well with a single graphics card that is less powerful than an 1080Ti.

[Uberdude]

I have read in a Chinese go forum that with the new Leela zero engine (0.16), it would take a 2x 1080ti setup to be able to enjoy the real speed of the new algorithms.

I am not sure why they say that. Leela Zero is already strong amateur strength with just a few visits (which you can easily get with no GPU at all). Uberdude's setup seems to be performing quite well with a single graphics card that is less powerful than an 1080Ti.

I think what they mean is this:
- LZ 0.16 has some optimisations that make LeelaZero faster
- Some of these optimisations are only available on modern top-notch graphics cards (e.g. reduced precision floating point arithmetic: basically by default GPUs used to do 32-bit floating point arithmetic, but if your neural network only needs 16-bit or even 8-bit then if you can pack 2 16-bit operations into where it used to do 1 32-bit one then you can go twice as fast)
- So if on some crappy old CPU 0.16 vs 0.15 is probably not much faster if at all
- On my 1060 GPU is quite a bit faster (how much exactly seems to vary)
- On 1080Ti or the next generation 2080s will be even more faster.

So "enjoy" means "derive the maximum benefit". I find less than the maximum benefit still enjoyable I mean even a 5-year old PC with LZ will in less than a second give you a good selection of candidate moves based on "shape intuition" comparable to that of a super-high dan player who didn't bother to do any reading or glance across the board for ladders.

[Gomoto]

So basically hydrogenpi has made 0 progress evaluating leela zero appropriatly in the last 4mo

[bernds]

I've tried to make an opening book for twogtp, consisting of a number of files with eight moves each, so that the programs would start from those positions. I've left it overnight, and it's not done, but so far the results are far more even, with #157 (playing as Black in every game so far, which you'd expect to be a disadvantage) winning slightly more often.

So, that experiment is complete now. Each program got to play the same initial position both with Black and White.

With #157 as Black: #157 17-14 #191
With #157 as White: #157 12-19 #191

In all, a narrow victory for #191, 33-29. Not enough to demonstrate that #191 has made significant progress. The earlier results suggest it knows something in the very early opening that #157 doesn't, but for analyzing arbitrary positions, I'd say there does not seem to be a big strength difference.

Possible sources of error - the machine wasn't completely unloaded, but the whole thing ran for more than 24 hours, so I'd expect errors from that to average out. The command line was shown above so if there was a setup error people should be able to spot it. I'd be happy if someone could try to reproduce the results.

Here is an example of an opening position that #157 managed to win with both colors.

game1.sgf

(260.04 KiB) Downloaded 840 times

game2.sgf

(306.19 KiB) Downloaded 791 times

[Charlie]

I've tried to make an opening book for twogtp,...

With #157 as Black: #157 17-14 #191
With #157 as White: #157 12-19 #191

Perhaps your opening book is biased to favour black.

If you ran the test for more iterations (so that each network gets several opportunities to play each opening as both black and white) we might have enough data to perform statistical tests to theorise on whether the colour or the network is more significant.

Another option would be to "harvest" an unbiased opening book -- perhaps from Leela Zero match games, available from the web site. You could select openings at move 8 that are most common (given symmetry) and most even for each colour, based on match game outcomes.