Leela 0.11.0 Beta version slower than 0.10.0 ?

[johnsmith]

I 've been having the exact same issue on my average laptop.

[jeromie]

I haven't tried running it, but I wonder if it's this item from the change log:

New, much stronger Monte Carlo evaluation by combining Policy Gradient Reinforcement Learning and Simulation Balancing.

That seems consistent with extra time being in the evaluation process.

[jin2983]

is twogtp.py available opensource? I can post results for comparison from a machine with a large gpu and linux.

[Satorian]

Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.

[Garf]

Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.

From the same release notes:
"More accurate, larger value network for 19x19."

Larger network would also take longer to compute.

[pookpooi]

This person said that Leela 11 beta is 2 stones stronger than Leela 10

https://www.reddit.com/r/baduk/comments ... nger_than/



By the way, Beta 5 is out, the changelog is

Windows version is now compiled with Clang/LLVM 5.0 instead of MSVC2017. This makes the Monte Carlo evaluations about 15% faster.

[jin2983]

I started on a 40 game series but 0.11 won the first 12 of them so I'll restart with 0.5 komi to get into 0-100% range.

Different from above, I used 7.5 komi as leela plays by chinese rules.

command:

Code: Select all

~/pachi/tools/twogtp.py --black '0.10.0/leela_0100_linux_x64_opencl --gtp --quiet --noponder' --white '0.11.0b5/leela_0110_linux_x64_opencl --gtp --quiet --noponder' --komi 7.5 --games 40

0.10 openCL benchmarks:

Code: Select all

Leela: benchmark
200000 games in  7.55 seconds -> 26490 g/s (3311 g/s per thread)
Avg Len: 451.69 Score: -1.961805

Leela: netbench
 2000 predictions in  1.45 seconds -> 1379 p/s
10000 evaluations in  3.44 seconds -> 2906 p/s

0.11b5 openCL benchmarks:

Code: Select all

Leela: benchmark
200000 games in 10.76 seconds -> 18587 g/s (2323 g/s per thread)
Avg Len: 418.58 Score: -2.588590

Leela: netbench
 2000 predictions in  3.11 seconds -> 643 p/s
10000 evaluations in  3.40 seconds -> 2941 p/s

Looking at benchmark results predictions are slower(explained by larger value net?) while evaluations are roughly equal(same sized policy net?).

As for playing strength / game quality, having a GPU seems almost a must.

First 5 games attached below:

[jin2983]

so far 0.11b5 as white has won 7 out of 8 games on 0.5 komi

[pookpooi]

beta 9 is out and changelog didn't mention any strength-related improvement
Judging by the number, if nothing wrong happen the completed version 11 will release soon.

[Krama]

I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

[johnsmith]

I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

I feel you. Too strong... even now (Leela 0.11 beta 10)...

[toannguyenthanh]

I think that Leela is stronger is the most important, slower is not a problem.

[Krama]

I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

I feel you. Too strong... even now (Leela 0.11 beta 10)...

You are a 6 dan on kgs? So that is like 4 dan EGF right?

How well can you fare against gpu version on unlimited simulations (let's say 30 min)

[Uberdude]

You are a 6 dan on kgs? So that is like 4 dan EGF right?

As an aside, I don't think many 4d EGF are 6d KGS, I'm certainly not. There are 6d EGF who are 6d KGS like Lukas Kraemer, or Cornel Burzo was 6d for a long time and just got back to 7d. KGS dan ranks are pretty tough these days. Ilya and Pavol (7d/1p) are only 7d KGS now.

[johnsmith]

You are a 6 dan on kgs? So that is like 4 dan EGF right?
How well can you fare against gpu version on unlimited simulations (let's say 30 min)

Well, I don't play on KGS, only on Tygem. And I'm 7d over there. I guess that's something like solid 5d EGF

Anyways, LEELA(B) on Tygem has 50:50 score against pros. That surely tells something about its strength. And it only plays against pros.
Personally, I like to play with RoyalLeela on OGS (thx to roy777 who has Ryzen 1700 CPU and 1080 Ti GPU), and I have 50:50 against Leela on 3 handicap stones; winrate on 4 handi is 100% though I struggle a little bit In even games RoyalLeela just smashes. Like Hulk.
On my personal average laptop, in even games I don't stand a chance either, though I have 100% score against 3 handi. And so far I won 2 games in Sunjung Baduk.