Leela 0.11.0 Beta version slower than 0.10.0 ?

johnsmith · #1

I 've been having the exact same issue on my average laptop.

jeromie · #2

I haven't tried running it, but I wonder if it's this item from the change log:

Quote:

New, much stronger Monte Carlo evaluation by combining Policy Gradient Reinforcement Learning and Simulation Balancing.

That seems consistent with extra time being in the evaluation process.

jin2983 · #3

is twogtp.py available opensource? I can post results for comparison from a machine with a large gpu and linux.

Satorian · #4

Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.

Garf · #5

Satorian wrote:

Given the release notes, it sounds like the playouts are a lot heavier than before, with a lot more attached to them. Playout count may go down, but overall level and playing quality should increase.

From the same release notes:
"More accurate, larger value network for 19x19."

Larger network would also take longer to compute.

pookpooi · #6

This person said that Leela 11 beta is 2 stones stronger than Leela 10

https://www.reddit.com/r/baduk/comments ... nger_than/

By the way, Beta 5 is out, the changelog is

Windows version is now compiled with Clang/LLVM 5.0 instead of MSVC2017. This makes the Monte Carlo evaluations about 15% faster.

jin2983 · #7

I started on a 40 game series but 0.11 won the first 12 of them so I'll restart with 0.5 komi to get into 0-100% range.

Different from above, I used 7.5 komi as leela plays by chinese rules.

command:

Code:

~/pachi/tools/twogtp.py --black '0.10.0/leela_0100_linux_x64_opencl --gtp --quiet --noponder' --white '0.11.0b5/leela_0110_linux_x64_opencl --gtp --quiet --noponder' --komi 7.5 --games 40

0.10 openCL benchmarks:

Code:

Leela: benchmark
200000 games in  7.55 seconds -> 26490 g/s (3311 g/s per thread)
Avg Len: 451.69 Score: -1.961805

Leela: netbench
 2000 predictions in  1.45 seconds -> 1379 p/s
10000 evaluations in  3.44 seconds -> 2906 p/s

0.11b5 openCL benchmarks:

Code:

Leela: benchmark
200000 games in 10.76 seconds -> 18587 g/s (2323 g/s per thread)
Avg Len: 418.58 Score: -2.588590

Leela: netbench
 2000 predictions in  3.11 seconds -> 643 p/s
10000 evaluations in  3.40 seconds -> 2941 p/s

Looking at benchmark results predictions are slower(explained by larger value net?) while evaluations are roughly equal(same sized policy net?).

As for playing strength / game quality, having a GPU seems almost a must.

First 5 games attached below:

jin2983 · #8

so far 0.11b5 as white has won 7 out of 8 games on 0.5 komi

pookpooi · #9

beta 9 is out and changelog didn't mention any strength-related improvement
Judging by the number, if nothing wrong happen the completed version 11 will release soon.

Krama · **#10**

I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

johnsmith · **#11**

Krama wrote:

I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

I feel you. Too strong... even now (Leela 0.11 beta 10)...

toannguyenthanh · **#12**

I think that Leela is stronger is the most important, slower is not a problem.

Krama · **#13**

johnsmith wrote:

Krama wrote:

I am just a mid kyu level player but holy detritus is Leela strong.
Bot easily gives me 6 handicap and I struggle.
I can't imagine what AlphaGo level is.

I feel you. Too strong... even now (Leela 0.11 beta 10)...

You are a 6 dan on kgs? So that is like 4 dan EGF right?

How well can you fare against gpu version on unlimited simulations (let's say 30 min)

Uberdude · **#14**

Krama wrote:

You are a 6 dan on kgs? So that is like 4 dan EGF right?

As an aside, I don't think many 4d EGF are 6d KGS, I'm certainly not. There are 6d EGF who are 6d KGS like Lukas Kraemer, or Cornel Burzo was 6d for a long time and just got back to 7d. KGS dan ranks are pretty tough these days. Ilya and Pavol (7d/1p) are only 7d KGS now.

johnsmith · **#15**

Krama wrote:

You are a 6 dan on kgs? So that is like 4 dan EGF right?
How well can you fare against gpu version on unlimited simulations (let's say 30 min)

Well, I don't play on KGS, only on Tygem. And I'm 7d over there. I guess that's something like solid 5d EGF

Anyways, LEELA(B) on Tygem has 50:50 score against pros. That surely tells something about its strength. And it only plays against pros.
Personally, I like to play with RoyalLeela on OGS (thx to roy777 who has Ryzen 1700 CPU and 1080 Ti GPU), and I have 50:50 against Leela on 3 handicap stones; winrate on 4 handi is 100% though I struggle a little bit

In even games RoyalLeela just smashes. Like Hulk.
On my personal average laptop, in even games I don't stand a chance either, though I have 100% score against 3 handi. And so far I won 2 games in Sunjung Baduk.

Pio2001 · **#16**

One of our club members is 7d on Tygem, but 1 kyu FFG (EGF).

Uberdude · **#17**

Pio2001 wrote:

One of our club members is 7d on Tygem, but 1 kyu FFG (EGF).

France is notorious for its under-ranked players (particularly young improvers), but also FFG rank != EGF rank, they have their own parallel internal rating system as well as submitting to EGD. Is this player 1 kyu in the EGF system, and has he gone about 50/50 in a tournament, such as the EGC, not just full of under-ranked French? I could believe a solid EGF 2d good at blitz is Tygem 7d, but not a 1k.

pookpooi · **#18**

johnsmith wrote:

Anyways, LEELA(B) on Tygem has 50:50 score against pros. That surely tells something about its strength. And it only plays against pros.

Very impressive. I also notice that Leela has 50% winrate against Fox 9d. Can we unanimously declare that Leela is the strongest freeware go AI? (and strongest go AI available to public right now, just before Zenith Go 7 release in upcoming months)

johnsmith · **#19**

pookpooi wrote:

johnsmith wrote:

Anyways, LEELA(B) on Tygem has 50:50 score against pros. That surely tells something about its strength. And it only plays against pros.

Very impressive. I also notice that Leela has 50% winrate against Fox 9d. Can we unanimously declare that Leela is the strongest freeware go AI? (and strongest go AI available to public right now, just before Zenith Go 7 release in upcoming months)

What is Leela's name on Fox? I'd like to check out those games.
P.S. FineArt also started playing and is stronger than ever - only 1 loss and 35 wins. It has kinda new style, it prefers high short shimari it seems.

pookpooi · **#20**

Leela测试

And yes, FineArt, DeepZenGo, CGI strength are another league, they're all 10d fox (with Ke Jie join as the only 10d human).
FineArt is able to maintain more than 60% winrate even giving 2-4 stones handicap to 9d. And to think that AlphaGo is likely to be 3 stones stronger than them is insane.

Leela 0.11.0 Beta version slower than 0.10.0 ?

Who is online