Is leela zero really that strong?

[betgo]

I heard it was beating professional players. However, I just downloaded it, and I have good chances against it taking a 4 stone handicap. I play the regular leela at that same handicap. It doesn't use or select joseki. It seems to play terrible against handicap stones. It seems to usually enter at the 3x3 point, which is usually terrible, simplifying things and allowing me to create a huge moyo.

[jlt]

Leelazero doesn't play well against handicap.

Also, its strength depends a lot on the number of visits. How many visits per move did you use?

[betgo]

Leelazero doesn't play well against handicap.

Also, its strength depends a lot on the number of visits. How many visits per move did you use?

What's a visit, and how do you configure that?

[AloneAgainstAll]

To imagine how strong currently LeelaZero is, take this into consideration:

There is a bot called BensonDarr (aka PhoenixGo), plays on FoxGoServer.In last 300 games against Fox 9d (including many pros) he scored 299 wins.In last 20 games against LeelaZero he scored 1.

I suspect this particular LZ is run on stronger hardware than BD, but anyway, LZ can be damn strong.Handi games are different thing, you need to tweak Leela to play good on handi, but 3-3 invasions are not that rubbish as you think, even in middle handi games.

[betgo]

--gtp --playouts 2500 --noponder --threads 4 -w C:\Users\Joe\Documents\leelaz\logs\ww.txt
time_settings 0 5 1

Above is how I have it configured. Is there a way to configure it to play stronger? I am using a 2 year old standard high end laptop.

You are obviously a stronger player than me. However, I know 3-3 invasions are playable in handicap games, but starting out by invading one corner then another in a 4-stone game is obviously not optimal.

[AloneAgainstAll]

There is specially tweaked LeelaZero bot on KGS named Geomancer (or geomancer2, not sure) - he is able to cope with high handi (including 9 stones handi if i am correct) - you can try it, i guess it will be much harder task to beat it that Leela on your laptop.Might be hard to get game with it, however i think it is at least worth to try.

[EdLee]

...PhoenixGo... plays on FoxGoServer. In last 300 games against Fox 9d (including many pros) it scored 299 wins. In last 20 games against LZ it scored 1.

FineArt and Golaxy are approx. two stones above top humans. dunno about LZ.
In a recent exhibition, a bot gave four women pros 2 stones and scored 4-0.
Male top pros have yet to play the top bots at 2-stones in public,
for various reasons (*cough*).
And yes, high handicaps (above 2 stones) are another story, as the bots haven't been sufficiently trained for those.

[betgo]

Even though it didn't know how to play a 4-stone handicap game, I didn't think leelaz on my computer played anything like the level of a top pro. I was surprised. Is there a way to configure it to play stronger? I was hoping to learn a lot playing 9-stone games against a top pro level program, but that doesn't seem possible.

[jlt]

--gtp --playouts 2500 --noponder --threads 4 -w C:\Users\Joe\Documents\leelaz\logs\ww.txt
time_settings 0 5 1

If I understand correctly, LeelaZero thinks for 5 seconds before playing a move. If you replace "time_settings 0 5 1" with "time_settings 0 15 1" it will take 15 seconds so will be stronger. You may also increase the number of playouts but this won't have any effect if your computer is too slow to reach 2500 playouts within 15 seconds. Whaterver, don't expect superhuman level while playing with more than 2 handicap stones.

[Uberdude]

Can you just remove the time settings parameters so it does 2500 playouts. That might then take 342 seconds a move and you will have learnt you have a slow computer and why it was weak before.

Also what network file are you using?

[golem7]

I can assure you it's very strong. Try playing even games or up to 2 handicap stones and see if you feel ahead at any point in the game.

The problem with higher handicap is: since leela expects the opponent to play good moves, it thinks all its own moves are losing moves (since the opponent is so far ahead with the handicap). So it doesn't differentiate between "good" and "bad" moves anymore and starts playing this weird stuff. It doesn't expect us to make that many mistakes
I think there is some work going on trying to correct this but I'm not up-to-date with the progress.

Other ways to increase strength (might have already been said):
1) use a dedicated graphics card !!!
2) give more time/playouts (much easier with (1))

But yeah, reduce the handicap (2 max) and there should be no more crazy stuff (except the occasional ladder hiccup).

[splee99]

Make sure your GPU can indeed do 2500 play out in 5 seconds. Usually you don't use time settings when payouts is specified, or just use time settings without payouts. You can also take out the --noponder setting.

[Charlie]

I watched In-Seong, Seong-Jin and Young-Sam team up against Leela Zero towards the end of last year. Leela Zero was being run on a gaming laptop but nothing particularly special as far as hardware is concerned. Nevertheless, the trio lost.

In the opening stages, the audience were laughing at Leela Zero for playing a move that looked like a ladder-reading mistake. In the aftermath, that very move turned out to be a good one -- just one that no human would have judged to be so.

How can we even question whether Leela Zero is "that strong" when there is concrete evidence to the contrary?

I heard it was beating professional players.

Leela Zero has defeated professional players. Even very old versions & networks.

I have good chances against it taking a 4 stone handicap. ... It seems to play terrible against handicap stones.

None of these AlphaGo-Zero-like bots play well with handicap stones on the board. They were trained to play even games with a fixed rule-set and komi value -- any other scenario is not what they were trained for and they will perform sub-optimally.

It doesn't use or select joseki.

This is patently not true. Leela Zero *does* indeed play some human-invented Joseki in even games, at least. There are also new Joseki that these new algorithms have popularized and that are now played by professionals.

It seems to usually enter at the 3x3 point, which is usually terrible ...

This generalisation is now considered obsolete theory. AlphaGo showed that invading at 3-3 is often a very good move, indeed. Most of the new bots and modern professionals play it, today.

... allowing me to create a huge moyo.

Again, obsolete. Modern thinking says that concrete territory is more valuable than large moyos and that sabaki is always possible. There are some exquisite examples of this in the AlphaGo Zero games. Whatever the case, these algorithms play as if they can live, anywhere, if necessary.

[lightvector]

I would be very interested to hear if you have a similar experience with KataGo, a bot that I've been working on.

You can actually play it right now on OGS, it's one of the many bots available there, and seems to be pretty popular as a handicap game opponent, judging by its game records: https://online-go.com/player/592684/

For running on a local machine, it's a more difficult to compile and use, requiring CUDA+CUDNN and a modern NVIDIA GPU (I am steadily working to remove that requirement), but if you have that, then it should be straightforward to compile on Linux from the GitHub source using the stated instructions, or on Windows 'alreadydone' has generously worked to produce some precompiled binaries.

Like Leela Zero, most of its training data is from even games, and possibly as a result it still likes doing lots of 3-3 invasions in handicap games, which is a very strong move in even games, but which, like you, I am slightly skeptical of for very high handicap. But unlike Leela Zero, it explicitly tries to maximize score in addition to just winning, so in a handicap game the search still has a reasonable idea of what to shoot for instead of just seeing all moves as almost 100% losing and indistinguishable, so it's able to often play well even in high-handicap: example 13x13 example 19x19. It is NOT as strong as Leela Zero due to having far less training, so at least in even games it is "merely" pro-ish level or slightly superhuman depending on hardware, rather than vastly superhuman. (Also a thing I'm steadily working on).

But there's a chance that it would provide the opponent that you're looking for, if you're interested in trying it, and as it's still under active development, feedback is of course welcome.

As a separate note, on OGS handicap stones in Chinese rules are free placement, so you can try putting some 3-4 or 3-3 stones if for learning purposes you want to get a greater variety of corner patterns than just 3-3 invasions, and presumably whatever GUI program you're using to play against bots locally should have a way to do this too?

Edit: Oh, and I just remembered there's also this issue from the LZ github where ihavnoid explores adding territory prediction and an explicit score maximization term to LZ as well, which might be of interest here too: https://github.com/leela-zero/leela-zero/issues/2331

[betgo]

Is there a way to configure it not to resign? When I play leelaz at 5 stones, it will resign when I think it still has good chances due to its skill advantage.

It also doesn't run if I take out the --noponder option.

As far as invading at the 3x3 point, it does it in situations where it clearly isn't good, like first move of a handicap game. This also simplifies the game and is not good handicap strategy.