Did Go Seigen cheat with LeelaZero?

[John Fairbairn]

Some recent posts motivated me to do at last something I've thought about doing for quite a while.

Mainly I was wondering how Go Seigen fared when compared to AI moves. But I was also still wondering about the alleged cheating case where a claim was made that someone used LZ by picking one of its three top moves. I wondered specifically how Go would fare when his move was compared to the top three.

For lack of time I did not let LZ run very long, but since I was also curious how feasible it was to run and use LZ in parallel with fairly quick time limits typical of servers, I treated that as a plus.

None of this is at all scientific, of course, but I was intrigued by a couple of unexpected things thrown up.

The game was Go Seigen as White against Rin Kaiho, chosen mainly because it had komi (but only 5.5. - GSG never used 6.5) but also because it was a significant game (Meijin; 1963-11-18). I looked only at the first 100 moves, i.e. 50 for each player.

Go had 37 of his 50 moves in LZ's top three; Rin had 34. There were no huge mistakes. Go won after 192 moves.

But that close parity masked some trends. Go was clearly more likely than Rin to choose better moves out of the three, because his winning ratio gradually climbed. Starting at 54%, he was at 64% by move 30 and by move 100 was at 84%.

The most fascinating thing for me was that quite often Go's move did not appear at all in LZ's first sweep of the board. But as time progressed his move did appear, and also typically move up to a high score. This phenomenon occurred just once for Rin. I tentatively infer from this that Go was seeing things at a deeper level. At any rate, his performance impressed me.

A further oddity was that on one occasion, Go's move did not appear at all in LZ's many candidates, even when I let it run a bit longer than usual, but once his actual move was played his winning ratio increased (I assume this means LZ approved). On another similar occasion Go's move was scored equal by LZ.

In addition, even when Go's move did not appear in the top three, he was almost always in the top four or five, and also, when his actual move was played, his winning ratio did not fall huge (max. about 4%). On a couple of occasions it instantly showed a big dip (10%) but as LZ searched more it re-adjusted that to 4% or less. Again a sign of Go seeing deeper?

I forgot to make a record but I think there were only two other occasions apart from the one above when Go's move did not appear at all. For Rin there were between 5 and 10.

The only thing this actually proved was that it is quite possible to cheat with LZ in a server game (not that I believed otherwise beforehand) even with fastish time limits, but there was quite a bit more that was suggestive. For example, if you want to play like Rin Kaiho, take LZ's early suggestions. If you want to play like GSG, let it run a bit longer.

Has anyone playing on the servers had a strong feeling yet that they are really playing Leela?

[Elom]

Thank you, this is evidence for what I've thought for a while; that Go Seigen is one of the most AI-like professionals to have played.

It seems to show that humans can understand the board the way AI do, as you could say Go Seigen was on the way there.

[quantumf]

I repeated the above exercise with Leela 11 (the software at the center of all the fuss in the Pandanet League). The results were very different: RK achieved a 40/50 (80%) match with the top 3 moves, while GSG achieved 33/50 (66%). Four of GSG's moves were not on Leela 11's radar (34/36/40/48), while two of RK's moves were not on the radar (29/37).

In terms of assessment, Leela 11 had RK ahead from the start, and at 60% as late as move 73. But by move 100, after that sequence in the corner, GSG was up to 70%.

I used Leela 11 on an i3 PC with a reasonably powerful graphics card. I tried to ensure that the top move had at least 50,000 playouts and that the top 3 moves had at least 10,000 playouts, but this wasn't always possible.

[Gomoto]

I think this is the game:

[Gomoto]

both players missed to press down 3-4 at and (10% and 15% loss)

(I learned to appreciate the true size of the press down by sensei Hayashi Kozo before the advent of AI)

[John Fairbairn]

I repeated the above exercise with Leela 11 (the software at the center of all the fuss in the Pandanet League). The results were very different: RK achieved a 40/50 (80%) match with the top 3 moves, while GSG achieved 33/50 (66%).

Thank you for doing this instructive alternative run. I certainly did not match your number of playouts, and from chess experience I'd always expect even bots to be different from each other, with some being significantly stronger.

But this raises questions about how strong Leela Zero is in the kind of setups most of us here seem to have. I have mixed views. On the one hand I could claim it is weak. I played LZ twice. The first time, I lost by 5 points. I expected to lose. I went in to the game with the sole purpose of defending my own groups and territories resolutely so that nothing died. I wanted to see how LZ coped. I was surprised at the result, but assumed LZ was acting in the old Monte Carlo way, just doing enough to win, not caring about the margin. It did occur to me also that I may have accidentally stumbled across a better way to play go and had become stronger in a trice. But yet another possibility is that my version of LZ was actually weak. My second game, which I won because I made a very long ladder LZ couldn't read (and which it stupidly played out to the bitter end), suggests this could be the case.

I have, however, tended to discount the latter possibility (I favour the MC-type explanation) because the results I get when looking at other games or positions discussed on L19 is that I seem to get similar sorts of results to other setups and machines, and (maybe above all) my LZ is coming up with all the sorts of moves that are typical of the AIs - early 3-3s, shoulder hits, disinclination to pincer, etc.

Also, and this was one reason I ran the Go Seigen test, my LZ seems to make similar sorts of moves to the best pros lots of the time. I was looking to see how GSG matched up to LZ (I thought rather well), but of course it works the other way round: we can see how well LZ matches up to GSG.

I infer therefore that even my short-run version of LZ, while clearly not the strongest around, may, in the absence of ladders, nevertheless be playing at pro level. Does this sort of reasoning make sense?

[Tryss]

Thank you for doing this instructive alternative run. I certainly did not match your number of playouts, and from chess experience I'd always expect even bots to be different from each other, with some being significantly stronger.

LZ (with current networks) is much (much) stronger than Leela 11 (and the playouts numbers are not comparable). With enough playouts, LZ is already stronger than humans.

What network did you use ? And how many playouts did you get when running at your time setting ?

[John Fairbairn]

What network did you use ? And how many playouts did you get when running at your time setting ?

I don't know how to find networks but I haven't updated after installation. Playouts here I kept in low thousands because I wanted to match the typical speeds of playing on servers. Even when I'm investigating more leisurely I use a max time of 15 minutes per move. But I find that not too much changes and so I assume that my LZ set to play fast is getting 80% or more of the moves found at higher settings. Maybe a rash assumption... In short, I'm at the Mickey Mouse end of the scale

[Gomoto]

LZ with low playouts has a very good intuition, but is not good at reading.

[quantumf]

LZ (with current networks) is much (much) stronger than Leela 11 (and the playouts numbers are not comparable). With enough playouts, LZ is already stronger than humans.

I'm sure this is true. I tend to stick to Leela 11 for two reasons - the integrated GUI, and the fact that the moves it suggest are way more understandable to me. In fact, playing through this particular game, I found GSG's moves quite bewildering and I was not surprised that Leela 11 often didn't consider them. John, or anyone else with Leela Zero, how well did Leela Zero match on moves 29, 34, 36, 37, 40 and 48 - the games moves that didn't even occur to Leela 11?

[Tryss]

how well did Leela Zero match on moves 29, 34, 36, 37, 40 and 48 - the games moves that didn't even occur to Leela 11?

The following is LZ #175 opinion after roughly 20k-25k playouts :

: Not on LZ radar, and she dislike this move ( -7% and still dropping)
: Not on LZ radar, and she dislike this move ( -4.2% but dislike it less with more playouts)
: Barely explored, and she think it's fine ( -0.2% so in the margin of error)
: Not on LZ radar, and she's not a fan ( -2.2% )
: LZ 4th choice (with 20k+ playouts), but she like it more after being played with +1.8%
: Barely explored, and she dislike this move (-6.5% )

[Kirby]

I tend to stick to Leela 11 for two reasons - the integrated GUI, ...

Lizzie seems to work well with Elf network weights. Does Leela 11 have some other GUI?

[dfan]

Lizzie seems to work well with Elf network weights. Does Leela 11 have some other GUI?

Yes, Leela 0.11 looks like the screenshot here: https://sjeng.org/leela.html