LeelaZero adventures on Fox

[Uberdude]

Recent musings about is LZ super-human on moderate hardware have prompted me to make an account on Fox to play as LeelaZero. It's called leelazero7 so that it's hopefully obviously I'm a bot and people can cancel the game (quite a few do) if they don't want to play a bot [I say "Hi, I am LeelaZero network #157 running on GTX 1060. If you don't want to play a bot feel free to quit." could anyone translate that to Chinese please?]. On Fox you have to start at 3d and can double rank promote if you win 18/20 at 3d or 20/20 at 5d+ and now I'm 10 wins at 7d. This thread is to record interesting games or things I notice.

My setup is a 3GB GeForce GTX 1060 GPU and running the best 15-block network #157 for now (I might switch to 40-block later if the extra strength is needed if/when I get to play top 9ds, or just as an experiment to see how good it is at lower playouts), running Lizzie and relaying the moves by hand (I've made 3 misclicks so far in 48 games, one was a taisha instead of knight press against 3-4 which was a serious mistake but soon recovered from, others were small endgame mistakes in positions winning by a lot). I provide some time management help/hindrance to LZ in that I choose when to play: I might take her top choice (by playouts, blue circle) after just a few seconds (could be ~1k playouts) if I think it's not likely to change or be important. In hard positions or where a 2nd/3rd choice with fewer playouts but higher winrate than the top playouts blue move is visible then I might let it analyse longer (still within game time settings, usually 30 sec byo-yomi) or play the higher winrate lower playouts move in Lizzie, let it analyse for as many playouts as the highest playouts one had and if the winrate remains higher then choose that move for the game (essentially accelerating LZ's switching to the better move, it would do it itself if left to its own devices, and such an approach could be implemented in LZ algorithmically and I expect it would increase its playing strength, a bodge for its too low IMO exploration behaviour). If it looks like LZ might blunder a ladder or other blindspot I won't stop it and use my go skill to correct its mistake, but will give it a bit more time. And if several choices are all very close I might pick the more interesting one. Average playouts per move are probably around 5k which is about 10 seconds, I will use opponent time to let LZ ponder if I think it's needed, else just live review the game and explore other choices for my own interest.

Summary so far: unsurprisingly all wins. Beats 3d/5d/7d with ease (usually >85% by move 50), though with the 7ds now there are some longer periods of play with a flatish instead of constantly declining winrate. Often beat 5ds by only 15-20 points if they didn't resign, it slacks off when leading (playing myself I usually beat Fox 5ds by more, I'm at 6d now and lose some). The 7ds seem to have more fighting spirit so it crushes them in early fighting. A 7d today was the first time LZ thought it was losing: it played an atari for a ladder that didn't work (even after 15k playouts), 7d extended, then LZ initially wanted to atari for the failed ladder but quickly realised it didn't work so compromise for a bad result (but not as bad as continuing failed ladder), see below. I've seen this mistake in this joseki in LZ's training games and it goes unpunished. A dozen moves later LZ was winning again though so no big deal (though because there was the non-working ladder atari on the board it wanted to play there initially for the next few moves, so I gave it more playouts (About 10k, 20s a move) so it would get over that delusion and play sensibly at top right). Another player also saw his winrate increase where LZ thought it could capture a cutting stone in a ladder but couldn't, but that was just making LZ go from winning a fair bit to winning a bit less.



P.S. does anyone have a list of known pro's usernames on Fox? I have this list from Tygem: https://docs.google.com/spreadsheets/d/ ... sp=sharing, maybe some are the same. I often see wonfun who I think is Weon Seongjin?

[pluspy1337]

Hey uber I just thought I'd comment about the moves in the corner at 20. I've noticed that some networks LZ157 and the early 40blocks, if memory serves, make that mistake consistently. I played some games using those LZ versions vs Elf and it happened quite a few times. The later 40 blocks seem to have corrected the issue, but it's funny that we see it here even in games vs humans. I had a theory that it was a hiccup caused by trying to integrate elf's self-training games but not having digested them completely yet, but who knows. Just saying tho, in case you want to upgrade to 187 *wink* *wink*

[Uberdude]

A few other interesting games.

With lower left pincer joseki AlphaGo and LeelaZero tend to prefer the 2nd line crawl over solid connect. LZ wanted to immediately start shenanigans in that area to exploit white's bad shape not answer lower right approach. Of course black could have done better in the fighting, but I thought this was a nice example of how to make solid connect bad.



Double 3-3 vs san-ren-sei, then attach on top. Move 52 was a sharp tesuji and an example of a promising alternative move (blue outline for low playouts but high winrate) which I played in Lizzie, quickly confirmed LZ liked it more than first feeling g14 block as it turns black's last move into a terrible shortage of liberties so could play quickly instead of needing to wait for 50k playouts or whatever and it to naturally rise to the most playouts.



This 7d did well to kill LZ's wall press group, but still a bad trade. And then LZ was merciless at lower left with using the thickness to make a submarine invasions into an attack and kill (White could have lived on left without ko but then lower side group almost dies, but this is do you want a 1% or 3% game decision). Also at move 39 I failed to spot a blue circle on f12 which it then liked more than the k12 hane I quickly played.

[Uberdude]

Hey uber I just thought I'd comment about the moves in the corner at 20. I've noticed that some networks LZ157 and the early 40blocks, if memory serves, make that mistake consistently. I played some games using those LZ versions vs Elf and it happened quite a few times. The later 40 blocks seem to have corrected the issue, but it's funny that we see it here even in games vs humans. I had a theory that it was a hiccup caused by trying to integrate elf's self-training games but not having digested them completely yet, but who knows. Just saying tho, in case you want to upgrade to 187 *wink* *wink*

For ladders, LZ and Elf seem to have quite different delusions: LZ tends to initially think all ladders work (for the capturing player, so sometimes sets up ladders which don't work), whereas Elf tends to think no ladders work, so plays moves which can get captured in a ladder. For this joseki LZ did't dream (at least with my ~20k playouts) of black escaping and just imagines black will play in the corner (like when ladder works for white) and then it will ponnuki capture. But actually I just checked Elf v0 and v1 and it started off wanting to play the ladder atari, but then realised it didn't work and switched to extend.

I'd actually quite like to stick with 157 and play a human strong enough to win after LZ makes a mistake like this Not sure such a human exists, let alone I'll be able to play them in a game they put their best effort into on Fox, but I can dream... Also as it's very early there's a lot of game left for LZ to catch up in, but if a human can get LZ to noob a ladder in some high-stakes middle-game fight I could see that as being how it loses.

[moha]

In hard positions or where a 2nd/3rd choice with fewer playouts but higher winrate than the top playouts blue move is visible then I might let it analyse longer (still within game time settings, usually 30 sec byo-yomi) or play the higher winrate lower playouts move in Lizzie, let it analyse for as many playouts as the highest playouts one had and if the winrate remains higher then choose that move for the game ... If it looks like LZ might blunder a ladder or other blindspot I won't stop it and use my go skill to correct its mistake, but will give it a bit more time.

Nothing wrong with this, but that setup may be stronger than LZ itself (fixing its weaknesses to some extent). I wonder how this would perform against vanilla LZ.

[Uberdude]

In hard positions or where a 2nd/3rd choice with fewer playouts but higher winrate than the top playouts blue move is visible then I might let it analyse longer (still within game time settings, usually 30 sec byo-yomi) or play the higher winrate lower playouts move in Lizzie, let it analyse for as many playouts as the highest playouts one had and if the winrate remains higher then choose that move for the game ... If it looks like LZ might blunder a ladder or other blindspot I won't stop it and use my go skill to correct its mistake, but will give it a bit more time.

Nothing wrong with this, but that setup may be stronger than LZ itself (fixing its weaknesses to some extent). I wonder how this would perform against vanilla LZ.

Yes, it could do, though sometimes when humans try to help an AI they can also make it worse. My idea is to act as an accelerator so I can grind through the easy wins faster: by doing this I can play a game in 20 minutes or so with an average of <5k playouts per move, and my "is this a good blue circle, and don't spend ages on the easy moves" behaviour should make LZ play as well as it would on more playouts, say 50k just pulling numbers out of my hat based on how (agonisingly) long it can take the promising blue circle moves to naturally become the #1 choice by playouts. Also it's possible my behaviour makes it worse if the winrate of the original most playouts move would increase beyond my promising blue outline circle move, or maybe if LZ really did 50k it would find an even better move that rises to the top. If someone else wants to pit vanilla LZ with similar hardware vs me as LeelaZero7 they are welcome .

[Tryss]

That could be fun ! I have a 970 gtx, so that should be roughly equal. How many playouts/s do you get with LZ 0.16 and the #157 network?

[Uberdude]

That could be fun ! I have a 970 gtx, so that should be roughly equal. How many playouts/s do you get with LZ 0.16 and the #157 network?

I'm still using LZ 0.15 because when I tried 0.16 with Lizzie there was some rendering problem with the most playouts move not getting the blue fill, and that's very useful. It would be nice to fix that when I go up to 40 blocks to get the speed boost (maybe the upcoming new Lizzie release will fix it, I guess some format change). On 0.15 I get ~323 playouts/sec with #157, and 123/s with 40-block #183.

[splee99]

I recommend to use #188 40b weight. On my PC with GTX1050 #157 doesn't have any chance using equal time per move.

[Uberdude]

I just did a little test on this ladder position at move 19 when white has to choose whether to cut (only good if has ladder) or hane in corner. 188 is indeed better:

188
35s, blue outline on corner hane
1:30 corner hane most playouts, 265 vs 236

157
starts off favouring corner hane but within 10s likes cut.
after 2 mins still no blue circle let alone fill on corner hane: cut is 57% with 2.4k and no expectation of black extend, vs corner 53% 602.

[Uberdude]

Well, that was weird: just had my first loss, and LZ thought it was winning the whole game but lost according to Fox by 17 points, and that was a no komi game as black against an 8d. It lost winrate from the lower side fighting, but then thought it recovered. Counting the game independently with CGoban and myself I get a black win by 17, a Fox bug? At the end of the game I clicked the Count button and then a dialog for automaric counting came back saying "W+17 'leelazero7'" so the colour of winner was wrong but name right. I, perhaps stupidly, clicked ok because another game I won on time displayed the wrong B/W colour so I assumed it was a similar UI glitch and wouldn't actually make the game result incorrect. Is it possible in Chinese client for a player to change the winner field?That's annoying to lose the chance to double promote to 9d.

loss winrate.PNG (149.71 KiB) Viewed 26505 times

[Gomoto]

Check your result in the games list to check for fox bug.

[Uberdude]

It shows as a loss there too, and in my win loss statistics and affects the display of how many games to win to rank up

[Gomoto]

Looks like a counting bug of the score estimator. When I load the game in fox the score estimator also shows W+17.

I once had a similar fox bug, but the result showed correct in the game list.

[EDIT: It was a game with special rules. Look further down in the thread for more info]

[Uberdude]

First real loss, LZ 157 made same ladder mistake in same joseki as before, and this 8d was good enough to keep the advantage. LZ did catch up a bit (according to its judgement, which I believe here) but then just after opp made a mistake I misclicked (crude [edit] q3 should have been r3 tesuji that LZ just managed to find) for -10% and I was impressed he maintained the lead against LZ's increasingly desperate attempts to catch up (I did little managing of LZ's moves and let it go crazy).