Life In 19x19

nbc44, were the games for your graph played in series of parallel? One effect that's been observed in the official LZ test matches is when the games are played in parallel and start around the same time (as they are with distributing the match tasks to LZ volunteers) then there can be a bias for the early games to be losses by the player who makes early blunders. So for example in Elf vs LZ matches when the games go to counting Elf tends to win, but Elf will blunder ladders and resign early so these data points have a bias to arrive earlier and makes LZ's winrate after 50 games be higher than it is after the full 400.

20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6 (8 wins as B, 6 wins as W)

Average game length : ~209 moves, average time used per side and per game : ~205"

Maybe I'll try a 20 game match with 2x1080TI, and 7 min games, i.e. 4-5 times more visits for both.

Vargo wrote:20 game match with GPU 1x1080 , 5 min per side and per game, no pondering.

#184 v. ELF v1
ELF wins 14 : 6 (8 wins as B, 6 wins as W)

So that's about 800 playouts for #184 and 1600 for Elfv1?

I looked at a few more of the official test match Elf v1 vs LZ #183 match at 1600 playouts and thought I'd characterise a few.
- Game #1: LZ 183 won by resign in 93 moves, Elf blundered itself in shortage of liberties and then walked into net harakiri. At move 61 it expects white to block instead of capture with the throw in as LZ and any 10k could find. Strangely it then planned to r18 block and then expected white to k19, so it could see this tesuji, but dreamed that it could get j15 kikashi first? - #402 (on previous post): LZ 183 won by resign in 92 moves (given the many 92/93 moves games I suppose that's the earliest they are allowed to resign), Elf got laddered. Elf thought it was doing slightly better (55%) at n15 extend. LZ's n13 is a move which surprised and impressed me (thought shares features with a move Blackie showed me at KPMC) and Elf similarly thinks it's good (it starts off not considering it much but by 1200 playouts thinks it's better than normal l17 defence), does this mean push and cut wasn't best? Some other games have same opening. But the big swing happens (q12 and q10 attachments surprised me but not Elf) with LZ starting the ko with s12 (Elf win collapse to 13%) instead of r13 (Elf win 45%). I suppose r13 is more often the better shape way to start the ko, but here it gives white decent local threats and a playable position. As for the ladder, at move 72 it thinks black will n8 atari allowing white to trade with p7 atari then o7 connect and ko (k15 is ignored threat), despite p12 atari for squeezy ladder having a few playouts and the ladder all the way to the edge of the board being the principal variation. - #25 LZ 183 won in 112 moves. Not obvious Elf blunder, LZ won an early fight and semeai; I was impressed Elf even managed to make a semeai it lost by 1 liberty instead of just simply dying. Same opening for a while as #402. - #59 LZ won in 93 moves. Elf got caught in an early ladder, game devolved to nonsense because not allowed to resign.Before white played atari for ladder Elf initially (1 playout) though it was ~65% but this dropped to about 60% by 1600. Before k13 on my weak PC Elf wanted to g16 peep, but k13 soon overtakes in winrate but not playouts. So this is a problem caused by Elf assuming ladders not working, wasting some of the limited playouts on some sente move prior to the failed ladder escape, and then going "hey wait a minute, if I run out this stone it's great for me" because Elf didn't have enough playouts remaining to read it to the edge of the board and capture. With just over 2k playouts it realizes it's a ladder and doesn't want to escape, so LZ is probably benefiting from being in a sweet spot in which it has enough playouts to read ladders whilst Elf doesn't. - #5 Elf won in relatively short 129 moves. Nice fight with a ko, Elf thought it was losing (40%) when ko started but it found a cool 1st line threat and LZ's g5 threat was no good. - #2 LZ won in 108 moves, but not Elf blunder: LZ push and clamp of 80 was a sharp tesuji. Elf resisted, led to a 30-move fight which LZ won by a liberty. I'd expect high dans to be able to read that as not too much branching, I could maybe on a good day, but probably wouldn't consider the clamp. Reviewing with Elf at 1600 playouts it thought not much deviation around black (Elf) 50-47% until e13 turn (did seem a bit thank you move to push behind on 4th line!) and at 2k playouts it decides f16 nice table shape is quite a bit better instead of going down to 42%. h15 cut I didn't understand (h17/k17 aim?) and Elf despite playing it Elf thinks it's bad (-10%) and just after 1600 realises e12 directly is better. But then it's back to almost even again when LZ doesn't j13 cut but k18. But with the LZ's push and clamp it did look at that a bit but soon after decides tenuki to s14 is better. As for the fight, Elf's prefers to push at h3 for g2 and expects white to g4 not g3: g3 leaving many cuts but taking a liberty and putting black in deep trouble. Once g3 blindspot played winrate drops. - #3 Elf won in 198 moves, nice game from both, impressive tesujis from Elf on right lower corner, and then seal the win with kill on left. - #76. LZ won in 92 moves, same opening 51 moves as #5, then same ko, "cool" 1st line threat was a fake theat and LZ didn't believe it this time! I tried analysing with LZ 183, and indeed on this game it quickly (<100 playouts) sees the threat as fake and wants to connect (but thinks winning even if answer threat), whereas with game 5 it took longer (about 400 playouts, so still within the 1600, I guess just bad luck in the real game).

Uberdude wrote:So that's about 800 playouts for #184 and 1600 for Elfv1?

Something like 800 visits for #184 and 2000 visits for Elfv1

Uberdude wrote:nbc44, were the games for your graph played in series of parallel? One effect that's been observed in the official LZ test matches is when the games are played in parallel and start around the same time (as they are with distributing the match tasks to LZ volunteers) then there can be a bias for the early games to be losses by the player who makes early blunders. So for example in Elf vs LZ matches when the games go to counting Elf tends to win, but Elf will blunder ladders and resign early so these data points have a bias to arrive earlier and makes LZ's winrate after 50 games be higher than it is after the full 400.

It was "validation.exe" test. 50,100,200,300,400 games per match, what are you offering? And what about of mathematical statistics?

Uberdude wrote:I wonder how does #183 do versus Elf v1 at time-parity? I saw it's now stronger (57% win) at visit parity in the test match: http://zero.sjeng.org/match-games/5bcaa ... 3e27abce47.

...

I was looking at this latest match as well last night (my time). It was interesting to compare it to the previous match with the 178 net vs ELF on September 20th. (This is about to scroll off the bottom of the list of test matches - grab it while it is hot!).

As we know, ELF favors black at the beginning of the game, in contrast to almost all other strong bots. In the earlier (178) match Black indeed won 54% of the games. Black won 194 out of 362 games with both ELF and the 178 net winning 97 times while White won 168 times with ELF winning 83 times and 178 winning 85 times.

However, in the latest match with the 183 net, the situation reversed and White won 54% of the games. Now White won 216 of the 402 games played with the 183 net winning 124 times and ELF winning 92 times. Black won 186 games with 183 winning 103 times and ELF winning 83 times. So 183 had it 55% White versus 45% Black while ELF had 53% White versus 47% Black.

In the earlier match 17 of the games were counted rather than resigned. ELF won all the counted games. In the latest match, however, the 183 net won 4 of the 7 counted games. Indeed ELF played out one game that it lost by 16.5 points. I did not look at any of the games. I wonder if this game contains some kind of life and death error?

Interesting observation about the colour win% changing.

ez4u wrote: It was interesting to compare it to the previous match with the 178 net vs ELF on September 20th. (This is about to scroll off the bottom of the list of test matches - grab it while it is hot!).

Saving this link which should keep working: http://zero.sjeng.org/match-games/5ba32 ... 3773940dd8

I wonder in the last #178 match how many of LZ's wins were Elf ladder/damezumari blunders. Obviously well done to LZ 183 for being able to keep roughly even with Elf long enough and to get into a position where Elf can mess up ladders, but these wins still feel a bit cheap to me (particularly as Elf would likely not make many of them if it was given same time instead of visits as LZ). So I wonder if we have gone from with 178 LZ getting say 10% wins from Elf blunders, 40% sensible wins, 50% sensible losses, to with 183 getting 10% wins from Elf blunders, 45% sensible wins, 45% sensible losses; or does 183 get 15% Elf blunder wins, 40$ sensible wins, 45% sensible losses.

Update:
As a quick check, I counted the number of 92 and 93 move games (all were LZ wins), assuming they are Elf ladder/liberty blunders*. In the 178 match there were 13 92 move (LZ B+res) 10 93 move (LZ W+Res). In the 183 match there were 19 92 move (LZ B+res) and 18 93 moves (LZ W+Res). So that's 5.7% cheap early wins in the 178 match, 9.2% for 183 match. So more than half of the match winrate improvement from 50.3 to 56.5 came from making Elf early blunder more.

* Characterising as blunders does sometimes diminish LZ's win, e.g. 178 game 11 below Elf got damezumarid, but that's because LZ played a nice loose net so to avoid that Elf would've needed to read that the net worked and played differently strategically beforehand to avoid giving LZ that possibility (actually playing around with Elf I think white can avoid capture, one line with n9 at l10 is seki). In fact when I analysed that game for 1600 playouts at move 46 Elfv1 wanted to first come out at n9 and then switched to p17 and didn't even consider s10 which is an aggressive move putting attack before defence and what got it into a net-able position. So that's a bit weird, but when played it does like s10 so I guess Elf just got randomly "lucky" that it gave a few playouts to s10 and discovered it liked it that in that instance it rose to the top of #playouts. On the other hand running out a stone captured in a ladder where tenuki would give a playable position is just Elf being super dumb and Elf losing rather than LZ winning.

14 game match between #184 and Elfv1
LZ015, GPU:2x1080Ti, 7 min per side per game, no pondering, komi 7.5 (Elfv1 is always W, sorry)

Elfv1 wins 12-2 ... Even with Elf always taking W, #184 seems far from Elf, at time parity.

(a 7 min game with 2x1080Ti is equivalent to ~8000 visits for Elfv1 and ~ 3200 visits for #184)

Uberdude wrote:So that's about 800 playouts for #184 and 1600 for Elfv1?

No chance: P.S. There are too many short games (~1/4).

I don't know what you mean.

Here is a very short test, anybody with a 1080 can do it :

I launched two 3 game matches #184 v Elf v1 (twogtp)

One match with --visits=801 for #184 , and --visits=2001 for Elf
and another match with -time 5 (minutes per side per game)

As you can see, the numbers are very similar : Global time per move is around 1.8 or 1.9" in both cases.
Two times 3 games is very few, but I don't want to run a 1000 game match, just to find the same numbers within 10%

Hmmm. I gather that the training games are all self play. Is it possible to have Elf vs. LZ training games? If so, I expect that both could benefit from that.

Some Elf games are/were (I think they are being thinned/phased out recently) used in the LZ training. I'm not sure if these are real Elf engine+weights self-play games released in some huge dataset by Facebook, the LZ engine with converted Elf weights playing against itself on LZ volunteer machines, or LZ engine with Elf weights vs LZ engine with LZ weights playing on volunteer machines, but suspect it's the middle one. Can anyone confirm? When these Elf games were added to the training pipeline LZ improved very fast initially as it absorbed the Elvish wisdom. You can see latest LZ has a more Elvish style; not so obsessed with 3-3 invasions of 4-4 but approaches them too, the 4 corners then 4 approaches fuseki liked by Elf is now also played by LZ (almost exclusively in the vs elf test matches I looked at, gets kinda boring). Also Elf is somewhat more human in not hating side extensions so much or playing hoshi pincers (kinda retro Japan style, you won't see Koreans do that since the 90s!). Whether LZ would have evolved like this without the influence of the Elf games we can't say, but I think it's fairly safe to say it has speed up that process.

Bill Spight wrote:Hmmm. I gather that the training games are all self play. Is it possible to have Elf vs. LZ training games? If so, I expect that both could benefit from that.

The nets are not trained on the games themselves but on search results in individual positions. So training on Elf vs LZ games would not be much different to training on some Elf games and some other LZ games (except slightly more variety in the types of positions occurring, and in game results for the value net). What matters is which net was used in the search at each position (whose turn it was).

The Elf games in the training are LZ selfplay with Elf weights AFAIK (the other options are not readily available).

The Elf vs LZ idea sounds most interesting for value net training but also very risky, may introduce some biases: game results would not reliably reflect the position itself anymore but also the strength difference.

In another thread, there was a very heated discussion about LZ 0.14 being stronger or weaker than LZ 0.15

But now, there's LZ 0.15.9999 (not a gag, you can download it HERE)
Discussion about V0.15.999 and V0.15.9999 HERE

V0.15.9999 is a test for V 0.16, and... it seems really stronger than V0.15

20 game match between
LZ 0.15.9999 network #184 and LZ 0.15 network #184
(3 min per side and per game, no pondering, GPU : 1x1080)
LZ 0.15.9999 wins 14-6
Big win...

Edit : another 40 game match 0.15.9999 v 0.15 (#184, same conditions as before)
LZ 0.15.9999 wins 22-18

So, after 60 games : LZ 0.15.9999 wins 36-24
Not bad


If someone wants the games...