Life In 19x19 http://www.lifein19x19.com/ |
|
Engine Tournament http://www.lifein19x19.com/viewtopic.php?f=18&t=13322 |
Page 10 of 20 |
Author: | q30 [ Sat Mar 31, 2018 1:34 am ] |
Post subject: | Re: Engine Tournament |
lightvector wrote: Yep. I had exactly this issue in mind when I posted this topic: forum/viewtopic.php?f=45&t=15314 As should be obvious from the discussion, there might indeed be a way, or indeed several ways, but it's not a trivial issue. What is the lack of next trivial way: to train such engines with komi = 7.5, when playing black, and with komi = 6.5, when playing white? |
Author: | as0770 [ Sun Apr 01, 2018 10:16 pm ] |
Post subject: | Re: Engine Tournament |
New entries: V-Run, which is little more than a random player, in League F and OnePunchGo in League D. Leela vs. AQ Code: 1. AQ 2.1.1 5/8 2. Leela Zero 0.13 CPU 85c6f2ad 3/8 League A: Code: 1. Leela Zero 0.13 CPU 85c6f2ad 18/20 2. Leela 0.11.0 15/20 3. Oakfoam 0.2.1 NG-06 11/20 4. Rayon 4.6.0 11/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League B: Code: 1. DreamGo 0.5.0 15/20 2. DarkForrest MCTS 1.0 12/20 3. Pachi 11.99 12/20 4. DarkGo 1.0 10/20 5. Ray 9.0.1 9/20 6. Mogo 4.86 2/20 League C: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League D: Code: 1. GNU Go 3.8 29/32 2. Hara 0.9 22/32 3. Matilda 1.25 19/32 4. Indigo 2009 19/32 5. Dariush 3.1.5.7 17/32 6. Aya 6.34 14/32 7. OnePunchGo 1.0 12/32 8. Fudo Go 3.0 9/32 9. JrefBot 081016-2022 2/32 League E: Code: 1. JrefBot 081016-2022 16/20 2. Iomrascálaí 0.3.2 12/20 3. SimpleGo 0.4.3 11/20 4. Crazy Patterns 0008-13 7/20 5. Marcos Go 1.0 7/20 6. AmiGo 1.8 7/20 League F: Code: 1. AmiGo 1.8 23/24 2. Beancounter 0.1 19/24 3. Stop 0.9-005 14/24 4. GoTraxx 1.4.2 11/24 5. CopyBot 0.1 7/24 6. V-Run 1 7/24 7. Brown 1.0 3/24 Configuration: Links: Best, Alex |
Author: | q30 [ Sat Apr 14, 2018 9:31 am ] |
Post subject: | Re: Engine Tournament |
I had compiled last Pachi version from 28.02.18. It became a bit stronger details. So the rating of generally available Go engines (i.e. without GPU support) is next: League A 1. Leela League B 2. Hiratuka 3. Pachi 4. Ray 5. MoGo |
Author: | pnprog [ Tue Apr 17, 2018 5:22 am ] |
Post subject: | Re: Engine Tournament |
I noticed this bot that was mentioned on Leela Zero github page: https://github.com/johnhuang-cn/FancyBing You might give it a try |
Author: | as0770 [ Tue Apr 17, 2018 7:15 am ] |
Post subject: | Re: Engine Tournament |
pnprog wrote: I noticed this bot that was mentioned on Leela Zero github page: https://github.com/johnhuang-cn/FancyBing You might give it a try Thanks! I'll try it after finishing tests of Pachi and Zen. |
Author: | as0770 [ Thu Apr 19, 2018 9:30 am ] |
Post subject: | Re: Engine Tournament |
New entry: Zen 7 via gtp4zen. Update: Pachi 12 and Leela Zero with the best 128x10 Network. In the future I will test the best Network of each size. Also there are some changes in the leagues, because of new entries I reduced the size. League A: Code: 1. Zen 7 16/24 2. AQ 2.1.1 13/24 3. Leela 0.11 10/14 4. Leela Zero 0.13 128x10 CPU 9/24 League B: Code: 1. Leela Zero 0.13 128x10 CPU 18/20 2. Leela 0.11.0 15/20 3. Oakfoam 0.2.1 NG-06 11/20 4. Rayon 4.6.0 11/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League C: Code: 1. DreamGo 0.5.0 15/20 2. Pachi 12 14/20 3. DarkForrest MCTS 1.0 12/20 4. Ray 9.0.1 9/20 5. DarkGo 1.0 8/20 6. Mogo 4.86 2/20 League D: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League E: Code: 1. GNU Go 3.8 18/20 2. Hara 0.9 11/20 3. Dariush 3.1.5.7 9/20 4. Indigo 2009 9/20 5. Matilda 1.25 9/20 6. OnePunchGo 1.0 4/20 League F: Code: 1. OnePunchGo 1.0 17/20 2. Aya 6.34 16/20 3. JrefBot 081016-2022 12/20 4. Fudo Go 3.0 10/20 5. Iomrascálaí 0.3.2 4/20 6. SimpleGo 0.4.3 1/20 League G: Code: 1. SimpleGo 0.4.3 17/20 2. AmiGo 1.8 15/20 3. Crazy Patterns 0008-13 13/20 4. Marcos Go 1.0 11/20 5. Beancounter 0.1 4/20 6. Stop 0.9-005 0/20 League H: Code: 1. Stop 0.9-005 14/16 2. GoTraxx 1.4.2 10/16 3. V-Run 1 7/16 4. CopyBot 0.1 6/16 5. Brown 1.0 3/16 Configuration: Links: Best, Alex |
Author: | as0770 [ Sat Apr 21, 2018 9:47 am ] |
Post subject: | Re: Engine Tournament |
And finally Zen 7 vs. Leela Zero GPU. Final: Code: 1. Leela Zero 0.13 128x10 GPU 5/8 2. Zen 7 3/8 League A: Code: 1. Zen 7 16/24 2. AQ 2.1.1 13/24 3. Leela 0.11 10/24 4. Leela Zero 0.13 128x10 CPU 9/24 League B: Code: 1. Leela Zero 0.13 128x10 CPU 18/20 2. Leela 0.11.0 15/20 3. Oakfoam 0.2.1 NG-06 11/20 4. Rayon 4.6.0 11/20 5. Hiratuka 10.37B (CPU) 4/20 6. DreamGo 0.5.0 1/20 League C: Code: 1. DreamGo 0.5.0 15/20 2. Pachi 12 14/20 3. DarkForrest MCTS 1.0 12/20 4. Ray 9.0.1 9/20 5. DarkGo 1.0 8/20 6. Mogo 4.86 2/20 League D: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League E: Code: 1. GNU Go 3.8 18/20 2. Hara 0.9 11/20 3. Dariush 3.1.5.7 9/20 4. Indigo 2009 9/20 5. Matilda 1.25 9/20 6. OnePunchGo 1.0 4/20 League F: Code: 1. OnePunchGo 1.0 17/20 2. Aya 6.34 16/20 3. JrefBot 081016-2022 12/20 4. Fudo Go 3.0 10/20 5. Iomrascálaí 0.3.2 4/20 6. SimpleGo 0.4.3 1/20 League G: Code: 1. SimpleGo 0.4.3 17/20 2. AmiGo 1.8 15/20 3. Crazy Patterns 0008-13 13/20 4. Marcos Go 1.0 11/20 5. Beancounter 0.1 4/20 6. Stop 0.9-005 0/20 League H: Code: 1. Stop 0.9-005 14/16 2. GoTraxx 1.4.2 10/16 3. V-Run 1 7/16 4. CopyBot 0.1 6/16 5. Brown 1.0 3/16 Configuration: Links: Best, Alex |
Author: | q30 [ Sat May 05, 2018 2:48 am ] |
Post subject: | Re: Engine Tournament |
I had compiled Ray (new version) and fought it with other engines details. The rating of Go engines with GTP and without GPU support is next: League A 1. Leela League B 2. Hiratuka 3&4.Pachi & Ray 5. MoGo |
Author: | as0770 [ Sun May 06, 2018 11:15 pm ] |
Post subject: | Re: Engine Tournament |
Updates: -Leela Zero 128x6 CPU in League B, Leela Zero 64x5 CPU in League C and Leela Zero 32x4 CPU in League H. -Leela Zero with ELF OpenGo Network in League A as CPU version with around 5 playouts/sec and with full strength and around 30 playouts/sec in the "Champions League". The performance of ELF OpenGo GPU is not quite as expected. It lost 2 games by 0.5 vs. Zen and one by a stupid ladder. Maybe I'll repeat the tournament with a komi of 7.5 but honestly I think a bot should be able to play also with a komi of 6.5. Zen can handle this and thus gets 2 points extra. Leela Zero won one game without komi or ladder problems, ELF resigned. The CPU version lost one because of a ladder but none because of the komi. Champions League: Code: 1. ELF OpenGo v0 12/16 2. Leela Zero 0.13 128x10 6/16 2. Zen 7 6/16 League A: Code: 1. Zen 7 11/16 2. ELF OpenGo v0 (CPU) 11/16 3. AQ 2.1.1 8/16 4. Leela 0.11 6/16 5. Leela Zero 0.13 128x10 (CPU) 4/16 League B: Code: 1. Leela Zero 0.13 128x10 (CPU) 22/24 2. Leela 0.11.0 19/24 3. Rayon 4.6.0 13/24 4. Oakfoam 0.2.1 NG-06 12/24 5. Leela Zero 128x6 (CPU) 12/24 5. Hiratuka 10.37B (CPU) 4/24 6. DreamGo 0.5.0 2/24 League C: Code: 1. DreamGo 0.5.0 18/24 2. DarkForrest MCTS 1.0 16/24 3. Pachi 12 16/24 4. Ray 9.0.1 11/24 5. DarkGo 1.0 10/24 6. Leela Zero 64x5 (CPU) 10/24 7. Mogo 4.86 3/24 League D: Code: 1. MoGo 4.86 18/20 2. deltaGo 1.0.0 14/20 3. Fuego 1.1 13/20 4. Michi C-2 1.4.2 8/20 5. Orego 7.08 5/20 6. GNU Go 3.8 2/20 League E: Code: 1. GNU Go 3.8 18/20 2. Hara 0.9 11/20 3. Dariush 3.1.5.7 9/20 4. Indigo 2009 9/20 5. Matilda 1.25 9/20 6. OnePunchGo 1.0 4/20 League F: Code: 1. OnePunchGo 1.0 17/20 2. Aya 6.34 16/20 3. JrefBot 081016-2022 12/20 4. Fudo Go 3.0 10/20 5. Iomrascálaí 0.3.2 4/20 6. SimpleGo 0.4.3 1/20 League G: Code: 1. SimpleGo 0.4.3 17/20 2. AmiGo 1.8 15/20 3. Crazy Patterns 0008-13 13/20 4. Marcos Go 1.0 11/20 5. Beancounter 0.1 4/20 6. Stop 0.9-005 0/20 League H: Code: 1. Stop 0.9-005 18/20 2. GoTraxx 1.4.2 14/20 3. V-Run 1 9/20 4. CopyBot 0.1 8/20 5. Leela Zero 32x4 (CPU) 7/20 5. Brown 1.0 4/20 Configuration: Links: Best, Alex |
Author: | luigi [ Mon May 07, 2018 6:14 am ] |
Post subject: | Re: Engine Tournament |
Why would you use 6.5 komi anyway? Aren't you using area scoring? |
Author: | Vargo [ Mon May 07, 2018 6:51 am ] |
Post subject: | Re: Engine Tournament |
I've tried various settings for LZ and the ELF weights. On my computer, it's very nearly unbeatable, with caveats : 1) It MUST play even games 2) Komi MUST be 7.5 3) I think there could be a kind of threshold for the playouts, a minimum of 20000 (?) to get this level. Even if LZ (latest network) is given quadruple thinking time per move, LZ_ELF wins every time...( time_settings 0 61 1 for LZ and time_settings 0 16 1 for LZ_ELF) I've run maybe 10-12 such games, and LZ hasn't won a single one. It's really uncanny. GPU 2x1080Ti, CPU i9-7920X PS. New best networks for LZ seem to pop up like crazy today (already 3 new ones !) is it related to the ELF weights ? LZ's strength graph is getting almost vertical |
Author: | pnprog [ Mon May 07, 2018 10:35 am ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: Updates: I think it would be more reasonable to wait for the ELF team to release a windows binary. ELF's network might be stronger, but there is also the fact that the original software run faster than Leela Zero from what I read (more play-outs per second than LeelaZero).
-Leela Zero 128x6 CPU in League B, Leela Zero 64x5 CPU in League C and Leela Zero 32x4 CPU in League H. -Leela Zero with ELF OpenGo Network in League A as CPU version with around 5 playouts/sec and with full strength and around 30 playouts/sec in the "Champions League". The performance of ELF OpenGo GPU is not quite as expected. It lost 2 games by 0.5 vs. Zen and one by a stupid ladder. Maybe I'll repeat the tournament with a komi of 7.5 but honestly I think a bot should be able to play also with a komi of 6.5. Zen can handle this and thus gets 2 points extra. Leela Zero won one game without komi or ladder problems, ELF resigned. The CPU version lost one because of a ladder but none because of the komi. |
Author: | jlt [ Mon May 07, 2018 11:27 am ] |
Post subject: | Re: Engine Tournament |
Vargo wrote: PS. New best networks for LZ seem to pop up like crazy today (already 3 new ones !) is it related to the ELF weights ? LZ's strength graph is getting almost vertical Yes, probably. See here and there. |
Author: | q30 [ Sat May 12, 2018 4:08 am ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: Is this right, or must be that one: https://github.com/pytorch/ELF |
Author: | q30 [ Sat May 12, 2018 5:08 am ] |
Post subject: | Re: Engine Tournament |
I had compiled last versions of Pachi_DCNN (required for its start additional soft is included: see link below). It's a bit stronger, than Pachi (details): League A 1. Leela League B 2.Hiratuka 3.Pachi_DCNN 4&5.Pachi & Ray 6.MoGo |
Author: | abcd_z [ Thu May 17, 2018 6:54 pm ] |
Post subject: | Re: Engine Tournament |
Quote: 2. Leela Zero 0.13 128x10 6/16 Why are you using an old model and an old version of Leela Zero? The weights have been 192x15 now for over a month, and the latest version of Leela is 0.15.
|
Author: | as0770 [ Sat May 19, 2018 1:33 pm ] |
Post subject: | Re: Engine Tournament |
abcd_z wrote: Quote: 2. Leela Zero 0.13 128x10 6/16 Why are you using an old model and an old version of Leela Zero? The weights have been 192x15 now for over a month, and the latest version of Leela is 0.15.I planed to use the best network of each size. Furthermore the Leela Zero team is now using ELF games to train the network, so I deliberate whether it makes sense to test "best" networks that are weaker than the ELF network. |
Author: | abcd_z [ Sun May 20, 2018 9:37 pm ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: I planed to use the best network of each size. Okay, but... why? Doing it that way misrepresents Leela Zero's actual, current strength.as0770 wrote: Furthermore the Leela Zero team is now using ELF games to train the network, so I deliberate whether it makes sense to test "best" networks that are weaker than the ELF network. Again, doing otherwise misrepresents Leela Zero's strength. Doing it the way you're currently doing makes it seem like LZ's current strength is weaker than it actually is.ELF has a certain level of strength, and Leela Zero has a different level of strength, and that gap will close over time until LZ reaches ELF's level of strength (and presumably exceeds it). Just because another network is stronger than LZ doesn't mean you should ignore the updates that close the gap. After all, isn't the whole point of this tournament to compare the strength of different go programs? as0770 wrote: Maybe I'll repeat the tournament with a komi of 7.5 but honestly I think a bot should be able to play also with a komi of 6.5. There are two problems with this. First, deep neural networks need to be trained from scratch in order to play at different komis. There's a paper that talks about an approach that would let go programs play at different komis, and I think Golaxy uses that approach, but AFAIK none of the programs you have use that. They would all need their architecture rewritten to accommodate that.Second, why are you using 6.5 komi? The easiest type of scoring for a go AI is area scoring, and under area scoring the result without komi is almost always even, so rulesets that use area scoring generally have a komi of 5.5 or 7.5. |
Author: | as0770 [ Sun May 20, 2018 10:16 pm ] |
Post subject: | Re: Engine Tournament |
abcd_z wrote: as0770 wrote: I planed to use the best network of each size. Okay, but... why? Doing it that way misrepresents Leela Zero's actual, current strength.as0770 wrote: Furthermore the Leela Zero team is now using ELF games to train the network, so I deliberate whether it makes sense to test "best" networks that are weaker than the ELF network. Again, doing otherwise misrepresents Leela Zero's strength. ELF has a certain level of strength, and Leela Zero has a different level of strength, and that gap will close over time until LZ reaches ELF's level of strength (and presumably exceeds it). Doing it the way you're currently doing makes it seem like LZ's current strength is weaker than it actually is.Just because another network is stronger than LZ doesn't mean you should ignore the updates that close the gap. After all, isn't the whole point of this tournament to compare the strength of different go programs? I won't test every network, so I have to do a cut somewhere. When ELF came up Leela won 7% against it. Ten Networks later it wins 10%. I think this says much more about the progress than a 16 game tournament. |
Author: | abcd_z [ Sun May 20, 2018 11:14 pm ] |
Post subject: | Re: Engine Tournament |
as0770 wrote: I won't test every network, so I have to do a cut somewhere. That's reasonable, but if you're going to do a tournament that tests the strength of Leela Zero it should include, at minimum, the most recent LZ network. Cut out other LZ networks if you have to.as0770 wrote: When ELF came up Leela won 7% against it. Ten Networks later it wins 10%. I think this says much more about the progress than a 16 game tournament. I have no idea what you mean by this.
|
Page 10 of 20 | All times are UTC - 8 hours [ DST ] |
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |