Life In 19x19 http://www.lifein19x19.com/ |
|
501 Opening Problems and a committee of AIs http://www.lifein19x19.com/viewtopic.php?f=15&t=16953 |
Page 1 of 2 |
Author: | xela [ Tue Oct 01, 2019 7:52 pm ] |
Post subject: | 501 Opening Problems and a committee of AIs |
I've started re-reading 501 Opening Problems and comparing the results with bot analysis. Now I know this unreasonable on a couple of levels:
So far I've been through the first 20 problems, and compared the book moves against the output of six different bots (using about a minute of thinking time for each bot). I'm thinking that if six different styles of training and analysis all come up with the same move, then there's a fair chance that it really is the best move. But if strong AIs disagree amongst themselves, there might be some interesting discussions ahead :-) The bots that I'm using at the moment are:
To me it makes sense to sort the problems into four categories:
I'm guessing the authors of 501OP were looking for positions with a clear-cut first move, so you'd hope to see most problems in class 1 or 2. My results for problems 1-20 were:
Below I'll post what I thought were the most interesting positions. (I'm certainly not going to go ahead and post the entire book, but I think five out of 501 is fair use.) |
Author: | Uberdude [ Tue Oct 01, 2019 8:14 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Interesting and methodological work xela, thanks, the committee of bots is something I was too lazy to apply rigourously. There was a thread with some similar bot analysis of book problems in the computer go section a while back with Rob's 'Making good shape' and then I looked at some Shuko problems with Elf. Could I suggest relabelling category 2 as 4, and shift 3 and 4 up to 2 and 3 so that they form a continuum of "book right" to "book wrong". |
Author: | xela [ Tue Oct 01, 2019 8:17 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Here are five positions where I was at least a little bit surprised by the results. All black to move (actually the first 106 problems in the book are all black to play, I'm not sure why the imbalance!) The book isn't specific about komi (I assumed 7.5 komi for the bot analysis; more likely 6.5 or 5.5 around the time the book was written). I'll wait a couple of days before revealing which problem is in which class, so you can form your own impressions first. |
Author: | xela [ Tue Oct 01, 2019 8:23 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Uberdude wrote: ...the committee of bots is something I was too lazy to apply rigourously... Indeed, creating SGFs for the positions and loading/reloading different files and software took more time than I'd planned for. If I wanted to do more of this, I'd be writing a bunch of scripts to automate the process. (Or contributing to the Kombilo source code to add engine integration there.) Then again, there's a fine line between laziness and procrastination. I could be using this time to solve tsumego problems and play games, or keep up with "real life"... |
Author: | xela [ Tue Oct 01, 2019 8:26 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Uberdude wrote: Could I suggest relabelling category 2 as 4, and shift 3 and 4 up to 2 and 3 so that they form a continuum of "book right" to "book wrong". I had in mind a different continuum: "clear-cut" to "confusing" :-) |
Author: | xela [ Tue Oct 01, 2019 8:28 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Uberdude wrote: There was a thread with some similar bot analysis of book problems in the computer go section a while back with Rob's 'Making good shape' and then I looked at some Shuko problems with Elf. Shape problems (and to a lesser extent, other sorts of tesuji) are tricky because the right answer (or urgency) can vary with context. I'd love to have a software tool that will drop a corner position into different full-board contexts to test whether the analysis does or doesn't change. |
Author: | gennan [ Tue Oct 01, 2019 11:20 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Uberdude wrote: Could I suggest relabelling category 2 as 4, and shift 3 and 4 up to 2 and 3 so that they form a continuum of "book right" to "book wrong". I agree that that order would make more sense. Category 2 seems to be the smallest, so I would say the book is doing pretty OK. Another interesting thing would be to use KataGo to evaluate how wrong the book is when it's wrong (how many points does it lose compared to the AI suggestions). |
Author: | Knotwilg [ Wed Oct 02, 2019 12:06 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
My unsollicited analysis: what I would play according to traditional theory and so I expect is the book answer (A) and what I think the/some bots will say (B) |
Author: | Uberdude [ Wed Oct 02, 2019 2:26 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
My analysis (WIP): |
Author: | Bill Spight [ Wed Oct 02, 2019 5:09 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Author: | Knotwilg [ Wed Oct 02, 2019 8:04 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
LZ on position 1 (3000 playouts) On position 3 On position 4 |
Author: | xela [ Thu Oct 03, 2019 3:48 am ] | ||
Post subject: | Re: 501 Opening Problems and a committee of AIs | ||
Position A: my results. Note: I'm not in any way saying that this is the "right" answer. I'm just an amateur player messing about with software that I only partly understand. Others may run different hardware/software/settings combinations and get different results. And I'm very happy for the stronger players on this board to critique my reasoning and interpretation.
|
Author: | xela [ Thu Oct 03, 2019 3:54 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Uberdude wrote: B: Class 1 but becomes class 3 with meta-reasoning. Pretty sure book wants to hane at head of 2 with a, and would have thought all bots would agree, but given xela selected this position as interesting I guess at least one of them doesn't want to play it, but probably most do. Or maybe he did want to include a unanimous agreement one as an example! Have you by any chance been watching (or reading) The Princess Bride? I'll get to this in a day or two, sorry to keep you in suspense! |
Author: | Knotwilg [ Thu Oct 03, 2019 4:10 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
LZ's variations on Position A, with my own comments (she speaks in sequences only) |
Author: | Bill Spight [ Thu Oct 03, 2019 9:29 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
xela wrote: It's interesting to see that some of the human-suggested moves are evaluated as pretty close to the bot moves, yet the bots don't explore them without prompting. Path dependency in learning, both in human and bots. The human learning is cultural, as it takes place over many lifetimes. One interesting thing is how close different alternatives are in winrate. We can't really say that one move is better than another. Ono of those moves, that is. Unlike some amateurs claim, the opening does matter. ![]() Edit: It seems to me that bots frequently make high extensions, particularly long ones. Probably for them those long extensions are not really territorial moves. |
Author: | Bill Spight [ Thu Oct 03, 2019 9:41 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
Knotwilg wrote: LZ's variations on Position A, with my own comments (she speaks in sequences only) Interesting that my "bot special" reduction got no first choice votes. Lizzie considered it but gave it only 129 playouts. With so few playouts I would not trust the winrate estimate. What happens when you force the play? (It will inherit the winrate estimate from Lizzie's top reply. That's what the Elf team did when a move got fewer than 500 playouts. ![]() |
Author: | xela [ Fri Oct 04, 2019 3:51 pm ] | ||
Post subject: | Re: 501 Opening Problems and a committee of AIs | ||
Position B: my results.
|
Author: | Bill Spight [ Fri Oct 04, 2019 6:31 pm ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
I also expected unanimity for ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Author: | xela [ Sat Oct 05, 2019 2:25 am ] | ||
Post subject: | Re: 501 Opening Problems and a committee of AIs | ||
Position C: my results.
|
Author: | Bill Spight [ Sat Oct 05, 2019 2:43 am ] |
Post subject: | Re: 501 Opening Problems and a committee of AIs |
For C, I am surprised that the book play is not the same as that of the bots. There is even a proverb about it. ![]() ![]() |
Page 1 of 2 | All times are UTC - 8 hours [ DST ] |
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |