Life In 19x19

Posted: **Mon Jun 04, 2018 2:57 pm**

I've been really enjoying using GRP in a slightly alternative way to look at what AI thinks about popular human moves.

It's no good looking at a move in just one circumstance as bots are really good at whole board thinking and game management so the choice of moves might be based on the specific circumstances. Automating GRP on the command line has helped me to review a large set of human pro games and pick out the moves I'm interested in to see what the bot thinks in each case.

A really usefull feature for this has been the ability to use GRP at the command line with a line like this:

Code: Select all

python leela_zero_analysis.py --range="21" mygame.sgf

The range parameter allows me to specify a range of moves for GRP to analyse, or in my case one specific move. Using this I can:

Use Smart go for windows to patern search for the move I'm interested in in the gogod database and export a list of all the relevant games and the move numbers where the patern search finds it's match
Make a little script to turn this list into a list of command line statements like the one above (I actually just used excell this time)
Put these statements in a .bat file and run. GRP runs through each game skipping straing to the interesting move.
Check out what the bot thinks without having to wait at each move! I also found r2sgf.py useful to convert to sgf so that I could quickly check through the games in SmartGo.

Not really the core intended use for GRP but I'm finding it really useful. Maybe other's can find interesting things with these great tools?

Posted: **Mon Jun 04, 2018 4:47 pm**

Too complex for me at the moment but very impressive innovation in the use of an already great tool!

Posted: **Sun Jun 10, 2018 9:38 am**

Hi pnprog,

I've also been using GRP in a non-intended way, to analyse games for bot-similarity and how many and how large mistakes the players make. Basically what I want is how much winrate vs bot's #1 move does the played human move lose (or rarely gain). At the moment this is a bit of a pain to get as:
- #1 you need to go 1-by-1 through each move, transcribing to your spreadsheet
- #2 you need to scroll down the textbox to see the win probabilities (I see you've addressed this in dev version)

This info is basically the raw data behind the win rate delta graph, so if you could somehow dump out the data for the whole game as text/file somewhere that'd be super useful, e.g. a CSV (I added a few bonus columns) like

Move number,Colour,Bot move,Bot winrate,Game move,Game winrate,Bot choice,Policy prob
20,W,h17,54.23,j17,53.5,2,5.12
21,B,h18,46.5,h18,46.5,1,45.32

Thanks.

(I have just realised that using absolute loss of percentage winrate is not such a good idea for measuring size of mistake : if you are 50% and drop 1% to 49%, that's not such a bad mistake as being at 15% and dropping to 14%, better to say the former is 1/50 = 2% relative loss of winrate and later is 1/15 = 7% loss of relative winrate).

Edit:
<mod>
I moved the posts about winrate deltas etc to a new thread so we don't distract from Go Review Partner too much.
</mod>

Posted: **Mon Jun 11, 2018 3:08 am**

I would also like to acknowledge that the author of GRP has been working on providing me exactly what uberdude asked for, a parser from rsgf to csv files that suits my type of analysis. Just today I am going to try out a hopefully final version of it that should speed up an analysis of the games. My view also is that regardless of the outcome of any current analysis we really need to develop methods and software to check if someone used an AI help or not. I am afraid we will face much more of "using AI accusations" in the future, even in regular games not played on internet. So, I really appreciate pnprog's help and work.

Posted: **Mon Jun 11, 2018 3:42 am**

Hi Uberdude,

Uberdude wrote:This info is basically the raw data behind the win rate delta graph, so if you could somehow dump out the data for the whole game as text/file somewhere that'd be super useful, e.g. a CSV (I added a few bonus columns) like
Move number,Colour,Bot move,Bot winrate,Game move,Game winrate,Bot choice,Policy prob
20,W,h17,54.23,j17,53.5,2,5.12
21,B,h18,46.5,h18,46.5,1,45.32

As AlesCieply mentioned above, yes I can certainly do that

One question: What is "Bot Choice"? Is that the rank of the player's move among all Leela's proposed moves?

Also, I need to clarify the methodology for calculating the winning probability for the move played by the player.
Let's take an example:

Code: Select all

Move 17, black to play
Leela's best moves:
D15: 54.1% 
O16: 53.4%
R6:  48.2%
J3:  47.5%
D5:  47.3%
D8:  46.8%

Player's move:
J3

Here, because J3 is part of Leela's best moves, then we could consider that the win rate for the move played by the player is 47.5%, but maybe it is not the best idea.

Another example:

Code: Select all

Move 17, black to play
Leela's best moves:
D15: 54.1% 
O16: 53.4%
R6:  48.2%
J3:  47.5%
D5:  47.3%
D8:  46.8%

Player's move:
K4

Now, K4 is not part of Leela's best moves, so how to evaluate K4 win rate? GRP solves this issue by having Leela play K4, then asking Leela for white next best move. If Leela says that white next best move is N3 with a 61.1% win rate for white, then GRP deduces that the win rate for black K4 was 38.9% (100% - 61.1%).

In fact, even if the player's move was part of Leela's best move, I use the opposite probability of the next move to estimate the player's move win rate. It allows Leela to use all her thinking time/power on the player move this way, not only a fraction of her playout, so the win rate is more accurate.

To continue with the example above:

Code: Select all

Move 17, black to play
Leela's best moves:
D15: 54.1% 
O16: 53.4%
R6:  48.2%
J3:  47.5%
D5:  47.3%
D8:  46.8%

Player's move:
D15

So here, the player's move and Leela best moves are the same. So delta will be 0

But if I now ask Leela for the best answer to black 17 at D15, maybe I will get:

Code: Select all

Move 18, white to play
Leela's best moves:
D15: 45.7%
D8:  43.8%
R6:  42.3%
D5:  42.1%

Here we can see that Leela's best answer is D15/45.7% which indicates that now, Leela re-evaluated black 17 at D15 at a win rate of 54.3% (100% - 45.7%) instead of the former estimation of 54.1%
This is because for this turn, all Leela's thinking power was focused on the continuation after black 17 at D15, so this new evaluation should be more accurate.

Because Leela's best move and the player best move are the same, then the delta is zero regardless of the probability, so I won't impact the histogram. But I just wanted to make the methodology clear.

The CSV can also incorporate both winrate values, it's not an issue.

One last precision, GRP does not rank Leela's moves based on probabilities, the moves have the same ranks as provided by Leela. So low ranked moves could have higher win rate than Leela's top move. The alphabetical order used by GRP truly reflects Leela's best moves order. I guess we are all ok with that?

Posted: **Tue Jun 12, 2018 2:38 am**

Thanks!

pnprog wrote: One question: What is "Bot Choice"? Is that the rank of the player's move among all Leela's proposed moves?

Yes.

That's a good point about the winrate changing with extra playouts after playing it, Ales noted that in his spreadsheet.

pnprog wrote: One last precision, GRP does not rank Leela's moves based on probabilities, the moves have the same ranks as provided by Leela. So low ranked moves could have higher win rate than Leela's top move. The alphabetical order used by GRP truly reflects Leela's best moves order. I guess we are all ok with that?

Yup, it's good to be clear.

Posted: **Wed Jun 13, 2018 2:24 am**

Hi!

For those interested, I implemented the tool to convert RSGF data into CSV files. I put the details on this other thread. It provides an example of batch file for windows that can be used to perform analysis of SGF file in batch, so you could be interested.

If this sort of analysis becomes popular, I will make it a standard feature of GRP

I will take this opportunity to give some news regarding GRP.

Around 2 months ago, I received a Chinese translations for GRP. I was really enthusiastic and added it right away into GRP. Then quickly, I was faced with ton's of issues regarding running GRP on native chinese+windows computers. Those errors appeared when trying to deal with Asian characters in SGF files, file names, interface, everywhere... I did not faced such issues when adding the Korean translation (nor the French or German) while they are complex characters encoding as well. And it was hard for me to troubleshoot them.

The Unicode Pandora box had just been opened, and it took me 2 months of work (and plenty of white hairs) until I was able to finally close it. But that was a good thing in the end, as I am now confident GRP will run properly on all sort of computers across the world. It seems GRP is the only software left in its category that can run on WindowsXP (yes, there are people running LeelaZero on Windows XP or Windows 2003). Admittedly, beside Lizzie and Sabaki-Leela, there are not many softwares in that category to start with

But I would not be surprised if there are still plenty of WinXp computers in use in many Asian children Go schools, or Go clubs.

Now that is issue is solved, I resumed the development, and I will try to release the next version this month.

Posted: **Fri Jun 15, 2018 6:29 am**

WOW,

Can't believe I just discovered this. I just wanted to say thank you for all the time you have spent on this. The design is simple and effective. Love the hover variations and the color coding for win rate. Dual view is also new to me. I can not offer much but is there a way to send a few $ your way as compensation? It won't be much but I think it is deserved.

I will PM you

Thanks.

Posted: **Sun Jun 17, 2018 7:59 am**

harumi wrote:WOW,

Can't believe I just discovered this. I just wanted to say thank you for all the time you have spent on this. The design is simple and effective. Love the hover variations and the color coding for win rate. Dual view is also new to me. I can not offer much but is there a way to send a few $ your way as compensation? It won't be much but I think it is deserved.

I will PM you

Thanks.

Haha, thanks for the proposal

Other users have proposed to donate as well, but it's not necessary: I am quite enjoying myself working on GRP at the moment, all of this is quite rewarding already

Posted: **Fri Jun 22, 2018 2:38 am**

Hi everybody!

I just watched the last game of the HayLee VS LeelaZero 8 games match, and in the video, HayLee mentioned the possibility to make a video using Lizzie or GRP.

So I decided to release a new version of GRP this week. There is not much new features in this version, it's mainly about new interface design, and other changes "under the hood", but in the same time, I don't want to make the same mistake as last time and wait for months to release a new version. I would better have more people use and test GRP, and provide new ideas instead. Also, the translations to Korean and Chinese are important steps for GRP

I am currently downloading the latest Ray (RLO, that uses ELF weights) to check support for the latest version and the latest versions of Leela to package everything. I will also propose a specific download for LeelaZero using ELF weight as well. Hopefully I can release this this weekend.

Posted: **Sat Jun 23, 2018 3:18 am**

pnprog wrote:So I decided to release a new version of GRP this week. [...] Hopefully I can release this this weekend.

Here it is: GoReviewPartner v0.12 !

The most visible changes are about the interface that should be easier to use, and the table that has been redesigned. The other big change is better support for localization (I am pretty confident that GRP will run smoothly on all sort of computers). The way the parameters are read/saved in the config file has been improve a lot as well, to avoid all sort of future headache for users.

I added a download package for Leela+ELF weight. Ray has been updated also, because it now uses LZ type of weights file (the one to download also uses ELF weights). I will make a separate post for my to do list for next release.

EDIT: I haven't updated the documentation yet, I guess I will do that fist

Posted: **Sun Jun 24, 2018 8:18 am**

Here is the updated documentation: http://yuntingdian.com/goreviewpartner/ ... on/doc.htm

It took me the whole day to update it, but I am happy with result. I use Markdown and Pandoc. I am sure there are still plenty of English mistakes inside, but it should be understandable

Posted: **Tue Jul 03, 2018 12:28 am**

Hi!
I just want to report that I added the possibility to play variation on the left side board. It now makes that board much more useful than it was before. Like it's possible to play a variation one the left, then display one of the bot variation on the right:

: screenshot.png (82.51 KiB) Viewed 19109 times

It's still not possible to use the bot on that left side, but still an improvement

Posted: **Wed Jul 04, 2018 11:29 pm**

Hi!

So I have been thinking hard about how to improve the interface, especially how to make that left side board more useful. This night I came up with the following idea (I even get up in the middle of the night to sketch my ideas on paper

) and I would like your opinions on it.

(the following pictures are patchworks of cropped screenshots, nothing coded so far)

So basically, I would like to use a "tab" system, à la Firefox:

: img0.png (30.52 KiB) Viewed 19045 times

At any time, there would always be at least those two tabs available:

The first one is the one above, that corresponds to the current right side goban. It shows the current variations from the computer. Two buttons are available Filter that would be used to filter what variations are displayed (the existing "display only of computer move is different than actual move", and an interval filter based on delta) and coloring (the existing "only winning variations in blue" and similar). The same "filtering options" would be made available when converting from RSGF to SGF.
The second tab that would always be available is the one below, equivalent to the current left side goban. This goban is mainly used to display the available maps (I plan to add a heatmap feature) and to play out variations manually. Not bot available on this tab:

: img1.png (55.03 KiB) Viewed 19045 times

Those two tabs would always be available (impossible to close them). When you click on "next move" (or "previous move" and others...) they move along with you. So they are always synchronized with the navigation bar and the left side panel (the panel with comments and table).

Then, there would be the possibility to add more tabs by pressing the "+" button, just like in Firefox.
This would add more tabs, that are equivalent to the "open position" goban:

: img2.png (58.56 KiB) Viewed 19045 times

Just like in the actual version, there would be the possibility to open as much tabs as one needs. Those tabs would offer the possibility to play against computers. I will find a way to only "load" the bot "on demand" so that opening more tabs does not eat too much memory (this has the disadvantage to make switching between bots slower). Just like in Firefox, it would be possible to close those tabs (but I haven't represented a button for that yet).

I am limited in the number of image per post, so I will continue in the next post

Posted: **Wed Jul 04, 2018 11:33 pm**

So, this is what the final interface would look like:

: img3.png (99.22 KiB) Viewed 19045 times

As you can see, it would be more compact, so easier to use on small screens.

Now, for people like me, who enjoy having both goban at the same time (I like to replay on the right the variations on the left) I would add the possibility to have two panels, like it's commonly used in some file explorers:

: img4.png (140.08 KiB) Viewed 19045 times

(I might lost the possibility to resize all three panel manually like it's actually the case, I am not sure)

This way, I think that would make GRP more useful, what do you think?

Life In 19x19

Announcing GoReviewPartner - v0.15 with Pachi and PhoenixGo

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.11.2 (with Live Analysis

Re: Announcing GoReviewPartner - v0.12

Re: Announcing GoReviewPartner - v0.12

Re: Announcing GoReviewPartner - v0.12

Re: Announcing GoReviewPartner - v0.12

Re: Announcing GoReviewPartner - v0.12