Dear all,

I am working on a program to use Leela to help review my games. And for that, I am parsing the standard output of Leela in console when running in gtp mode. I am not so clear with the meaning of some Leela's messages. In fact, I am not so clear with most of the output, so I am coming with a lot of questions.
Let me show an example (using Leela 6.2 under Linux).

Here I run Leela in gtp:

Then in the terminal, I input some game moves using gtp commands, as follow:


Note I did not mention anything specific about komi; so this would give black a advantage in Leela's eye. Also; this game if from Henning W. as published on that thread: http://lifein19x19.com/forum/viewtopic.php?f=4&t=13669.
At that point, the go board position should be like this:

Click Here To Show Diagram Code

[go]$$c Current game position, black to play
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . X . . |
$$ | . . . . . . . . . . . . . . X X O O . |
$$ | . . . O . . . . . . . . . . . O . . . |
$$ | . . . . . . . . . . . . . . . O . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . . X X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . . . . . . . . . . . |
$$ | . . X . . O . . . . . . . . . X . . . |
$$ | . . X O . . . . . . . . . . . . . . . |
$$ | . . O . O . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

Then, I ask Leela to play the next move with that gtp command:


Here is Leela output:

So now (for what I understand), Leela proposes to play n16 as the next move. And the expected follow up is as follow: o15 o13 o16 o17 s18 n15
And other good move that Leela considered is M17, with that follow up: o16 o17 r9

For those two possibilities, what are the meanings of the complete lines? for example, for N16, what the meaning of "U: 51.35%", "R: 51.71%: 29322" and "N: 64.1%" in


The other potential moves (S18, O15, O16, N17) have U and R value at 0%, what the meaning for that? Are those moves that were immediately refuted by Leela (bad moves)?

Regarding the fist line output:
How to interpret it? 37.5% of chance to win for black according to Monte Carlo search ?

Those 3 lines:
Are they potential variations on the main line of play, N16?

And finally, for that line:
The score of 51.35% is from the U value above (my guess), then what about that "from 51.29%"? Is that the winning rate estimated by Leela prior to play N16?

Ok, so now the situation on the board is as follow:

Click Here To Show Diagram Code

[go]$$c Leela just played N16
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . X . . |
$$ | . . . . . . . . . . . . . . X X O O . |
$$ | . . . O . . . . . . . . # . . O . . . |
$$ | . . . . . . . . . . . . . . . O . . . |
$$ | . . . . . . . . . . . . . . . . O . . |
$$ | . . . . . . . . . . . . . . . X X . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . X . . . . . . . . . . . . . . . |
$$ | . . X . . O . . . . . . . . . X . . . |
$$ | . . X O . . . . . . . . . . . . . . . |
$$ | . . O . O . . . . . . . . . . . . . . |
$$ | . . . . . . . . . . . . . . . . . . . |
$$ ---------------------------------------[/go]

If I now input the following gtp command:
I got:


W+6.5 is and estimation of Leela result at this point of the game?

Thanks in advance for your replies. I am coming with a lot of questions, but I feel that there a lot of potential from learning from Leela's output, for players several stones weaker than Leela (like me). I am trying to make the extration of such information automatic and displayed in an adequate way to review games.

Best regards,

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Please note that Leela GUI works well on GNU/Linux under Wine. I'm attaching below an illustrative screenshot of the running analysis for the game you mention. (This is with Leela 0.7.0, Wine 1.9.21.) So, using native Leela GUI under Wine might be an easy out-of-the-box option to analyse your games on GNU/Linux... Otherwise you can get the meaning of some of the console output variables by comparing the two, for example see "N %" and "Net prob %".

I can get the vanilla Leela to work with PlayOnLinux, but Leela070_OpenCL crashes. Have you had success running that in Wine? Do you know the dependencies?

Many thanks for the reply and pictures!

From what I can see on your picture, the "other potential moves" (d6, m4, o4, r13, r14, o17 and n17 in your running analysis) are in fact moves that where given too low probabilities to be played by the neural network and thus were not analysed with MC search, but not necessarily bad moves.

So the "Win%" column probably corresponds to the "U:" variable in console and the "Net Prob%" column is "N:" variable.

I still do not understand the "R:" variable meaning, and I now guess it is probably linked with "from" in the last console message "28871 visits, score 51.35% (from 51.29%) PV: n16 o15 o13 o16 o17 s18 n15". For this one signification, I will have to send an email to Leela creator.

Haha, I am a happy user of the terminal, but there is more to it: The program I have been working on is intended to help me replay/analyse my games and operates this way:

It takes a sgf file, and for each game positions, it asks Leela to play its best move, and parses from the terminal the different sequences proposed by Leela. It also gets the score estimate from Gnugo, as this score estimation is more "human friendly" than what is proposed by Leela (because Gnugo does not try to win by a very small margin/high win probability, Gnugo gives a real territory estimation). With all that information, it builds a new sgf file with all variations in the game tree, and territory estimations/win rates in the comments.

As I give a long thinking time to Leela (30s or 60s per moves) it takes a (very) long time, and I usually run that during the night or while I am not busy at the computer. For that reason, working from the terminal is best to make it an automatic process, and in this case, it seems there is more information available from the terminal than from the interface (I did not try Leela 0.7.0 yet, maybe the console message are different?).

The sgf file I get is huge, with too many variations to be worked with using usual sgf editor. I usually use Quarry for that, and there may be better editors for huge game trees; but still, I think it is too much variations and too much "computer mouse work" to be able to enjoy the review. Have a try with the one enclosed, result of analysis from the original example to get a feeling.

So next, I built a visual interface to help work with that sgf file. It displays two gobans: on the left side goban, the game position with actual game move represented with the letter A (and score estimate by gnugo); on the right side goban, the different variations proposed by Leela, A, B, C, D... from the best win probability to the lowest one:


Move the mouse pointer over one of those letters and the complete variation is displayed with Leela win rate estimation (or actual game 5 next moves for the left side goban):


That's it for now, but I also plan to add the possibility to open the current game position into a third goban (like a popup) with the possibility to try other moves or variations against Gnugo or Leela. Maybe I also could use those "other potential moves" that were not searched further with MC search to display additional "hot area" the same way as Leela interface is doing. But too much information sometime is not so good. I will give it a try.

Leela is several stones stronger than I am, so reviewing my games this ways help me find missed opportunities, point out some of my recurrent mistakes, and give me some exposure to the more influence oriented Leela's way of playing. And all that with minimum "mouse work", so the review can go pretty fast.

This is all made with python, I will release the code when it's working well. Anyway, Leela is an amazing program, and having a gtp version available provides even more opportunities to harness this power.

Best regards,

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Fantastic work, pnprog. Please do share your code once you're ready! Preferably on github or the like so others can iterate.

Sounds awesome! I've thought about doing something similar in Python, so I'm happy this is being worked on already. Now let's hope Leela gets a value net at some point and gets even stronger and can provide even better analysis. A variable bias/filter between win rate and expected territory would also be a nice option for Leela.

I am having some trouble to use leela in an automatic manner and would appreciate your help.
To review a game in sgf format I first turned it into a sequence of gtp commands such as (to take your example):



then calling ./leela070gtp_x64 < input_moves.txt > output.txt

Leela plays through the game correctly but then starts displaying the same board again and again at a very fast rate. My guess is that it reads the end-of-file character repeatedly. Do you know how to solve this?

I am also surprised by the fact that Leela uses the standard error channel to display the board and its analysis result (instead of standard output). Is this usual?

_________________
La victoire est un hasard, la défaite une nécessité.

It's not unusual.
Note that you can redirect stderr using 2>.

Excellent; seems I hadn't done my homework properly. Thank you for the help.

_________________
La victoire est un hasard, la défaite une nécessité.

Hello all!

Just to say that I made some of the modifications I was mentioning above (possibility to open a position in popup to play out variations) and put the result on github.

To be honest, it is unusable for now without having to dig into the python code (to change leela executable path and other things) and running it from the command line. But at least the code is available now. In fact, I was closed to lose this project data due to a miss manipulation on my computer, so it urged me to put everything on github even if it's not usable yet.

I'm new to all this git/github thing, so I will need some time to get comfortable with this way of working, but I plan to make the whole thing easy to use soon. All the recent developments about AlphaGo/DeepZen are very exciting, it's probably a matter of years before pro level bots become available for study for most players.

Best regards,

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

I did email the author about some other things and while I was at it, I asked about this. The reply was:

You are confusing name with function. In this case, the language provides for two output files that you do not have to open as files (always available). I lost my C books in the 2006 house fire, so just going by memory, but I think all that is different is "buffering*. So standard error releases each record written to it immediately (with a "buffered" file, the physical write takes place only when the buffer has been filled or at the end (the last partial gets written). You don't send error messages to standard output because in a case of abnormal termination, you might not get to see them.

In other words, error messages should be sent to standard error but that does not mean the only things you can send there are error messages.

You can of course have more outputs, but those would be files you had to define and explicitly open.

Thanks for asking.

So I guess I will just ignore this one.

Best regards,

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

Leela's GTP replies come on stdout, and the diagnostic output comes over stderr. The diagnostic output is not valid GTP so it would confuse GTP interfaces if sent over the same channel.