Why is Zenith Go 7 > GTP Bots for analyzing my games?

[Gomoto]

One feature I am really missing when I am analyzing with GTP bots.

With Zenith Go I can watch the engine thinking. (During the evaluation the current best move is shown on the board and the move alternatives are constantly updated in a list with the attached values).

(The Leela desktop program offers a similar function.)

But with GTP only bots like AQ I have to wait for the thinking process to finish.

For now I use Zenith for a quick first review of my games. At critical moves I double check with AQ GPU, ...

Perhaps there is a way to watch AQ while thinking and I am not aware of it yet?

[pnprog]

Hi,

The GTP protocol is just not designed for that, there is no way around using GTP.

Some bots offer specific features, but they are not standardized anyway.

GTP protocol is almost 20 years old, and was not design with this use case in mind.

As the author of GoReviewPartner, I am facing the same limitations and it's frustrating.

I hope the group of people who defined GTP 15 years ago could work together again to issue a new version taking into account all the ways bots can be used for game analysis. But maybe it's too late, maybe a new working group should be started from scratch.

[as0770]

I hope the group of people who defined GTP 15 years ago could work together again to issue a new version taking into account all the ways bots can be used for game analysis. But maybe it's too late, maybe a new working group should be started from scratch.

I am not very familiar with gtp, but isn't there a new standard? This is the output by Leela Zero with win rates, best moves and pv:

Code: Select all

Playouts: 3383, Win: 90.58%, PV: L5 C15 G17 J17 C16 D15 C11 E17 E18 E16 C18 G16 F17 F16 H17 H16 J18
  L5 ->    4438 (V: 91.57%) (N: 18.83%) PV: L5 C15 G17 J17 C16 D15 C11 E17 E18 E16 C18 G16 F17 F16 H17 H16 J18
  L2 ->     493 (V: 90.92%) (N:  7.95%) PV: L2 N8 N9 M9 M8 M10 N7 C15 G17 J17 C16 D15 C11 E17
  N7 ->     142 (V: 79.50%) (N: 30.10%) PV: N7 L5 K5 K4 O6 P6 K3 K6 L7 K4 P5 O6 K3 O8 N9 K4
  N8 ->      50 (V: 80.71%) (N:  9.38%) PV: N8 L5 L2 K6 J7 K5 K4 K7
  M9 ->      33 (V: 84.37%) (N:  3.96%) PV: M9 L5 K4 K6 J7 L7 K7 N8 N9 M8
  O8 ->      26 (V: 79.63%) (N:  5.45%) PV: O8 L5 K5 K4 N7 O6 K3 K6 L7 K4
 J17 ->      19 (V: 78.66%) (N:  4.26%) PV: J17 L5 L2 K6 L7 K5 K4 J7
  O3 ->      12 (V: 81.99%) (N:  2.15%) PV: O3 L5 K5 K4 O4 P5
  N9 ->      10 (V: 80.63%) (N:  1.83%) PV: N9 L5 K5 K4 L2 K3
 D14 ->       9 (V: 80.87%) (N:  1.79%) PV: D14 L5 K5 K4 J17
 H17 ->       7 (V: 79.82%) (N:  1.27%) PV: H17 L5 K5 K4
 D15 ->       7 (V: 77.31%) (N:  1.73%) PV: D15 L5 K5 K4 N7
  O6 ->       6 (V: 83.08%) (N:  1.01%) PV: O6 L5 K5 K4
 K17 ->       6 (V: 79.79%) (N:  1.29%) PV: K17 L5 K5 K4 K15
  O4 ->       6 (V: 75.36%) (N:  1.93%) PV: O4 O3 P3 L5 K5 K4
 M14 ->       5 (V: 83.02%) (N:  0.76%) PV: M14 L5 K5 K4
 N10 ->       4 (V: 83.37%) (N:  0.56%) PV: N10 L5 K5 K4
 C14 ->       4 (V: 81.44%) (N:  0.56%) PV: C14 L5 K5 K4
  P5 ->       4 (V: 76.42%) (N:  0.99%) PV: P5 L5 K5 K4
5282 visits, 1338942 nodes, 3615 playouts, 225 n/s

I've seen this also by other engines.

[Gomoto]

Every engine has the info always available during processing time. The interesting question: Is there a learning and analyzing tool available, that shows the info during process time to the user. This is a big time saver during my review time.

[pnprog]

I am not very familiar with gtp, but isn't there a new standard? This is the output by Leela Zero with win rates, best moves and pv:

In fact, those infos are not part of the GTP protocol.

In details, the program (for example Sabaki) sends GTP commands to the bot (the bot is a terminal application) by writing those commands on the bot "standard input" (sdtin), then reads the bot answers from the bot "standard output" (stdout). Additionally, the bot can send a number of messages on a third channel, the "standard error" (stderr) but this channel is more intended for developers to print warning and debug information. But even if the standard error channel is not really intended to be read and parsed by another program, it is totally possible (and GRP does just that). Note that all three channels are one way only.

Typically, for convenience, when a program is ran inside a terminal, both the standard output and standard error are printed together, and it gives the impression that there are only one channel, only one flow of information, which is not the case. In Sabaki (or is it gogui?), I noticed the text from the standard error appears in grey.

So for example, Sabaki writes the following command on Leela standard input:

Code: Select all

genmove black

Then Leela starts crunching numbers and run MC simulation and send some intermediate results its standard error:

Code: Select all

Playouts: 3383, Win: 90.58%, PV: L5 C15 G17 J17 C16 D15 C11 E17 E18 E16 C18 G16 F17 F16 H17 H16 J18
  L5 ->    4438 (V: 91.57%) (N: 18.83%) PV: L5 C15 G17 J17 C16 D15 C11 E17 E18 E16 C18 G16 F17 F16 H17 H16 J18
  L2 ->     493 (V: 90.92%) (N:  7.95%) PV: L2 N8 N9 M9 M8 M10 N7 C15 G17 J17 C16 D15 C11 E17
  N7 ->     142 (V: 79.50%) (N: 30.10%) PV: N7 L5 K5 K4 O6 P6 K3 K6 L7 K4 P5 O6 K3 O8 N9 K4
  N8 ->      50 (V: 80.71%) (N:  9.38%) PV: N8 L5 L2 K6 J7 K5 K4 K7
  M9 ->      33 (V: 84.37%) (N:  3.96%) PV: M9 L5 K4 K6 J7 L7 K7 N8 N9 M8
  O8 ->      26 (V: 79.63%) (N:  5.45%) PV: O8 L5 K5 K4 N7 O6 K3 K6 L7 K4
J17 ->      19 (V: 78.66%) (N:  4.26%) PV: J17 L5 L2 K6 L7 K5 K4 J7
  O3 ->      12 (V: 81.99%) (N:  2.15%) PV: O3 L5 K5 K4 O4 P5
  N9 ->      10 (V: 80.63%) (N:  1.83%) PV: N9 L5 K5 K4 L2 K3
D14 ->       9 (V: 80.87%) (N:  1.79%) PV: D14 L5 K5 K4 J17
H17 ->       7 (V: 79.82%) (N:  1.27%) PV: H17 L5 K5 K4
D15 ->       7 (V: 77.31%) (N:  1.73%) PV: D15 L5 K5 K4 N7
  O6 ->       6 (V: 83.08%) (N:  1.01%) PV: O6 L5 K5 K4
K17 ->       6 (V: 79.79%) (N:  1.29%) PV: K17 L5 K5 K4 K15
  O4 ->       6 (V: 75.36%) (N:  1.93%) PV: O4 O3 P3 L5 K5 K4
M14 ->       5 (V: 83.02%) (N:  0.76%) PV: M14 L5 K5 K4
N10 ->       4 (V: 83.37%) (N:  0.56%) PV: N10 L5 K5 K4
C14 ->       4 (V: 81.44%) (N:  0.56%) PV: C14 L5 K5 K4
  P5 ->       4 (V: 76.42%) (N:  0.99%) PV: P5 L5 K5 K4
5282 visits, 1338942 nodes, 3615 playouts, 225 n/s

Then finally print the result on its standard output:

Code: Select all

= L5

Now, the GTP protocol only works using standard input and standard output and ignores the standard error channel that is then left free to use by developers as they wish. The AQ standard error text is quite different and also contains some useful informations (there is a screen shot there). Ray also shows different informations on standard error channel. Leela Zero standard error output is very similar to Leela but still different.

So if each bot has its own "recipe" for what to display on the standard error, then there is no way to make a general program that would work for all bots to extract those information.

Regarding the GTP protocol in itself, the minimal set of command a bot must implement is :

Code: Select all

protocol_version
name
version
known_command
list_commands
quit
boardsize
clear_board
komi
play
genmove

With the 2 main commands used to play with a bot at the end:

• play: to indicate the opponent played one stone, example: play black d4
• genmove: to ask the bot to play the game next move, example: genmove white

So a standard GTP program like Sabaki or gogui can work with all GTP bots without exception, but can only ask them to perform a set of limited action (you play, I play, you play, I play...).

In the opposite, a specialized program like SabakiLeela or GoReviewPartner can only work with a limited number of programs (Leela/LeelaZero for SabakiLeela) (Leela/LeelaZero/AQ/Ray/GnuGo for GRP) but can interact better with those bots. Although this require to "craft" a dedicated way to communicate with all of them separately).

What we would need is a set of GTP standard commands that would look like:

• start_thinking: the bot start to think about the next best moves, and collect data, example: start_thinking black
• stop_thinking: the bot stop to thinking about the next best moves, example: stop_thinking
• best_moves N: the bot provide it's own proffered N moves, example: best_move 5
• best_sequences N M: the bot provide it's own proffered N sequences of length M, example: best_sequence 5 10
• show_heat_map
• show_influence_map
• show_territory_map
• show_winrate_map

Then a program like Sabaki could issue command in this order: start_thinking, then wait one second, then stop_thinking, then immediately best_moves 5, best_sequences and so on. At that point, Sabaki can display all those information in a nice visual way for the user. Then again start_thinking, wait for one second, then stop_thinking and so on... and those informations would be refreshed every second as the bot thinks longer and longer.

[pnprog]

This is a big time saver during my review time.

If your motivation is about saving time during the review, then I suggest you try GoReviewPartner: It works by performing the analysis (getting alternative moves, sequences and stats) of all your SGF file moves automatically (in one shot) and save them into an SGF like file.

The analysis can take quite a lot of time (depending on how long time you want to bot to think about each move) but since it is automatic, it is not really wasting your time (you can do something else in the meantime).

You can get an idea of what it does by checking the screen-shots in the documentation.

[Gomoto]

In fact I like the Leela Zero GTP output much more over the AQ output, this is exactly what I am looking for.
Now I only want the feature to stop the thinking process by the user and not only by the time settings.

The Leela Zero output is very readable in Sabaki during thinking time of the engine.

And Leela Zero is playing good go even at short thinking times <= 20 sec on my box.

[Gomoto]

Yes, commands like start_thinking and stop_thinking would be very helpful to allow the user to go through a game at a user defined pace. Skip moves that are not that interesting and stay longer at interesting moves. Only Zenit 7 does provide this service currently for my analysis.

[lemonsqueez]

One feature I am really missing when I am analyzing with GTP bots.

With Zenith Go I can watch the engine thinking. (During the evaluation the current best move is shown on the board and the move alternatives are constantly updated in a list with the attached values).

But with GTP only bots like AQ I have to wait for the thinking process to finish.

Perhaps there is a way to watch AQ while thinking and I am not aware of it yet?

gogui has a live graphics feature which allows gtp engines to display things graphically while they're thinking, check if AQ uses it maybe. It's a gogui extension so not all gtp engines will have it, but it's been around long enough you might be lucky.

You can try with Pachi in gogui for example:

• Programs -> Attach -> Pachi
• Tools -> Analyze Commands -> Live gfx = best sequence -> Run

That will display current best sequence while it's thinking.

If there are no graphics commands listed in the analyze window then the engine doesn't support it.

[spook]

Even the formats of LeeLa and LeeLa Zero are not entirely the same.

They both use the following notation:

Code: Select all

T19 -> 361 (...) PV: T19 S18 R17

But LeeLa Zero wraps the entire thing between "~begin" and "~end" keywords.

I personally also like Zenith a lot for its user friendlyness.
But tools like LeeLa GUI, Lizzie, GoReviewPartner are improving.

I would love to see tools that allow you to combine opinions of multiple bots.
But before we will see those kind of tools, a lot of small issues need to be resolved first.
To name a couple:

• To run Zenith in GTP mode.
  Komi support for LeeLa Zero.
  Downloadable ELF OpenGo builds for different operating systems.

[dfan]

Even the formats of LeeLa and LeeLa Zero are not entirely the same.

They both use the following notation:

Code: Select all

T19 -> 361 (...) PV: T19 S18 R17

But LeeLa Zero wraps the entire thing between "~begin" and "~end" keywords.

That's just the ponder hack from the patch that lets Leela Zero work with Lizzie.