Analysing move quality. Kind of, for 9x9 games (using pachi)

[RBerenguel]

Pachi (http://pachi.or.cz/) is an open source UCT go playing program. According to its authors, its strength is around 7d KGS for 9x9, 2d for 19x19. I am interested in the 9x9 part, since (as I said elsewhere) I try to improve my level of play, and if my computer can help me, well, I'll use it.

Pachi comes with a set of tools (in the folder /tools) and among them there is sgf-analyse.pl, which should give you an analysis of your own games. Problem is, I couldn't make it work and my knowledge of perl is just barely enough to understand what it does but not how to fix it.

So I rewrote sgf-analyse in awk (requires GNU awk for the co-process pipe operator)

Code: Select all

BEGIN {
    total=4
    commands[1]=""
    move[1]=""
    engine="-e uct"
    plays="-t =1000"
    pachi_exec="./pachi"
    pachi=pachi_exec " " engine " " plays " -d 0 dynkomi=none"
    file="/tmp/gorate"
    system("rm " file)
    pachi_exec "  -a 2>&1 | head -n 2 | tail -n 1\n" | getline
    printf "Using " plays " plays and " engine " engine.\n" $0 "\n\n"
    printf "For each move evaluate the board and generate a next\nmove. Evaluating the board and generating a move give\ndifferent values, even for the same move.\n\nGenerating also /tmp/gorate with format:\n Pwin generated, Evaluate value | -1 , Exact match move? (i.e. like 0.55, -1,  1)\n\n"
}

NR<=4{
    commands[NR]=$0
}

NR==3{
    move[1]=$3
}
NR==4{
    move[2]=$3
}

# Will process output from sgf2gtp, repeating
(NR>4){
    total=total+1
    mvc=NR-3
    commands[NR]=$0
    #printf commands[NR]
    move[mvc]=toupper($3)
    tomove[mvc]=$2
    #printf "Move: " move[NR] "\n"
}

END {
    for(j=7;j<=total;j++){ #Starts at 5 and evaluates starting at j=Move to study+3
	mvc=j-3
	count=count?0:1
	color="unknown"
	steps=0
	while (color=="unknown"&&steps<3){
	    steps=steps+1
	    for (i=1;i<=j-1;i++){
#		printf "DEBUG: Issuing command #"i": " commands[i] "\n"
		printf commands[i] "\n" |& pachi
	    }
#	    printf "DEBUG: Issuing command: evaluate " tomove[mvc] "\n"
	    printf "pachi-evaluate " tomove[mvc] "\n" |& pachi
#	    printf "DEBUG: Issuing command: genmove " tomove[mvc] "\n"
	    printf "genmove " tomove[mvc] "\n" |& pachi
#	    printf "DEBUG: Issuing command: pachi-result\n"
	    printf "pachi-result\n" |& pachi
#	    printf "DEBUG: Issuing command: quit\n"
	    printf "quit\n" |& pachi
	    countreply=0
	    score=""
	    while(pachi |& getline >0){
#		printf "DEBUG: " $0 "->" $1 " " $2 " " $3 "\n"
		if ($2!=""){
		    # format color move playouts pwin dynkomi for pachi-result
		    color=$2
		    genmove=$3
		    pwin=$5
		    if($4==""){
#			print "DEBUG: $4 is null: " $0 " " $1 " " $2
		    }
		    # For pachi-evaluate, move and pwin 
		    if(color==move[mvc]&&$3!=""){
#			printf "DEBUG: Match evaluate"
			score=genmove
		    }
		    if($1==move[mvc]){
#			printf "DEBUG: Match evaluate-2"
			score=$2
		    }		
		}
	    }
	    close(pachi)
	}
	printf pwin ", " >> file
	if(score=="")
	    printf "-1, " >> file
	else
	    printf score ", " >> file
	if(move[mvc]==genmove)
	    printf " 1\n" >> file #Correct hit
	else
	    printf " 0\n" >> file #Wrong hit
	if(score!=""){
	    printf mvc ": Original " tomove[mvc] " move: " move[mvc] " has possible P(win) " score ". Suggested " color " move: " genmove " P(win): " pwin "\n"
	}else{
	    printf mvc ": Original " tomove[mvc] " move: " move[mvc] ". Suggested " color " move: " genmove " pwin: " pwin "\n"
	}

    }

    close(pachi)
}

Here's a run of it against one of Go Seigen - Miyamoto Naoki's 9x9 games. You'll see some "unknown" in there. Not sure why it happens, one of these spurious things that just make you wonder if programming is made for sane minds.

Code: Select all

tools/sgf2gtp.pl < ~/Downloads/Pro9x9-1.sgf | gawk -f ~/Dropbox/Codi/awk/pachi-eval.awk
Using -t =1000 plays and -e uct engine.
Pachi version 10.00 (Satsugen)

For each move evaluate the board and generate a next
move. Evaluating the board and generating a move give
different values, even for the same move.

Generating also /tmp/gorate with format:
 Pwin generated, Evaluate value | -1 , Exact match move? (i.e. like 0.55, -1,  1)

 4: Original B move: F4 has possible P(win) 0.520. Suggested black move: D7 P(win): 0.60
 5: Original W move: G4 has possible P(win) 0.415. Suggested white move: G4 P(win): 0.41
 6: Original B move: G3 has possible P(win) 0.528. Suggested black move: G3 P(win): 0.59
 7: Original W move: F3 has possible P(win) 0.446. Suggested white move: F3 P(win): 0.51
 8: Original B move: E4 has possible P(win) 0.478. Suggested black move: E4 P(win): 0.58
 9: Original W move: G2 has possible P(win) 0.430. Suggested white move: H3 P(win): 0.47
10: Original B move: H3 has possible P(win) 0.569. Suggested black move: H3 P(win): 0.56
11: Original W move: H2 has possible P(win) 0.389. Suggested white move: H2 P(win): 0.50
12: Original B move: E3 has possible P(win) 0.585. Suggested black move: H4 P(win): 0.57
13: Original W move: F2 has possible P(win) 0.408. Suggested white move: F2 P(win): 0.48
14: Original B move: G5 has possible P(win) 0.559. Suggested black move: E7 P(win): 0.55
15: Original W move: H4 has possible P(win) 0.400. Suggested white move: H4 P(win): 0.43
16: Original B move: H5 has possible P(win) 0.530. Suggested black move: D7 P(win): 0.61
17: Original W move: J3 has possible P(win) 0.380. Suggested white move: J3 P(win): 0.44
18: Original B move: F7 has possible P(win) 0.527. Suggested black move: D8 P(win): 0.58
19: Original W move: G6 has possible P(win) 0.390. Suggested white move: G6 P(win): 0.43
20: Original B move: G7 has possible P(win) 0.549. Suggested black move: E6 P(win): 0.59
21: Original W move: H6 has possible P(win) 0.389. Suggested white move: H6 P(win): 0.42
22: Original B move: E6 has possible P(win) 0.519. Suggested black move: E7 P(win): 0.62
23: Original W move: B5 has possible P(win) 0.417. Suggested white move: B4 P(win): 0.44
24: Original B move: H7 has possible P(win) 0.566. Suggested black move: C3 P(win): 0.57
25: Original W move: J5 has possible P(win) 0.398. Suggested white move: J5 P(win): 0.45
26: Original B move: B7 has possible P(win) 0.504. Suggested black move: B4 P(win): 0.61
27: Original W move: C3 has possible P(win) 0.380. Suggested white move: C3 P(win): 0.46
28: Original B move: C4 has possible P(win) 0.565. Suggested black move: C4 P(win): 0.60
29: Original W move: B4 has possible P(win) 0.388. Suggested white move: B4 P(win): 0.46
30: Original B move: C2 has possible P(win) 0.529. Suggested black move: D2 P(win): 0.55
31: Original W move: C7 has possible P(win) 0.442. Suggested white move: C7 P(win): 0.49
32: Original B move: C8 has possible P(win) 0.509. Suggested black move: C6 P(win): 0.57
33: Original W move: D3 has possible P(win) 0.471. Suggested white move: B3 P(win): 0.54
34: Original B move: D2 has possible P(win) 0.454. Suggested black move: D2 P(win): 0.47
35: Original W move: B2 has possible P(win) 0.492. Suggested white move: B2 P(win): 0.59
36: Original B move: E2 has possible P(win) 0.513. Suggested black move: E2 P(win): 0.54
37: Original W move: D7 has possible P(win) 0.409. Suggested white move: C6 P(win): 0.51
38: Original B move: B6 has possible P(win) 0.628. Suggested black move: C6 P(win): 0.63
39: Original W move: E7 has possible P(win) 0.242. Suggested white move: C6 P(win): 0.37
40: Original B move: C6 has possible P(win) 0.792. Suggested black move: D6 P(win): 0.80
41: Original W move: F8 has possible P(win) 0.267. Suggested white move: F8 P(win): 0.40
42: Original B move: G8 has possible P(win) 0.719. Suggested black move: G8 P(win): 0.74
43: Original W move: B8 has possible P(win) 0.186. Suggested white move: H9 P(win): 0.30
44: Original B move: E8 has possible P(win) 0.520. Suggested black move: A8 P(win): 0.83
45: Original W move: D8 has possible P(win) 0.328. Suggested white move: D8 P(win): 0.48
46: Original B move: F9 has possible P(win) 0.544. Suggested black move: F9 P(win): 0.55
47: Original W move: C9 has possible P(win) 0.478. Suggested white move: A8 P(win): 0.48
48: Original B move: E9 has possible P(win) 0.492. Suggested black move: A8 P(win): 0.61
49: Original W move: D9 has possible P(win) 0.356. Suggested white move: A8 P(win): 0.49
50: Original B move: D4 has possible P(win) 0.447. Suggested black move: A8 P(win): 0.63
51: Original W move: B3 has possible P(win) 0.351. Suggested white move: B3 P(win): 0.43
52: Original B move: B1 has possible P(win) 0.568. Suggested black move: A8 P(win): 0.66
53: Original W move: A8 has possible P(win) 0.280. Suggested white move: D6 P(win): 0.54
54: Original B move: A2 has possible P(win) 0.493. Suggested black move: A2 P(win): 0.71
55: Original W move: C1 has possible P(win) 0.284. Suggested white move: A6 P(win): 0.31
56: Original B move: D1 has possible P(win) 0.730. Suggested black move: D1 P(win): 0.71
57: Original W move: A3 has possible P(win) 0.246. Suggested white move: A7 P(win): 0.38
58: Original B move: A5 has possible P(win) 0.668. Suggested black move: A5 P(win): 0.78
59: Original W move: A7 has possible P(win) 0.173. Suggested white move: H9 P(win): 0.38
60: Original B move: A6 has possible P(win) 0.618. Suggested black move: C5 P(win): 0.85
61: Original W move: H9 has possible P(win) 0.317. Suggested white move: H9 P(win): 0.35
62: Original B move: H8 has possible P(win) 0.673. Suggested black move: G9 P(win): 0.72
63: Original W move: F1 has possible P(win) 0.154. Suggested white move: A1 P(win): 0.30
64: Original B move: C5 has possible P(win) 0.859. Suggested black move: C5 P(win): 0.81
65: Original W move: A1 has possible P(win) 0.101. Suggested unknown move: pachi-result P(win):
66: Original B move: C1 has possible P(win) 0.888. Suggested black move: J7 P(win): 0.89
67: Original W move: E1 has possible P(win) 0.096. Suggested white move: J7 P(win): 0.10
68: Original B move: A2 has possible P(win) 0.827. Suggested black move: G9 P(win): 0.87
69: Original W move: J7 has possible P(win) 0.108. Suggested white move: J7 P(win): 0.21
70: Original B move: J8 has possible P(win) 0.917. Suggested black move: J8 P(win): 0.89
71: Original W move: A1 has possible P(win) 0.038. Suggested unknown move: pachi-result P(win):
72: Original B move: B9 has possible P(win) 0.854. Suggested black move: G9 P(win): 0.97
73: Original W move: A9 has possible P(win) 0.029. Suggested white move: A9 P(win): 0.16
74: Original B move: A2 has possible P(win) 0.870. Suggested black move: G9 P(win): 0.98
75: Original W move: G9 has possible P(win) 0.065. Suggested unknown move: pachi-result P(win):
76: Original B move: J9 has possible P(win) 0.912. Suggested black move: J9 P(win): 0.92
77: Original W move: A1 has possible P(win) 0.023. Suggested unknown move: pachi-result P(win):
78: Original B move: G9 has possible P(win) 0.992. Suggested black move: G9 P(win): 0.97
79: Original W move: J6. Suggested white move: D6 pwin: 0.01

And here's a plot of the win-rates of real moves (according to pachi-evaluate) and of the generated game by pachi (according to pachi-result):

Screen Shot 2014-05-26 at 19.38.56.jpg (116.61 KiB) Viewed 8000 times

I used gnuplot for the plot, here's the plotting command:

Code: Select all

plot "/tmp/gorate" every 2::1 using 0:2 with lines title "Black WR-move", '' every 2::2 using 0:2 with lines title "White WR-move", '' every 2::1 using 0:1 with lines title "Black WR-gen", '' every 2::2 using 0:1 with lines title "White WR-gen"

It would be cooler hith Highcharts.js but I was too lazy for it (at least now.) So, almost done with one of the pieces of the 9x9 analysis suite (the other one I have kind of done is an opening move % classifier.)

[oca]

oh... pachi can be remoted via GTP... that seems nice... thx for your post ! I will give pachi a second look

[RBerenguel]

oh... pachi can be remoted via GTP... that seems nice... thx for your post ! I will give pachi a second look

Yup, it can and it's "relatively" straightforward. There's also a "shape finding" thing about it (don't remember exactly the name, but it outputs a lot of shape/pattern information that can then be cross-referenced... I think I interacted with it via the Acme text editor, so it was relatively easy to do) which is what I told you in the thread about naming moves, it's more or less what the gostyle app used.

I just updated the script, now it generates another output file that can then be parsed into an SGF (with another awk script) so the result is more "visual"

[RBerenguel]

Since editing the first post is a little cumbersome (it's very long) here's the updated script. Now it generates another file goratemv with has the format originalmove, eval-value, suggestedmove, pachi-result, coincide:

Code: Select all

BEGIN {
    total=4
    commands[1]=""
    move[1]=""
    engine="-e uct"
    plays="-t =3000"
    pachi_exec="./pachi"
    pachi=pachi_exec " " engine " " plays " -d 0 dynkomi=none"
    file="/tmp/gorate"
    file2="/tmp/goratemv"
    system("rm " file)
    system("rm " file2)
    pachi_exec "  -a 2>&1 | head -n 2 | tail -n 1\n" | getline
    printf "Using " plays " plays and " engine " engine.\n" $0 "\n\n"
    printf "For each move evaluate the board and generate a next\nmove. Evaluating the board and generating a move give\ndifferent values, even for the same move.\n\nGenerating also /tmp/gorate with format:\n Pwin generated, Evaluate value | -1 , Exact match move? (i.e. like 0.55, -1,  1)\n\n"
}

NR<=4{
    commands[NR]=$0
}

NR==3{
    move[1]=$3
}
NR==4{
    move[2]=$3
}

# Will process output from sgf2gtp, repeating
(NR>4){
    total=total+1
    mvc=NR-3
    commands[NR]=$0
    #printf commands[NR]
    move[mvc]=toupper($3)
    tomove[mvc]=$2
    #printf "Move: " move[NR] "\n"
}

END {
    for(i=1;i<4;i++){
	printf move[i] ", , , , -1" >> file2
    }
    for(j=7;j<=total;j++){ #Starts at 5 and evaluates starting at j=Move to study+3
	mvc=j-3
	count=count?0:1
	color="unknown"
	steps=0
	while (color=="unknown"&&steps<5){
	    steps=steps+1
	    for (i=1;i<=j-1;i++){
#		printf "DEBUG: Issuing command #"i": " commands[i] "\n"
		printf commands[i] "\n" |& pachi
	    }
#	    printf "DEBUG: Issuing command: evaluate " tomove[mvc] "\n"
	    printf "pachi-evaluate " tomove[mvc] "\n" |& pachi
#	    printf "DEBUG: Issuing command: genmove " tomove[mvc] "\n"
	    printf "genmove " tomove[mvc] "\n" |& pachi
#	    printf "DEBUG: Issuing command: pachi-result\n"
	    printf "pachi-result\n" |& pachi
#	    printf "DEBUG: Issuing command: quit\n"
	    printf "quit\n" |& pachi
	    countreply=0
	    score=""
	    while(pachi |& getline >0){
#		printf "DEBUG: " $0 "->" $1 " " $2 " " $3 "\n"
		if ($2!=""){
		    # format color move playouts pwin dynkomi for pachi-result
		    color=$2
		    genmove=$3
		    pwin=$5
		    if($4==""){
#			print "DEBUG: $4 is null: " $0 " " $1 " " $2
		    }
		    # For pachi-evaluate, move and pwin 
		    if(color==move[mvc]&&$3!=""){
#			printf "DEBUG: Match evaluate"
			score=genmove
		    }
		    if($1==move[mvc]){
#			printf "DEBUG: Match evaluate-2"
			score=$2
		    }		
		}
	    }
	    close(pachi)
	}
	printf pwin ", " >> file
	printf move[mvc] ", " >> file2
	if(score==""){
	    printf "-1, " >> file
	    printf "-1, " >> file2
	}
	else{
	    printf score ", " >> file
	    printf score ", " >> file2
	}
	printf genmove ", " pwin ", ">> file2
	if(move[mvc]==genmove){
	    printf " 1\n" >> file #Correct hit
	    printf " 1\n" >> file2 #Correct hit
	}
	else{
	    printf " 0\n" >> file #Wrong hit
	    printf " 0\n" >> file2 #Wrong hit
	}
	if(score!=""){
	    printf mvc ": Original " tomove[mvc] " move: " move[mvc] " has possible P(win) " score ". Suggested " color " move: " genmove " P(win): " pwin "\n"
	}else{
	    printf mvc ": Original " tomove[mvc] " move: " move[mvc] ". Suggested " color " move: " genmove " pwin: " pwin "\n"
	}

    }

    close(pachi)
}

And this script can take a file like that and convert it to an SGF you can follow. For now, komi, rules and players are missing, since there is no way to pass directly this data from the sgf2gtp script through the awk files. Next step (for me, at least) about these scripts is wrapping them between (probably) a simple shell script that can create all necessary files and information

Code: Select all

# (;GM[1]FF[4]CA[UTF-8]AP[CGoban:3]ST[2]
# RU[Japanese]SZ[9]KM[0.00]
# PW[test]PB[test]
# ;B[gc]CR[cc]C[Main move with circle for other considered move]
# ;W[gg]CR[cg]C[Main line follows with next move and letter])

BEGIN {
    printf "(;GM[1]FF[4]CA[UTF-8]AP[yhdeksan]SZ[9]"
    printf "RU[Japanese]KM[0.00]" # Fix later
    printf "PW[test]PB[test]\n" # Fix later
    color="W"
    FS=", "
    split("IHGFEDCBA", letters, "")
}

{
    color=color=="W"?"B":"W"
    split($1, gtpmove, "")
    failsafe=gtpmove[1]=="J"?"I":gtpmove[1]
    move= failsafe letters[gtpmove[2]]
    printf ";" color "[" tolower(move) "]"
    if($5!="-1"){
	if (!$5) {
	    split($3, gtpmove2, "")
	    failsafe=gtpmove2[1]=="J"?"I":gtpmove2[1]
	    genmove= failsafe letters[gtpmove2[2]]
	    
	    printf "CR[" tolower(genmove) "]"
	    comment="Main line move: " $2 "\n\nSuggested move: " $4
	}else{
	    comment="Main line move coincides with suggestion: " $4
	}
	
	printf "C[" comment "]\n"
    }
}

END {
    printf ")"
}

Attached a sample from a very lousy 9x9 game I lost in OGS recently (yeah, I played pretty horribly, didn't need a computer to tell me )

Edit: actually, better inlining it!

[oca]

I really, really like that idea...

Side note: pachi seems quite easy to remote from node.js

Code: Select all

var spawn = require('child_process').spawn;
var child = spawn("..\\bin\\pachi.exe ",["-e","uct","-t=2000"]);
var resp = [];
var current = 0;

child.stdout.on('data', function (data) {
    resp.push(data.toString().trim());
	onResponse(cmds[current],resp[current]);
	current ++;
	if (current == cmds.length) {
		child.stdin.end();
	}
});


function onResponse (req, resp) {
	console.log("req : "+req+" -> response : "+resp)
}

var cmds =  [
	"boardsize 9",
	"clear_board",
	"komi 7.5",
	"genmove black",
	"genmove white",
	"genmove black"
]

cmds.forEach(function(cmd) {
	child.stdin.write(cmd+"\n"); // execute the command
});

D:\my\go\pachi\oca>node test.js
req : boardsize 9 -> response : =
req : clear_board -> response : =
req : komi 7.5 -> response : =
req : genmove black -> response : = G7
req : genmove white -> response : = E7
req : genmove black -> response : = E5

D:\my\go\pachi\oca>

[daal]

What's up with move 6? The main line move and the suggested move have different values, but they are just mirrored. Kind of funny that pachi considers its own move better.

[RBerenguel]

What's up with move 6? The main line move and the suggested move have different values, but they are just mirrored. Kind of funny that pachi considers its own move better.

The values for "eval" and "generated" almost never coincide, even for the same move. Lovely Monte Carlo stuff

[RBerenguel]

This is not as good as I want, yet. I was going to create a neat Highcharts visualisation... Then I realised Highcharts is only free for NC use, and I need to use something similar to this for my job (similar=dynamic graphs from data with interactivity....) So... rolled up my sleeves and did some quick hack (very quick, very hackish) in D3, but I'm pretty much still a D3 newbie (I was somewhat proficient like 2 years ago, but then I discovered Highcharts and never looked back... until now, damn :/)

Animated gif follows between the hide tags (since it's quite wide.) Uses jgoboard and D3. Shows evals of moves, not suggestions. I need to think exactly WHAT I want to plot there. The idea in any case is having the plot of values and the game and be able to move around.

Sooner or later, this will include a beautiful button "pattern" that will look for the current board situation among all 9x9 games I have and let me check them from just within my browser.

test.gif (819.98 KiB) Viewed 7860 times

[Satorian]

I really like the SGF export idea. Would be brilliant to take a 19x19 game, leave Pachi running over night to give it 8-10 hours to analyze, and then review the results per move.

Hope you keep this project going!

[RBerenguel]

I really like the SGF export idea. Would be brilliant to take a 19x19 game, leave Pachi running over night to give it 8-10 hours to analyze, and then review the results per move.

Hope you keep this project going!

Supposedly pachi is not strong enough for it to be really, really meaningful. Of course, you can always rise the playout value to huge numbers and just let it smash the game tree, but for 9x9 it would already be overkill expecting it to be stronger than 5-7d amateur, much, much less for 19x19, where the playout time also increases by a lot.

There's also the playstyle: pachi (or any other MCTS-based engine) shines in tactical situations. In opening and endgame it can blunder greatly (or at least not follow standard patterns and just play its own, weird game.) So for a 19x19 game I'd restrict it for moves 50+, at the very least.

For 9x9 it's not that significant, after move 4-5 the game is already tactical.