NordicGoDojo wrote:
I have analysed a considerable number of games with KataGo looking at the size of a player's 'average mistake' – not in terms of winrate-%, but points, as KataGo is able to do. I think this method is more robust than looking at the winrate, because even a very small mistake can cause a big winrate change when the game is close.
Good point. In addition, while evaluation in terms of points only is theoretically problematic, because it ignores the value of having the move (sente), it is much closer to how humans evaluate positions than winrate estimates are.
Quote:
After analysing a few players, Ke Jie's average mistake seemed to be in the range of -0.5 points per move. For my own games, I got around -0.8 points; then for Shūsaku I got around -1.2 points, at which point I started to get suspicious. Then I checked a few European 6d players, who came to around -1.5 points; and then I came upon a game by Lukas Podpera 7d and Tanguy Le Calvé, which had an average mistake of only -0.3 points per move for both players.
Investigating further, I realised that the size of the average mistake depends on the 'nature' of the game: fighting-oriented games inevitable lead to higher average mistakes and peaceful games lead to lower average mistake. This is why Ke Jie's -0.5 points per move is impressive. Even if we analysed winrate-% rather than KataGo-points, I believe we would get the same conclusion.
Interesting. I think for this kind of approach we need to profile a large number of players, not just in terms of average or median mistakes, but of the whole distribution of errors. Furthermore, since the value of sente changes during the game, we should look at profiles of errors at different stages of the game, which might also reflect different styles of play.
Ideally, we would be able to come up with a finite number of profiles of honest play. Then not fitting any of those honest profiles would be evidence of possible cheating. I.e., you got some 'splainin' to do.