Using a bot to estimate rank/rating

[HermanHiddema]

Just an idea I had: Would it be doable to use a go playing program to estimate a player's playing strength?

As I see it, you would set up a monte carlo go program so that it would not be trying to win, but would be trying to keep the game close. (i.e. keep the estimated winning percentage close to 50%). The program would then keep track, on each of its moves, of how "suboptimal" (how much worse than the program's best move) that move was.

The higher the average "suboptimality" of the program, the weaker the opponent. Of course, you would need to find a way to translate suboptimality to an actual, real life, rank. This could perhaps be done by having the bot play against many players of know rank to find a distribution of suboptimality vs. playing strength.

This wouldn't work, of course for players stronger than the bot, but with the current strenght of MC programs, it would still be useful for a large percentage of the playing population. Also, for players stronger than the bot, you could perhaps do something with how quickly the estimated winning percentage dropped (e.g if it dropped to 10% after 100 moves, that player is stronger than if it dropped to 10% after 150 moves).

This could, if it works, also be an interesting way of ranking new players on a go server. It would, theoretically, only require one game to get at least a rough estimate, and additional games would refine that estimate.

[hyperpape]

This wouldn't work, of course for players stronger than the bot, but with the current strenght of MC programs, it would still be useful for a large percentage of the playing population. Also, for players stronger than the bot, you could perhaps do something with how quickly the estimated winning percentage dropped (e.g if it dropped to 10% after 100 moves, that player is stronger than if it dropped to 10% after 150 moves).

This seems to be a bit of a stretch--players of equal strength may be quite different in their strength in the opening or middlegame.

The overall idea seems nice.

[entropi]

I remember having seen on KGS remarks like "rank bot, get a rank here". Especially useful for players with a "?".

A big problem with chess bots is that they are too strong and for simulating lower level players they have to make artificial mistakes, which sometimes makes their play very strange. But for Go, we are lucky to have the handicap system.

[Mike Novack]

Not yet but soon?

A better way would be to have a continuously variable bot. But for this I think we need to wait for the bots to get just a bit stronger (why will soon become obvious) and also a bit weaker but still MC. As soon as you have a bot that can play at two different (well defined) levels both MC it should be possible to have it play anywhere in between by a "slider" controlling the probability that the stronger evaluator will be used or the weaker on each move.

That's rather like a human player who is weaker than the top level, Sometimes makes the best move but sometimes makes a move not as strong.

Doing this not going to be worth while till there is a bot offering two levels of MC play around 3-5 ranks apart (on the same machine, under the same time controls)

[daniel_the_smith]

Searching for the move that wins half the time isn't a concept that can be applied recursively (just about every move wins 50% with random play!).

I think, instead, you'd have to modify the MCTS algorithm. Every iteration, for the root level only, explore the "best" option and the "most average" option (unless the two are the same move, in which case only explore it once). It might require some careful tuning to actually work that into a real algorithm, but it seems doable. You need to have comparable numbers of playouts on the "average" move and the "best" move, so that you can meaningfully compare them.

After a game, you should end up with a curve which would tell you not only how bad the player's average move is, but how likely the player is to make an especially good or terrible move. You'd have to get a lot of sample games vs. opponents of known strength to calibrate this.

Also, I expect games against such a bot would teach you horrible habits...

[xed_over]

Also, I expect games against such a bot would teach you horrible habits...

no it won't

[Magicwand]

we currently have a bot that are strong as professionals in 9x9 game.
so using bot to estimate rank is very fisable idea.

[jts]

Also, I expect games against such a bot would teach you horrible habits...

no it won't

Thanks for weighing in. I'm glad I know what to think now.

[xed_over]

Thanks for weighing in. I'm glad I know what to think now.

glad to be of service

[gaius]

Having a bot that tries to keep the game 50/50 seems like a strange idea to me. Imagine that the user misses a tricky tsumego where the bot could kill a 100-point group with one tesuji. The bot will then not actually kill it, of course, because it is trying to keep the game close. So the position could stay on the board until the endgame, and at every move the "suboptimality" will be huge. In the end, the player might be rated 30k just for misreading this one tricky tsumego!

Also, what you would probably end up with is a bot that refuses to destroy the other player's thin shapes and that consistently plays timid, submissive moves. I might be wrong, but this seems like an odd idea to me...

[daniel_the_smith]

What gaius just said is what I was thinking when I said it would be a horrible bot to learn from...

[HermanHiddema]

Having a bot that tries to keep the game 50/50 seems like a strange idea to me. Imagine that the user misses a tricky tsumego where the bot could kill a 100-point group with one tesuji. The bot will then not actually kill it, of course, because it is trying to keep the game close. So the position could stay on the board until the endgame, and at every move the "suboptimality" will be huge. In the end, the player might be rated 30k just for misreading this one tricky tsumego!

Also, what you would probably end up with is a bot that refuses to destroy the other player's thin shapes and that consistently plays timid, submissive moves. I might be wrong, but this seems like an odd idea to me...

If the tsumego is hard enough, the bot might miss it 9 moves out of 10 as well, so it would only penalize you for it on 10% of moves. As the tsumego gets easier, it'll find it more often, and you'll be penalized more severely, which is reasonable.

Of course the system would work better with more games, where one bad performance gets evened out.

[HermanHiddema]

What gaius just said is what I was thinking when I said it would be a horrible bot to learn from...

The purpose of said bot would definitely not be to learn from it.