Just an idea I had: Would it be doable to use a go playing program to estimate a player's playing strength?

As I see it, you would set up a monte carlo go program so that it would not be trying to win, but would be trying to keep the game close. (i.e. keep the estimated winning percentage close to 50%). The program would then keep track, on each of its moves, of how "suboptimal" (how much worse than the program's best move) that move was.

The higher the average "suboptimality" of the program, the weaker the opponent. Of course, you would need to find a way to translate suboptimality to an actual, real life, rank. This could perhaps be done by having the bot play against many players of know rank to find a distribution of suboptimality vs. playing strength.

This wouldn't work, of course for players stronger than the bot, but with the current strenght of MC programs, it would still be useful for a large percentage of the playing population. Also, for players stronger than the bot, you could perhaps do something with how quickly the estimated winning percentage dropped (e.g if it dropped to 10% after 100 moves, that player is stronger than if it dropped to 10% after 150 moves).

This could, if it works, also be an interesting way of ranking new players on a go server. It would, theoretically, only require one game to get at least a rough estimate, and additional games would refine that estimate.

This seems to be a bit of a stretch--players of equal strength may be quite different in their strength in the opening or middlegame.

The overall idea seems nice.

_________________
Occupy Babel!

I remember having seen on KGS remarks like "rank bot, get a rank here". Especially useful for players with a "?".

A big problem with chess bots is that they are too strong and for simulating lower level players they have to make artificial mistakes, which sometimes makes their play very strange. But for Go, we are lucky to have the handicap system.

_________________
If you say no, Elwood and I will come here for breakfast, lunch, and dinner every day of the week.

Not yet but soon?

A better way would be to have a continuously variable bot. But for this I think we need to wait for the bots to get just a bit stronger (why will soon become obvious) and also a bit weaker but still MC. As soon as you have a bot that can play at two different (well defined) levels both MC it should be possible to have it play anywhere in between by a "slider" controlling the probability that the stronger evaluator will be used or the weaker on each move.

That's rather like a human player who is weaker than the top level, Sometimes makes the best move but sometimes makes a move not as strong.

Doing this not going to be worth while till there is a bot offering two levels of MC play around 3-5 ranks apart (on the same machine, under the same time controls)

Searching for the move that wins half the time isn't a concept that can be applied recursively (just about every move wins 50% with random play!).

I think, instead, you'd have to modify the MCTS algorithm. Every iteration, for the root level only, explore the "best" option and the "most average" option (unless the two are the same move, in which case only explore it once). It might require some careful tuning to actually work that into a real algorithm, but it seems doable. You need to have comparable numbers of playouts on the "average" move and the "best" move, so that you can meaningfully compare them.

After a game, you should end up with a curve which would tell you not only how bad the player's average move is, but how likely the player is to make an especially good or terrible move. You'd have to get a lot of sample games vs. opponents of known strength to calibrate this.

Also, I expect games against such a bot would teach you horrible habits...

_________________
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com

no it won't

we currently have a bot that are strong as professionals in 9x9 game.
so using bot to estimate rank is very fisable idea.

_________________
"The more we think we know about
The greater the unknown"
Words by neil peart, music by geddy lee and alex lifeson

Thanks for weighing in. I'm glad I know what to think now.

glad to be of service

Having a bot that tries to keep the game 50/50 seems like a strange idea to me. Imagine that the user misses a tricky tsumego where the bot could kill a 100-point group with one tesuji. The bot will then not actually kill it, of course, because it is trying to keep the game close. So the position could stay on the board until the endgame, and at every move the "suboptimality" will be huge. In the end, the player might be rated 30k just for misreading this one tricky tsumego!

Also, what you would probably end up with is a bot that refuses to destroy the other player's thin shapes and that consistently plays timid, submissive moves. I might be wrong, but this seems like an odd idea to me...

_________________
My name is Gijs, from Utrecht, NL.

When in doubt, play the most aggressive move

What gaius just said is what I was thinking when I said it would be a horrible bot to learn from...

_________________
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com

If the tsumego is hard enough, the bot might miss it 9 moves out of 10 as well, so it would only penalize you for it on 10% of moves. As the tsumego gets easier, it'll find it more often, and you'll be penalized more severely, which is reasonable.

Of course the system would work better with more games, where one bad performance gets evened out.

The purpose of said bot would definitely not be to learn from it.