topazg [76]:

while i am not completely decided which side of this argument i support, your claims based on the statistics are interesting, i shall look more into those systems proposed as a substitution for ELO (or GoR)

_________________
Spilling gasoline feels good.

I might be wrong, but probably not.

I can see Topazg's argument, but I think one pretty important factor has been omitted from the analysis. The chance for blunders by the 9p/1d/higher ranked player. It's much more likely for a 20k player to punish such a blunder in comparison to a random play bot. I don't know how much this would shift the odds in favour of the 20k player in comparison to the random bot, but I suspect the effect will be quite large because the only realistic way for 20k to beat 1d/9p will be from blunders. This is partly corroborated by my experience against much weaker players - my losses against them are almost always due to blunders.

In other words, I think the discussion so far has been about the chance of the 20k/random bot playing near 1d/9p strength instead of the chance for 1d/9p to play at the 20k level (if only for one crucial move) :p

Exactly. But a 20k player, in my experience, will NOT play any move on the board. They will only play 20k moves, and will frequently/usually not play the right moves.

I've just had a look for the source material that supports my argument, and it's gone from Jeff Sonas' Chessmetrics site There's still a fair bit on the 'net from him about the discussions over appropriate K values and fudge factors (FIDE decided to keep Elo in the end, but made a few adjustments to the way it worked), but not the full broad spectrum results data the showed such a clear trend line.

Much of his work now seems to be around standardising chess ratings over time so you can compare players from different eras, which whilst interesting, isn't the point of this discussion.

I corresponded with him a couple of times a while back, so if I can't find it with a bit of further digging, I'll send him an email. What we really need is actual statistics of serious even games of Go between widely varying strengths of players. Play 1000 swiss tournaments with anyone from 6d to 20k and you should get enough data for trend analysis to support one side or the other - we just need to organise those tournaments



I agree, and this is definitely an important point. I'm pretty sure that this is precisely why losses between 1d and 7k players happen. I think the course of a Go game is probably too long for a 20k to manage to secure a victory due to a 1d blunder or 5 (unless the 1d is hopelessly drunk or something), but I'm pretty convinced a 9p wouldn't make enough of the sorts of blunders required to lose to a 20k. The stronger the player, the fewer (and smaller) the blunders there are.

For the sake of argument, if we modelled Go to be a 0.5 win for White with perfect play on both sides, the argument still holds with a perfect-bot playing black and the chance of losing to a 20k or a random-bot.



I agree. I suspect it is precisely the blinkered tunnel vision, that partial knowledge and the attempt to apply it provide, that allows a random bot to occasionally succeed where the 20k can't. I have never met a player, even of mid dan strength, that doesn't concede that they frequently simply have blind spots to certain moves and ideas. Random bots have no blind spots.

Saying it fits the curve well is actually a pretty weak statement. One would need to know a lot about what tests were used, whether alternative curves were checked, and so on.

_________________
Occupy Babel!

Some of you might want to read this: http://lesswrong.com/lw/mp/0_and_1_are_ ... abilities/

It's not possible for a 20k to have a literally 0 chance of beating a 9p. We know 9p players occasionally keep the game close and occasionally self atari near the end. This is vastly more probable than random play producing a 1d game.

_________________
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com

Proof?

EDIT: Probabilities of 0 do exist aplenty. What's the probability of rolling a 7 on a standard 6-sided die?

It's on the same order as random play beating a 1d.

(You could be mistaken about what kind of die you're rolling, or many other similar weird cases.... You could even be hallucinating it.)

_________________
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com

Hahaha, oh come on, even you know that's close to trolling

I spoke to Jeff, and he said he'd get back to me on that data, but in the meantime, he's running the following:

http://www.kaggle.com/c/ChessRatings2

Apparently FIDE intend on considering adopting the winner (by which I suspect they mean not adopting) - perhaps the EGF could for Go?

EDIT: And an interesting quote attributed to Arpad Elo (interesting given the debate over rank deflation / inflation):

"The true challenge (of the rating system administrator) is maintenance of the integrity of the ratings in his pool, so that from one year to the next, or from one decade to the next, a given rating will represent essentially the same level of chess proficiency." Arpad Elo - "The Rating of Chessplayers, Past and Present" 1978.

To quote from that article:

"However, in the real world, when you roll a die, it doesn't literally have infinite certainty of coming up some number between 1 and 6. The die might land on its edge; or get struck by a meteor; or the Dark Lords of the Matrix might reach in and write "37" on one side."

This argument can quickly get into the ridiculous, and rarely does it not involve one side trolling the other. You can end up arguing that there is a probability of 1 that there are a negative number of possible outcomes from rolling said die, but seriously, what kind of theoretical abstractity is this?

We must make some sorts of abstraction from actual play--either that or get Robert to write rules about what happens when a player suffers a heart attack during the game. If you don't do that, then what are the odds the 9p resigns in disgust against the random bot?

_________________
Occupy Babel!

That's not a probability, that is a certainty. If we can enumerate all the possible outcomes, then it is easy to say what is and isn't possible.

To make this apply in this case, you would have to enumerate all possible games that a 20k and a 9p can play, an show that none of them are a victory for the 20k.

Apparently not according to Daniel's linked article

The linked article isn't relevant to the discussion. The author was consider Bayesian inference, where one has a prior estimate of something (e.g., chess skill) and then updates the estimate based on new information. The author points out that it would take an infinite amount of data to conclude that someone will beat someone else with probability 1. We aren't doing Bayesian statistics here, although that is what underlies most of the rating systems that are out there.

In our case the question originally posed seems to be estimating whether a random monkey can beat a top pro. Clearly the answer is yes; in principle the monkey could find a sequence of moves that would.

Then the question shifts to estimating a probability, which then brings people to compare to non-random monkeys (us for example). At this point you have to have a model for the play, and even then you can't come up with a definitive conclusion.

For example, we might model a biased random monkey, which selects better moves more often than the random monkey. Such a monkey would be able to beat the random monkey more often than not, but also has a non-zero probability of beating a pro. On the other hand, we could consider another type of monkey (or human) that never selects the worst 10% moves, but crucially never selects the top 3% moves or strategies either. Such a player would show up somewhere in a rating system as being better than the random monkey, but lacking optimal moves might never be able to beat a 9p.

If we model a player as actually considering all available moves and selecting one using a biased method, then such a player could beat a pro. If we model a real player as selecting good moves, but having a blind spot that somehow prevents moves from being considered then the probability of beating a pro might well be zero, even if the player is better than a monkey.

First, assume a spherical monkey in simple harmonic motion...

Or to simplify things (which I like doing), you could simply blindfold the person that the monkey happens to be playing. That could dramatically improve the chances of the monkey winning. Then you have two players playing essentially random moves.

I suppose this isn't an honest method, but nobody said it had to be.

_________________
Slava Ukraini!

And how do you make sure that the blindfolded person would play legal moves rather than just random moves.

_________________
http://tchan001.wordpress.com
A blog on Asian go books, go sightings, and interesting tidbits
Go is such a beautiful game.

OK, maybe I should avoid posting from my phone as it seems to make my posts terse and trollish sounding.

I do realize I'm being incredibly pedantic on the point of zero not being a probability, but I don't think I'm wrong. If you do bayesian updating, there's quite literally no amount of evidence that will cause you to ascribe 0 probability to a hypothesis. There's no amount of data that topazg can provide that will cause me to believe the true number is literally zero and not zero plus epsilon. (I'm prepared to accept that epsilon is arbitrarily tiny, though.)

Both of the cases we're talking about here have epsilon probability, we're just debating the relative sizes of the epsilons.

I accept the argument that the 20k can be better on the whole but still miss the top x% of moves required to beat a 9p. I have two reasons why it doesn't sway me: it's unclear whether 20ks actually function that way*, and it doesn't change the fact that the 9p can still have a brain fart; I believe there's a famous video of self-atari by one...

[*] I think it's something on the order of 90% likely that 20ks indeed cannot generate the top x% of moves. However, I can't believe that there's a 0.000000....00001 chance of a 20k beating a pro based on a proposition that I'm so relatively unsure about.

Anyway, I'm tired and have been working overtime so probably you should mostly ignore everything I'm saying.

_________________
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com

Another thought just occurred to me. To reliably reject the top 3% of moves, don't you have to have a pro-level move rejection module in your brain?

_________________
That which can be destroyed by the truth should be.
--
My (sadly neglected, but not forgotten) project: http://dailyjoseki.com

No, it doesn't require a pro-level move detection module, so long as the claim is that it isn't only the top 3% that get rejected. The 20k reliably rejects the top 3% of moves, the 12th-17th%, the 28-34th%.

Seriously, I think there are just going to be certain kinds of tenukis, certain kinds of tesuji moves that will never be seen by the 20k. There will occasionally be death-in-gote hallucinations that won't affect him, because he doesn't have the knowledge that I misapply in a game.

_________________
Occupy Babel!