How little or much can a single number express about relative player strengths?

1) The number can only describe what is measured. If tournament games are measured but non-tournament games are not measured, then it will not be noticed if different players are weaker or stronger in or outside tournaments. Same if some tournaments are but others are not measured.

2) Even if the numbers suggest a transitive ordering of the players because the numbers themselves have a transitive order, the players cannot be ordered transitively and linearly. This is so beacuse regular cyclical domination like A beats B beats C beats A occurs.

3) A multiple of a double round robin played within a reasonably short time (so that some players do not improve significantly yet) is the best tournament system for getting numbers: They are the number of wins. Double round robin ensures that each player gets each colour equally often. Even so, the numbers do not express a player's general strength but only the one under the particular tournament's conditions like komi, rules and thinking times.

4) A series of regularly occurring tournaments with the same conditions does not assess equally meaningful numbers for all players because some players will not be able to play all rounds or all occurrences of the tournaments. Rather some players' information will be more significant than others'.

5) A set of different tournaments in that all players of the same population play all games do not compare their strengths but only a partial aspect of their strengths because the tournaments will have a particular distribution of tournament conditions, which favour some of the players who play the strongest under them and which will be a disadvantage for others.

6) It is very hard to compare well players with (very) different numbers of played games.

7) Different tournament conditions cannot be expressed well by a single number. Rather every player can have different strengths under different conditions like being weak at lightning and strong at slow games or like being weak at 6.5 komi but strong at 7.5 komi or like being weak under area scoring but strong under territory scoring.

8) If numbers are determined by anything more complicated than simple number of wins, then (rather) arbitrary parameters (and system methods) are required and strengths depend also on how the ranking system is defined rather than only on how a player performs. To make things at least a bit fairer again, a space of various sets of parameters leading to a space of various ranking systems should be tested and expressed as a space of numbers per player rather than a single number.

9) Connecting different player populations (like from different countries) to each other is very difficult because a) relatively few inter-population games are played, b) different players play different numbers / percentages of games as inter-population games, c) tournament conditions in different populations differ, d) the measurement problems in one population occur also on the inter-population scale.

10) Theory of non-trivial ranking systems is being researched more than already well understood.

Conclusion: A world ranking (rating) system relying on a single determined number per player is nothing but an illusion.

Robert, 2 is a non-sequitor. A transitive ordering is just a type of logical relation. It requires an interpretation to know what is says about the players. You simply assume that it must say "player a will beat player b more than 50% of the time." But this is not the only thing you can mean when creating a rating system--nor is it what you should mean.

_________________
Occupy Babel!

It is not a question of whether the ranks are correct or accurate, but rather whether they make the fans spend more money.

_________________
Patience, grasshopper.

But you would agree, wouldn't you, that if a given ranking system says X is better player than Y, and Y is better player than Z, but Z beats X more often than not, that would be a distinctly odd result, right?

Personally, no, I wouldn't. It's pretty common actually. A ranking system for me should approximately order people by their overall results. Say we have 101 players who play every other player 100 times, totalling 10,000 games per player. I would expect the rating to prioritise people with an overall better record, but I could happily believe that 3rd lost to 5th in their vs games 35-65 or something. Some people just find some opponents harder than others.

There was a curious case, some years ago, of a British 4dan who was always just getting close to promotion to 5dan and he would lose to the same 2dan in tournaments. He actually lost 4 games in a row (the fifth was a jigo) before he got the measure of him and got promoted. Sometimes a person's style can be a real bug-bear.

Best wishes.

_________________
No aji, keshi, kifu or kikashi has been harmed in the compiling of this post.
http://www.gogod.co.uk

It sounds weird in the case of three players, but it's not at all an issue when you have a hundred--you will have cases of circular dominance. Of course if there were a circle involving one hundred players, you will not have effective ratings. Similarly, if most players alternate between 9 dan and 1kyu playing strength, you will have a great deal of trouble. But we're lucky--things don't turn out this way. (The same can be said for scientific practice in general. If nature is sufficiently uncooperative, you will never discover anything).

_________________
Occupy Babel!

I completely agree with the last point; intransitivity paradoxes are rarely a problem in the real world, at least not for practical purposes. I don't think we'd ever have difficulty assigning handicap stones, for example. But once you get beyond "practical purposes," what's the point of ranking players if transitivity doesn't hold, in general? We agree that if 1 in 3 rankings are intransitive, rankings are pointless; if 1 in 100 are instransitive, the rankings work quite well, with an asterisk. But people in general tend to underestimate the likelihood of transitivity difficulties.

(I should note that this is a separate, and less important, issue than the dimensionality problem I was discussing with Monadology earlier.)

We want ratings to tell us who is stronger than whom. Yet we have situations like this: Player A has a rating 100 points higher than player B yet over their many games together player B has forced A to a two stone handicap. Whether you say A is stronger than B or vice versa depends on your definition of stronger. There is no question that A has a higher rating, of course, but does that alone mean he is stronger?