It is currently Thu Mar 28, 2024 11:03 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 75 posts ]  Go to page 1, 2, 3, 4  Next
Author Message
Offline
 Post subject: EGF Rating System Commission Report 2020
Post #1 Posted: Sat Jul 11, 2020 9:29 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
At the Annual General Meeting 2019 of the EGF, proposals were put forward by the Belgian, Dutch and British EGF members to modify the EGF rating system.
See items 13.a and 13.b in the Minutes of the AGM 2019.
In short, these were the proposals that were put forward:
- (B) Lower the rating floor of the EGD to 40k to accomodate weaker tournament players, such as novice youth players. The current rating floor at 20k is too high.
- (NL) Improve parameters of the system (amongst other things, to fix apparent long term rating deflation and to match expected winrates to observed winrates)
- (UK) Allow downward rating resets to accomodate players who clearly declined (possibly after a long hiatus from EGF tournament play).

Instead of voting for each of those proposals, a commission was formed to evaluate these proposals and present recommendations to the AGM 2020:
- Toby Manning (UK) represented the British Go Association and their proposal and he was the convenor of the commission.
- Ales Cieply (CZ), who created the Rating system about 20 years ago, was a commission member as an expert on the rating system.
- Dave de Vos (NL, this is me) represented the Dutch Go Association and their proposals, also he is an investigator of the rating system and a youth trainer.
- Michael Silcher (B) represented the Belgian Go Federation and their proposal, also he is a youth trainer.
- Wilhelm Buehler (D) was a commission member as an active local EGF tournament organiser.
- Geoff Kaniuk (UK) served the commission as an observer/advisor, being a long time investigator of the rating system and a rating officer of the BGA.

The interim EGD manager Lorenz Trippel provided the commission with historical data from the EGD.
Dave de Vos used this for statistical analysis and simulations to allow evaluation of the impact of several proposals.

After almost a year of discussion and investigation, this is the Commission Report with recommendations from the commission, which will be presented at the AGM 2020 to be held online on Tuesday 28th July 2020 at 19:00 CET.

Please feel free to comment, ask questions or criticise. I'll try to respond as best as I can.


This post by gennan was liked by 2 people: betterlife, Bonobo
Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #2 Posted: Sat Jul 11, 2020 6:20 pm 
Lives in sente
User avatar

Posts: 914
Liked others: 391
Was liked: 162
Rank: German 2 dan
A few questions, mostly out of curiosity:

Did you do something like a statistical analysis to determine some measurement for how “good” the system is? “Good” might be something like how well it predicts new results.

Did you consider using a radically different system, like WHR (whole history rating)?

_________________
A good system naturally covers all corner cases without further effort.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #3 Posted: Sat Jul 11, 2020 7:55 pm 
Lives in gote

Posts: 653
Location: Austin, Texas, USA
Liked others: 54
Was liked: 216
Quoting the changes to the math:

Quote:
We now have a lot more data (from 20 years of results). This has been examined in detail and we are proposing some amendments.

The reasons for amending these parameters are:.

* To correct rating deflation (change and increase the bonus per game)
* To fit winrate expectations to observed winrates (modify the parameter "a")
* improve the mathematical model of rating updates (switch from Elo model to Bradley-Terry model)
* decrease rating volatility of low ratings (decrease the parameter "con" and remove the loss limit as it becomes mostly redundant as a result).

We have analysed all this data and we recommend a change to the parameters. Our interim recommendations are as follows:

* correct deflation: bonus = 0.0017 * (2900 - rating) / (1 + exp[(rating - 2150) / 200]) - 0.3)
* rating’ = rating + con * (Sa - Se) + bonus
* winrate fitting: a = (3200 - rating) / 6
* Use the Bradley-Terry model: Se = 1 / (1 + exp[B(rating2) - B(rating1)])
where B = -6 * ln(3200 - rating), which is the integral of 1/a ]
* Volatility reduction: con = 0.13 * a


So they did do some analysis of the data.

Side question for my curiosity: I thought the Elo model and the Bradley-Terry model are mathematically equivalent, except for the scale. The scale change can be seen in the parameter "B". Am I wrong? If I'm right I think the point of using BT is to be simpler and less confusing, getting rid of the traditional base 10 exponent and 400 divisor, which makes less sense with all the surrounding changes such as the winrate scale fit parameter etc. Is that right?

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #4 Posted: Sat Jul 11, 2020 8:06 pm 
Lives in sente

Posts: 757
Liked others: 114
Was liked: 916
Rank: maybe 2d
Yeah, as far as I know Bradley-Terry-like models and classical Elo-like models are isomorphic. Classical Elo ratings (i.e. those defined by logistic winning odds differences which are generally used across a wide variety of games, not the weird Go-specific ones that are defined by stone differences) are simply the logarithms of Bradley-Terry gammas, with a certain scaling.


This post by lightvector was liked by: dfan
Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #5 Posted: Sun Jul 12, 2020 1:40 am 
Lives in gote

Posts: 309
Liked others: 3
Was liked: 41
Rank: 5 dan
When the EGF rating system was implemented, its purpose was to enable defining the super group for the European Championship. As far as I know, the purpose has not been redefined. If any changes are to be made, the purpose should propably get redefined.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #6 Posted: Sun Jul 12, 2020 1:57 am 
Gosei

Posts: 1494
Liked others: 111
Was liked: 315
I think one key point should be agreed by the EGF members. If you are going to use the EGF database, everyone should agree how to use it. That means that EGF resets are done in the same manner, entry grades are done in the same manner, etc. Then as Matti says, you can define the purpose of the EGF rating system as being to define the ratings of all players belonging to EGF member federations.

Maybe you can get rid of the Club Country code confusion as well. The ratings database is used to count membership numbers per country, but this is not correct. A quick suggestion: Club, Country (to which club belongs), Nationality (primary).

_________________
North Lecale

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #7 Posted: Sun Jul 12, 2020 3:02 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Harleqin wrote:
Did you do something like a statistical analysis to determine some measurement for how “good” the system is? “Good” might be something like how well it predicts new results.

Yes, all kind of statistics were extracted from the EGD hisorical data to analyse the current system and allow comparison with different modified systems.
Many of those group data by rating classes or declared grades as well as by time periods.
The results are tables on which diargrams can be made to show if and how those indicators evolve over time.
For example:
- Number of players
- Number of games
- Difference between declared grade and rating grade
- Rating change per game
- Standard deviation of ratings
- Elo points per rating grade (using logistic regression on game results)
- Promotions
- Demotions
- Rating resets
- Newcomers
- ...

Harleqin wrote:
Did you consider using a radically different system, like WHR (whole history rating)?

Not really.

It's not really clear what the benefits would be and if the EGD really needs those.
WHR and Glicko2 still compute regular Elo ratings, so that still leaves the question of how to convert between Elo ratings and GoR ratings (winrates <=> rating grades).
WHR and Glicko2 don't have any intrinsic support for handicap, which is the basis of grades in go. Handicap grades are the basis for the EGD by defining 100 GoR per grade and per full handicap stone.

One important feature which the EGD has and which those systems don't have, is to take declared grades of newcomers and rating resets at face value. But these grades often come from real life club grades and this helps a lot to keep the EGD rating grades fairly well aligned to real life grades (although nowadays more newcomers come from online play and it's not really clear how this affects the accuracy of declared grades in the future).

Preliminary statistics from handicap games in the EGD indicate that the distance between EGD grades is quite consistent with handicaps. The amount of handicap games in the EGD is a bit low for accurate handicap statistics, but the average error doesn't seem to be much more than 1 grade. So as far as I can tell, the overall grade distribution in the EGD seems to be quite good and holding up quite well over the years. But investigation in this area is still ongoing.

Also, a disadvantage of a more sophisticated system like WHR or Glicko2 is that the system would become a black box. You won't be able to calculate by hand anymore how your rating will change from a game result.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #8 Posted: Sun Jul 12, 2020 4:38 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
yoyoma wrote:
Side question for my curiosity: I thought the Elo model and the Bradley-Terry model are mathematically equivalent, except for the scale. The scale change can be seen in the parameter "B". Am I wrong? If I'm right I think the point of using BT is to be simpler and less confusing, getting rid of the traditional base 10 exponent and 400 divisor, which makes less sense with all the surrounding changes such as the winrate scale fit parameter etc. Is that right?
lightvector wrote:
Yeah, as far as I know Bradley-Terry-like models and classical Elo-like models are isomorphic. Classical Elo ratings (i.e. those defined by logistic winning odds differences which are generally used across a wide variety of games, not the weird Go-specific ones that are defined by stone differences) are simply the logarithms of Bradley-Terry gammas, with a certain scaling.

The Bradley-Terry model has this relation for the probability p that player A beats player B: logit(p) = BA - BB.

The Elo formula is a special case when B is a linear function of rating: B = R * ln(10) / 400 + C, where R is the Elo rating and C is an arbitrary constant.
In that case, the Bradley-Terry formula collapses to logit(p) = (RA - RB) * ln(10) / 400.
From there you can derive the Elo fomula p = 1 / (1 + 10((RA - RB)/400)).

But GoR doesn't have a linear relationship between B and rating, so it's not really proper to use the Elo formula.


This post by gennan was liked by: dfan
Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #9 Posted: Sun Jul 12, 2020 4:46 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Matti wrote:
When the EGF rating system was implemented, its purpose was to enable defining the super group for the European Championship. As far as I know, the purpose has not been redefined. If any changes are to be made, the purpose should propably get redefined.

I didn't know that and it didn't come up in the commission.
But if that was still the sole purpose of the EGD, why does it include DDK ratings (which are also referenced more and more over the years by players as well as tournament organisers)?

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #10 Posted: Sun Jul 12, 2020 5:09 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Javaness2 wrote:
I think one key point should be agreed by the EGF members. If you are going to use the EGF database, everyone should agree how to use it. That means that EGF resets are done in the same manner, entry grades are done in the same manner, etc.

There is no agreed upon set of policies in this regard. Every EGF member has its own policies. Personally I consider the EGD more as an EGF service to its members, that each member is free to use.
Unifying policies of different EGF members would have been a very daunting task, as every EGF member would probably want to keep their policies in place as they consider it the best. I think the EGF is not as powerful an entity as the FIFA of FIDE which can enforce very specific policies upon their members.
Anyway, I think policy unification was beyond the scope of this commission.
I think such a commission would have to contain different people and more representatives from different countries. And I expect it would take a long time to reach an agreement (if that's even possible). I haven't heard of any proposals or attemps made towards this in the past. Does any of you know?

Javaness2 wrote:
Then as Matti says, you can define the purpose of the EGF rating system as being to define the ratings of all players belonging to EGF member federations.

I think de facto this has become the main purpose of the EGD (even if there never was an offical statement).

Javaness2 wrote:
Maybe you can get rid of the Club Country code confusion as well. The ratings database is used to count membership numbers per country, but this is not correct. A quick suggestion: Club, Country (to which club belongs), Nationality (primary).

Personally, I have never heard about confusion related to the country code. I can imagine that the country code usually means the nationality of the club and not the nationality of the person. But why is that a problem?
Anyway, this topic didn't come up in the commission.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #11 Posted: Sun Jul 12, 2020 5:54 am 
Dies in gote

Posts: 26
Location: Karlsruhe/Germany
Liked others: 1
Was liked: 6
Javaness2 wrote:
Maybe you can get rid of the Club Country code confusion as well. The ratings database is used to count membership numbers per country, but this is not correct. A quick suggestion: Club, Country (to which club belongs), Nationality (primary).



The nationality of a player is normaly not asked at a weekend tournament - it's only relevant for national championships.

The Country code is always the nation, where the club is located, it has no effect on the rating.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #12 Posted: Sun Jul 12, 2020 6:38 am 
Gosei

Posts: 1494
Liked others: 111
Was liked: 315
Nobody expects the country code to have an effect upon rating, but if it is used to wrongly decide things about membership fees, or pair go representation, that's not a great thing. I accept that it isn't a terrible thing, because the numbers are normally almost the same.

_________________
North Lecale

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #13 Posted: Sun Jul 12, 2020 8:49 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
Javaness2 wrote:
Nobody expects the country code to have an effect upon rating, but if it is used to wrongly decide things about membership fees, or pair go representation, that's not a great thing. I accept that it isn't a terrible thing, because the numbers are normally almost the same.

I'm no expert in these matters, but this is what I know:

The EGF is really quite a different organisation than the AGA.
The members of the EGF are national go associations, not individual players.

Inclusion of individual players in the EGD has no relation to any membership of those individual players. Participating in a European tournament is all that is needed to be included in the EGD. Being a member of a national association may or may not be a requirement for participation (in the Netherlands it mostly isn't).

The EGF members (national go associations) pay membership fees to the EGF. Individual European go players only pay a membership fee to the national go associations. National associations don't report this information to the EGF on the level of names of those individuals.
EGF members only report the total number of members they have in each year (this is not determined by the number of EGD players with that country code).
The EGF membership fee depends on this reported number and it is also used to determine the number of votes of each EGF member representative in votings of the AGM (as shown on page 1 and 2 of the Minutes of the AGM 2019). So this can change from one year to another.
EGF members that don't pay their EGF membership fee, forfeit their right to vote in the AGM.

National championships (including pair go championships and such) and subsequent delegation to World championships are mostly a matter of national go associations. The EGF is not directly involved in this AFAIK. The EGF is mostly involved in European championships and European pro tournaments.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #14 Posted: Sun Jul 12, 2020 10:12 am 
Gosei

Posts: 1494
Liked others: 111
Was liked: 315
Apologies for going down this little side road. For the Pair Go check https://www.eurogofed.org/egf/pairsystem.htm
pairgorules wrote:
(ii) Twice the number of players of that country in the last issue of the full European rating list valid on 7th December of the previous year with Go Rating at least 1000. The nationality of a player on the rating list is decided on by the rating list maintainer. Every player on the rating list can have only one nationality. Mistakes or gaps in the rating data can only be corrected if they are reported to the rating list maintainer by 31st December of the previous year.
Maybe a better way to solve this is to change the rules for Pair Go selection so that countries are selected on the basis of a combination of how much pair go they play, and how well they play pair go.

I am a bit out of touch on the membership figures. I know Germany has protested for the last few years that various countries were not paying their correct membership fee. From memory, the 200 minimum fee is only allowed if the EGF membership list says it is okay for them to pay that.

Thinking about it a bit more, isn't it the case that IGF membership rates are also set from the EGF list - I think you can find the applicable rules for that in the EGF AGM minutes archive. (Try circa 2014)

I don't want to try to exclude people from having ratings, I'm simply concerned/perplexed that the database is (still) used to make decisions based on a dodgy assumption in the Country field.

gennan wrote:
I'm no expert in these matters, but this is what I know:

The EGF is really quite a different organisation than the AGA.
The members of the EGF are national go associations, not individual players.

Inclusion of individual players in the EGD has no relation to any membership of those individual players. Participating in a European tournament is all that is needed to be included in the EGD. Being a member of a national association may or may not be a requirement for participation (in the Netherlands it mostly isn't).

The EGF members (national go associations) pay membership fees to the EGF. Individual European go players only pay a membership fee to the national go associations. National associations don't report this information to the EGF on the level of names of those individuals.
EGF members only report the total number of members they have in each year (this is not determined by the number of EGD players with that country code).
The EGF membership fee depends on this reported number and it is also used to determine the number of votes of each EGF member representative in votings of the AGM (as shown on page 1 and 2 of the Minutes of the AGM 2019). So this can change from one year to another.
EGF members that don't pay their EGF membership fee, forfeit their right to vote in the AGM.

National championships (including pair go championships and such) and subsequent delegation to World championships are mostly a matter of national go associations. The EGF is not directly involved in this AFAIK. The EGF is mostly involved in European championships and European pro tournaments.

_________________
North Lecale

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #15 Posted: Sun Jul 12, 2020 10:47 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
@Javaness: Sorry, I didn't know about those EGF pair go rules and the significance of the country code in the EGD.
But perhaps it's something to address in a different topic in the EGF subforum, because it's outside the investigations and recommendations of the commission and I know very little about this.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #16 Posted: Sun Jul 12, 2020 1:08 pm 
Dies in gote
User avatar

Posts: 59
Location: Zürich, Switzerland
Liked others: 162
Was liked: 15
Rank: EGF 1 Dan
Universal go server handle: betterlife
gennan wrote:
@Javaness: Sorry, I didn't know about those EGF pair go rules and the significance of the country code in the EGD.
But perhaps it's something to address in a different topic in the EGF subforum, because it's outside the investigations and recommendations of the commission and I know very little about this.

These pair go rules were outdated the day they were made. IMO they are a good example on how to not make rules by defining details which can never work for longer time or would have to be adapted to the reality all the time... all the same in core they do what they should, give some guideline on how to handle the selection process for the international amateur pairgo championship in Tokyo!
The same counts for the super group at the open european championship and actually it's not only there where the EGD ratings are taken into account, it happens like this in a dozen of other occasions, like for example the grand slam qualification.
I agree the EGD itself doesn't say much about it's purpose. There is some info on https://europeangodatabase.eu/EGD/About_EGD.php, it's far from complete. Me personally as manager of the EGD, i would be happy to have a clear goal and purpose, a more detailed description which for example would also help to make decisions on what to do when the calculation aglorithm changes! It should be clear if the EGD wants to preserve old ratings or if it always retrospectivley recalculates everything from the start in 1996.


This post by betterlife was liked by: gennan
Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #17 Posted: Sun Jul 12, 2020 1:40 pm 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
betterlife wrote:
... a more detailed description which for example would also help to make decisions on what to do when the calculation aglorithm changes! It should be clear if the EGD wants to preserve old ratings or if it always retrospectivley recalculates everything from the start in 1996.

This is one of topics where the commission couldn't reach a concensus for a recommendation.

- Preserving old ratings means that it will take a long time before changes to the algorithm will having a noticable effect (including the change to counter deflation).
- Recalculating all the ratings from the start in 1996 will mean that everybody's rating (and rating history) will change suddenly.

So both rating preservation and rating recalculation have disadvantages and we couldn't reach agreement about which is the lesser evil.
So we'll be delegating this decision to the AGM without a recommendation from the commission.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #18 Posted: Tue Jul 14, 2020 3:04 am 
Dies with sente

Posts: 111
Liked others: 9
Was liked: 23
In general these seem to be acceptable recommendations.

Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #19 Posted: Fri Jul 17, 2020 6:04 am 
Lives in gote

Posts: 476
Location: Netherlands
Liked others: 270
Was liked: 147
Rank: EGF 3d
Universal go server handle: gennan
It was mentioned that the formulae cited in the commission report don't help much to judge the impact of those modifications.
So here is a hypothetical rating list with EGD ratings recalculated (from 1996-01-01 until 2020-03-15) by the current formulae: http://goratings.eu/RatingList

That list also reflects the commission recommendations to have the Entry Grade floor and the rating floor at 30k.

Some formulae have been updated after the report was made (work on those is still continueing).
Code:
a = (3300 - rating) / 7
  and conversely B = -7 * ln(3300 - rating)
con = ((3300 - rating) / 200)^1.6
bonus = ln(1 + exp((2300 - rating) / 80)) / 5

The Grade column shows the Entry Grade of players and the RGrade column shows the Grade computer from the rating.


This post by gennan was liked by: Bonobo
Top
 Profile  
 
Offline
 Post subject: Re: EGF Rating System Commission Report 2020
Post #20 Posted: Thu Jul 23, 2020 3:55 pm 
Lives in gote

Posts: 418
Liked others: 9
Was liked: 83
Rank: kgs 5 kyu
KGS: Pio2001
Hi,
Thank you for the information and all the hard work.

Something that would need to be stated is if there is a grade limit. Some say that the highest possible grade is 7 dan. Other say that it is 8 dan. Other say it's 9 dan.
The page about the rating system (https://www.europeangodatabase.eu/EGD/E ... system.php) doesn't tell about any limit.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 75 posts ]  Go to page 1, 2, 3, 4  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group