Has anyone done human/bot winrate correlation 'experiments'?

[Yakago]

I was wondering if anyone had examined how AI (leela/katago) evaluation correlates to human winrates at different levels.

What I am imagining is an SGF database of online amateur games with rank and game result information (and other things perhaps)

Then sample enough positions and analyze with an AI to see its evaluation.

Then we can ask questions like
"In positions where the AI evaluated one side to be favored 70-75%, how often did that side win the game?"
"Does this vary depending on rank of the players?"
etc..

[jann]

Note tho that bot evaluation is not even guaranteed to correlate to bot winrate (OC in practice it usually is).

But in theory an eval of say 70% not necessarily means 70% of hypothetical games won from the position. Even if it is a good estimate it could also mean a 70% probability of all games won from there (eg. if there is an unreadable fight or large capturing race going on, which will almost always be won by the same side but the bot cannot confidently read which side it is).

[Bill Spight]

Then we can ask questions like
"In positions where the AI evaluated one side to be favored 70-75%, how often did that side win the game?"
"Does this vary depending on rank of the players?"
etc..

There are people here who can correct me if I am wrong, but, AFAIK, such statistics do not even exist for a bot's own winrate estimates in self play.

[mhlepore]

My intuition tells me that in human games, being 85% at move 200 would yield a higher ultimate win % than being 85% at move 50.

The more moves that remain, the more humans can screw things up and lose games that are supposed to be winnable. But it would be nice to test with actual data.

[ez4u]

My intuition tells me that in human games, being 85% at move 200 would yield a higher ultimate win % than being 85% at move 50.

The more moves that remain, the more humans can screw things up and lose games that are supposed to be winnable. But it would be nice to test with actual data.

Realize that your intuition may be lying to you here. An 85% winrate at move 200 may be (probably is?) a half point lead. However, an 85% winrate at move 50 will be something quite different. Indeed katago's current 40-block calculates a winrate for Black of around 87% for the empty board with komi set to zero and "Japanese" rules.