It is currently Thu Oct 31, 2024 3:53 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 5 posts ] 
Author Message
Offline
 Post subject: Has anyone done human/bot winrate correlation 'experiments'?
Post #1 Posted: Mon Jun 01, 2020 12:16 pm 
Dies in gote

Posts: 53
Liked others: 2
Was liked: 12
I was wondering if anyone had examined how AI (leela/katago) evaluation correlates to human winrates at different levels.

What I am imagining is an SGF database of online amateur games with rank and game result information (and other things perhaps)

Then sample enough positions and analyze with an AI to see its evaluation.

Then we can ask questions like
"In positions where the AI evaluated one side to be favored 70-75%, how often did that side win the game?"
"Does this vary depending on rank of the players?"
etc..

Top
 Profile  
 
Offline
 Post subject: Re: Has anyone done human/bot winrate correlation 'experimen
Post #2 Posted: Mon Jun 01, 2020 1:04 pm 
Lives in gote

Posts: 445
Liked others: 0
Was liked: 37
Note tho that bot evaluation is not even guaranteed to correlate to bot winrate (OC in practice it usually is).

But in theory an eval of say 70% not necessarily means 70% of hypothetical games won from the position. Even if it is a good estimate it could also mean a 70% probability of all games won from there (eg. if there is an unreadable fight or large capturing race going on, which will almost always be won by the same side but the bot cannot confidently read which side it is).

Top
 Profile  
 
Offline
 Post subject: Re: Has anyone done human/bot winrate correlation 'experimen
Post #3 Posted: Mon Jun 01, 2020 1:21 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Yakago wrote:
Then we can ask questions like
"In positions where the AI evaluated one side to be favored 70-75%, how often did that side win the game?"
"Does this vary depending on rank of the players?"
etc..


There are people here who can correct me if I am wrong, but, AFAIK, such statistics do not even exist for a bot's own winrate estimates in self play.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: Has anyone done human/bot winrate correlation 'experimen
Post #4 Posted: Mon Jun 01, 2020 2:04 pm 
Lives in gote

Posts: 389
Liked others: 81
Was liked: 128
KGS: lepore
My intuition tells me that in human games, being 85% at move 200 would yield a higher ultimate win % than being 85% at move 50.

The more moves that remain, the more humans can screw things up and lose games that are supposed to be winnable. But it would be nice to test with actual data.

Top
 Profile  
 
Offline
 Post subject: Re: Has anyone done human/bot winrate correlation 'experimen
Post #5 Posted: Mon Jun 01, 2020 5:53 pm 
Oza
User avatar

Posts: 2408
Location: Tokyo, Japan
Liked others: 2346
Was liked: 1332
Rank: Jp 6 dan
KGS: ez4u
mhlepore wrote:
My intuition tells me that in human games, being 85% at move 200 would yield a higher ultimate win % than being 85% at move 50.

The more moves that remain, the more humans can screw things up and lose games that are supposed to be winnable. But it would be nice to test with actual data.

Realize that your intuition may be lying to you here. An 85% winrate at move 200 may be (probably is?) a half point lead. However, an 85% winrate at move 50 will be something quite different. Indeed katago's current 40-block calculates a winrate for Black of around 87% for the empty board with komi set to zero and "Japanese" rules.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group