It is currently Thu Oct 31, 2024 4:52 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 147 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
Author Message
Offline
 Post subject: Re: On handling online cheating with AI
Post #61 Posted: Tue Jun 02, 2020 8:25 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 8.

Click Here To Show Diagram Code
[go]$$B Move 131
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . X . O . . X X X . X O O . . . |
$$ | . . . X X O . X X O O O X X X O O O . |
$$ | . . X , O . O O O , O X . . O X X O a |
$$ | . . X O . X X . O O O X . . O . . X . |
$$ | . . X . . X O O . . . O X X . X X . . |
$$ | . X O X . X X . . . . O . . . . . . . |
$$ | . X O O O . . . . . . . . . . . . . . |
$$ | . X O . O X . . . . . . . . . . O . . |
$$ | . . X O O . . . . , . . . . . , . . . |
$$ | . . X . . . . . . . . . . . . . . . . |
$$ | . . X O . O . . . . . . . . . . O . . |
$$ | . . . O . . . . . . . . . . X . . . . |
$$ | . . . . . X . X X X . . . . X O . . . |
$$ | . . O . O X X O X O X X . . . O . . . |
$$ | . . . O . X O . O O O X . X O , . . . |
$$ | . . O X X X O O O . X O . X O X O . . |
$$ | . . O . . X O . O . O X X . X O . . . |
$$ | . . . . . . . . . . O . . X . b . . . |
$$ ---------------------------------------[/go]

Who played the hane, who played the atari?
Both are sente, OC.

The winrate difference is 0.1%. Elf's play got 57k rollouts, the human play got 433.

Elf played the atari.

For the record, here is Elf's main variation.

Click Here To Show Diagram Code
[go]$$B Moves 131 to 131
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . X . O . . X X X . X O O . . . |
$$ | . . . X X O . X X O O O X X X O O O . |
$$ | . . X , O . O O O , O X . . O X X O . |
$$ | . . X O . X X . O O O X . . O . . X . |
$$ | . . X . . X O O . . . O X X . X X . . |
$$ | . X O X . X X . . 9 . O . . . . . . . |
$$ | . X O O O . . . . . . . . . . . . . . |
$$ | . X O . O X . . . . . . . . . . O . . |
$$ | . . X O O . . . . , . . . . . 5 6 . . |
$$ | . . X . . . . . . . . . . . . . 8 . . |
$$ | . . X O . O . . . . . . . . . 7 O . . |
$$ | . . . O . . . . . . . . . . X . . . . |
$$ | . . . . . X . X X X . . . . X O . . . |
$$ | . . O . O X X O X O X X . . 3 O . . . |
$$ | . . . O . X O . O O O X . X O 4 . . . |
$$ | . . O X X X O O O . X O . X O X O . . |
$$ | . . O . . X O . O . O X X . X O 2 . . |
$$ | . . . . . . . . . . O . . X . 1 . . . |
$$ ---------------------------------------[/go]

Black plays kikashi against the bottom right corner and right side before attacking the White group on the top. The top right corner was not on Elf's radar.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #62 Posted: Tue Jun 02, 2020 8:28 am 
Judan

Posts: 6230
Liked others: 0
Was liked: 793
Adin,

"What a human investigator is looking for is mostly plays that are highly unusual for the rank of the player (or for any human players)."

Such an investigator would be prejudiced by his own incompetence. Some human players that do not cheat can and do (even frequently) play moves highly unusual for humans or for humans of certain ranks.

"And other stuff which I will not detail here in case cheaters are reading."

If "justice" relies on secrecy, it is oppression.


This post by RobertJasiek was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #63 Posted: Tue Jun 02, 2020 8:40 am 
Dies in gote

Posts: 28
Liked others: 0
Was liked: 2
Rank: 1 kyu
A few months ago I played a game on another server than KGS. I got crushed. Looking at opponent user info his latest about one hundred games were all wins. He had skyrocketed from a very stable 2k to 4d and showed no sign of stopping. I contacted the server administration and they won't do anything since there is no policy regarding AI cheating. And meanwhile instead of complaining or taking action people get into the most abstract philosophical discussions involving 0.5% win rate. Oh well.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #64 Posted: Tue Jun 02, 2020 8:45 am 
Oza
User avatar

Posts: 2420
Location: Ghent, Belgium
Liked others: 359
Was liked: 1020
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
6/8 so far;
heuristics:
- moves with more visual appeal are human; moves that require reading to understand what they could accomplish in the first place, are bot's
- ajikeshi is human; sente is bot's (assuming I can see the difference)
- development is human; stability (or undermining it) is bot's

If I had consistently applied 2 & 3, I'd be closer to 8/8

Eventually this kind of study would lead to "better move's are bot's" :) I won't get there I'm afraid, but a pro might. Which is the point of this thread.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #65 Posted: Tue Jun 02, 2020 8:46 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 9

Click Here To Show Diagram Code
[go]$$Wc Move 186
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . . . . . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O . |
$$ | . . X , O X O O O , O X . . O X X O X |
$$ | . . X O X X X . O O O X . . O . . X . |
$$ | . . X X O X O O . . . O X X X X X . . |
$$ | . X O X O X X . . X . O . . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X . . O O X . . . . X O X . |
$$ | . . X O O . . . . , X . . . . X X O O |
$$ | . . X . . . . . X . . . . . O O O X . |
$$ | . X X O . O . a . X . . b O . . O O . |
$$ | X O O O . . O . . . O . . . X . . . . |
$$ | . X . . . X X X X X O . O . X O . . . |
$$ | . . O . O X X O X O X X . . X O . . . |
$$ | . . . O . X O . O O O X . X O O . . . |
$$ | . . O X X X O O O O X . X X O . O . . |
$$ | . . O . . X O . O . O X X . X O O . . |
$$ | . . . . . . O . . . O X . X . X . . . |
$$ ---------------------------------------[/go]

Who played the peep, who played the narabi?

The winrate difference is 0.3%. Elf's play got 28k rollouts, the human's play got 97.

Elf played the narabi, with sente, as it threatens to cut off the three Black stones. Later in Elf's mainline, White played the peep with sente.

In the actual game, White played the peep with sente, but later, when it played the narabi, it was not sente.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #66 Posted: Tue Jun 02, 2020 8:50 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Adin wrote:
A few months ago I played a game on another server than KGS. I got crushed. Looking at opponent user info his latest about one hundred games were all wins. He had skyrocketed from a very stable 2k to 4d and showed no sign of stopping. I contacted the server administration and they won't do anything since there is no policy regarding AI cheating.


I'm with you on that. :)

Quote:
And meanwhile instead of complaining or taking action people get into the most abstract philosophical discussions involving 0.5% win rate. Oh well.


It's not philosophical on my part, it's scientific. If we are going to base the question of cheating with AI on whether plays look like AI plays or humans plays, we ought to have some idea what we are talking about.

Comparing plays with small winrate differences is a way to control for the quality of the plays. That's standard scientific practice.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #67 Posted: Tue Jun 02, 2020 9:14 am 
Dies in gote

Posts: 28
Liked others: 0
Was liked: 2
Rank: 1 kyu
Knotwilg wrote:
Which is the point of this thread.

The point of this thread is to hopefully make people realize that AI cheating in online is already a very real problem and it gets worse every day. And that action is very much needed but right now very little is done. We really need automated detection but I heard of nobody working on that.

Meanwhile admins and tournament organizers are usually afraid to take action even in obvious cases because of that 1% chance they might be wrong, which they certainly should accept and do take action! And the community of players instead of rallying against cheaters would rather endlessly and uselessly debate all the fine details and philosophical subtleties. There are many cases of blatant cheating that just get away with it.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #68 Posted: Tue Jun 02, 2020 9:20 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 10.

Click Here To Show Diagram Code
[go]$$Wc Move 212
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , O X O O O , O X . . O X X O . |
$$ | . . X O X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . . O X X X X X . X |
$$ | . X O X O X X . O X . O . . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . . . a X O X . |
$$ | . . X O O . . X . , X . . . O X X O O |
$$ | . . X . . . O . X . X . . . O O O X . |
$$ | . X X O . O . O X X O . O O . . O O . |
$$ | X O O O . . O . . . O . . . X X O . . |
$$ | . X b . . X X X X X O X O . X O . . . |
$$ | X O O . O X X O X O X X . . X O . . . |
$$ | . X . O . X O . O O O X . X O O . . . |
$$ | . O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . . . . . O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]

Who played atari against four stones, who made the ko?

The winrate difference is 0.4%. Elf's play got 23k rollouts, the human's play got 221.

Elf played the atari against 4 stones. Neither player made the other play any time soon.

Game continuation.

Click Here To Show Diagram Code
[go]$$Wcm 12
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , O X O O O , O X . . O X X O . |
$$ | . . X O X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . . O X X X X X . X |
$$ | . X O X O X X . O X . O . . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . . . . X O X . |
$$ | . . X O O . . X . , X . 3 . O X X O O |
$$ | . . X . . . O . X . X . . . O O O X . |
$$ | . X X O . O . O X X O . O O . . O O . |
$$ | X O O O . . O . . . O . . . X X O . . |
$$ | 2 X 1 . . X X X X X O X O . X O . . . |
$$ | X O O . O X X O X O X X . . X O . . . |
$$ | . X . O . X O . O O O X . X O O . . . |
$$ | . O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . . . . . O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]


After :b13: filled the ko, :w14: made an eye without the atari against the four stones.

Click Here To Show Diagram Code
[go]$$Wcm12 Elf's mainline
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , O X O O O , O X . . O X X O . |
$$ | . . X O X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . . O X X X X X . X |
$$ | . X O X O X X . O X . O . . . 2 . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . 5 . 1 X O X . |
$$ | . . X O O . . X . , X . 6 . O X X O O |
$$ | . . X . . . O . X . X . . . O O O X . |
$$ | . X X O . O . O X X O . O O . . O O . |
$$ | X O O O . . O . . . O 3 . . X X O . . |
$$ | . X . . . X X X X X O X O . X O . . . |
$$ | X O O . O X X O X O X X 4 . X O . . . |
$$ | . X . O . X O . O O O X . X O O . . . |
$$ | . O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . . . . . O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]


:w12: played the atari against four stones, and then :w14: took kikashi agains the Black group in the bottom right. Then :w16: was a different jump in the center, allowing :b17:. The ko was not on Elf's radar.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #69 Posted: Tue Jun 02, 2020 9:27 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Adin wrote:
Knotwilg wrote:
Which is the point of this thread.

The point of this thread is to hopefully make people realize that AI cheating in online is already a very real problem and it gets worse every day. And that action is very much needed but right now very little is done. We really need automated detection but I heard of nobody working on that.


It's not an easy task to find and validate an algorithm to do that.

Quote:
Meanwhile admins and tournament organizers are usually afraid to take action even in obvious cases because of that 1% chance they might be wrong, which they certainly should accept and do take action! And the community of players instead of rallying against cheaters would rather endlessly and uselessly debate all the fine details and philosophical subtleties.


If we are going to deal with online cheating we have to develop correct methods and procedures. What the hell do you think I am doing here? Just because you do not realize the difficulty of the task does not make the effort useless.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


This post by Bill Spight was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #70 Posted: Tue Jun 02, 2020 9:42 am 
Dies in gote

Posts: 28
Liked others: 0
Was liked: 2
Rank: 1 kyu
Bill Spight wrote:
What the hell do you think I am doing here?

I'd guess you're the only one who knows that. Perhaps proving why Reddit replaced traditional forums.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #71 Posted: Tue Jun 02, 2020 9:54 am 
Judan

Posts: 6230
Liked others: 0
Was liked: 793
"his latest about one hundred games were all wins. He had skyrocketed from a very stable 2k to 4d and showed no sign of stopping."

This might be a case of AI cheating or something else. A different possible case: He was 4d but a victim of the rating / ranking system and stuck at 2k. Then he got a series of opponents whom he could always beat with dusciplined play.

Without further investigation, one cannot know which case of explanation applies.

For reference, when stuck at a too low KGS rank (ca. 1 or 2 ranks below my supposed skill rank of ca. 50% wins), disciplined playing and expected to beat opponents yield winning series of 10 or 20 games. Those series stop when eventually I get tired after too many games on a day, lose my discipline out of frustration of too slowly changing rating, out of forgetting about the fast time setting (when underrated, a long time setting would be unbesrably frustrating), meet the exceptionslly motivated usually weaker but then very well playing opponent or meet the hard to beat opponent (e.g. another victim of the rating system). Otherwise, such series could last much longer.


This post by RobertJasiek was liked by: Harleqin
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #72 Posted: Tue Jun 02, 2020 10:44 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 11

Click Here To Show Diagram Code
[go]$$Wc Move 226
$$ ---------------------------------------
$$ | . . . . . . . . . . . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , O X O O O , O X . . O X X O . |
$$ | . . X O X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . O O X X X X X . X |
$$ | . X O X O X X . O X . O X . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . . . . X O X . |
$$ | . . X O O . . X X , X . O . O X X O O |
$$ | . . X . . . O O X . X . . . O O O X . |
$$ | . X X O . O . O X X O O O O . . O O . |
$$ | X O O O . . O . . . O X . . X X O . . |
$$ | X X O . X X X X X X O X O O X O . . . |
$$ | X O O . O X X O X O X X . X X O . . . |
$$ | X X O O . X O . O O O X . X O O . . . |
$$ | a O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . . b . . O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]


Who played the atari, who played the hane?

The winrate difference is 1.6%. Elf's play got 18k rollouts, and the human's play got 255.

Elf played the 1 pt. sente at a. Locally, it is plainly superior to the hane, which gains less than 1 pt., since Black has less than 1 pt. of territory at stake locally. In any event, the sente is not a mistake.

First, let's look at Elf's main variation.

Click Here To Show Diagram Code
[go]$$Wcm26 One pt. plays
$$ ---------------------------------------
$$ | . . . . . . . . 5 3 4 . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , W X O O O , O X . . O X X O . |
$$ | . . X W X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . O O X X X X X . X |
$$ | . X O X O X X . O X . O X . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . 6 . . X O X . |
$$ | . . X O O . . X X , X . O . O X X O O |
$$ | . . X . . . O O X . X . . . O O O X . |
$$ | 2 X X O . O . O X X O O O O . . O O . |
$$ | X O O O . . O . . . O X . . X X O . . |
$$ | X X O . X X X X X X O X O O X O . . . |
$$ | X O O . O X X O X O X X . X X O . . . |
$$ | X X O O . X O . O O O X . X O O . . . |
$$ | 1 O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . . . . . O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]

White takes the 1 pt. sente with :w26: and then takes the 1 pt. gote on the top side. Note that B 231 connects three Black dragons, so that Black does not have to capture the two :wc: stones at the cost of 1 pt.

Now let's look at the actual game.

Click Here To Show Diagram Code
[go]$$Wcm26 Actual game
$$ ---------------------------------------
$$ | . . . . . . . 5 4 6 . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , W X O O O , O X . . O X X O . |
$$ | . . X W X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . O O X X X X X . X |
$$ | . X O X O X X . O X . O X . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . . . . X O X . |
$$ | . . X O O . . X X , X . O . O X X O O |
$$ | . . X . . . O O X a X . . . O O O X . |
$$ | 8 X X O . O . O X X O O O O . . O O . |
$$ | X O O O . . O . . . O X . . X X O . . |
$$ | X X O . X X X X X X O X O O X O . . . |
$$ | X O O . O X X O X O X X . X X O . . . |
$$ | X X O O . X O . O O O X . X O O . . . |
$$ | 7 O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . 3 1 2 . O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]


After :w26: - :w28: in the bottom left, Black gets the 1 pt. play on the top side and the White takes the sente in the bottom left. This takes us up to the next example, so I will continue from there.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


Last edited by Bill Spight on Tue Jun 02, 2020 11:02 am, edited 1 time in total.
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #73 Posted: Tue Jun 02, 2020 10:48 am 
Gosei
User avatar

Posts: 1769
Liked others: 180
Was liked: 492
RobertJasiek wrote:
"his latest about one hundred games were all wins. He had skyrocketed from a very stable 2k to 4d and showed no sign of stopping."

(...)

For reference, when stuck at a too low KGS rank (ca. 1 or 2 ranks below my supposed skill rank of ca. 50% wins), disciplined playing and expected to beat opponents yield winning series of 10 or 20 games.

(...) such series could last much longer.


Adin didn't say on which server the player had a 100-game winning streak, but if it's on Fox, Tygem or IGS, winning 100 games in a row would result in being promoted or double-promoted many times, and I think this is almost impossible to achieve without cheating.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #74 Posted: Tue Jun 02, 2020 11:06 am 
Oza
User avatar

Posts: 2420
Location: Ghent, Belgium
Liked others: 359
Was liked: 1020
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
Adin wrote:
Bill Spight wrote:
What the hell do you think I am doing here?

I'd guess you're the only one who knows that. Perhaps proving why Reddit replaced traditional forums.


There have been very thoughtful responses to your question. No flaming, not hating. Bill has started a tangent to explore if individual moves can be easily found out as bot-copied. I must admit I find the tangent somewhat overtaking the main discussion but still trying to help out with it. It doesn't warrant the enmity with which you met it.

Both of you are helpful volunteers in the Go community. There's not enough of us to engage in heated debate - though I've been guilty of that myself.


This post by Knotwilg was liked by: gennan
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #75 Posted: Tue Jun 02, 2020 11:18 am 
Dies in gote

Posts: 28
Liked others: 0
Was liked: 2
Rank: 1 kyu
I'm sorry if I've been a bit blunt about it. But this belongs to another thread. Posting 12 messages with diagrams and then maybe getting responses to them really disrupts the main discussion.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #76 Posted: Tue Jun 02, 2020 11:19 am 
Gosei
User avatar

Posts: 1769
Liked others: 180
Was liked: 492
Just a note on Bill's problems: for almost every board position, I preferred the human move. Perhaps Bill's point is that in some situations, it may happen that moves A and B have similar winrates, but A is much more bot-like than B. This might be detectable by training a neural network on human games, but that's hard work.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #77 Posted: Tue Jun 02, 2020 11:42 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Example 12

Click Here To Show Diagram Code
[go]$$Wc Move 234
$$ ---------------------------------------
$$ | . . . . . . . O X X . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , O X O O O , O X . . O X X O . |
$$ | . . X O X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . O O X X X X X . X |
$$ | . X O X O X X . O X . O X . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . . . a X O X . |
$$ | . . X O O . . X X , X . O . O X X O O |
$$ | . . X . . . O O X . X . . . O O O X . |
$$ | X X X O . O . O X X O O O O . . O O . |
$$ | X O O O . . O . . . O X . . X X O . . |
$$ | X X O . X X X X X X O X O O X O . . . |
$$ | X O O . O X X O X O X X . X X O . . . |
$$ | X X O O . X O . O O O X . X O O . . . |
$$ | O O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . O O X b O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]

Who played the atari on four stones, who made the ko?

The winrate difference is 1.9%. Elf's play got 65k rollouts, and the human's play got 99.

Elf played the atari on 4 stones. It may not be a 1 pt. sente, but it is not too early to make this play. Besides, making the ko gains only ⅔ pt. locally, and there are bigger plays on the board.

Continuing the discussion from the last example.

Click Here To Show Diagram Code
[go]$$Wcm34 Actual game
$$ ---------------------------------------
$$ | . . . . . . . O X X . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , W X O O O , O X . . O X X O . |
$$ | . . X W X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . O O X X X X X . X |
$$ | . X O X O X X . O X . O X . . . . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . 4 . . X O X 2 |
$$ | . . X O O . . X X , X . O . O X X O O |
$$ | . . X . . . O O X a X . . . O O O X 3 |
$$ | X X X O . O . O X X O O O O . . O O . |
$$ | X O O O . . O . . . O X . . X X O . . |
$$ | X X O . X X X X X X O X O O X O . . . |
$$ | X O O . O X X O X O X X . X X O . . . |
$$ | X X O O . X O . O O O X . X O O . . . |
$$ | O O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . O O X 1 O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]

:w34: gains ⅔ pt. locally, but also takes away the potential eye there. If Black has to capture the :wc: stones, because there is no eye here, then this move gains more than ⅔ pt., and :w26: may have been OK, or even superior to the 1 pt. play on the top side.

However, :b35: takes his 1 pt. sente on the right side and then :b37: plays the hanging connection. So Black will not have to capture the :wc: stones. Note that Black also had a possible eye at a. A full analysis would take some time and effort. But it appears that :w34: gains less than 1 pt. and that White 226 was an error costing a fraction of a point. Whether that matters to the score is another question, OC. ;)

Click Here To Show Diagram Code
[go]$$Wcm34 Elf's mainline
$$ ---------------------------------------
$$ | . . . . . . . O X X . . . X X O . . . |
$$ | . . . . X . O . O X X X . X O O . O . |
$$ | . . . X X O O X X O O O X X X O O O O |
$$ | . . X , W X O O O , O X . . O X X O . |
$$ | . . X W X X X . O O O X . . O . . X O |
$$ | . . X X O X O O . . O O X X X X X . X |
$$ | . X O X O X X . O X . O X . . 2 . X . |
$$ | . X O O O . . . O X . X . . O X O X . |
$$ | . X O . O X X . O O X . 3 . 1 X O X . |
$$ | . . X O O . . X X 4 X . O . O X X O O |
$$ | . . X . . . O O X . X . . . O O O X . |
$$ | X X X O . O . O X X O O O O . . O O . |
$$ | X O O O . . O . . . O X . . X X O . . |
$$ | X X O . X X X X X X O X O O X O . . . |
$$ | X O O . O X X O X O X X . X X O . . . |
$$ | X X O O . X O . O O O X . X O O . . . |
$$ | O O O X X X O O O O X . X X O . O . . |
$$ | . . O X . X O . O . O X X . X O O . . |
$$ | . . O O X a O . . . O X . X X X O . . |
$$ ---------------------------------------[/go]

:w34: takes the sente against the four Black stones. Then :w36: increases White's territory and makes :b37: worth 1 pt. (If White plays the atari at a Black has to play at 37 to make an eye.) Once more it appears that White lost a fraction of a point in the game.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #78 Posted: Tue Jun 02, 2020 11:45 am 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
Adin wrote:
I'm sorry if I've been a bit blunt about it. But this belongs to another thread. Posting 12 messages with diagrams and then maybe getting responses to them really disrupts the main discussion.


Well, that's your opinion, and you are welcome to it. But, as I indicated, if matching the bot's moves is an indicator of cheating, then not matching them is relevant to the discussion. IMO, of course. :)

And the discussion of online cheating has a history, both here and in other forums. Several people seem to believe that matching some number of some bot's suggestions, in itself, is enough to decide the matter. But this is something that we actually know very little about. And people without a scientific background are unaware that confirmatory evidence, which is what matching is, is very weak. So instead of simply repeating once again that matching is very weak evidence, I decided to gather some scientific data. OC, it is not enough to settle anything, but it is a start. :)

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.


This post by Bill Spight was liked by: Bonobo
Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #79 Posted: Tue Jun 02, 2020 12:14 pm 
Honinbo

Posts: 10905
Liked others: 3651
Was liked: 3374
What about the data so far? It is next to nothing, OC, only 12 examples from a single game. (BTW, the game is GoGoD 1979-11-08a, Hashimoto Utaro, 9 dan, (W) vs. Ohira Shuzo, 9 dan. Ohira won by ½ pt.) I came up with criteria that allowed me to collect the data in an unbiased way: fewer than 500 rollouts for the human moves, with a winrate difference of 2.0% or less.

It is fairly easy to learn the preferences of bots in the opening. Most pros now play nearly perfect openings, according to the bots. And Elf did seem to play a more accurate endgame than Hashimoto. Other than that, my impression is that the bots like to play sente. In one example, both the peep and the narabi are sente, but Elf got in both of them in its mainline variation, while the human only got in one of them in the actual game. A number of people have already noticed that bots like to play sente, perhaps to a fault.

Elf calculates differently from other bots, so a winrate difference of 1% might be better with them. Still, I think that comparing the human play to a bot's top choice when the human play gets few rollouts but still has a small winrate difference is a good research method.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Top
 Profile  
 
Offline
 Post subject: Re: On handling online cheating with AI
Post #80 Posted: Tue Jun 02, 2020 12:14 pm 
Oza
User avatar

Posts: 2420
Location: Ghent, Belgium
Liked others: 359
Was liked: 1020
Rank: KGS 2d OGS 1d Fox 4d
KGS: Artevelde
OGS: Knotwilg
Online playing schedule: UTC 18:00 - 22:00
I got 10/12. In the last 4 examples I found it much easier to determine the better move and so I would only think the bot played the other move for some statistical win probability maximizing reason. But it didn't.

Now I admit, what's the point here? Me calling out the bot move 10/12 doesn't say a lot I guess. Suppose more people can do it, what does that prove? That this forum has good insight in what bots play? That this forum is strong on average? What kind of benchmark is this establishing?


This post by Knotwilg was liked by: Bill Spight
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 147 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group