All positions are Black to play. The last play is marked.

First position:




Second position.



Position 3


Position 4


Position 5


Position 6



Edit: Added the times for White's previous plays, in response to Javaness2.

Edit: Added choices and % losses from Ales Cieply's analysis with Leela 11.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Position 7


Position 8



Edit: Added plays and percentages from the analysis by Cieply's Leela 11.

Position 9


Position 10


Position 11


Position 12


----
No more positions. These 12 positions include all 10 which took Carlo 49 seconds or more, plus a couple of others. OC, we don't know why they took so long, but Botvinnik's idea that they were relatively difficult for him is a good working hypothesis.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Just to state the obvious, the kgb metric is fine up to a point. It would be nicer to have not just the time B spend on Move X but also the time W spend on the previous move (or moves if they were forced). What we can't tell is when B or W left the board to go to the bathroom, or nipped off to the kitchen to get a biscuit, or called up a professional on the telephone. As we don't know about such behaviour, we have to be careful about the kgb.

_________________
North Lecale

Bill,
nice topic for analysis.

I have to write again regarding time, since it is online game, we don't know what players were doing.
You might say that he was thinking for some time, but he could be reading news like I do, getting up, phone, etc.

Same goes for fast plays, it does not necessary mean that he didn't think on it.
If you run Leela in analysis mode (open empty sgf and start from beginning), you can go forward some moves. Also Leela will show you next moves on the bar, or in analysis window. Therefore, if opponent spent even little time thinking (10+ s), next moves could already be prepared.

During analysis of important moves I did almost similar as alleged cheater would do.

----

And finally, trying to go move by move and think would it be possible that 4d+ would play them is not good enough.
He could play most of them, but we have to take a look at bigger picture.
Main question is overall similarity of play to Leela, in suspicious and live games.

In 2 supsicious games, there is several important tenukis. He played all of them top Leela's choice (apart from one forcing sequence).
Theoretically, he could do it.
But how he could not hit one in WAGC games?

And finally, how all of his moves in 2 games were within 2% of Leela's top choice?
Human players rarely can do it, in short fighting games. Not in long games.

I think it is really worthwhile to pursue this approach. However, in the Italian appeal they demonstrate that Leela is not quite deterministic when deciding on its top move (or sometimes even on whether the move is among the top 3). Thus, my first question to your presentation would be - did you check whether in these chosen positions Leela "always" suggest the same move? I would run Leela let's say 5 times on each chosen position and with various playouts settings. We do not know for sure what computer CM used. We can just assume the information provided by the Italians (about 100k nodes in 30s) is OK. When playing with Leela I found that the estimated winrates are more settled than the Leela's top move suggestions, which also led me to choosing the type of analysis based on delta histograms rather than looking at selected significant moves. Both approaches have their merits and flaws ...

That's why I said what Frejlak's Leela said and what Bojanic's Leela said. Bojanic's rsgf file also gives win rate estimates.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Good to know.

Looking at the rsgf files, it seems to me that the variability in the estimated win rates crucially depends upon the number of rollouts. (Big Duh! ) So to use Leela for evaluation, not for play, I would want to have the same number of rollouts for each possible choice. Preferably at least 10,000 each.

Edit: Today's top bots are optimized for play, not for evaluation.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

Interesting idea.



Claro que si.



Ditto.



Ditto.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

It is not good enough to demonstrate innocence. But that is not where the burden of proof lies.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

For moves I considered important, I watched how analysis progressed, and noted when some choice first appeared.

There is so much to say about this analysis, so I am going to make some renarks in some random order.

1. We do know the specifications of that computer: unless you have reasons to think somebody is lying which I find disturbing. I even find it disturbing that it is suggested not knowing something that is known.

Anyway is has an NVIDIA GTX 960M, which is the dominant factor in speed. CPU and memory are far less important. The speed gain over the CPU is a factor of at least 10.

2. The time taken to move contains a wealth of information that has not been used in a thorough analysis. In the Italian appeal only the weaknesses of the original "98% method" were shown, but they failed to provide additional evidence that goes against the allegation of cheating. A statistical analysus can show that both players speeded up, resp. slowed down in the same periods: i.e. they could both experience the same situations as difficult or easy. This is IMO strong evidence of not cheating, because it is very difficult to simulate that behaviour.

3. It is an error to use Leela 0.11 to evaluate a situation when you investigate cheating with the same program because that will easily lead to circular reasoning. The win rate given by Leela 0.11 are very different from AQ (which existed at the time of the game) and from Leela Zero and Leela ELF.

4. Position 5: black 51 was played in 2 seconds, too fast to consult a computer. You could object that he could have peeked earlier, but both players were playing at a rather high speed in this sequence, so there was very little time to use an AI. Furthermore if you would use the Leela application in analysis mode it does NOT suggest this move until after white 50. In other words, there is firm evidence that B could not have cheated to play this move.

5. Position 6: Black 71 @ J9. Here we see the limits of the evaluation function of Leela 0.11, because it does not "see" the whole forced sequence that follows, where the difference increases from 2 to 10%. AQ sees a 8% difference immediately. So that classifies White 70 as a very bad move.

6. Edit again: sorry - I made an error in the move number.

thanks.

The following diagram shows AQ's evaluation after Blacks move. This diagram shows how Blacks "winning %" changes after his answer to Whites moves. So if it jumps up- White's move was not so good and Black replied took advantage from that.

The diagram is from 11-04 this year, when AQ was sill about as strong as LeelaZeo and stronger than Leela 0.11

It is wrong to use Leela in analysis of case where it is suspected that it was used?
Interesting reasoning. What is next, to use some random player's games instead of Metta's?


Analysis window of Leela.
Please note that on bottom bar before W50 was played you can see L17 suggestion almost immediately.

Therefore, black had enough time, and your "firm evidence" is wrong.

And this is the diagram of game Metta-Ben David in Leela 0.11, which shows remarkably small differences.


This diagram is very remarkable on it's own - black made very few mistakes, that were rather small.
It is difficult how to explain not only such small deviations, but stream of Leela's top choices in human game of this length.

My Leela 0.11 suggests L17 as third choice after M17 (1st) and E13 (2nd).

Is there any chance that Leela 0.11 has some hardware optimization? If I am right its code is not open source.

First, we are discussing if this move could be known in advance. And it could.
Second, choice of moves varies during analysis. Now it was there from the start, then it fell out for some time, and when I checked it was there at 50k.

As I said before, your analysis is sloppoy and in this case you just confirmed what I wrote.

I said that "It is an error to use Leela 0.11 to evaluate a situation when you investigate cheating with the same program because that will easily lead to circular reasoning." It seems you just dont understand that or don't read carefully enough. I was mentioning evaluation and that has nothing to do with the case.

The second part of your remark is clearly meant as an insult. Oh well, I am immune to that.



Your "analysis window" is not an analysis window that a cheater would have used because it uses "no ponder". And (2) 3000- someting evaluation that your window shows is nothing on the GTX 960M that Carlos has. That's a fraction of a second on that machine and then it immediately continues to better alternatives.


Then about my AQ diagram



No it does not. In the AQ evalaution Black gains a plus after white 70, gains more and then gives it away again. About 5% gain is left. In your diagram, the Leela evaluation it is still at 15% at that point.

So thats why you should not rely on the evaluation function of that 1 AI. This diagram shows that maybe B had an advatage, but it was not big at move 124. Just the same 5% he had becuse of the bad white 70.

Now, which cheater would take his score up 15% and then down 15% again within 40 moves? This is very much evidence agaisnt cheating.

And then, how can a game that see-saws like this raise suspicion of cheating? Are high dan players really capable of positional evaluation to detect cheating? Why and when did Fragman think something was fishy?

As explained by someone here or in other threads, they fill a complain against that particular game beacuse that was the game with the most similarity rate according to a measure took by some self-called "experts". So maybe it was just an excuse for presenting a formal appeal by one of the team captain.

Maybe because Carlo used Leela and not the AQ that came out in March 2018? According to Leela there are not so big jumps over those 40 moves, so Carlo could not have realized that his winrate goes up and down that much. He might also not used Leela for all his moves.