Actually, the time required on semi-decent hardware would be a matter of hours. A day at the most. Here is what you do:

1. Download all the SGF file. How many can there be? Even for a prolific online player, I guess there are only hundreds -- not thousands -- of games to check.
2. Write a script that opens each SGF file (any number of libraries exist for this purpose)
3. For moves 150 to the end, do the following:
a) Pass the previous board position to Leela 0.11 and Leela Zero, respectively. (Easily done with GTP)
b) Wait for a certain number of playouts.
c) Compare the suggested moves and find a large discrepancy in win-rates for the actual move between the two bots.
d) Whenever a large difference in win-rates is found, pass the variation from the bot that yields the larger win-rate to that that returned the lesser and look to see if that variation contains a "refutation"

But why? What does it prove? This experiment shows only one fact -- that Leela 0.11 and Leela Zero are completely different -- and that fact has absolutely no bearing on this case.

The fact that two bots are different does not indicate that a human player cheated. How can it?

(@Fenring: I think you mean "flawed". "flawless" is actually a synonym for "perfect")

Here is my analysis/commentary on the Carlo vs Chris game, which combines my impressions as a 4d kibitzer with post-game analysis with Leela Zero #145. In summary, Carlo did better in a pretty normal modern opening with an early 3-3 and big shimari, Chris played flexibly but Carlo just solidly captured 2 stones on right for a good result. Chris then found some sharp local moves in upper right fighting and when Carlo made a natural looking mistake of connecting some stones in atari they all died and game was even to good for Chris. I considered the lower side black group in danger but neither player played in that area for ages which was odd (LZ wanted to many times for both). Chris then spoilt his lead with a liberty-reducing push and Carlo found an excellent kosumi to exploit the shortage of liberties which meant Chris was just running out some weak stones but Carlo's adjacent ones were safe and there was a long but natural pushing battle. Bad aji in the left side meant Carlo reduced it and then play finally turned to the lower side, Chris lost a bunch in the lower left corner to secure his group. In endgame at top Chris lost sente so Carlo got to defend the middle and played a nice but not so amazing tesuji to connect up more efficiently and then won by resign.



Here is the winrate graph from Leela Zero #145 on 6k playouts (Chris was black so wins towards the top). It mostly agreed with my feelings watching the game, but I was surprised how good LZ though the game was for Chris after the upper right fighting as I thought he had made it even to a little good, but not more good for him than it had been for Carlo prior to that.


Edit: it has been suggested to me that this winrate graph could be skewed in favour of Chris (black) because the game file shows an incorrect komi of -5.5. LeelaZero does not read the komi from the file but always uses 7.5 komi Chinese counting. The league uses Japanese 6.5 komi so this isn't totally correct, but the winner is usually the same under the two rulesets so will only make a significant difference in a very close game.

Well, if someone with a "semi-decent hardware" is willing to give it a try, I can workout something to automatize this sort of analysis. (but my computer does not qualify as semi-decent hardware, so somebody else would have to run it on his own computer)


Well, let's see the results first. If it happens that, out of 20 games (2000 end game moves analysed), there is only that specific moves already mentioned on Reddit that matches all those criteria, then, certainly, finding such move would be difficult (whatever the methodology used).

I should add this: it might end up being be relatively easy to spot such moves once one uses an automatic analysis script designed only for this purpose (if each game has 1 or 2 such moves). But we won't know what methodology was used by this anonymous Redditor to discover about this move. Without such tool, it's could still be very hard to spot such move.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

I see.

And to whose analysis you referred here?
viewtopic.php?p=232635#p232635

Looking at Uberdude's analyses of the game Carlo played against Chris and at the game itself, move 156 looks pretty normal to me. Looking at the times in the SGF makes it seem even more normal.

Carlo spent nearly 4 minutes on move 154. Does anyone honestly believe that he needed that long for that hane? No way -- he was obviously thinking about the group at G18 which only has one eye and pretty much zero chance of making another one. When black descended to H18, Carlo played J18 in two seconds -- he had prepared it.

Can a human 4-dan find J18? Yes, sure. I could find it and I'm only 3 kyu! (I'm pretty damn sure I could find it in a game, too. This sort of exact reading is really my only real strength.)

What's there to read? There are basically only about three possible local moves for white and failing to connect or at least make life would be game-over. Sure, there are many cutting points and weaknesses but 4 minutes is also a long time.

It would find the move that the anonymous troll highlighted, for a start. That would demonstrate that it is entirely plausible to find such a move with a script.

Even if it were to be the only one, I have described a methodology that could be used to find such a move. The existence of even one such methodology shows that, given that the moves have already been played, finding "suspicious" ones is easily done.

The existence of those moves still proves nothing!

Regarding this case of alleged cheating, the witch-hunters have demonstrated that they are quite content to expend all sorts of effort to condemn Carlo and so I can well believe that someone would bother to write and run such a script although I cannot identify with their motivations.

(2 bots, 10 seconds a move, 2000 moves = a bit more than 11 hours of compute time = "over night". In practice, not all games go to 250 moves and some wouldn't even pass 150, there are probably less than 2000 games and one would probably stop upon finding their "evidence" so checking all of them would be improbable.)

(Simba, by admitting that he finds this thread too long and tedious to go back and actually read the whole thing (#564), has been struck off the role of diligent-witch-hunters despite his vehemence. )

How possible is it for a 6 dan player to play like a 4 dan?

Fair enough. But who know, we might come across some interesting results while doing this, the kind that would be unrelated to the existing case, but could be useful in some other contexts, like designing a "cheating detection tool" for future cases, or whatever. A bit like basic research VS applied research.

As for me, my motivations are linked with my work on developing GoReviewPartner: I am interested in exploring in what ways computers could change the way to play and learn Go. I am having fun developing this tool, and for this specific case (searching for moves wrongly considered as mistake by Leela, but evaluated as good moves by Leela Zero), I already have tools that can perform 90% of this.


My computers evaluate ~300 playout per second with Leela, and probably less than 100/s with latest Leela Zero, so I would prefer to pass on this one.

_________________
I am the author of GoReviewPartner, a small software aimed at assisting reviewing a game of Go. Give it a try!

We discussed this matter earlier.
4 dan played could play 6d moves, but basically, he would play same amount of 2d moves. And he would also play nice sum of 4d moves.
I played games against 6+d where they made less than 4d moves, and I punished them for it.
That is simply how humans play.

But it is impossible for any human to play consistently like program.

Uberdude noticed that Carlo played some moves like LZ, but they were not impossible to find.
Of course, but sheer number of such moves and their combination is what makes a difference.

I tried to make the review fairly objective in terms of the comparisons to LeelaZero's moves as I planned to interpret that evidence and make conclusions about cheating in a later post. If you read it you will see there are quite a few moves of Carlo's I found somewhat suspicious, none a big red flag but little suspicions can accumulate. There were also some of Chris's good moves that LZ liked you could find suspicious. I agree Carlo played more consistently in that game, so apart from that connecting atari and related sequence there were no major mistakes. A concern I have is that if someone applied the same approach to Chris as he is happy to apply to Carlo then there is a fair chance that he himself could be convicted of cheating with an AI (which I don't believe he did, and so wish not to happen). I will explain more later.

Whether this is so depends on what exactly you mean, so please explain. 100% same moves? Same playing style? Something else?

Those are motivations that I support without reservation and, were I not working on my own stuff in that vein, I think that the experiment might be worthwhile.



You keep writing that, but why? Why must he necessarily play 2 dan moves to match any 6 dan moves?

At best, one could argue that a player that consistently played 6 dan moves would be 6 dan but a single game might definitely be biased. There's no guarantee at all that an even distribution of strong and weak moves fall into one game.

The 98% matching evidence is the original one of Leela's top three match in the range, moves 50 - 150.

At first I chalked it up to statistical naivete, but as more time goes on I suspect that the evidence was chosen for effect. Not maliciously, or as part of a witch hunt, but more like, Wow, look at this! The manufactured 98% match impressed a lot of people. Matching only the top choice, for instance, would not have been as impressive.

Edit: Or perhaps the 98% matches came from a belief that Metta was cheating by using Leela to pick all his moves. Matching one of Leela's top three choices and restricting the range to moves 50 - 150 produced results that fit that belief.

Edit2: I see I made a typo in what you quoted. I meant 51 - 150, not 51 - 100.

Edit3: I have found out that the 98% analysis was presented with the original complaint. That fits with the idea of presenting the evidence in the strongest and most favorable way for your side. That may be appropriate when filing a complaint, but that is not the same as doing a scientific investigation. The statistics sucks.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I expect that, overall, a 4 dan will make rather more 2 dan plays than 6 dan plays. That does not mean that the 2 dan plays are all mistakes, however.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

I'm going to step out of things at this point; the main reason I came onto here was to highlight the censorship and make sure that the Reddit thread was in everyone's awareness. I'm satisfied that this has been done at this point. I hope that appropriate lessons have been learnt by the administration, and I appreciate as a community owner myself that sometimes some administrative decisions can be difficult. Mistakes are fine, so long as we learn from them.

I'm spending most of my time here trying to explain and reexplain things to some who unfortunately and repeatedly fail to understand what's being said or claimed, while others are arguing that "X doesn't prove he cheated" when no one on my side is arguing for X in the first place. I really don't have the patience or time to keep going with that.

This sort of forum format clearly proves tricky for some people to keep up to date with all the posts, and to keep everything neatly tied together in their minds. I can appreciate that this is a very long discussion, but there is very little point in continuing to debate in favour of oranges when the response is "no, pears are awful, how can you favour pears!"

So yes - my final words here at least for the foreseeable future:

- I still believe without even the slightest shadow of a doubt that CM cheated against me, and find the opposite viewpoint to be generally very naive and idealistic. Don't get me wrong - I wish this wasn't true, and I wish we weren't dealing with this. But we are. All is not well; don't turn a blind eye or others will learn that they can get away with this sort of thing against a mountain of evidence from many directions and will be more likely to do it in the future in light of that.

- I won't touch PGETC again if he isn't convicted and thrown out as per the rules, nor will apparently the entirety of Serbia based on what someone said a few pages ago, and I'm sure a few other strong players won't either. If the officials would rather keep this cheater in exchange for a bunch of legitimate players, then that's a shame but it's their prerogative. I do however want to make very clear that I do not in any way support any action against the Italian team as a whole (other than having CM's games all counted as losses). Yes, the rules stipulate they should be heavily punished too, but I think this would be horrifically unfair.

- I also would want to see a proper process for this kind of situation constructed in advance of next season. I would be interested in helping with this, to a point, but am too busy to be the main organiser of this.

Bojanic: If the appeals committee refuse to restart things after their EGC excuse, please contact me directly by email and I'll open a new complaint against him.

Lukan: Thanks. I appreciate what you've said, and you're probably the only one on this forum strong enough to actually have any kind of grasp of understanding how ridiculously overpowering his play was - even you say it'd be far too strong for you, and you're stronger than I am.

Generally: This thread has contained me speaking from a business/professional standpoint - do not take anything personally. I hold no personal ire against any of you, no matter how poorly I view you in this context; I understand that you're simply trying to debate for your own beliefs rather than coming from a place of deliberate malice.

Simba over and out.

Oh, Lukas this is terrible, are you sure? I saw him last week and he was quite serene, he won the tournament here in my city, Bologna .

We spent a good time together. Are you sure, who tell it to you?

Honestly, I did not see anyone not understanding what you were claiming, just people finding that this claims were clearly wrong.
I find it scary than some basic logic mistakes persist like this in the discussion.

As said properly by Jan.van.Rongen, we are in phase 3 of witch hunt. The phase of disinformation, propaganda and fake news.

Sorry, I was mistaken. I have now found the facts. The original "98% analysis" was submitted by the Israelian team captain Amil Fragman together with his offcial complaint on 2017-12-13. Together with two other "independent" analyses.

But they used exactly the same method (50K simulations, no pondering) - and of course the results were very similar to each other.

These data and this method were then used (without discussion?) again in the rest of the procedure. Your only role was to ask an independent analyst (Lukas Podpera) and you were also consulted about the results of these analyses and asked whether you supported the verdict. Which you did - I think.

So IMO my question is still valid: what was your reaction to the much more detailed and much more solid Italian expert report?

Actually, in Bologna we had an unexpected guest Hayashi Kozo 6p sensei that filled me with happiness:
http://bolognagoclub.altervista.org/fot ... ogna-2018/

Having the lunch together, he told us of our ranking system that is the best in the world. I wanted to say it to Ales that I know has developed it and thanks him for the work he has done for the European go community.

I hope we could have better times to meet e to know each other, may be in Pisa if you came next July.

I apologize if I were rude in some e-mail, but this is my character that in Italian language I can try to mitigate, but in English it results much more difficult to me.

Best regards, Mirco.