The Shodan Go Bet

[SpongeBob]

if he went 4-0, the performance result for the bot during THIS test, seems to be below 1d. (that's not to say that we can verify this hypothesis with just one test, it's just that a 1d result would be winning 1.3 games.)

I was going to raise this question, given the name Shodan-Bet and John's strength of 1k at the time when the bet was made, if the bot would qualify at least as a shodan. Your statistical reasoning supports my feeling that the bot is not quite there, yet.

[Joaz Banbeck]

His last move of game 4 looks like a blunder. He plays a 1 point gote move which he cannot defend without converting a 10-point semeai into a seki. But C19 is clearly sente and bigger. A 2k should see that. I'm assuming that he does. He seems openly contemptuous toward it.

And it does not seem like even 1D to me.

[Li Kao]

At the end of game 4 Tromp had time pressure, which might explain some weak moves. And the program seems to have misjudged the life&death of the lower corners and thus thought for most of the game that it were ahead.

[Harleqin]

His last move of game 4 looks like a blunder. He plays a 1 point gote move which he cannot defend without converting a 10-point semeai into a seki. But C19 is clearly sente and bigger. A 2k should see that. I'm assuming that he does. He seems openly contemptuous toward it.

If you mean Black's move, you may have missed that White had "thrown in" a stone there just before, so that Black had no other sensible move.

[Mike Novack]

if he went 4-0, the performance result for the bot during THIS test, seems to be below 1d. (that's not to say that we can verify this hypothesis with just one test, it's just that a 1d result would be winning 1.3 games.)

I was going to raise this question, given the name Shodan-Bet and John's strength of 1k at the time when the bet was made, if the bot would qualify at least as a shodan. Your statistical reasoning supports my feeling that the bot is not quite there, yet.

No, as shapenaji was clear to point out, an insufficient number of games from which to draw any* statistical inference even were the games between a human 1 dan and a 2 dan. We need to also remember that the expectation that a 1 dan will win 1/3 of the games against a 2 dan is empirical and based upon human players. It is entirely possible that the deviation between games is either greater or lesser for computer relative to human. I suspect the latter (that it is more likely that the human player sometimes puts together a great game well above his/her average performance than that the computer manages that feat).

* Consider an honest coin. If flipped four times the most likely (single) outcome is 2-2. But it is more likely than not the outcome will be one of the other possibilities (odds (5:3). In other words, looking at this in reverse, we should not express an opinion about the honesty of the coin based upon four trials because we do not expect the most likely result for an honest coin.

[Magicwand]

i can not understand why people still think machine has a chance against human player.
in my opinion machine is around 3k if the player expose their weakness.

i remember 20 years ago i played against manyface 9 stone handycap.
my whole object were to kill every group on the board.
I WAS SUCCESSFUL exposing!

killing everything on the board is hard but winning is too easy.
unless machine make major update on their weakness it is too easy beat machine.

[pwaldron]

killing everything on the board is hard but winning is too easy.
unless machine make major update on their weakness it is too easy beat machine.

Why don't you put your money down? John Tromp made a $1000 bet about the strength of computers in 2010. What are you prepared to wager regarding the strength of computers in 10 years?

[Magicwand]

killing everything on the board is hard but winning is too easy.
unless machine make major update on their weakness it is too easy beat machine.

Why don't you put your money down? John Tromp made a $1000 bet about the strength of computers in 2010. What are you prepared to wager regarding the strength of computers in 10 years?

i dont mind putting 1G. 10 years later computer can not beat me in serious game for sure.
but who will monitor my bet?

[LocoRon]

No, as shapenaji was clear to point out, an insufficient number of games from which to draw any* statistical inference even were the games between a human 1 dan and a 2 dan. We need to also remember that the expectation that a 1 dan will win 1/3 of the games against a 2 dan is empirical and based upon human players.

* Consider an honest coin. If flipped four times the most likely (single) outcome is 2-2. But it is more likely than not the outcome will be one of the other possibilities (odds (5:3). In other words, looking at this in reverse, we should not express an opinion about the honesty of the coin based upon four trials because we do not expect the most likely result for an honest coin.

One thing I'm not certain you've taken into account with this statistical approach is that for the purposes of this bet, the bot and John were treated as equals--white received komi (7.5, I believe), and they alternated between black and white. For your statistical "1d vs 2d with 1d winning 1/3 of the games", the 1d would always be black and there would be no komi.

[robinz]

For your statistical "1d vs 2d with 1d winning 1/3 of the games", the 1d would always be black and there would be no komi.

Really? I always thought the situation where the weaker player plays black with no komi, when the players are 1 rank apart, was the standard way of arranging what is effectively a "1 stone handicap" - so in theory would be expected to give the two players equal winning chances, as in any handicap game. I thought the figure of 1/3 winning chance for the weaker player applied only to even games (with whatever is currently thought to be the "correct" value of komi).

I'm not 100% sure about this (I'm relatively new to go), but it seems logical to me. Can anyone else comment? (I don't care whether it's to back me up or tell me I'm wrong )

[John Fairbairn]

Why don't you put your money down? John Tromp made a $1000 bet about the strength of computers in 2010. What are you prepared to wager regarding the strength of computers in 10 years?

I'm not sure I follow the argument that you have to put down money just to have an opinion. David Levy's famous bet in the chess computer world was nothing at all to do with putting his money where his mouth was or even with his opinions on chess computers. He made a very calculated gamble that the publicity from his bet would easily offset the cost if he lost. In fact, he miscalculated - it was worth far, far more than he dared hope, partly because chess computers took longer than he expected to make progress.

I don't think John Tromp had any such ulterior motive, but he very kindly gave a lot of us a good deal of entertainment nevertheless.

On a different tack, I was looking at a 1973 Go Review today and there was a survey of computing projects by Stuart Dowsey. I was surprised to see that a program playing strength of 10 kyu was being claimed, and that a tsumego project by Bill Mann of Massachusetts could "solve shodan problems without too much difficulty". If that's true, there's been much less progress in the last almost 40 years than I previously thought.

[hyperpape]

I agree that it's silly to say that you have to bet to have an opinion. The advantage of betting is that it slows down your thought processes, and makes you ask yourself "did I really mean the thing that I just said?" But if you're disciplined, you can do that without a bet.

The other advantage is that this bet resulted in a much cleaner comparison than some previous events.

David Levy's famous bet in the chess computer world was nothing at all to do with putting his money where his mouth was or even with his opinions on chess computers. He made a very calculated gamble that the publicity from his bet would easily offset the cost if he lost. In fact, he miscalculated - it was worth far, far more than he dared hope, partly because chess computers took longer than he expected to make progress.

I'm probably slow, but can you explain what you mean by saying it was worth more than he dared hope?

[John Fairbairn]

I'm probably slow, but can you explain what you mean by saying it was worth more than he dared hope?

I can't remember his exact words, but it was along the lines of getting publicity over more years than he had expected, and that publicity, if he'd had to pay for advertisements or hiring PR companies or taking people out to dinner, would have cost an awful lot more.

[yoyoma]

Why are you guys trying to shut down another epic bet?

Who will step up and go against Magicwand? Let's look forward to 2021!

[Mike Novack]

i can not understand why people still think machine has a chance against human player.
in my opinion machine is around 3k if the player expose their weakness.

i remember 20 years ago i played against manyface 9 stone handycap.
my whole object were to kill every group on the board.
I WAS SUCCESSFUL exposing!

killing everything on the board is hard but winning is too easy.
unless machine make major update on their weakness it is too easy beat machine.

20 years ago !!! (in this matter 2 years can be a long time)

You think ManyFaces is playing at around 3k? How about expressing this in terms of handicap stones? How many stones do you think you could allow Manyfaces 12.021 playing on a "standard" computer (4-5 ECU)? I have MFOG 12.021 here at home running on a computer of that power. Get to play lots of games against it. Now I am a much weaker player than you are but I know both how many stones I need to have a fair chance against the program compared to how many I need against various human players of known rank at our local go club.

In other words, how many stones can you offer a human 3k and still win? (and at one more would lose most of the games). How about a human 1k or a human 1 dan?