It is currently Thu Mar 28, 2024 5:10 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 96 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

Who will win?
Poll ended at Mon Dec 27, 2010 12:07 pm
John Tromp 61%  61%  [ 22 ]
The Computer 19%  19%  [ 7 ]
Too close to call 6%  6%  [ 2 ]
Kasparov switches games to defend humanity 14%  14%  [ 5 ]
Total votes : 36
Author Message
Offline
 Post subject: Re: The Shodan Go Bet
Post #61 Posted: Tue Dec 28, 2010 3:57 pm 
Oza

Posts: 3647
Liked others: 20
Was liked: 4626
My own impression was that the games were not a great advertisement for the computer, but they were not much of an advertisement for human play either. I think we need to remember just how weak a 2-dan amateur really is.

MFOG did a very good job of passing the Turing test early on, but since it appears to have a large store of human knowledge in the form of fuseki and joseki libraries, that's possibly deceptive.

I would also expect John Tromp to learn something about the computer's weaknesses from these two games (in fact, I had the sense that he was testing it even in Game 2), whereas the computer will learn nothing from them. However, I understand that the program did not have all the hardware it expected and that may be changed for the other games.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #62 Posted: Tue Dec 28, 2010 4:29 pm 
Lives in sente

Posts: 774
Liked others: 137
Was liked: 155
hyperpape wrote:
I'm not sure what you meant, tapir, but just to be clear, Tromp didn't play J14 and J16 in the latter game. The computer played as White.


I meant J15 and J17 :) John Tromp was clearly testing the bot here - as he was winning that game already. But J18 in the first game was played in the very beginning - and I certainly would not play such a move against a human opponent I consider as being of my strength (maybe it's just that I am weak): What I want to say, John Tromp plays as if he utterly disrespects the playing strength of the bot and for now he stands unrefuted.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #63 Posted: Tue Dec 28, 2010 6:24 pm 
Lives in sente

Posts: 1037
Liked others: 0
Was liked: 180
I think we are seeing about what we should exect to be seeing.

The hardware for the first two games wasn't all that much more powerful than what Fotland would call a "standard machine" and so the program wouldn't be much over 1 dan. We should expect Tromp to be able to beat a 1 dan and as soon as he has a significant advantage in the game MFOG will begin to play wildly "go for broke" since it only considers won or lost, not by what margin.

Ignore that the bot playing this program on KGS has a 2 dan rating. I believe that is running on a machine about six times more powerful.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #64 Posted: Tue Dec 28, 2010 6:50 pm 
Lives in gote
User avatar

Posts: 499
Location: Germany
Liked others: 213
Was liked: 96
Rank: Fox 3D
GD Posts: 325
John Fairbairn wrote:
My own impression was that the games were not a great advertisement for the computer, but they were not much of an advertisement for human play either. I think we need to remember just how weak a 2-dan amateur really is.

Come on, take it easy ... John Tromp is quite a strong player, from my perspective and I guess also for the majority of the folks watching.

_________________
Stay out of my territory! (W. White, aka Heisenberg)

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #65 Posted: Tue Dec 28, 2010 8:17 pm 
Tengen

Posts: 4380
Location: North Carolina
Liked others: 499
Was liked: 733
Rank: AGA 3k
GD Posts: 65
OGS: Hyperpape 4k
Mike Novack wrote:
The hardware for the first two games wasn't all that much more powerful than what Fotland would call a "standard machine" and so the program wouldn't be much over 1 dan.


Fotland's proprietary term, or can I call it a standard machine too?

_________________
Occupy Babel!

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #66 Posted: Wed Dec 29, 2010 6:13 am 
Oza

Posts: 3647
Liked others: 20
Was liked: 4626
Quote:
Come on, take it easy ... John Tromp is quite a strong player, from my perspective and I guess also for the majority of the folks watching.


No. This is scientific research not a vanity contest. If this was medicine instead of go, I hardly think you would be tempted to go to a 2-dan amateur "doctor". In fact, when you consider the number of mistakes amateur dan players make (and I'm one of them), if this was driving a car instead of go, there'd be constant pile-ups.

Just think for a moment how small a proportion of his life a 2-dan amateur has spent on studying go.


This post by John Fairbairn was liked by: logan
Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #67 Posted: Wed Dec 29, 2010 6:20 am 
Lives with ko

Posts: 281
Location: France
Liked others: 69
Was liked: 25
Rank: yes
and to drive ? I bet he used more time on go than on driving a car...

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #68 Posted: Wed Dec 29, 2010 6:46 am 
Lives in sente

Posts: 1037
Liked others: 0
Was liked: 180
Perhaps all creators of go playing software, especially if this is software for sale, should indicate on what power machine any claims of the strength of the software are based. Fotland does tell if you ask. And it is why he calls MFOG 12.021 "1 dan" even though the bot playing on KGS holds a 2 dan rank.

Is his definition of "standard" reasonable?
Well it's about the strength of a good current "consumer grade" laptop or a "professional grade" one from several years ago*. Not unreasonable to expect that his consumers might have available a machine of that power. But we should note that this might be more power than a current "consumer grade" netbook would have. Anybody considering buying any software needs to find out how it can be expected to perform on their available hardware.

Is "standard" universal?
Of course not. A different creator of software might target a different consumer market as primary. But as I noted, should tell you. In my opinion it would have been wrong/deceptive for Fotland to have called MFOG 12.021 "2 dan" just because it might be able to play at that level on KGS when running on a workstation many times more powerful than he could expect his customers to possess. But notice that this isn't quite the same as the "system requirements" (minimum machine for some software application). For example, MFOG 12.021 would run on a weaker machine, just slower (for the same time control, weaker) but this would not affect the non-playing aspects of the application (problems, for example).

Is there a problem? (for the consumers)
Yes, I think so. But we can scarcely blame the developers for the lack of understanding among users that the fact that their old desktop or a netbook performs perfectly well browsing the internet or doing other non computationally intesive tasks doesn't mean it will work well compared to a more powerful machine when a "crunch" is involved. Probably >99% of users never use their computers for a computationally intensive task.

Can you call it a "standard machine"?
Of course. It's up to you to decide what sort of machine you consider adequate for go playing software. Look, this can be important when we compare go software. The programs using MCTS are computationally intesive but the ones using just an AI are often not and need comparatively less computer power. This affects the relative strength we would expect if the machine power is dropped below what the MCTS program needs but which is adequate for the AI. Thus I would expect the difference in strength between MFOG 12.021 and gnugo to be more if both were running on my T61 compared to both running on this old desktop (fine for browsing and email but only about 1/4 of a machine "standard" for MFOG 12). That's because the gnugo would remain at the same strength but the MFOG 12 would play weaker.



* For example, my Lenovo ThinkPad T61's would qualify as "standard" for MFOG 12.021 (bought reconditioned from Lenovo, they are quite a few years old but still a match for most current "consumer grade" machines)

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #69 Posted: Wed Dec 29, 2010 7:06 am 
Lives with ko
User avatar

Posts: 295
Location: Linz, Austria
Liked others: 21
Was liked: 44
Rank: EGF 4 kyu
GD Posts: 627
Tromp won the third game, this time against MFOG running on the Amazon cloud, using 26 ECU.

Seems like the game was closer this time, and the computer lost in the endgame. That's unexpected, I always thought a precise endgame was the strongest part of computers ;).

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #70 Posted: Wed Dec 29, 2010 7:10 am 
Oza

Posts: 3647
Liked others: 20
Was liked: 4626
Quote:
running on the Amazon cloud, using 26 ECU.


What is the significance of this for non-computer people, please? E.g. is it Ferrari-level computer compared to the family-saloon level most of us have on our desks?

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #71 Posted: Wed Dec 29, 2010 7:21 am 
Lives in gote
User avatar

Posts: 643
Location: Munich, Germany
Liked others: 115
Was liked: 102
Rank: KGS 3k
KGS: LiKao / Loki
1ECU corresponds to about 1GHz singlecore. So 26 ECU about as strong as single CPUs get(Something like a hyperthreaded 3GHz Quadcore).
http://aws.amazon.com/about-aws/whats-n ... instances/

_________________
Sanity is for the weak.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #72 Posted: Wed Dec 29, 2010 7:37 am 
Lives with ko
User avatar

Posts: 295
Location: Linz, Austria
Liked others: 21
Was liked: 44
Rank: EGF 4 kyu
GD Posts: 627
20 ECU is roughly equal to a fast 8-core CPU. Afaik, that's currently the fastest consumer single CPU system that you can buy.

Interpolating that, 26 ECU would be slightly faster than a 10-core CPU.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #73 Posted: Wed Dec 29, 2010 11:45 am 
Lives in gote

Posts: 653
Location: Austin, Texas, USA
Liked others: 54
Was liked: 216
Tromp swept the computer 4-0. See below games 3 and 4. Game 3 the computer did several squeezes, and eventually created a large moyo. I didn't count to see if the computer had a chance if it held onto it, but Tromp broke into it anyways and won. Game 4 Tromp killed a group in the lower left corner. The computer made some fights and managed to kill some stones in the upper left area, but really Tromp was in control the entire game.





This post by yoyoma was liked by 2 people: shapenaji, wms
Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #74 Posted: Wed Dec 29, 2010 12:15 pm 
Lives in sente
User avatar

Posts: 1103
Location: Netherlands
Liked others: 408
Was liked: 422
Rank: EGF 4d
GD Posts: 952
if he went 4-0, the performance result for the bot during THIS test, seems to be below 1d. (that's not to say that we can verify this hypothesis with just one test, it's just that a 1d result would be winning 1.3 games.)

_________________
Tactics yes, Tact no...

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #75 Posted: Wed Dec 29, 2010 2:31 pm 
Lives in sente
User avatar

Posts: 914
Liked others: 391
Was liked: 162
Rank: German 2 dan
I must say that John Tromp seems to play quite calmly and with very few blunders for a 2 dan. His last tournament in the European Go Database is from 2005, and he has a rating of 2231 from then. I would hazard a guess that he is really 3 dan or even stronger now.

_________________
A good system naturally covers all corner cases without further effort.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #76 Posted: Wed Dec 29, 2010 3:15 pm 
Lives in gote
User avatar

Posts: 499
Location: Germany
Liked others: 213
Was liked: 96
Rank: Fox 3D
GD Posts: 325
shapenaji wrote:
if he went 4-0, the performance result for the bot during THIS test, seems to be below 1d. (that's not to say that we can verify this hypothesis with just one test, it's just that a 1d result would be winning 1.3 games.)

I was going to raise this question, given the name Shodan-Bet and John's strength of 1k at the time when the bet was made, if the bot would qualify at least as a shodan. Your statistical reasoning supports my feeling that the bot is not quite there, yet.

_________________
Stay out of my territory! (W. White, aka Heisenberg)

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #77 Posted: Wed Dec 29, 2010 3:16 pm 
Judan
User avatar

Posts: 5539
Location: Banbeck Vale
Liked others: 1103
Was liked: 1456
Rank: 1D AGA
GD Posts: 1512
Kaya handle: Test
His last move of game 4 looks like a blunder. He plays a 1 point gote move which he cannot defend without converting a 10-point semeai into a seki. But C19 is clearly sente and bigger. A 2k should see that. I'm assuming that he does. He seems openly contemptuous toward it.

And it does not seem like even 1D to me.

_________________
Help make L19 more organized. Make an index: https://lifein19x19.com/viewtopic.php?f=14&t=5207

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #78 Posted: Wed Dec 29, 2010 3:34 pm 
Lives in gote
User avatar

Posts: 643
Location: Munich, Germany
Liked others: 115
Was liked: 102
Rank: KGS 3k
KGS: LiKao / Loki
At the end of game 4 Tromp had time pressure, which might explain some weak moves. And the program seems to have misjudged the life&death of the lower corners and thus thought for most of the game that it were ahead.

_________________
Sanity is for the weak.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #79 Posted: Thu Dec 30, 2010 4:28 am 
Lives in sente
User avatar

Posts: 914
Liked others: 391
Was liked: 162
Rank: German 2 dan
Joaz Banbeck wrote:
His last move of game 4 looks like a blunder. He plays a 1 point gote move which he cannot defend without converting a 10-point semeai into a seki. But C19 is clearly sente and bigger. A 2k should see that. I'm assuming that he does. He seems openly contemptuous toward it.


If you mean Black's move, you may have missed that White had "thrown in" a stone there just before, so that Black had no other sensible move.

_________________
A good system naturally covers all corner cases without further effort.

Top
 Profile  
 
Offline
 Post subject: Re: The Shodan Go Bet
Post #80 Posted: Thu Dec 30, 2010 7:25 am 
Lives in sente

Posts: 1037
Liked others: 0
Was liked: 180
SpongeBob wrote:
shapenaji wrote:
if he went 4-0, the performance result for the bot during THIS test, seems to be below 1d. (that's not to say that we can verify this hypothesis with just one test, it's just that a 1d result would be winning 1.3 games.)

I was going to raise this question, given the name Shodan-Bet and John's strength of 1k at the time when the bet was made, if the bot would qualify at least as a shodan. Your statistical reasoning supports my feeling that the bot is not quite there, yet.


No, as shapenaji was clear to point out, an insufficient number of games from which to draw any* statistical inference even were the games between a human 1 dan and a 2 dan. We need to also remember that the expectation that a 1 dan will win 1/3 of the games against a 2 dan is empirical and based upon human players. It is entirely possible that the deviation between games is either greater or lesser for computer relative to human. I suspect the latter (that it is more likely that the human player sometimes puts together a great game well above his/her average performance than that the computer manages that feat).

* Consider an honest coin. If flipped four times the most likely (single) outcome is 2-2. But it is more likely than not the outcome will be one of the other possibilities (odds (5:3). In other words, looking at this in reverse, we should not express an opinion about the honesty of the coin based upon four trials because we do not expect the most likely result for an honest coin.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 96 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group