The KGS 3 dan rating is probably based largely on people just trying out a bot for fun, but the bot itself is always playing at maximum skill. If the humans would always play at maximum skill, I doubt whether the bot would even get up to 1 dan.

_________________
My name is Gijs, from Utrecht, NL.

When in doubt, play the most aggressive move

Well, the same could be said for any kgs player, wether bot or human.

Eh, I think there's a difference, what I would LOVE to see, is a bot in a blind test, people don't know right off the bat if they're playing a bot or not. A sort of Turing rank test.

There are a number of things at issue with its rank:
A) People are looking for something different when they play a bot, hence they play differently
B) If bots make mistakes, they usually make them systematically, hence if you can beat a bot once, you can usually beat it indefinitely, likewise with losing.

_________________
Tactics yes, Tact no...

I believe this was already done, by a manual operator inputing all the moves. I don't remember where I read this, though.

I know people who play bots because they they feel they can leave the game when something comes up without annoying somebody. Such players would tend to boost the ratings of the bots. It looks like Zen wins many games on time or resign...

Theodor

Surely someone exists who is both curious about how a bot would fare in a blind competition and has available hardware (out of all the bots currently playing on KGS, none of their controllers is curious?). Of course this is akin to a collective action problem--I'm saying this but don't have a computer that I could use to run a bot.

_________________
Occupy Babel!

That's what is so interesting about this bet! You can be sure John Tromp will be trying.

I have mixed feelings about the whole thing. I like to track the progress of the computers, and I see it as progress of humankind, through our ability to get more and more things done through technology. But on the other hand it's sad to see another game slowly getting overtaken by bots (not that Go has reached that point yet).

So for humankind's progress, my heart is behind the computer (strange thing to say! ). But I think Tromp will win this time. In a few years I think it will swing the other way.

Some misconceptions

a) The bots win a disproportionate number of games on "time".
Well yes of course, this is what you should expect to see, because unless there is a bug in the program the computer shouldn't ever lose on time. It "knows" how much time it can afford to spend "thinking" before making a move.

b) Until "weaknesses discovered"
Well this is more or less dated. An AI playing based on "go knowledge" might not be able to "see" something (has no code for that) but the strongest programs of the last couple years are all using MCTS to select moves and that algorithms doesn't depend on "go knowledge".

c) Why relatively stronger with less time?
This is actually deceptive and true only in a relatively narrow region of time. But it seems that way because these are the time ranges at which we usually play. The programs using MCTS are stronger in absolute terms given more time and would be very strong given unlimited time. At "blitz" speeds too short for the human to read anything out the bot does better against the human compared to speeds slow enough to allow the human reasonable to read. Were the time greatly increased above that errors in reading by the human would limit improvement with the additional time so this trend would reverse.

d) This rating based on playing against whom?
Well naturally selection is involved (who is willing to play the bot). But it isn't quite fair for any of us to say "not that strong" if we aren't willing to back that up by our own trial. Also very important to remember that our experience doing this form a year or two back might not be very relevant and that our experience playing against program A says little about what program B can or cannot do.

3) The machine matters
Since time is a factor, so is the power of the machine on which the program is running. This is very significant in challenges like this and significant when people say "I don't have a machine capable of runnign one of these bots" (odd, what are you using to access this forum -- well if a borrowed machine, one at the library, etc. where you aren't allowed to install software, I guess that would apply).

So this does affect what you would experience as the strength of the program running on your own machine at home and at least for the programs that are being sold you should be told "on what power machine at what time settings". Thus for example while MFOG 12.021 running as a bot on KGS has at the moment a rating over 2 dan that's on a machine far more powerful than you would have -- but Fotland is honest about that and calls this version 1 dan on a "standard machine" and will explain what that means. That program would not play at 1 dan on this old desktop from which I am writing this (but would on my laptop which is about 4 times more powerful).

I'm not sure this is true. My understanding is that the playouts are directed to a degree by go knowledge. MCTS doesn't pick a random move from the 19x19 board, it picks from a set of suggested moves based on the current position. In fact, I'm under the impression that this is one of the main areas of research - finding effective ways to apply go knowledge to make useful suggestions.

Just because the bot doesn't use go knowledge, that doesn't mean a bot can't have a specific exploitable weakness.

For example, a naive implementation of MC playout will have a blind spot involving ladders, because it is extremely unlikely to randomly play out the ladder far enough to actually hit the ladder breaker. For that you at least need to make sure the bot always tries capture, atari and extend-from-atari moves, which is application of specific go-knowledge, and in that case it *removes* a specific weakness .

I mean more general weaknesses, weaknesses of style, not specific shape weaknesses. If you find a style that beats a computer, that style will KEEP working.

This was a problem for the best chess computers way up until they finally established dominance, GM's could play a "closed" game which made the look aheads fiendishly difficult.

The systematic weaknesses are in the way such engines over/undervalue certain kinds of positions.

_________________
Tactics yes, Tact no...

soon computer will be strong in below areas but
1) computer will overpower any human in end game.
2) computer will know many many josekies and known variations of it.
3) computer will read small and closed life and death perfectly.

but they will not be able to read well if the board become too complicated.
they will not be good at counting thickness.

in the game of go it only take one move to lose the game.
that is why i am saying that i can not lose to computer in my lifetime.

_________________
"The more we think we know about
The greater the unknown"
Words by neil peart, music by geddy lee and alex lifeson

don't know how old you are, but looking at your picture and you being 4d kgs I think you will need some serious studying to be still ahead of the computer in 30-40 years.

_________________
Laurens
Go school: freegame's Teaching School
Author of: The Next Move a book for 15-3kyu players.

I guess we have the next go bet here. What about a game every year starting from 2011 against one of the bot playing best at that time?

Although I'm in the camp that welcomes research into computer go and is pleased by progress, I'm also in the camp that believes current programs are probably significantly overranked, for various reasons - people not being serious when playing them, not analysing the program's weaknesses, etc. I also believe that computers that do score well do so often because they have strengths that work against amateurs but that would not work against professionals. In particular, they seem to have a pretty powerful endgame. I reckon even dan players among amateurs lose 10-20 points per game in this area. Most pros, however, claim to play a pretty good endgame, and mistakes are usually only of the order of a point or so.

There is also the point that the weaknesses of programs have not been properly studied yet. I know, for example, that when I have played programs, I have often tried to engineer unusual positions such as semeais and sekis which are both difficult enough perhaps not to have been taken account of in the programming and which require extreme precision, something Monte Carlo may be bad at. My experience is that programs play utterly stupid moves in these cases. I am sure a pro who dedicates himself to studying these weaknesses would do even better.

The argument about computer weaknesses is often countered by the claim that they exist but are sorted out over time. This has certainly been true in chess. The horizon effect, lack of random play and taking the program out of the book were all strategies that even weak amateurs could use for a long time but are now irrelevant. Among chess pros, the strategy of playing close games with as few tactics as possible has also been demolished.

But I do wonder if this really will apply to go (except in the very long term). The nature of go is such that it involves several battles over an entire game (and can be made to have even more). One mistake in chess is usually fatal against a computer. In go you get to fight again. But on the computer side, the very programming strategies that have been devised to deal with the huge branching factor involve a large measure of randomness and therefore (I presume) mistakes. You could argue that, because it is go, mistakes don't have to be fatal for the computer no more than they are more the human, but I suspect there might be a major difference. The computer would be unaware that a mistake has been made, but a pro would be aware. The computer finds it hard to change its behaviour, the human is designed to create coping strategies in new environments. Intuitively, I also feel a pro would be able to find enough precision to punish a mistake whereas a computer of the current type would not.

I also have reservations about pro reactions to computer go so far. I think there has been a large measure of politeness, or maybe noblesse oblige. One exception may have been when a program called Erica won the World Computer Olympiad recently and so got to play young Fujisawa Rina on six stones. She made a monkey out of it, relying on a large semeai strategy incidentally. She made it plain beforehand that she wasn't going to go easy, as previous pros have apparently done, although she hadn't made any special study of computer go. This game doesn't seem to have appeared elsewhere so I give it here as another GoGoD Christmas present (thank TMark as he transcribed it).

[sgf-full](;SZ[19]FF[3]
PW[Fujisawa Rina]
WR[1d]
PB[Erica program]
HA[6]
RE[W+R]
US[GoGoD95]
AB[dd][pp][dp][pd][dj][pj]
;W[fq];B[fp];W[gp];B[gq];W[ep];B[eq];W[fo];B[fr];W[fp];B[dq];W[gr];B[hq]
;W[er];B[hr];W[fs];B[dn];W[kp];B[kq];W[lq];B[mq];W[jq];B[lp];W[kr];B[lo]
;W[mr];B[nq];W[qf];B[pf];W[pg];B[qg];W[pe];B[of];W[qe];B[oe];W[qd];B[qc]
;W[pc];B[od];W[rg];B[qh];W[rh];B[qi];W[qb];B[oc];W[rc];B[fc];W[cl];B[dl]
;W[dk];B[ck];W[ek];B[cm];W[cj];B[bl];W[bk];B[ci];W[cl];B[el];W[ck];B[fk]
;W[ej];B[fi];W[fj];B[gj];W[gi];B[hi];W[gh];B[gk];W[hh];B[ih];W[ig];B[jg]
;W[ii];B[jh];W[hj];B[ei];W[di];B[bm];W[ch];B[dh];W[dg];B[dj];W[eh];B[if]
;W[kj];B[fh];W[di];B[fg];W[fl];B[fm];W[gl];B[hk];W[ik];B[hl];W[gm];B[gn]
;W[hm];B[il];W[jl];B[im];W[hn];B[in];W[ho];B[ji];W[jn];B[jk];W[jj];B[ij]
;W[co];B[do];W[ik];B[io];W[kk];B[hg];W[ij];B[gg];W[hi];B[hp];W[go];B[jm]
;W[km];B[kn];W[jo];B[ip];W[ln];B[ko];W[jp];B[lm];W[kl];B[mn];W[cr];B[bq]
;W[ro];B[dr];W[qq];B[qp];W[rp];B[pq];W[qr];B[qn];W[rn];B[kc];W[ob];B[nb]
;W[bd];B[cd];W[bc];B[mk];W[mj];B[lj];W[li];B[lk];W[ki];B[cb];W[cf];B[ce]
;W[be];B[lh];W[mh];B[nj];W[mi];B[qm];W[lg];B[oa];W[pb];B[mg];W[kh];B[ds]
;W[le];B[ld];W[ke];B[lf];W[kf];B[rr];W[rq];B[es];W[fr];B[pr];W[qs];B[ak]
;W[aj];B[jd];W[ef];B[bi];W[al];B[eg];W[dh];B[de];W[df];B[mf];W[jf]
)[/sgf-full]

As regards pros going easy on computers, I do know this happens. I can't speak for every case, obviously, but many years ago, when I went to Japan to help market the British-made computer Shogimaster (program written by David Levy's team that had won world chess programming titles), we had the benefit of this sort of behaviour.

We had already encountered, in various visits to major Japanese companies that we hoped would take on production, and to universities doing shogi research, a resentment that a non-Japanese team had made such a product with a previously unheard of level of play. But one evening in the Shogi Renmei (where George Hodges and I were billeted for the week in the pros' overnight quarters) we were invited to the poshest playing room and a senior pro deigned to play our program, giving a four piece handicap (which meant he was treating us as potential dan level). Not only that, he let us win. Very soon after someone at the Renmei arranged for a reporter from a major Japanese newspaper to interview us about this scoop of beating a pro at our first attempt (and we even got paid for the interview!).

I regard that as pure altruism, and I have seen that sort of behaviour countless times. I therefore factor it in when I see pro-program games in go.

Although I'm a computer go fan, I will be rooting for John Tromp, by the way.

Return to (initial) question?

I think it is close to certain that go playing computer programs will not have the strenth of a high ranking pro any time soon and I think that remains a true statement even if there is another breakthrough equivalent to the last.

But that wasn't the question, was it? Not high rated pro but amateur ~3 dan.

As an aside, am I alone in thinking that the latest computer tournament didn't really tell us much about the comparative strength of the top three finishers? Yes of course, there were rules in place that would define a "winner" in a situation like that* so clearly Erica "won" the contest according to the rules.

* A round robin where the results were each of the top three finishers lost exactly one game to each other and won all the rest of its games.

Mike: sounds like what you need is a thread in the Go Rules forum. They'll definitely have something to say.

_________________
Occupy Babel!

Why? I wasn't expressing an opinion that anything was wrong with the torunament rules. We like to have a well defined winner of tournaments and so have rules in place to ensure that and since all the contestants know the rules this is perfectly fair.

No matter what the rules (or whether a human or computer competition) the results sometimes allow us to draw conclusions about the strength of the contestants and sometimes the results don't. I was expressing an opinion that in this case the outcome was inconclusive with regard to the strength of the top three programs. I wasn't saying that Erica shouldn't have won the tournament, just that its winning didn't didn't let us conclude stronger than Zen or MFOG.

Is it just me?

Don't others here find what people are saying even mildly odd? I don't mean the statements given as conclusions as much as the reasons presented.

Are the bots playing at the strength their ratings indicate?
Well perhaps not, but instead of people saying things like "probably the people playing the bots aren't really trying" (that would imply the conclusion) would expect to see a reason for this belief. After all, if I didn't believe the bot was as strong as its rating and I were playing it I don't think I'd want to lose to it under conditions where by what I believe my own rating to be I shouldn't.

We aren't seeing here postings of personal experience. Why not? My own observations are useless in this regard but surely some of you have ratings similar to some of these bots. What happens when you play against them? I know that if I believed they had grossly inflated ratings I'd want to confirm that for myself.

i will conclude with my confidence.

i can not give 5 stone handycap to human kgs 3d.
but i am pretty sure i can beat computer 3d in 5 stone handy game.

conclusing: computer is over rated

_________________
"The more we think we know about
The greater the unknown"
Words by neil peart, music by geddy lee and alex lifeson