The current state of various go playing programs

[judicata]

Judicata, do you feel familiar enough with using Crazy Stone on your laptop to reveiw it?

I really haven't played it enough yet, but I'll be happy to review when I have.

The window size issue is really annoying. It offers, in theory, three "Window Sizes": Small (800x600), Medium (1024x768), and Large (1280x1024). I say "in theory" because the Large option is disabled when using my 1280x1024 external monitor. And the medium size is disabled on my 1366x768 laptop monitor (Thinkpad X220, FYI), which leaves me with a tiny board and actually affects my ability to focus. If the window is actually larger and it disables the higher resolutions to protect me from myself (i.e. having the window extend beyond the viewable screen), then shame on the program designers.
[EDIT: See update and work-around here. ]

As for the actual substance of the program: I've won and lost on both level 6 and level 7 (out of 10), I haven't tried 8-10 yet, and haven't played many games. It also seems to seek control of the center, at almost any cost (perhaps by design to capitalize on its fighting ability).

EDIT: Although I haven't provided much info yet, here is some relevant system info: Intel Core i5 2.5ghz, Windows 7 64-bit, 4gb Ram.

[yoyoma]

You are right that these programs do not save the playouts in memory. But the tree search portion can still grow very big very fast. 3GB in 30 seconds!

Try to keep this "user"

Do include in your reviews of specific programs "uses lots of memory" but don't assume this would be true for other programs. Emeraldemon is running fuego for a Malkovich giving it 30 minutes per move! (that's an awful lot of playouts) He doesn't seem to be having any memory problem. As a pro I would ask the Pachi crew what sort of an abstract data structure they are using for the tree. There obviously can't be more nodes than playouts if each playout adds a node and I wouldn't think it should take all that many bytes to store the data of a node.

Judicata, do you feel familiar enough with using Crazy Stone on your laptop to reveiw it?

And Emeraldemon, feugo?

I believe Feugo and most all modern MCTS programs, when they run out of memory start pruning the tree. So yes the program will continue to run, but there could be an impact to the playing strength of the program. I don't have much experience with Fuego, but I saw some instructions for compiling it that gave a switch to allow you to use ~3.5GB of memory instead of "only" 2GB. http://fuego.sourceforge.net/fuego-doc- ... mingw.html

I guess my point is that reviews should include how much memory the machine has and/or how much they gave to the program. For that matter it should include all settings relevant to playing strength.

ETA: I should have posed all these memory issues as questions. I haven't used these programs much myself, I'm just an interested observer. I saw a few references saying they can use a lot of memory, but I have no idea how much difference it makes if they are limited to less.

[Mike Novack]

EDIT: Although I haven't provided much info yet, here is some relevant system info: Intel Core i5 2.5ghz, Windows 7 64-bit, 4gb Ram.

For future references --- in a case like this were there is a very large range of crunch power within a processor family perhaps better to specify in terms of number of cores and speed of yours rather than just a family designation (the most powerful i5's and i7's are several times more powerful than the weaker ones).

Again, let's try to keep this "user" and not speculate on details of implementations. There are usually possible tradeoffs between speed and space and a wise choice of structures can possible greatly reduce space and whether structures no longer in use are collected or abandoned (and again that is a tradeoff). Leave those things for us "pros" to discuss.

[judicata]

Again, let's try to keep this "user" and not speculate on details of implementations. There are usually possible tradeoffs between speed and space and a wise choice of structures can possible greatly reduce space and whether structures no longer in use are collected or abandoned (and again that is a tradeoff). Leave those things for us "pros" to discuss.

I'm not entirely sure what you mean, but I'll speculate on anything I think may be useful for people reading a review. Are you referring to my speculation that Crazy Stone is designed to seek center influence to capitalize on its fighting ability? If so, I respectfully disagree--a layperson is in a position to offer such speculation. I try to make it clear that I'm speculating, and comments are always offered simply for what they're worth.

Of course people shouldn't pretend to be an expert on something they're not, but anyone from novices to hobbyists to professionals can comment on what they observe and speculate. Also, I have a propensity for anti-elitism, so your comment may have hit a nerve .

I added my system info as an afterthought because people seem interested in that. I don't know how it affects the program. Here's some additional info: Intel Core i5 2520m 2.5GHz, 2-core.

[emeraldemon]

Ok, my Fuego review:

Availability: Fuego is free and open source, with semi-active development: (http://fuego.sourceforge.net/). On the other hand, it currently requires building from source, which I guess puts it out of reach for many. If the phrase "./configure; make; make install" has meaning to you, you should be fine, although you will also have to install boost if you don't have it already.

GUI: Fuego uses Go Text Protocol (GTP), so it can be used with any GUI that knows that protocol, but they suggest downloading the also free GoGui program (http://gogui.sourceforge.net/). GoGui is a java app that should just run without problems for most users. If you use GoGui, Fuego can take advantage of some special features like setting Fuego's options through menus & buttons rather than command line, and showing the variations it considers in real-time, which I find interesting. Otherwise the UI is nice but unremarkable, I'd say.

Hardware: there is an option to set the amount of memory fuego uses, and Fuego will use up everything you give it before it starts pruning. If your computer grinds to a halt, try lowering the memory allocation. I haven't done any tests, but since more memory means a larger search tree, it's safe to assume the program will be weaker the less memory it gets. It is multi-threaded, so it can take advantage of your 2 or 4 core machine to compute faster.

Like many (most? all?) computer programs, Fuego only understands area scoring. It also only understands Canadian byo-yomi. Fuego will use the time you give it, so you can adjust playing strength somewhat by increasing or decreasing the time settings. If you don't set time, Fuego defaults to 10 sec/move. There isn't a "difficulty" parameter, as some programs have.

Now what people really care about: how it plays. It is much stronger on smaller boards than 19x19: if I had to guess I would say something like 5k-1k on 19x19, 1d-4d on 9x9. It's a bit difficult to judge strength, since to my knowledge there isn't any rated Fuego bot anywhere. I set up an unrated bot to play some on KGS, and 4k-5k seemed about right for the medium-fast settings I was using (25/5:00 Canadian byo-yomi).

A few words about the style: it only cares about winning and losing, not point differential. If Fuego thinks it is far behind, it will make increasingly desperate moves and then resign. This may be frustrating for beginners who want to play a game to the end. Similarly when it is ahead it will play protective moves that reduce its lead but ensure the win, so losing by 0.5 or 1.5 against Fuego is pretty standard.

As for usage, I primarily play it on 9x9 with fast-ish settings when I want to kill some time & practice my reading. I typically take black and a reverse komi and it's a good challenge for me.

[Mike Novack]

A few words about the style: it only cares about winning and losing, not point differential. If Fuego thinks it is far behind, it will make increasingly desperate moves and then resign. This may be frustrating for beginners who want to play a game to the end. Similarly when it is ahead it will play protective moves that reduce its lead but ensure the win, so losing by 0.5 or 1.5 against Fuego is pretty standard.

That behavior will be true for almost all MCTS based programs. It also happens to be the super rational way to play go since there is no bonus for winning by more than 0.5 point and no consolation for losing "but at least it was close". As long as these programs simply counts wins vs losses ignoring by what margin in searching out the line with the highest winning percentage, this is how they will play.

BTW -- you must not turn off "allowed to resign" if that is an option when the program is MCTS. If the game ever becomes hopeless the algorithm will make ridiculous moves if forced to move. Having the computer resign instead of playing to the end is mainly an irritaion for beginners but no MCTS program is weak enough to be an opponet for beginners even at a 9 stone handicap.

Because this is a case of a "free software" application that is not distributed by its developers as an executable we need to discuss its cost differently. Let's call it for the moment undefined/unkown. People who can compile their own code can get it for free. For "end users" it costs whatever it takes to get somebody to compile it for them which could be anything from thank you to a decent bottle of wine, etc. Probably, if there were enough interest, somebody would step forward and make executables available (along with the source code; that's a license requirement for "free software") for nominal charge if not free. The point being (a major concept of "free software") that they could not charge excessively for this service or somebody else would undercut them. I'm just guessing, but say $10 might be about right (including packaging and shipping).

[judicata]

Because this is a case of a "free software" application that is not distributed by its developers as an executable we need to discuss its cost differently. Let's call it for the moment undefined/unkown. People who can compile their own code can get it for free. For "end users" it costs whatever it takes to get somebody to compile it for them which could be anything from thank you to a decent bottle of wine, etc.

At least if you're on windows, there is a pre-packaged executable with GoGui here on the Fuego project site.

[Mike Novack]

I was going by what Emeraldemon who had just reviewed the program said. It is not uncommon that free software projects refuse to make for MS operating systems (politics). In fact there is at least one free software project where that is what they do (provide a wide range of free apps for installation under MS all in one place*). However this refusal to recognize reality (see, my politics showing) has diminished the last few years and more and more projects provide run execs for Windows.

BTW -- besides initially posting reviews this forum topic would be a good place to post changes of this sort.


* "vetted" apps they consider ready for the real world, adequately stable versions, etc. The sort of thing that many "end users" can't seem to understand. I am on a number of support lists for some of these apps and you would be amazed at the number of users reporting problems not understanding that as users they should never have installed a beta or even worse, an alpha version!

[Mike Novack]

I'm not entirely sure what you mean, but I'll speculate on anything I think may be useful for people reading a review. Are you referring to my speculation that Crazy Stone is designed to seek center influence to capitalize on its fighting ability? If so, I respectfully disagree--a layperson is in a position to offer such speculation. I try to make it clear that I'm speculating, and comments are always offered simply for what they're worth.

Of course people shouldn't pretend to be an expert on something they're not, but anyone from novices to hobbyists to professionals can comment on what they observe and speculate. Also, I have a propensity for anti-elitism, so your comment may have hit a nerve .

Nope. We should report on how we see the programs behave at the user level. You gave an example of that. I was referring to speculation about what details of implementation might be responsible the behavior. As to the degree professionalism required to comment on that, well there are program design/coding issues that novices and hobbyists can understand and some that would challenge even most folks making their living in the cypher mines. This is one of the latter, a tough one even for the pros. Why do I say that? A little unfair advantage on my part. See, all the players in this game (those trying to write implementations) have access to the academic papers behind the algorithm. Nevertheless, even one of the people who was a leader in the race before the MCTS algorithm came along has failed to get his implementation of a MCTS version strong enough, fast enough. I know because I did volunteer beta testing for him and had to tell him "you are at least 2-3 stones behind". Part of that might be at the algorithm tweaking level (where to prune, etc.) but part is probably elegance in choice of data structures and method.

My "line of country" (though not with game playing software). Many were the times I ended up having to rewrite a program (designed and written by another pro) and sometimes managed almost two magnitudes improvement in performance (same language, just different design and coding). It is very hard to teach this (given this problem, how to recognize what are the data structures and algorithms that make an elegant/efficient solution possible). BTW, this does affect my attitudes about what is reasonable or unreasonable in terms of the price of software. In other words, I know what I expected to get paid per hour of this level of work.

[pookpooi]

I heard that almost all monte-carlo programs are set to play more aggressive and a little bit greedy, so that the game will not be come boring by computer winning 0.5 point. Weaker but funnier!

[Mike Novack]

Misconception?

The MCTS programs aren't set to play in some particular way. The "choose the next move" evaluator isn't some kind of go knowledged based artificial intelligence. Leaving aside a great many important details (and the devil is in the details) the idea is that the computer is trying a large number of (semi) random games from each of the possibilites and then making the move which had the highest percentage of wins.

Think of it this way. If from a given position, with move A there is a higher percentage of wins than with move B when two very strong players (euqally storng) try 10,000 games we would conclude that move A was better than move B. Well how about if two very weak players tried those 10,000 games as long as they were equally weak.

[prokofiev]

Mike: Are the academic papers you refer to about Go and MCTS or just MCTS? If the former, could you post a link to some?

Do you know: Are some sort of "local tsumego reading" modules implemented in such programs as well? How does that interface with the MCTS?

Sorry for gumming up your reviews thread!

[Mike Novack]

Mike: Are the academic papers you refer to about Go and MCTS or just MCTS? If the former, could you post a link to some?

The former, but I'm not sure what help you need in finding the academic papers. Keep in mind that several of these programs are essentially academic projects even if now offering a commercial version.

For a starting point, go to the Crazy Stone main site. Or since it was his initial paper that started this, just look for papers by Rémi Coulom. Look at the other sites of the programs that were initially academic projects. And once you have identified the journals where these sorts of papers are being published you can search those for the topic.