It is currently Tue Oct 27, 2020 10:43 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 8 posts ] 
Author Message
Offline
 Post subject: KataGo Versus Golaxy - Kata's got game
Post #1 Posted: Mon Oct 19, 2020 2:03 pm 
Lives in sente
User avatar

Posts: 843
Liked others: 302
Was liked: 343
Bob Song recently ran a series of games between high-powered versions of Katago versus Golaxy. As many of you know Bob is huge supporter/fan of Golaxy. Given all the development time and money that has gone into Golaxy, combined with the recent World AI Go qualifiers, I think he expected Golaxy to dominate. I certainly did. But the picture wasn't so simple.

Before I show the games let me give a few notes:
1) I didn't run them, and can't speak to any config parameters
2) Golaxy's public engine is not as strong as its tournament engine, though it is 50 blocks, which theoretically allows greater strength than katago's 40 blocks.
3) I believe that in terms of playouts, each 6x of Golaxy speed is equivalent to a 2080ti in katago playouts, so approximately 1400 playouts per second. Golaxy has not publicly disclosed their hardware.
4) It is unreasonable to draw conclusions about which engine is stronger from a small number of games.

Game 1:
White: Golaxy Pro AI 24x speed - averaged 5.6k playouts per second on 50 block engine
Black: Katago 1.61 with 8 2080ti gpu's - averaged 8k playouts per second on 40 block engine

Time: One hour base time, 60 second byo-yomi



Notes:
    *Katago used most of its time each move. Golaxy would often play very fast, presumably when it was certain a particular move was right. Both bots had pondering on, meaning they were thinking on the other player's turn.
    *Katago believed it drew ahead somewhere by move 97 (that's where my much slower katago finds it).
    *Golaxy had a blind spot somewhere on the upper right fight. Before the exchange it thought white had a 63% chance of winning versus Katago's thinking white had a 31% chance. Once the exchange happened Golaxy agreed with Katago and claimed 35.5%
    *Move 149 really surprised Bob and me. We thought its purpose was to say, "I could connect and fight, but instead I'll play this and claim the left side." But white both lived and took away the left side. Further analysis with Katago showed it could neither kill white, nor claim the left side, so I guess it was all okay. I believe Golaxy thought it lead from this point.
    *Katago thought it lead for a long time after this, until maybe move 188, then its winning percentage continuously declined. Golaxy was very confident and stayed that way until near the end.
    *I think the ko at 235 was key to complicating the game and giving black a chance.
    *By 243 katago's believed it lead.

Thanks to Bob Song for running this and the other games. More to come soon, including some where Golaxy had the faster rig.


Attachments:
Golaxy 24x-Katago 8x2080ti.sgf [3.13 KiB]
Downloaded 549 times

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

This post by wineandgolover was liked by 2 people: ez4u, Limeztone
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #2 Posted: Tue Oct 20, 2020 1:50 pm 
Lives in sente
User avatar

Posts: 843
Liked others: 302
Was liked: 343
Game 2: Kata Gets White - A Coast-to-Coast Victory

So in game one, katago 8x2080ti eked out a 1.5 win against Golaxy 24x playing black. Both AI's prefer white with 7.5 Komi, so how would Katago fare with white?

White: Katago 1.61 with 8 2080ti gpu's - averaged 8k playouts per second on 40 block engine

Black: Golaxy Pro AI 24x speed - averaged 5.6k playouts per second on 50 block engine
Time: One hour base time, 60 second byo-yomi



Notes:
* Katago believed it lead throughout the whole game, increasing its lead from 0.8 points to 1.3 points by around move 50.
* I remember one of AlphaGo's creators saying that once it thought its lead was 70%, the game was over. I don't think this holds true when different AI's and their different blind spots play, but anyway, by move 64, Katago believes its winning percentage is over 71% with a slightly better than 2 point lead.
* For me, Golaxy's move 75 was a surprise, but katago also saw it as the first choice. Both katago and Golaxy generally agree with each others' moves, even those that surprise humans, which should really be the subject of a whole other post. White's lead is now slightly over 3 points.
* By move 84, Katago believes its lead is over 4.5 points, and winning percentage over 80.
* There is something to be learned with white's play starting at move 86 in the upper left. Katago never takes gote, but plays a series of forcing moves, leading to a way out, if not safety. I suspect these moves also increase ko threats, which will prove useful soon.
* The sequence starting at 90 settles white's "weak" group on the right, putting pressure on black's weak groups and further increasing white's perceived margin and winning percentage.
* By move 100, Golaxy needs to find something to kill and encloses the upper left in a very large way. Katago isn't impressed, believing it has a 7 point lead, and 84% winning percentage. Golaxy agrees, giving itself only 20%
* By move 126, Golaxy seems to have subdued the upper left, but white is threatening the upper right. (11 points, 92%)
* Through 146, black saves the upper right, but white gains tremendous thickness in sente, and commences the attack in the lower right. (15 points, 93%)
* 158 The ko that marks the beginning of the end. Though white threatens to capture the whole string, half will do, making an interminable series of ko threats. Black resigns before white even makes much use of his corpse in the upper left.

This seems a good example of the leading side just slowly but continually building its lead until the losing side has to try something drastic. It was an interesting game, but in our next games, Golaxy upgrades to 60x power, and the games get even more interesting.


Attachments:
KataGo 8x2080ti-Golaxy 24x.sgf [1.6 KiB]
Downloaded 444 times

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders


Last edited by wineandgolover on Tue Oct 20, 2020 3:11 pm, edited 2 times in total.

This post by wineandgolover was liked by: Limeztone
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #3 Posted: Tue Oct 20, 2020 2:36 pm 
Beginner

Posts: 4
Liked others: 0
Was liked: 1
Rank: KGS3dan
Hello, Thank you for placing the interesting information.
I have a few questions to ask, please let me know.

1. What does "24 x speed" mean? Is there any difference in computational resources from KataGo's 8x2080Ti?

2. There is an upgrade to 60x power. Is Golaxy still training?

3. Currently, Golaxy is a paid service, but if you know the level of computational resources used, please let us know. Is KataGo more valuable if you use a high performance GPU?

Thank you

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #4 Posted: Tue Oct 20, 2020 3:06 pm 
Lives in sente
User avatar

Posts: 843
Liked others: 302
Was liked: 343
hope366 wrote:
Hello, Thank you for placing the interesting information.
I have a few questions to ask, please let me know.

1. What does "24 x speed" mean? Is there any difference in computational resources from KataGo's 8x2080Ti?

2. There is an upgrade to 60x power. Is Golaxy still training?

3. Currently, Golaxy is a paid service, but if you know the level of computational resources used, please let us know. Is KataGo more valuable if you use a high performance GPU?

Thank you


1. Golaxy doesn't disclose their hardware. 24x refers to how much computing power Golaxy is providing. You pay for their service on a sliding scale. I am told that 3x, their base level provides 700 visits per second, which is already pretty strong. 24x is eight times faster, and therefore stronger. I am told that a 2080ti gets a bit better than 1000 visits per second on katago's current 40 block engine. By that math Golaxy is getting 5600 visits per second and katago is getting 8000, but Golaxy has a 50 block engine versus katago's 40. So the comparison isn't exactly apples-to-apples, but should be reasonable.

2a. 60x Golaxy is super fast, approximately 14,000 visits per second. Within each AI, faster corresponds with stronger, because it can explore wider and deeper in the same time.

2b. Yes, Golaxy is still training, and I am told that their publicly available net is not the same as the one they used in the recent qualifying rounds of the recent World Go AI championship.

3a. Golaxy pricing is complicated. In general you pay more for increased speed. If you use it a lot, you can get some discounts. I don't have the details handy, but it isn't super-expensive. Here is their English site: https://www.19x19.com/engine/index

3b. Yes, katago is better with a faster GPU. What you need is dictated by your purpose. Katago with almost any GPU is strong enough to find all of my mistakes. Midrange GPU's will help you quickly see professional mistakes. If you want to look at powerful bot games, faster is better.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #5 Posted: Tue Oct 20, 2020 9:29 pm 
Beginner

Posts: 4
Liked others: 0
Was liked: 1
Rank: KGS3dan
Thanks for sharing the details! :)

Considering that Golaxy is 50b and KataGo is 40b, 5600visits/s and 8000visits/s look somewhat reasonable. I haven't checked many of the validation results, but based on the results of the competition and your reports during this time, I get a sense of how good KataGo is.

I'm using an RTX2060 and I'm getting about 500 visits/s with the KataGo 40b. I am an amateur 5d or so, so this much performance is more than enough for me.

By the way, KataGo has 40x384, and I'm also interested in how this compares to Golaxy.

Thank you.


This post by hope366 was liked by: sweatypie
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #6 Posted: Wed Oct 21, 2020 3:29 pm 
Lives in sente
User avatar

Posts: 843
Liked others: 302
Was liked: 343
Game 3: Golaxy Upgrade - A One-Sided Affair

So after losing two games at a super-human 24x, Golaxy went to god-level at 60x for the next match. Could Katago up?

White: Golaxy Pro AI 60x speed - averaged 14k playouts per second on 50 block engine.
Black: Katago 1.61 with 8 2080ti gpu's - averaged 8k playouts per second on 40 block engine

Time: Two hour base time, 60 second byo-yomi



Notes:
* With Chinese rules and a Komi of 7.5, white starts with a 60% chance of winning, and a 0.8 point lead, according to Katago
* Is the ladder at 20 good? (I know the answer, it's a rhetorical question)
* With 23 and 25 black takes gote in the upper right and lower left corners, which seems unusual. Of course the upper right is half dead, and katago thinks the score is that same as at move 1.
* I can't make my version of katago agree with move 33. It prefers the 3-3 invasion, by 1.3% and 0.2 points at 850k playouts. That loss is sustained in the followup moves.
* Move 41, half-sacrificing the lower left corner to build the right is pretty cool. (black 38%, -1.2 points. White accepts the offer.
* For move 44, at low playouts, katago thinks white should have invaded at R13, one of the few moves the bots have disagreed on so far. (2.9%, 0.3 point difference)
* 49 and 51 are both cool. Does 51 kill the corner and/or help the right side? (Again, rhetorical)
* Katago also sees the equally cool 52
* Golaxy's 54 feels human and mean, but with 55, Katago thinks its a little better off than the start of the game (43.3%, -.6 points)
* With 56 to 67, Katago says, "this is fine" and believes 68 is a mistake making the game even. After 79 Golaxy has a very different opinion, thinking its lead is 72%
* Katago is fine with Golaxy's move 80, but prefers some forcing moves starting with O16 first. But it doesn't make a big difference in its analysis. White leads by a bit (58%, 0.4)
* White's tenuki at 82 is a complete surprise to Katago on my system at 200k playouts. Rather it sees it, but thinks white should resolve the upper right first. (8%, 0.7 point difference). I think it's just a sequence difference, and by resolving the upper right first, Katago realizes white leads (60.4%, 0.7). Two moves later, Katago soon sees that white 82 leads to the same result. 82 looks like black 51, but it kills the black corner outright.
* White 92 shows that black 51 allowed a ko for life. Do I hear Golaxy laughing? (white leads 62.4%, 0.9 points)
* White 96 is confident and cool, and katago sees it coming. (63.3%, 0.9 points)
* Is Katago's 105 a horizon pushing move? I don't see how it helps, but maybe it pushes the final analysis of loss a little further away?
* By move 108 Katago thinks its behind by more than 77% and 2.6 points.
* Katago resigns after 132, thinking its behind (>90%, >6 points)

This game was a lot like the prior, but more so. White started with a small lead, and gradually increased it. There were no big errors, no fireworks, just inevitable doom.

Golaxy 60x is strong! I wonder if even playing white, whether single machine katago can keep up with this distributed monster. Stay tuned.


Attachments:
Golaxy 60x-Katago 8x2080ti.sgf [1.12 KiB]
Downloaded 340 times

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

This post by wineandgolover was liked by 3 people: ez4u, Limeztone, sweatypie
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #7 Posted: Wed Oct 21, 2020 5:12 pm 
Lives in sente
User avatar

Posts: 843
Liked others: 302
Was liked: 343
A quick update on Golaxy pricing from Bob Song.

The minimum 3x speed, which gives you 700 playouts/second, costs 9 CNY (1.35 USD) per hour. The max 120x speed, which gives you 28000 playouts/second, costs 412.8 CNY (62 USD) per hour. If you are a heavy user, you can a VIP card for discounts up to 25%. There are many other permutations and discounts, but this gives a good idea.

The minimum 3x speed is very strong. According to the co-developer Dr. Xiaochuan You, the speed for Golaxy 6x is as fast as a single 2080ti (1400 v per second.

Unless you are studying for an international championship or reviewing/playing top-bot games, I can't imagine a need for 60x or 120x speeds.

https://www.19x19.com/engine/index

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders


This post by wineandgolover was liked by: sweatypie
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Versus Golaxy - Kata's got game
Post #8 Posted: Thu Oct 22, 2020 5:15 pm 
Lives in sente
User avatar

Posts: 843
Liked others: 302
Was liked: 343
Game 4: Katago Gets White And It's a Brawl.

In the prior game, Golaxy and its 60x distributed engine played white and crushed Katago. Is taking white enough to let Katago catch up? This game is the most interesting so far, with neither side willing to back down and take a close loss. I've even added a few variations to the sgf.

White: Katago 1.61 with 8 2080ti gpu's - averaged 8k playouts per second on 40 block engine

Black: Golaxy Pro AI 60x speed - averaged 14k playouts per second on 50 block engine. Distributed architecture

Time: Two hour base time, 60 second byo-yomi



Comments:
* My low-power Katago did a better job of predicting Golaxy's moves than KataGo's moves. Katago struggled to agree with some of its moves, even after being shown them.
* Move 4, Oooh, diagonal opening. This could be fun. Do bots often choose diagonal openings?
* Move 19, It's really intriguing how Katago shifts from upper right to upper left and back again over the next few moves.
* Move 28, Low-power katago prefers P18, by 3% and 0.4 points at 200k playouts. I guess more playouts yields quite a shift.
* Move 42, It felt like Golaxy pushed white around, but Katago is fine with this shape, thinking it leads (62.4% winning chance, 0.8 point lead)
* White 48 surprises low-visit Katago, with B17 and E8 getting the overwhelming majority of the visits. Through 100k playouts, white 48 had only 45 visits and was 2% and 0.2 points behind the other candidates. Katago instantly like it when shown. Finding unlikely moves is the power of many playouts.
* white 68: low playout Katago does not prefer this move, far preferring N6, then M6. Even at 700k playouts, Q8 is 1.6% and 0.3 points behind. But this moves sets the table for the coming fight, clearly splitting black in two.
* White 78 is a cool move, either gaining a partial base as in the game, or gaining sente before running to the center, as in the variation. Katago thinks white leads (64.9%, 1.4 points)
* White 84 What the hell, a quick forcing move, who cares about the fight.
* White 88 I suspect that white sees the fighting overflowing to this side of the board, and it decides to get a quick forcing move in. Interestingly, at 300k playouts, katago doesn't seem certain that it's truly forcing, and black could tenuki at p9 (39% winning rate), in many variations dominating the lower right, and losing out on the bottom or the left. I played a ton of variations starting at p9, all varying between 30-45% winning rates for black, with drastically different looking outcomes. Perhaps this chaos would have been a better choice for Golaxy.
* White 98 is hard to find at low playouts. At 100k playouts, Katago thinks it is 8% and 2 points worse than O5, and thinks it's a losing move.Even when shown P5, it takes katago 8000 playouts to realize it's good.
* White 106 is somewhat hard for katago to find, katago preferred L9 for at least 200k visits. It took 50k playouts before it agreed with the move, after it was played.
* 108 Despite the fact that both black and white have two weak groups, Katago's estimate of score hasn't changed much (60%, 2.8 points). Given that each has one group sealed in, and one group running, it's hard to see it ending peacefully.
* Black 111 seems necessary to live with the bottom group, but white 112 pretty much kills black's lower right group. See the variation for black's best attempt to save the group in ko.
* Black 113 and 115 are cool. When katago avoids the ko in the lower right, these moves enclose white, and add a new group to the fight. I think white 88 saw this coming.
* black 143 - Are this, 145 and 147 horizon extending moves? They seem a waste of ko threats.
* white 158 creates two cuts, and turns black's attack into a fight.
* black 171 creates a huge ko, but black needs larger threats than white.
* if black 173 captures the ko at F1, white captures the middle black group with P11 and leads by 72%, 1.4 points. Golaxy chooses to fight on, instead.
* from 174, both sides make outside groups safer, fill outside liberties, and exchange monster game winning ko threats.
* Golaxy resigns after white 244. If black answers this threat, it does not have a threat big enough. See the variation, in which white throws away the upper left, but still wins handily. 99.5%, 9.5 points

I am not going to say Katago is stronger than Golaxy. But I am delighted to see that it can play well even against a super-strong distributed version of Golaxy.

Time for a wee bully-pulpit:

There are at least three ways to measure an algorithm's (bot) strength.

1) Which is most efficient to develop? Google's Deepmind spent a fortune building AlphaGo. They dedicated resources, both hardware and engineering that few others if any could match. Now we've reached the point where a mostly solo developer, David Wu, renting cloud GPU's is able to challenge multinational corporations. Pretty cool. It would be fun to host an AI Go Challenge that included development in the process.

2) Which bot is strongest without hardware restrictions? This current World Go AI Championship is following this format. I guess its fine, but not really meaningful. The reality is that a worse engine on better hardware can win. If one of the finalists in this year's contest comes with a bank of A100's to compete against 2080's, it's going to do very well. So what does this format prove? Well deep pockets can make the strongest bot. I think both Golaxy and FineArt will continue to do very well, so long as they retain funding, even if their algorithms fall a little behind. (Note, I have no reason to believe they've fallen behind, and I'm not saying they have.)

3) Which bot is strongest on equivalent or identical hardware? For me this is a real contest. Let each developer equip identical machines, and see who comes out on top. I also think individual games are somewhat meaningless. Just as I wouldn't claim katago is stronger than Golaxy despite winning three out of four of these recent games, I find the existing championship format of very few games somewhat meaningless. I'd like to see each bot play other bots many times. They aren't human, there is no reason to restrict bots to one game a day.

Ah well, that's enough for today. I liked this game!

Next - What happens if you let Katago play with ten cards?


Attachments:
Katago 8x2080ti-Golaxy 60x commented.sgf [5.53 KiB]
Downloaded 202 times

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

This post by wineandgolover was liked by 3 people: Dusk Eagle, Limeztone, sweatypie
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group