It is currently Fri May 07, 2021 11:50 am

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 117 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6
Author Message
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #101 Posted: Wed Mar 17, 2021 5:23 am 
Beginner

Posts: 6
Liked others: 0
Was liked: 0
Where do u take those 60b nets? I searched all over https://katagotraining.org/networks/ and didn't find any. And how much stronger this 60b in comparison with 40b 697?

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #102 Posted: Wed Mar 17, 2021 8:04 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
while there are only two networks 60 blocks:
https://media.katagotraining.org/upload ... 221.bin.gz
https://media.katagotraining.org/upload ... 314.bin.gz

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #103 Posted: Wed Mar 17, 2021 12:01 pm 
Beginner

Posts: 6
Liked others: 0
Was liked: 0
Yeah, I know, but those links are not to the site, but download links. And where do you find them? I couldn't find anything about 60b nets on the katago site (https://katagotraining.org/networks/). Does someone upload them on github or what? Is there any other source where I can look for new ones when they eventually come, other than you and go4thewin who uploaded previous 60b net?

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #104 Posted: Wed Mar 17, 2021 4:04 pm 
Lives in gote

Posts: 660
Liked others: 108
Was liked: 799
Rank: maybe 2d
Join the discord server:
https://discord.gg/bqkZAz3

60b is not ready to officially take over 40b because it's not strong enough yet given the extra compute cost, these are nets that are posted to see how it's progressing. Join discord if you care about news or discussions of these things, or you want to know when cool stuff is posted (nets, experimental branches of code, stats, test results, games)

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #105 Posted: Thu Mar 18, 2021 9:33 am 
Beginner

Posts: 6
Liked others: 0
Was liked: 0
That's awesome, thanks!

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #106 Posted: Sat Apr 03, 2021 4:32 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
New b60 net https://media.katagotraining.org/upload ... 898.bin.gz

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #107 Posted: Mon Apr 19, 2021 6:06 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
KataGo v1.8.2
"This is a minor release mainly of interest to contributors to KataGo's distributed run, or to users who run KataGo self-play training on their own GPUs. This release doesn't make any changes to the KataGo engine itself, but it does fix an issue that was believed to be limiting the diversity in KataGo's self-play games. Switching to this version should, over the long term of training, improve KataGo's learning, particularly on small boards, as well as enable a few further parameter changes in the future once most people have upgraded, which should also further improve opening diversity."
https://github.com/lightvector/KataGo/releases

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #108 Posted: Sun Apr 25, 2021 4:58 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
New b60 nets
http://media.katagotraining.org/uploade ... 932.bin.gz
https://media.katagotraining.org/upload ... 424.bin.gz
https://discord.com/channels/4170221623 ... 8804732928

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #109 Posted: Mon Apr 26, 2021 8:09 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
KataGo b60c320-20210424 playouts 1, komi 0 - CS Zero 9d:


Attachments:
CS Zero - KataGo.sgf [1.7 KiB]
Downloaded 287 times

This post by And was liked by: go4thewin
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #110 Posted: Mon Apr 26, 2021 2:45 pm 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
KataGo b60c320-s275 playouts 1 - CS Zero 9d H2:


Attachments:
CS Zero - KataGo.sgf [1.78 KiB]
Downloaded 273 times

This post by And was liked by: go4thewin
Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #111 Posted: Wed Apr 28, 2021 5:34 am 
Oza
User avatar

Posts: 2341
Location: Tokyo, Japan
Liked others: 2244
Was liked: 1317
Rank: Jp 6 dan
KGS: ez4u
And wrote:
KataGo v1.8.2
"This is a minor release mainly of interest to contributors to KataGo's distributed run, or to users who run KataGo self-play training on their own GPUs. This release doesn't make any changes to the KataGo engine itself, but it does fix an issue that was believed to be limiting the diversity in KataGo's self-play games. Switching to this version should, over the long term of training, improve KataGo's learning, particularly on small boards, as well as enable a few further parameter changes in the future once most people have upgraded, which should also further improve opening diversity."
https://github.com/lightvector/KataGo/releases


The new v1.8.2 seems to run much slower than 1.8.1 on my machine. I am running contribute on windows 10 with an Nvidia GTX1650 super (no tensor cores, no FP16) using the opencl version. Moving from 1.8.1 to 1.8.2, I am seeing: -33% on plays per second, -30% on nn evals per second, and +61% on seconds per game. This is from the information in the log files.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #112 Posted: Wed Apr 28, 2021 6:27 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
I compared versions of eigenavx2 - 1.8.2 10% slower, several moves 20% slower, watched over 100 moves. The default_gtp.cfg files are the same, network 20b, 10 playouts :sad:

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #113 Posted: Wed Apr 28, 2021 6:50 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
OpenCL version 1.8.2 is 2.5 times slower! but there are differences in opencltuning files (GT 610)
EDIT I replaced the opencltuning file, nothing changed. 1.8.2 is also 2.5x slower! The default_gtp.cfg files are the same, network 20b, 10 playouts

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #114 Posted: Wed Apr 28, 2021 6:58 am 
Oza
User avatar

Posts: 2341
Location: Tokyo, Japan
Liked others: 2244
Was liked: 1317
Rank: Jp 6 dan
KGS: ez4u
When I run the cuda version on google colab, I do not see any difference. If anything, 1.8.2 might be a little faster. However, that was with quite short runs versus over 24 hours when I looked at the opencl versions on my machine.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #115 Posted: Wed Apr 28, 2021 7:01 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
obviously something is wrong :)

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #116 Posted: Thu Apr 29, 2021 4:06 pm 
Lives in gote

Posts: 660
Liked others: 108
Was liked: 799
Rank: maybe 2d
We've figured out what's going on - it's not a matter of the version, but apparently on some GPUs KataGo's tuning is noisy enough that it it has an occasional chance of picking to not use FP16 storage when FP16 storage is beneficial, so on those GPUs, every time you manually re-tune you have a chance of getting different performance.

I'm not entirely sure how to fix it without making the OpenCL tuning process heavier and slower. But definitely something I'll think about for a future release - @ez4u thanks for the report and being so careful to check and notice things. :tmbup:

Also, is there a place in the docs where I can write or fix to avoid giving the impression that people have to manually retune each version? Although, I guess it was really good that you did anyways, otherwise we wouldn't have known about this quirk about tuning. :)

Top
 Profile  
 
Offline
 Post subject: Re: KataGo Distributed Training and new networks
Post #117 Posted: Mon May 03, 2021 6:58 am 
Lives in sente
User avatar

Posts: 754
Liked others: 153
Was liked: 134
KataGo s790 playouts 1 - CS Zero 9d H2:


Attachments:
CS Zero - KataGo.sgf [1.48 KiB]
Downloaded 78 times
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 117 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group