I've made google colab notebook image which can contribute to katago training.
You can join the contribution without GPU now.
Just check the link below.
https://colab.research.google.com/drive ... sp=sharing

This looks cool. I’d love to know more. To start...

What gpu's does this use?

Is it free? For how long?

Is there some sort of limitation that might affect other google services?

Does it run in the background?

Thanks.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

edit: Is it possible to make a script like this to train the 15b net on the new s663 40b data? thanks!

If one is non-technical, and has never messed with the chrome console, and doesn't want to screw things up, where exactly in the chrome console should one paste this? Top? Bottom, embedded somewhere, doesn't matter?

Also, I assume you mean. "it turns off after 12 hours or less UNLESS you paste..."?

Finally, I agree that it is really simple. Even I got it running easily. If you want to help make the strongest open-sourced go engine even better, please run this in the background when you use your computer. Highly recommended!!!!!

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

Yes, sorry about that. In the picture below, paste the following code where the bottom most > sign is. Ill delete the previous redundant post.

I am trying this out also. It is indeed simple to do.
I am running it in Firefox and for whatever reason, it does not shut down by itself after 90 minutes. Just now I came back after about 3 hours and it's still running. Great job! Thanks

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

To answer my own questions.

1. I’ve connected to a Tesla T4 each time. Getting around 390nn evals/ second

2. It’s completely free. It uses Google Colab, a free machine learning tool. It runs in the browser, so it’s platform independent.

3. It does not affect other google services. There is a limitation within Colab in that it will stop working for heavy users. Looking on discord, it seems running it for twelve hours every other day avoids any problems.

4. Yes it runs in the background. Google provides the CPU's and GPU's. All you need is a google drive account and a browser.

If you’d like to run it more stably, for longer, and probably get assigned a better GPU, you can consider Colab Pro, which costs $10 per month with a US or Canadian address. That seems pretty reasonable versus buying and powering your own Tesla V100. I might try Pro soon to check out it’s performance. I’d love to hear if anybody else has already done so.

Again, I encourage anyone who wishes to help make katago stronger to consider running this completely free utility in the background.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

I followed your instruction and got an error as following...

Starting KataGo training...
2021-02-28 22:39:59+0000: Distributed Self Play Engine starting...
2021-02-28 22:39:59+0000: Attempting to connect to server
2021-02-28 22:39:59+0000: isSSL: true
2021-02-28 22:39:59+0000: host: katagotraining.org
2021-02-28 22:39:59+0000: port: 443
2021-02-28 22:39:59+0000: baseResourcePath: /
2021-02-28 22:39:59+0000: KataGo v1.8.0
2021-02-28 22:39:59+0000: Git revision: 8ffda1fe05c69c67342365013b11225d443445e8
2021-02-28 22:39:59+0000: Running tiny net to sanity-check that GPU is working
2021-02-28 22:39:59+0000: nnRandSeed0 = 10486611865130445872
2021-02-28 22:39:59+0000: After dedups: nnModelFile0 = katago_contribute/kata1/tmpTinyModel.bin.gz useFP16 auto useNHWC auto
terminate called after throwing an instance of 'StringError'
what(): OpenCL error at /home/dwugcloud/data/kata/cpp/neuralnet/openclhelpers.cpp, func err, line 263, error CL_PLATFORM_NOT_FOUND_KHR

When the process is running on colab, I am seeing these security warnings constantly in the console

Is this something that should be fixed or can we just ignore it?

Meanwhile I am currently increasing the "maxSimultaneousGames" at the bottom of the script. Going from 8 (default) to 12 jumped "nn evals" from around 380/second to around 470/second.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

My startup looks like this...

As far as I understand what we are doing here (questionable right there! ), should not be using OpenCL for anything. You should be using CUDA instead.
At the very beginning of the output from your run do you see...

This is what I get every time.

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

My errors are fixed by changing notebook setting. Setting hardware accelerator to "GPU' lets colab use TeslaT4.

Now it works fine.

The story so far...

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

Yeah, GPUs really like it when you have large batches to run in parallel, and more games helps with that.

The one thing I would caution is - please don't make the number of simultaneous games too large compared to the number of games you are playing in a given run before you shut it down or it shuts itself down - ideally make sure the total number of games you're getting per session would be at least 10x or 20x the number of simultaneous games.

The reason is if the total is too small, such that the games are coming in relatively few "waves" before it gets killed, it will create a bias towards short games in the data - because in the last wave, disproportionately short games will be the ones that finish and get uploaded and not the longer ones. Games that were on small boards, or that had fewer fights and were more peaceful, or that initialized starting from later positions, etc. will be favored over the configured and desired distribution.

Yeah, I also saw that every time until last night.

I ran the Colab script very late last night, and it assigned me to an A100, so I was excited. (Note, this information is wrong, and corrected in subsequent posts) But it didn't use CUDA, instead opting for OpenCL. I waited a half hour and it hadn't finished any games, so I quit and went to bed. I can't guarantee that my interpretation of what happened is perfect, but I think I'm right. Maybe the cat hit a kill switch. I should have taken a screenshot, sorry.

I assume the A100 should support CUDA, right?

Is there a known reason why OpenCL should fail on Colab?

Should I change the very beginning of the script currently set to KATAGO_BACKEND="AUTO"
to read KATAGO_BACKEND="CUDA"?

Anyway, today I'm back on a good old T4 and it chose CUDA again. Following recommendations, I increased the number of games to 16, and it's chugging along nicely. (530'ish nn evals /sec).

Thanks!

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

At the beginning of the script, probably we can take this

and make it this?

_________________
Dave Sigaty
"Short-lived are both the praiser and the praised, and rememberer and the remembered..."
- Marcus Aurelius; Meditations, VIII 21

After a few searches, I now suspect I was connected to a P100, not A100. A huge difference of course.

I'd still guess that P100 supports CUDA though.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

Again to answer my own question.

I was so excited to contribute to KataGo that I used the script a lot and Google put the breaks on my use of Colab. (Don't run it all the time, if you want to use it for free, folks)

I've upgraded to Colab Pro, because I'm still excited to help. And paying $10/month for the duration of the project is a hell of a lot cheaper than buying and running a powerful GPU.

Since upgrading, I've 100% been assigned P100 GPU, which I believe is supposed to be better than the T4. Perhaps that is so, but not for this job, with this Colab script.

After several tests on the P100, in which I forced it to use CUDA, it worked, but was significantly slower than OpenCL. So the script is right to use OpenCL.

Unfortunately, the P100 running OpenCL is also slower than the T4 GPU running CUDA. I am currently getting around 340 nnevals/second with 8 simultaneous games, and 385 nnevals/second with 16. (versus 380 and 525 nnevals/sec for the T4 running CUDA)

Unfortunately, with Colab, I believe you have to take the GPU you are given. And because I am a good paying customer, I consistently get the "superior" P100.

The good news is that I seem to be able to run scripts in two browser windows, and it disconnects far less often. And so far, Google hasn't told me to back off.

I still strongly recommend every go player try it. It's kind of like the SETI project for go players, except that Google is running the GPU's, so all you need is a simple browser tab. It's also easy, I'm running the script from the first post, just with my user name and password, no other changes. And it's free for those of you less obsessive than me.

_________________
- Brady
Want to see videos of low-dan mistakes and what to learn from them? Brady's Blunders

If you don't want to train katago, but just play rating games, would this work?
https://github.com/portkata/KataGo/blob ... ames.ipynb
probably have to replace cuda with auto.

Notebook image's been updated to use Katago v1.8.1 today.

KataGo v1.8.2 engines have released today.

So I've updated colab image to use new KataGo version.

You may need to copy colab image again so far.

Thanks.