KaTrain Questions

[RobertJasiek]

That a command was copied on itself was my suspicion:) I will try your second echo syntax. Again, I will guess that "name" is a parameter, for which to insert one of KataGo's GTP commands.

katago.exe analysis -model MMMM -config CCCC -analysis-threads XXXX -override-config "homeDataDir=DDDD"

[...] I don't know if "-analysis-threads" or the rest is necessary.

Assuming -analysis-threads XXXX -override-config "homeDataDir=DDDD" is partially or fully needed, can you please give an example for this command part?

[RobertJasiek]

Each of the following commands creates the same output.

Code: Select all

C:\echo>echo -h | katago.exe gtp -model b18.bin.gz -config gtp_custom.cfg

C:\echo>echo -version | katago.exe gtp -model b18.bin.gz -config gtp_custom.cfg

C:\echo>echo -model | katago.exe gtp -model b18.bin.gz -config gtp_custom.cfg

C:\echo>echo -config | katago.exe gtp -model b18.bin.gz -config gtp_custom.cfg

Code: Select all

KataGo v1.13.0
Using NewZealand rules initially, unless GTP/GUI overrides this
Creating context for OpenCL Platform: NVIDIA CUDA (NVIDIA Corporation) (OpenCL 3.0 CUDA 12.1.107)
Using OpenCL Device 1: NVIDIA GeForce RTX 4070 (NVIDIA Corporation) OpenCL 3.0 CUDA (Extensions: cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_d3d10_sharing cl_khr_d3d10_sharing cl_nv_d3d11_sharing cl_nv_copy_opts cl_nv_create_buffer cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_device_uuid cl_khr_pci_bus_info cl_khr_external_semaphore cl_khr_external_memory cl_khr_external_semaphore_win32 cl_khr_external_memory_win32)
Loaded tuning parameters from: C:\echo/KataGoData/opencltuning/tune11_gpuNVIDIAGeForceRTX4070_x19_y19_c384_mv11.txt
Initializing board with boardXSize 19 boardYSize 19
Loaded config gtp_custom.cfg
Loaded model b18.bin.gz
Model name: kata1-b18c384nbt-s6386600960-d3368371862
GTP ready, beginning main protocol loop
? unknown command

[RobertJasiek]

The following citation of katago analysis -help might help somebody:

katago analysis -help

DESCRIPTION:

Run KataGo parallel JSON-based analysis engine.

USAGE:

analysis -config <FILE> ... [-model <FILE>] [...other flags...]


Where:

-config <FILE> (accepted multiple times)
(required) Config file(s) to use, can be one or multiple files (see
analysis_example.cfg or configs/analysis_example.cfg).

-model <FILE>
Neural net model file. Defaults to: (dir containing
katago.exe)/default_model.bin.gz

-override-config <KEYVALUEPAIRS> (accepted multiple times)
Override config parameters. Format: "key=value, key=value,..."

-analysis-threads <THREADS>
Analyze up to this many positions in parallel. Equivalent to
numAnalysisThreads in the config.

-quit-without-waiting
When stdin is closed, quit quickly without waiting for queued tasks

--, -ignore_rest
Ignores the rest of the labeled arguments following this flag.

-version
Displays version information and exits.

-h, -help
Displays usage information and exits.

[kvasir]

That a command was copied on itself was my suspicion:) I will try your second echo syntax. Again, I will guess that "name" is a parameter, for which to insert one of KataGo's GTP commands.

"name" is a GTP command that outputs the name of the engine. "list_commands" is another one that outputs the list of commands.

Assuming -analysis-threads XXXX -override-config "homeDataDir=DDDD" is partially or fully needed, can you please give an example for this command part?

From Task Manager (and I also tested it in the Katrain).

Code: Select all

C:\Users\Thor\Desktop\katago\katago-v1.13.1-trt8.5-cuda11.2-windows-x64\katago.exe analysis -model C:\Users\Thor\Desktop\katago\models\b18c384nbt-optimisticv13-s5971M.bin.gz -config C:\Users\Thor\Desktop\katago\katago-v1.13.1-trt8.5-cuda11.2-windows-x64\analysis_config.cfg -analysis-threads 12 -override-config homeDataDir=C:\Users\Thor/.katrain

For whatever reason '-analysis-threads' is 12, maybe this is a workaround in case 'numAnalysisThreads' is not in the CFG file.
'homeDataDir' is set to a directory that Katrain uses various data, I guess this is useful if you want different versions of KataGo to share things like TensorRT plan files and the results of OpenCL tuning.

[RobertJasiek]

Do you think my second echo outputs indicate a working KataGo?

list_commands: I see, will study that much later...

From my current tests, it appears that KaTrain expects several parameters submitted by the referred CFG file or by additional command paramaters. The gtp_custom.cfg has too few of those parameters KaTrain expects so I have become creative and copied analysis_example.cfg to analysis_config.cfg then set the following Override command:

C:\echo\katago.exe analysis -model C:\echo\b18.bin.gz -config C:\echo\analysis_config.cfg


KataGo engine is ready.

Processes:

Code: Select all

"C:\baduk\KaTrain\KaTrain.exe" 
C:\Windows\system32\cmd.exe /c "C:\echo\katago.exe analysis -model C:\echo\b18.bin.gz -config C:\echo\analysis_config.cfg"
\??\C:\Windows\system32\conhost.exe 0x4
C:\echo\katago.exe  analysis -model C:\echo\b18.bin.gz -config C:\echo\analysis_config.cfg

Playing. GPU load 97%. It works! Many thanks for enabling the start!

So it seems KaTrain is not buggy after all but a description of the syntax in its manual is essential.

Now, a fundamental problem remains: in the Windows command line, gtp_custom.cfg has been created for a purpose: better tuning of KataGo. So far instead only the renamed analysis_example.cfg works. Apparently, it will be necessary to merge both. Since the file is a) read by KaTrain and b) submitted as a parameter to KataGo when calling it, then this should achieve both: get KaTrain going and tuning KataGo. So tuning will be even more difficult. Besides getting the essential parameters right, one must also edit and merge both CFG files.

Anyway, the main insight is: a GUI and KataGo communicate in one of a few possible different modes, among which there are gtp and analysis. Some GUIs run on gtp mode while other GUIs run on analysis mode. KaTrain uses the latter.

How could earlier eager users of KaTrain figure out this without description in manuals?

[kvasir]

Do you think my second echo outputs indicate a working KataGo?

Yes, the error is about the command being unknown. Prior to printing that error some information about loading the model and other things is printed in the output. I also saw that you managed to get it to play moves in KaTrain.

Now, a fundamental problem remains: in the Windows command line, gtp_custom.cfg has been created for a purpose: better tuning of KataGo. So far instead only the renamed analysis_example.cfg works. Apparently, it will be necessary to merge both. Since the file is a) read by KaTrain and b) submitted as a parameter to KataGo when calling it, then this should achieve both: get KaTrain going and tuning KataGo. So tuning will be even more difficult. Besides getting the essential parameters right, one must also edit and merge both CFG files.

You can use 'katago.exe genconfig ...' to create a good config. It will ask some questions first and then run a benchmark to search for good parameters that affect performance.

I guess you could run it like this to create a new config 'C:\echo\new_analysis_config.cfg'. I haven't tested this exact command line, but I think you can figure it out.

Code: Select all

C:\echo\katago.exe genconfig -model C:\echo\b18.bin.gz -output C:\echo\new_analysis_config.cfg

You will have to add 'nnMaxBatchSize = xx', where 'xx' is some number, to the config file or there will be an error when you start KaTrain. Maybe this is something that is unique to the TensorRT version, since it is really the only thing that is missing. I guess 'nnMaxBatchSize' can be same number as 'numSearchThreads' but it is not clear to me. In my case I see that KaTrain specifies '-analysis-threads 12' on the command line, my config has 'numSearchThreads = 64' but I don't really know if it matters if nnMaxBatchSize is any number between 12 and 12 * 64 (I don't think I will run out of memory). I have always just set it to be same as numSearchThreads.

How could earlier eager users of KaTrain figure out this without description in manuals?

KaTrain comes with a working KataGo and you can download new versions in the program. You can also download different versions (OpenCL, Eigen, ...) but when it comes to CUDA and TensorRT they are not available in this way for Windows at least. For CUDA and TensorRT you have to install lot of libraries, like you have done, before they will run and there is no way around that. Still, the KataGo versions that you can download in KaTrain really work out of the box.

I think the documentation for KataGo and KaTrain is actually surprisingly good. If you look for it then you can find many text files in the GitHub repositories, that are documentation about various things, and KaTrain even has a manual.

[RobertJasiek]

KaTrain comes with a working KataGo and you can download new versions in the program.

My use of programs differs from some of their programmers' directed intentions: I use go AI software offline for PC security reasons. I prefer to download files manually under a separate online Windows user account.

I think the documentation for KataGo and KaTrain is actually surprisingly good. If you look for it then you can find many text files in the GitHub repositories, that are documentation about various things

Oh. It seems I need to dig deeply. So far, I have mostly noticed the main manuals of every GitHub project, and those are totally missing relevant information.

A few days ago, my opinion was that installing and using go AI well is 20 times as difficult as installing a large ordinary software suite. Meanwhile, I rather think it is 30 or 40 times as difficult.

I know, there are the quick installers to get something running. However, it does not make sense to only get things running somehow when different KataGo versions and tuning affect speed apparently more than the difference between slowest and fastest GPUs of a generation.

[RobertJasiek]

REFLECTION OF AN EARLY ATTEMPT

Having revisted various information of this week, I notice that there was an early hint to a command line like this:

Code: Select all

<path>\katago.exe analysis -model <path>\<model_name>.gz -config <path>\analysis_config.cfg -analysis-threads 12
-override-config homeDataDir=C:\Users\<username>/.katrain

KaTrain reported errors. At that time, I lacked background knowledge to judge whether the errors were meaningful or, as suggested on KataGo's information page, the result of the GUI not knowing how to interpret KataGo's error messages. Meanwhile, I have got some background knowledge and find KaTrain's error messages useful if they are specific (a char(0) error still tells me nothing).

At that time (and still now), the missing syntax description was (and remains) the major hurdle.

At that time, I think there might have been two causes for the errors: 1) analysis_config.cfg might have been missing; 2) the parameter -analysis-threads 12 should not have been used if analysis_config.cfg existed with the number of threads already declared there.

At that time, however, I knew nothing about possible causes for the errors. Instead, I had to guess causes. I thought:
- I gave the parameter analysis a chance but since it failed, I should revert back to trying the paramter gtp according to KataGo's information page.
- The parameter analysis seemed to be for also specifying an "analysis" type of config file, such as analysis_config.cfg. Since various sources of information suggested to specify gtp_custom.cfg, I thought that this would be more likely together with the allegedly expected gtp parameter.


UNCLEAR SUCCESS OF COMMAND LINE

Code: Select all

<path>\katago.exe analysis -model <path>\<model_name>.gz -config <path>\analysis_config.cfg -analysis-threads 12
-override-config homeDataDir=C:\Users\<username>/.katrain

Whether this works seems to depend on whether -analysis-threads 12 is needed or number of threads already specified in analysis_config.cfg. It is unclear whether -override-config homeDataDir=C:\Users\<username>/.katrain is needed.


SUCCESSFUL BUT COMPLICATED COMMAND LINE

Instead of my earlier reported successful command line, the following is an alternative but complicated one and with an error the user must ignore:

Code: Select all

<path>\katago.exe analysis -model <path>\<model_name>.bin.gz -config <path>\analysis_config.cfg -override-config homeDataDir=C:\Users\<user_name>\.katrain

This lets KaTrain tune, then output this error: ERROR: ERROR: Could not find any configuration that worked

Despite the error, then playing with GPU load 96% works.

[Gomoto]

Isn't it possible that the system uses the existing configuration without overwriting it due to the error?