katago Handicap games

[Vargo]

Katago 1.3.1 wins every time at H9 against GnuGo (~5k), so, I tried higher handicaps.
At H10, it won by 20+
At H11 it won by resignation
At H12 it won by 2
at H13 (!) it lost by 10, then lost by 15, and won the 3rd game (below)

(Sabaki 0.43.3, laptop with GTX 1660Ti, komi=0, time_settings 0 7 1 (7sec/move) network=g170-b20c256x2-s1039565568-d285739972.txt.gz )

Against CrazyStoneDL (set at 5 dan)
it won at H6, H7, and H8 (below)

woaw !!!

[Vargo]

Same kind of results with gtp4zen7
Sabaki 0.43.3, laptop with GTX 1660Ti, komi=0
time_settings 0 7 1 (7sec/move) network=g170-b20c256x2-s1039565568-d285739972.txt.gz for katago 1.3.1

According to this site , gtp4zen parameters are :

zen7 5K : -t1 -r5 -s800 -n1 -o2.4 -p0.3
zen7 1D : -t1 -s1800 -n1 -o1.3 -p0.35 (and -r5)
zen7 5D : -t1 -s2700 -n2 -o1.0 -p0.55 (and -r5)

with these settings,
zen5k loses by resignation at H12
zen1d loses by resignation at H9
zen5d loses by resignation at H8

The games :

I'll try higher handicaps, and upload lost games too.

[Vargo]

Two handicap games with free placement of the handicap stones :
Gnugo plays n consecutive moves before the first white move.

GnuGo seems stronger if it can choose the places for the handicap stones.
At H12, GnuGo wins 3 times, with katago resigning (too?) soon. So I changed the resignThreshold to -0.9999
But katago lost this game :

At H10, with free placement by GnuGo, katago wins :

[Vargo]

After several 20-game matches at very high handicap (katago v1.3.1 v. Gnugo 3.8)
I find the results very surprising...
For example, at H17 (!) Katago wins ~20% of the games (twogtp 1.5.1, 3sec/move and resignThreshold = -0.9999 for Katago, desktop with GTX 1080)
Here is one :

At H15, on a laptop, katago wins about 50% of its games on Sabaki (7 sec/move with a 1660 Ti)

Is Katago absolutely incredible, is Gnugo getting weaker, or is it me, doing something stupid ?
Could someone please try it, thank you.

If you want the same places as I had for the handicap stones, download h15.sgf
1) In Sabaki, "File", "Open..." (h15.sgf)
2) "Engines" "Attach..." , Gnugo for B, Katago for W , and "OK"

[gennan]

If GnuGo is 5k EGF, then 15 stones handicap for 50% looks quite reasonable to me. It suggests KataGo is 10-11d EGF, meaning it should be able to give strong pros 2-3 stones handicap, which is consistent with actual game results of strong pros vs strong AI.

I'm even pleasently surprised by this consistency.

[gennan]

Experimenting with KataGo 1.3.1, I updated my table of point values of handicaps (without komi). (previous instance: viewtopic.php?f=15&t=16992&p=249931&hil ... ap#p249931)
Also in this respect, I'm pleasantly surprised by the improved linearity even including higher than 9 stones handicap.

Code: Select all

Handicap  Estimated  Ideal 
1           6          7
2          20         21
3          34         35
4          48         49
5          60         63
6          74         77
7          87         91
8         103        105
9         115        119
10        127        133
11        139        147
12        153        161
13        170        175
14        186        189
15        206        203
16        219        217
17        222        232

The estimated points value is based on KataGo's evaluation.
The ideal points value is based on the assumption that perfect komi is 7 points.
I used traditional fixed handicap placement and for handicaps higher than 9 I used 3-3 points for handicaps 10-13 and 10-3 points for handicaps 14-17.

The discrepancy from perfect linearity is less than 6% (except for the 1st entry, but there it's only 1 point).

To me this suggests there exists a nice and simple probability distribution of errors made by players of a specific rank. This might be useful to create a nice statistical model for rating systems based on ranks. It may even be possible to estimate a player's rank by analyzing a few of his games to find his typical error distribution.

The anchor of such a rating system would be perfect play (no errors). We don't know exactly which rank that is, but if we assume that "God" is barely able to give strong AI a 2 stone handicap, she would be about 12d-13d EGF (EGF rating 3250).

Actually I've been working on such a statisical model lately.

[Vargo]

According to this site, there are several patterns for very large handicaps.
With the Japanese pattern, Katago can win some games up to 20 stones against Gnugo.
At 21 stones , even with different parameters, I've not seen it win.

Katago 1.3.1 , Gnugo 3.8 , Gogui-twogtp 1.5.1, maxVisits=1000 and resignThreshold=-0.999 for Katatgo

Two games won by Katago :

H19 :

H20 :

[moha]

To me this suggests there exists a nice and simple probability distribution of errors made by players of a specific rank. This might be useful to create a nice statistical model for rating systems based on ranks. It may even be possible to estimate a player's rank by analyzing a few of his games to find his typical error distribution.

The anchor of such a rating system would be perfect play (no errors). We don't know exactly which rank that is, but if we assume that "God" is barely able to give strong AI a 2 stone handicap, she would be about 12d-13d EGF (EGF rating 3250).

You may also be interested in this, your idea sounds similar.

[And]

Katago 1.3.1 "g170 20 block s1.42G" (maxVisits=300, resignThreshold=-0.999) & AmiGo1.8 (~11k) H25:

https://sourceforge.net/projects/amigogtp/

[Vargo]

A chain of H9 games (komi 0.5, Sabaki 0.43.3)

KataGo 1.3.2 (16k visits, 20bs191) wins against KataGo 1.3.2 (1 playout, 20bs191) W+R

KataGo 1.3.2 (1 playout, 20bs191) wins against GnuGo 3.8 W+20.5

GnuGo 3.8 wins against AmiGo W+194.5

And maybe a very weak bot would lose at H9 against Amigo

[And]

And maybe a very weak bot would lose at H9 against Amigo

may be LZ 008 0.1k? (https://github.com/breakwa11/GoAIRatings), but against him LZ 004 0.01k?

[And]

Katago 1.3.5 "g170 30 block s2.27G" (maxVisits=50, resignThreshold=-0.999) & AmiGo1.8 (~11k) H41:

[And]

It is interesting that KataGo networks 15, 20 and 40 blocks do not want to play with such a handicap H41. pass and immediately or after a certain number of moves because of this lose. 30 block network somehow differently "sees"