zen7 30 second : leela-zero 30 second, Zen7 won consistently

[kyotosato]

I haven't checked it out yet, but i notice that in the web, some users had problem about time setting not working. the author said that the file gtp4zen.lua needs to be taken out of the directory where the gtp4zen.dll is located (or deleted if you don't use it), otherwise whatever you set in arguments or initial commands will be overwritten by the default (as long as the .lua file existed in the directory). my guess -z 6 maybe not work (default is -z 7, zen 7.) maybe i am wrong.
I use sabaki 0.33.2, and did not update it every time asked. it's my habit not to update any software until it is proven to be stabilized (time will tell.) hope this is not the reason.

[kyotosato]

pnprog:
Go GridMaster is just what I need for my android devices. thanks to your creation, a decent and nice product. i just wonder why you didn't continue to go further to AI 19x19?

[Vargo]

the author said that the file gtp4zen.lua needs to be taken out of the directory where the gtp4zen.dll is located (or deleted if you don't use it)

Bingo !
That was it, you just have to delete this file.
I've run two games between gtp4zen (zenV6) and AQ2.1.1 (H2, AQ takes white)
One game on a quadcore i7 and a gtx965, and one game on a 12-core i9 and 2x1080Ti
AQ won both at CGOS time settings, AQ took more time per move, so it's not very scientific, but still, it seems to be around 2-3 stones stronger than zen6. Thank you for this software, it's much easier to run matches with gtp than by hand !

[kyotosato]

would you mind post CGOS time settings in detail? so test may run in sync results will be more meaningful for getting some feeling of the strength for leelaz, AQ, zen6 or zen7.
Zen 6 or 7 if run by setting seconds than by dans will be more accurate i guess. it is not like 30s=5dan as far as i experienced. few months ago i used zen6 using 120 seconds (max strength) to play realtime in yahoo japan with a nihonkiin certified pro 2dan (so he/she said), zen6 won with 8.5 points, it took more than 3 hours. that was zen6 era, now i use zen7 and employ the method i mentioned above to play online. all games were saved, total around ~500 games.

[hydrogenpi7]

I have critizied Leela Zero a lot in the past but your test is 100% invalid and wrongly

First, you used only the 32 bit version which means it canot even use more than 4GB RAM.

Typically on my tests AQ uses 9GB ram and LZ uses 30GB ram, so that right there is basically a show stopper for your test.

Then you use a weak CPU with no GPU... which come on, is not a fair test... everybody has at least a gtx 1080ti these days. Any test or benchmark that doesn't include a >= gtx 1080 isn't a real world test imho

Run it again on 4X titan and lets see whether Zen7 can beat Leela Zero.


btw your system had no gpu, what is the point of seting playouts to half a million yet giving it only 30 seconds thinking time? must be a joke right?! it wouldn't even hit the limit even if you gave it 30 hours thinking time per move

as an "AI practitioner" you should know this is a rigged test in Zen7's favor. I'm willing to bet if they put AlphaGoZero on a Pentium I against Zen7, Zen7 would also win. proves nothing tho

[tchan001]

How can it be wrong when he is only reporting the results of his tests on his own system. As he provides details of the configuration of the computer and of the settings he uses, you are responsible for making your own judgments on how useful the results are for yourself. Of course, you are free to offer results from tests done on your own system and settings which may differ significantly from what he achieved.

[kyotosato]

comments noted, it is still a CPU/GPU prone problem/discussion, as i said result(s) is inclusive. anyone use LZ plays with in realtime with real 9dan players (pro or ama?) i don't mean online LZ bots. I did a lot games with 8-9 dans in fox and temgen.
for among go computer programs, my test is just to test strength of them for my own use. As an AI practitioner, my interests are NN, optimization, algorithm, subjective statistics among others..

[Vargo]

Three handicap games between LZ and Zen6 (Sabaki 0.33.4, win 10, 2x1080Ti, i9-12 core, 10" per move, no pondering)

H2, H3, and H4 : gtp4zen6 (B) v LZ13_5d6d9 (W)

Results : LZ wins at H2 and H3, and resigns after only 52 moves at H4... which is inconclusive I think. It mostly shows that LZ doesn't handle high handicaps well.

command line for gtp4zen :
-z 6 -t 12 -T 10 -s 40000

command line for LZ13 :
--gtp -w 5d6d9.txt --noponder -r 10 -t 4 --gpu 0 --gpu 1
time_settings 0 10 1

-komi set at 0.5

-It seems that Zen6 doesn't ponder : high usage of CPU and no GPU usage during zen's time, followed by high GPU usage and not much CPU by LZ.

-I don't have Zen7, but it's probably at least 1 stone stronger than Zen6.

- Only thing missing... gtp4CSDL , because CSDL has a very nice and precise kyu/dan setting

The games : H2 ,H3,H4

[kyotosato]

From my experience, Zen7 is about 2 stones or more stronger than Zen6. Two reasons: 1. end game has significantly improved, 2. opening has significantly improved. the opening recognizes mini-chinese--variation, and knows how to deal with it, zen6 doesn't. I used zen6 to play with 8dan in wildfox, 7 out of 10 lost because of opponents employing mini-chinese-variation.
30 second setting is a better setting, less than 30 seconds may have undershooting problem, and more than 30 seconds is ok, but if zen6 or zen7 uses 120 seconds may have overshooting problem. if you use the way i mentioned in my previous post to play Zen6 or Zen7 with opponents, you will notice that suggested spots/moves may change often after 30 seconds or more. zen6 7dan (or zen7 9dan) setting takes about 45s +-5s on my XP/pentium4 box.
the vision/GPU is a way to let AI has the ability to jump out of local maximum if needed. human players have that ability on behalf of vision, can easily have global view. opening of a game may benefit the most from vision. This is why i feel gpu matters.

[Vargo]

That's interesting! You have zen6 and zen7, I would be very much interested in a zen6 v zen7 match, it should be easy with gtp4zen.
As for the time, 10 sec per move with 12-core, rapid access and a lot of ram is probably at least as much as 30" on an older computer. I'm not sure but I think I read somewhere that zen6 or 7 couldn't use more than 10 cores (?)
Anyway, I'm looking forward to the future versions of LZ, Zen, AQ, etc

[kyotosato]

under same setting, same box, Zen6 vs. Zen7 i tried few times, Zen6 has no match (early in the game zen6 winrate drops steadily, then it is bloodshed.) zen6 always loss. My post is to see if any AI go accessible (by me for free or commercially available) could beat Zen7 or future zen. Current CrazyStone computer version is 7dan max, i have it and it is no match to zen7 (9dan max) or zen6 (7dan max.) i am looking forward CrazyStone new version commercial.
why i am so interested in Leela-zero, one is it is freely available, the other is because sooner or later LZ will surpass Zen7. they are two different animals. one is improving daily, and the other is fixed (discretely fixed). i have no objection of this possible outcome, and alphago master against alphago-zero matches have already proven that. the fact also proves that human heuristic experience is no match to AI heuristic experience. that means human guided search may give way to machine less-guided search. I do know why late Hawkins and Martin Heidegger both against AI development, however I do feel AI must continue but with caution.

[Mike Novack]

How can it be wrong when he is only reporting the results of his tests on his own system. As he provides details of the configuration of the computer and of the settings he uses, you are responsible for making your own judgments on how useful the results are for yourself. Of course, you are free to offer results from tests done on your own system and settings which may differ significantly from what he achieved.

Well yes, BUT ..... these programs have minimum hardware requirements for adequate performance. Just saying "equal" does not actually cut it if the hardware requirements are different. Imagine a short race between a Tour de France competitor and a seven year old child on EQUAL bikes (both sized for that seven year old child)

[Vargo]

I've just run another H3 game between LZ13 and Zen6 (Sabaki 0.33.4, win 10, 2x1080Ti, i9-12 core, no pondering, this time with 30" per move)

LZ wins after taking the 4 corners.

command line for gtp4zen :
-z 6 -t 12 -T 30 -s 80000

command line for LZ13 :
--gtp -w 5d6d9.txt --noponder -r 10 -t 4 --gpu 0 --gpu 1
time_settings 0 30 1


THE GAME

I am almost to the point where I'll buy Zen7 just to test the difference in strength between V6 and V7

[Javaness2]

black passed mid game...

[Vargo]

Yes, at move 114... that's weird...