CPU vs GPU

[Kirby]

How important is GPU to use Elf for analysis?

Presumably, you can get more play outs faster, but if Lizzie shows, say 50k playouts on a CPU vs. 50k playouts on a GPU, using Elf's network, is the evaluation the same?

I.e. if I let Elf think for long enough on a CPU, do I get the same quality analysis as GPU?

[Tryss]

It doesn't matter if you use a GPU or a CPU. The result is the same (or at least, should be)

Obviously, a CPU will be much slower.

[Kirby]

Thanks, that's what I thought. I haven't made the time to learn how elf works in detail, so I wanted to double check.

Has there been any study of move variation across playouts? At what number of playouts does elf typically converge to what it believes to be the best move? Does this number change as the game progresses?

[jlt]

Not sure that with an ordinary computer we are able to see convergence.

viewtopic.php?p=234917#p234917

[Kirby]

Hm, if there is no convergence maybe it's worthwhile to get a better machine with a GPU.

[Gomoto]

I use my desktop at home (10k-100k if I am interested in a move.)

But even with low visit count on my laptop (<100) you are able to spot mistakes in your game. It is really a great tool already with low visits.


desktop: shows you strong moves that are difficult to refute

laptop: shows you good candidate moves and most (>95%) of your own mistakes


But if you have the spare money and you are very interested in go, a good hardware is a reasonable choice.

[dfan]

Thanks, that's what I thought. I haven't made the time to learn how elf works in detail, so I wanted to double check.

Has there been any study of move variation across playouts? At what number of playouts does elf typically converge to what it believes to be the best move? Does this number change as the game progresses?

My recollection is that doubling the number of visits generally results in approximately the same increase in strength, whether that is going from 100 visits to 200 or 100,000 visits to 200,000. There is no reliable point at which the networks will always converge to an answer that will not change with further visits.

[Kirby]

laptop: shows you good candidate moves and most (>95%) of your own mistakes

I get the general sentiment that you are conveying, but I don't understand how this assertion can be confidently made with high confidence.

-How do you know there aren't mistakes it didn't catch?
-How do you know they are real mistakes if there is no convergence? Maybe another 10k playouts will give a different answer.
-Where does 95% come from?

I'm becoming warmer to the idea of using Elf, LZ, and crew for new ideas. But to give confident bounds on optimality is still a leap for me.

[Tryss]

-How do you know there aren't mistakes it didn't catch?

It's sure that there are mistake it didn't catch. If it didn't miss any mistakes, it would have solved go.

See it like that : if you eliminate all mistakes pointed by LZ (with high enough playouts), you'll be the strongest go player on earth. That seems good enough to me.

[Bill Spight]

laptop: shows you good candidate moves and most (>95%) of your own mistakes

I get the general sentiment that you are conveying, but I don't understand how this assertion can be confidently made with high confidence.

-How do you know there aren't mistakes it didn't catch?

The concept of a winrate (different from 1 or 0) is based upon making mistakes. Obviously, if both you and the program make the same mistake, it won't catch it.

-How do you know they are real mistakes if there is no convergence?

In theory there is convergence in the limit, as the estimated winrates approach 1 or 0.

-Where does 95% come from?

I'm becoming warmer to the idea of using Elf, LZ, and crew for new ideas. But to give confident bounds on optimality is still a leap for me.

The bots do not produce any error estimates or confidence bounds. I have met resistance to the idea of generating any.

[Bill Spight]

Has there been any study of move variation across playouts? At what number of playouts does elf typically converge to what it believes to be the best move? Does this number change as the game progresses?

My recollection is that doubling the number of visits generally results in approximately the same increase in strength, whether that is going from 100 visits to 200 or 100,000 visits to 200,000. There is no reliable point at which the networks will always converge to an answer that will not change with further visits.

For AlphaGo Teach, its team settled on a figure of 10,000,000 simulations as good enough. Not that they published any error rates or confidence intervals.

This is a general area I have been interested in for a long time. In 1968, as a student, I attended a conference of the New England Psychological Association. One speaker talked about his research in training physical performance of various complex tasks. For each task you might have different ways, different devices and algorithms to measure the person's performance. You could make your measurements more sensitive by altering the algorithms. He addressed the question of when to stop increasing the sensitivity of the algorithm.

In the case of the winrate estimates of Leela Zero, Elf, et al., we may consider them with different settings as different algorithms with different levels of sensitivity. One stopping criterion is to stop when the differences in winrates between successive settings become random. I have only compared Leela Zero at setting of 100k visits and 200k visits on a single game review. At those settings for that game it's not even close to meeting that criterion.

With neural networks my guess is that the main factor is the time taken to run the computer. Choose a setting that takes a reasonable amount of time and go with that.

Here is another approach. Suppose that with Elf we think that a delta of 10% or more is a likely mistake. Having run Elf with a certain number of visits, note the probable mistakes and run Elf with double that number of visits. If the delta for a certain move identified as a probable mistake is greater than its delta at the lower setting, take that as confirmation that it is actually a mistake.

Edit: Edited for clarity and improved memory.

[Gomoto]

Perhaps my remarks were misleading.

I did not intend to sound like 95% of all mistakes made are eliminated.

I did intend to compare the usefullness of desktop (100k visits) and laptop (100 visits). To my astonishment I am able to identify with my laptop around 95% of the mistakes compared to the mistakes I can identify with my desktop.

(I review my games after real life tournament on the road with the laptop, and then I review further when I am back at home on my desktop. And I cant remember I missed a critical mistake on the laptop, that I only discovered on the desktop. That is all I wanted to say. Surly 100k visits are preferable, but already 100 visits are a great tool for reviewing games. 100 visits show you good candidate moves that you missed perhaps in your game.)

A further remark:
If I want to really evaluate a move with AI I never take the shown percentage for the move. I always enter the variations and evalute the move from the endpoints of the variations. I think for now it is the wrong concept to think if one increases the visits one get substantial more reliable result. The result is much more reliable when you continue the variations and compare these.

[Kirby]

Thanks, Gomoto. That way of phrasing your thoughts makes a lot more sense to me.

At my level of play, using Elf to review a game on my cheap laptop is probably enough.

I have to admit, part of the reason i made this thread was to convince myself to buy a high end computer.

Sadly, I'll have to find another excuse.

[Uberdude]

I have to admit, part of the reason i made this thread was to convince myself to buy a high end computer.

Sadly, I'll have to find another excuse.

If you want Elf to not epic fail at ladders.

[Kirby]

Haha, yeah. Btw, I am referring to Elf now because I started using the elf network weights.

Is there any reason to use LeelaZero's weights?

Elf is stronger, right? I suppose you can get more ideas on a position by using both.