Life In 19x19

Are there any interested parties, i.e. people working on Go software, reading this?

While working on a few things in q5go I've run across a few situations where the SGF file format is inadequate. The major issue is that it cannot be safely extended: any extra properties a program might add which go beyond the specification could be misunderstood by another tool that chooses the same property name with a different meaning.

The imminent problem is that I'd like a way to save LZ evaluations. There is V[], specifying a value, but it's defined as an estimated score (negative for White). I suppose I could set V[0] to represent 50% winrate to and adjust everything else accoringly to a range of +/-50. But it would be nice to also save the visit count that was used to arrive at that evaluation, and the komi used by the program (which could be different from the game, and an application could have an option to flip between 7.5 and -7.5 for engines with fixed komi).

Ideally there'd be a namespace which programs could use to define their own properties, such as Zxx[]. A program would have to verify AP[] first to see whether it understands these or not. This would of course only be useful for properties that can safely be ignored. For others, the issue of interoperability is less critical - for example, for toroidal boards I've chosen to simply use a different (and invalid, according to the spec) GM, to make sure the file isn't misinterpreted by other tools. In an ideal world we'd perhaps also have another namespace (Yxx?) for non-standard but also non-ignorable properties.

So, what to do with the immediate problem? Overload V[] or add a new ZV[val:visits:komi]?

Marcel Grünauer wrote:I solved this for my project by wrapping SGF in a JSON object. That JSON object also contains metadata and other custom properties. I call this "SGJ" (Smart Game JSON).

Hmm. That's of course not very backwards compatible - the problem I'm looking at is "how do I write an SGF file that has some annotations for analysis, but can still be understood by other editors?". Ideally we'd get a sufficiently large group of authors to agree on a backwards-compatible extension to FF[4] SGF. We could also try to invent a new file format (there seems to have been some sort of half-developed XML spec), but unless a lot of editors implement support for it, it won't have the kind of interoperability I'm looking for.

Consider to contact Anders Kierulf.

bernds wrote:Are there any interested parties, i.e. people working on Go software, reading this?

I am interested!

For GoReviewPartner (which really just does that: save LZ evaluations), I use a set of non standard SGF properties, and I just changed the file extension to RSGF (for Reviewed SGF).

So the resulting file can be opened with a standard SGF editor, but the data won't be available. The properties are used to record data such win rate, policy network value, monte carlo values, number of playouts, reading depth... I stick to 3 or 4 letters properties to make it obvious they are non standard.

I would be happy to switch/contribute to a more standard format, if it can cover all the needs of GoReviewPartner.

I think the developer of Sabaki would be interested as well, see: https://github.com/SabakiHQ/Sabaki/issues/487

Marcel Grünauer wrote:As far as I can see on https://www.red-bean.com/sgf/ there are no properties - general or game-specific - that start with the letter "X".

In email, "X-headers" are custom headers that are added to the standard mail headers (for example, "X-Spam-Level"), and likewise "X-properties" could, if developers of editors and viewers agree to it, be custom properties.

So you could have XWR[] for win rate, for example.

I'd be happy with that, it's essentially the thought process I had with "Z" instead of "X". The only thing is,

It would then be helpful to have a registry of common X-properies so that developers don't use two different X-properties for the same concept.

X-properties should be documented like the standard properties are on https://www.red-bean.com/sgf/properties.html - that is, what values (or range of values) do they take, is it a single or a list value etc.

Editors should ignore any X-property that they don't understand and just preserve them between reading and writing the file.

All this sounds a lot like standard properties, if they are documented and editors agree on them. And SGF already documents that unknown properties should be preserved, the problem is how to safely add new ones. Of course, getting properties documented would be ideal, but the SGF standard seems to not have been touched in over a decade (other than a "current practice" page by aeb), which is why I started a thread here. But I'll send an email to Arno, Anders and aeb, to see if any of them have any input.

But the problem that X-properties should be trying to solve is to make it possible to add extra data to an SGF file without fear of name clashes. So what we really need is for the X-namespace to require that before interpreting any properties in it, the client must check against AP.

Maybe the win-rate isn't the best thing to place into X then, since it's generally useful. Ideally we'd have a new documented property. Since WR is taken, I'd propose VR[wr{:visits{:komi}}] (wr:float, win percentage for Black, range 0-100, visits: integer, komi: float). The latter elements can be omitted entirely, but the form [wr::komi] is also valid.
If this is added to any SGF document, I'll use it - otherwise I think it'll be XWR as suggested by Marcel, with the same interpretation.

Does this kind of plan work for everyone here?

a few scattered thoughts:

this innocent-sounding topic is possibly the single most important issue this forum could address, because it is an issue that will impact developers and users alike for decades to come.

a glamce at https://www.red-bean.com/sgf/properties.html gives me the impression that noble though the sgf effort was, it has become a pig's breakfast, much as many iso standards also are, and the London road system is.

things evolve, and nature has a way of weeding out the unworkable, but that takes time, so it is sensible and timely for interested parties to try to work out a standard that will be both backwards-compatible and forwards-flexible.

sgf seems to be based on a 2-character code; the bit that caught my eye were the lines:

Game Info Properties AN, BR, BT, CP, DT, EV, GN, GC, ON, OT, PB, PC, PW, RE, RO, RU, SO, TM, US, WR, WT
Miscellaneous Properties FG, PM, VW

This might provide the opportunity needed to include additional information that developers want to be able to transmit wihout making the message uninteliigible to old readers. There must be, among that pile of codes, one that none of them use, so by using it, they won't be disturbed - they will merely be blind to it, which is fine, since they don't know what to do with it anyway.

but i don't vote for such a solution, because there's sure to be a squabble about which code should mean what, and there may be instead a much cleaner and more workable approach:-

at the os level, files have a special eof character code - does sgf also have a special "here endeth the lesson" code?

if so, therein - ie after it - lies the best place to stuff whatever information your writer wants to create, which could be as non-standard as you like - whereas this means that not every reader would necessarily be able to read it, i don't see that as a bad thing, for if you write something people like, other writers will copy you, and if they don't, they won't. An analogy might be what happened to English newpapers, which all eventually switched from broadsheet to tabloid - ironically, just before they are about to become extinct!

LZ creates additional point data, which everyone is going to want to display some of (so far, far too much of, in my unhumble opinion!) but there are other writers, such as OGS and baby irgobot, that write data that spreads across, and between, points as well.

So having an extra basket after the "end of sgf" code into which a writer can put anything they want would keep everyone happy.

So i reckon the best way to extend sgf is to not extend sgf, but to extend the file containing sgf instead.

If anyone is serious about possibly developing FF[5] of SGF, https://www.red-bean.com/sgf/email_list.html is the proper place of discussion.

Not all sgf editors read sgf files correctly. In fact, I'm told that very, very few do. Even the Eidogo one used on this forum seems to choke on quite a few things.

So extending the format seems to be a recipe for more confusion. A fresh start using e.g. xml may be wiser.

One simple solution for the OP seems to be to use the C[ ] property. The info he wants can be considered a kind of comment anyway, but he can also bracket it in some coded way within the C[] text so that the info can be used or extracted programmatically.

RobertJasiek wrote:FF5

the FF4 crowd have already put in a lot of time, sweat and determination, and have already got their hooks into chess and backgammon as well and are no doubt Scrabbling to get Monopoly too, so unless you have a Trumpwall to keep them out, you may as well join'em 'cos you sure cant beat'em.

Children are taught to obey their parents, but i have often thought it should be the other way round, so i asked irgobot what language she would like to speak.

i was a little taken aback when she replied "WTF".

But slowly it dawned on me that she was following in the time-honoured tradition of using a three letter word; like SGF, FF4, BNF, ISO, URL, WWW, WFF, WTO, EEC, ABC, BBC. CBC, CNN, FRS, LIB, LAB, CON (the clue is in the name), MI5, CIA, WW3, ACT, GOD, and NWO, which are so much more respectable and easier to remember than offensive 4-letter words such as KISS, OHMS, CTRL, BNFE, NATO, 1984, and BEAN.

WTF is shorthand for "Well-Trained Format", which seems entirely apposite, for there is nothing so delightful as a well-trained dog which only barks when you tell it to.

And, by some fluke, it turns out that SGF insists on starting with "(" and ending with ")", so all old readers will ignore anything that comes after that in a .sgf file containing WTF code after the last SGF ")".

POC! -> TTFN

John Fairbairn wrote:Not all sgf editors read sgf files correctly. In fact, I'm told that very, very few do. Even the Eidogo one used on this forum seems to choke on quite a few things.

So extending the format seems to be a recipe for more confusion. A fresh start using e.g. xml may be wiser

That's assuming everyone would implement the new format, and implement it correctly. If people don't get SGF correct, why would we expect this to be different with a new format? And in the interim phase when people start using a new format we'd have lots of software that suddenly cannot be used anymore, which is not an ideal situation.

So I think FF[4] is by now too entrenched to replace it with something new, so we should look for ways to extend it with a minimum of breakage.

The idea of using C[] is also something that had occurred to me, but it is somewhat inelegant...

Saving evaluations seems like a good candidate for a property that a number of programs would use and could agree on. Using VR seems like a good choice. My recommendation would be to stick to defining that one property: get all interested parties to work out and agree to an extensible definition for that property. That will be hard enough; no need to open the can of worms of changing the definition of SGF.

If people can get to an agreement on that one property, document it and add a link to it on the FF[4] SGF page. That would be a realistic and practical improvement for programs dealing with computer evaluations. Anybody not interested in those values will just roundtrip that property, as they should with any properties they don’t know about.

So just to add a bit to the discussion, here is the list of "non standard" properties the GoReviewPartner uses to record extra information:

• BWWR: black/white win probability for a variation. For the bots of the AlphaGo Lee generation (Leela original, AQ), it's a calculation based on the following properties.
• MCWR: Monte Carlo win probability for a variation. The bots prior to AlphaGo Lee (Pachi, Oakfoam...) use that as win rate estimate
• RAVE: RAVE(x%% : y) Still in use by many bots, especially one 9x9 or 13x13 where no neural networks have been trained
• EVAL: "Evaluation" for this variation (something only original Leela original uses (on 9x9, 13x13), but could be used as a place holder for any other non standard position evaluation.
• VNWR: Value network black/white win probability for a variation. The bots of the AlphaGo Zero generation (LZ, PhoenixGo) directly use that as win rate estimate. The others use this as part of their win rate estimation.
• PNV: Policy network value for a variation
• PLYO: Number of playouts used to estimate a variation. Here, one should probably differentiate between visit count and playout as they are not the same, but so far I did not really get the difference myself...
• BKMV: Book move (indicates the move was picked from fuseki dictionary)
• UBS/LBS: the upper/lower bounds on the score estimation (used together with "ES")
• MRD/ARD: Max and average reading depth (how far ahead the bot was reading from this position or in this variation when selecting a move)
• BOT/BOTV: bot name and version. Currently only used on the root node, but more and more people ask for GoReviewPartner to support the analysis of one game by several bots, so this could be used for each variation as well
• CBM: computer best move. I use this to clarify what the best computer answer in that position. Something that is sometime not that clear. In fact, I miss something to label the variations from best to worst (higher win rate variation may not be considered the best if not confirmed by a high number of play outs), and use this as a temporary work around.
• IBM/IWM: black or white influence map (the list of points that black or white control)
• TBM/TWM: black or white territory map. There was a reason I did not use TB/TW but I don't remember right now
• HTM: heat map (a value for each points, so not the same "format" as IBM/IWM or TBM/IWM)

I also use a RSGF properties to give some general informations on the review (date, software used, command line of the bots...)

All those properties are 3 or 4 letters, so all SGF editor can open the files, they will just ignore those properties.

Now, one issue I see coming is to decide when some information should be stored at the variation level (first node of the variation) or the node level (data are available at each node). So far I stick to variation level.

pnprog wrote:So just to add a bit to the discussion, here is the list of "non standard" properties the GoReviewPartner uses to record extra information:

• BWWR: black/white win probability for a variation. For the bots of the AlphaGo Lee generation (Leela original, AQ), it's a calculation based on the following properties.
• MCWR: Monte Carlo win probability for a variation. The bots prior to AlphaGo Lee (Pachi, Oakfoam...) use that as win rate estimate
• RAVE: RAVE(x%% : y) Still in use by many bots, especially one 9x9 or 13x13 where no neural networks have been trained
• EVAL: "Evaluation" for this variation (something only original Leela original uses (on 9x9, 13x13), but could be used as a place holder for any other non standard position evaluation.

I think to some degree things which use winrates/playouts could all use the same property, in conjunction with the existing AN property (or some other mechanism, see below) to show which program was doing the reviewing. I don't know what RAVE or EVAL is, but note that SGF already defines GW/GB/V which may be usable.

• VNWR: Value network black/white win probability for a variation. The bots of the AlphaGo Zero generation (LZ, PhoenixGo) directly use that as win rate estimate. The others use this as part of their win rate estimation.
• PNV: Policy network value for a variation
• PLYO: Number of playouts used to estimate a variation. Here, one should probably differentiate between visit count and playout as they are not the same, but so far I did not really get the difference myself...

So I think VNWR together with PLYO is what I am looking for. Do I understand correctly that PNV is the likelyhood a move is chosen, rather than an evaluation?

• BKMV: Book move (indicates the move was picked from fuseki dictionary)
• UBS/LBS: the upper/lower bounds on the score estimation (used together with "ES")
• MRD/ARD: Max and average reading depth (how far ahead the bot was reading from this position or in this variation when selecting a move)

No good ideas about BKMV or UBS/LBS, they may be necessary if one wants to record that type of information. As for reading depth, isn't that kind of implicit in the variation lines added? Generally, things like BKMV and reading depth aren't all that critical: I imagine there is no need for the viewer to take that information and use it for something, so these could really just be placed in a comment field.

• BOT/BOTV: bot name and version. Currently only used on the root node, but more and more people ask for GoReviewPartner to support the analysis of one game by several bots, so this could be used for each variation as well

There is AN which should be the same thing. But having games annotated by multiple bots could be a real thing. So I would propose that AN should be allowed in variations, to show who came up with the line, and that we allow multiple evaluations. If we take the VR proposal, it would be extended to "VR[wr{:visits{:komi{:botname}}}]"

• CBM: computer best move. I use this to clarify what the best computer answer in that position. Something that is sometime not that clear. In fact, I miss something to label the variations from best to worst (higher win rate variation may not be considered the best if not confirmed by a high number of play outs), and use this as a temporary work around.

I was planning to just add variations in the order I receive them from the bot and assume the first one will be considered best. Doesn't LZ always pick the one with the highest number of visits? I don't really see a need for this property since you can order the variations from best to worst.

• IBM/IWM: black or white influence map (the list of points that black or white control)
• TBM/TWM: black or white territory map. There was a reason I did not use TB/TW but I don't remember right now
• HTM: heat map (a value for each points, so not the same "format" as IBM/IWM or TBM/IWM)

No idea what to do about these, but I'm not particular interested in them myself. I'm glad you didn't use TB/TW though: q5go interprets these to indicate a node that was actually scored and switches the user interface around a bit.

Now, one issue I see coming is to decide when some information should be stored at the variation level (first node of the variation) or the node level (data are available at each node). So far I stick to variation level.

Depending on what you mean by "each node" I think that's sensible. At least inside a variation that was made for a computer line we need no duplicate information. A more interesting question is whether the win rate for the best variation should be attached to its parent node as well or whether that should be left implicit.

One thing I'm planning to do is to mark computer variations with FG, with a title containing a string describing the evaluation. In q5go that will bring up a second board where the user can choose which line to display.

bernds wrote:

• CBM: computer best move. I use this to clarify what the best computer answer in that position. Something that is sometime not that clear. In fact, I miss something to label the variations from best to worst (higher win rate variation may not be considered the best if not confirmed by a high number of play outs), and use this as a temporary work around.

I was planning to just add variations in the order I receive them from the bot and assume the first one will be considered best. Doesn't LZ always pick the one with the highest number of visits? I don't really see a need for this property since you can order the variations from best to worst.

No, in general bots do not always select the move with highest number of visits. Several older MCTS bots occasionally selected a move with lower visits sometimes if the lower visits move had a combination of policy prior and current winrate statistics such that the move would be statistically extremely likely to eventually become the move with highest visits. This could happen, for example, if the highest-visits move was recently refuted, causing its winrate to plummet, such that even though still highest visits, MCTS would for the forseeable future be putting near 0 further visits into it and the second-highest visit move would be mathematically highly likely to overtake it soon.

Also there have been some branches and experiments with Leela Zero by some people selecting the move based on statistical upper and lower confidence bounds, which although not official, in some cases seem to have promise for being slightly better than choosing the move with highest visits.

bernds wrote:

pnprog wrote:So just to add a bit to the discussion, here is the list of "non standard" properties the GoReviewPartner uses to record extra information:

• RAVE: RAVE(x%% : y) Still in use by many bots, especially one 9x9 or 13x13 where no neural networks have been trained

I don't know what RAVE or EVAL is, but note that SGF already defines GW/GB/V which may be usable.

Slight tangent:

While I am not sure what the value in the RAVE field means, RAVE in MCTS search is a technique for filtering good move candidates.

Roughly, when a move is present in (many) winning MCTS playouts, it is inferred that it might be a good move. As a result the the probability to try that move even earlier in the search is increased.