The two proposals, <CG_id> and <%stub> are examined.

<(1) Addition of a CD_id tag>

Format: <[CG_id "tid.gid.w_pid.b_pid"]>

In discussing additional new header tags beyond the 7-mandatory from the DGT extended PGN proposal comes this:

<
3.Proposals
3.1 Introduction

These additions are not problematic as the standard concedes the possibility of adding new tags to the header (outside the mandatory seven tag roster)

>

http://www.enpassant.dk/chess/palvi...

So everybody adds PGN headers, and rather wantonly. Ours is prefixed with <CG_>, a terse notation, but fairly well established. It's purpose is to avoid collisions with other extended tags, which it does.

Generally, the extended tags are hidden away from the user in various viewers. This is just as well. It can usually be easily found, with just an extra step or two.

So this addition should be innocuous.

<(2) Addition of stub games, denoted by comment %stub etc.>

The exact details of the comment haven't been finalized, but the use of the keyword %stub, borrowing from the DGT extensions, is unique enough to avoid collisions with any user comments.

The game consists of PGN headers, the %stub comment and the result. The purpose of using a comment rather than a simple header is two-fold.

First, viewers usually do display comments when presenting a game. Users should be aware that the game is a stub.

Secondly, the reason for the stub game can be included in a comment. For instance, if the game were a forfeit, or lost in history. Additionally, partial known moves, opening classifications or final positions can be included inside the comment.

Fixing up <Hotel Bali (2002)> reveals a missing game on <CG>:

http://www.chessgames.com/perl/ches...

So I intend to submit the following:

<

;; <Granero Roca, Antonio -- Amura, Claudia (R2) 41 1-0>

[Event "Superstars Hotel Bali"]
[Site "Benidorm ESP"]
[Date "2002.11.29"]
[Round "2"]
[White "Granero Roca, Antonio"]
[Black "Amura, Claudia"]
[Result "1-0"]
[WhiteElo "2342"]
[BlackElo "2360"]
[ECO "D07"]
[EventDate "2002.11.29"]
[CG_id "39472.0.50585.24807"]
[PlyCount "81"]

1.d4 d5 2.c4 Nc6 3.Nf3 Bg4 4.Bf4 Bxf3 5.gxf3 Nf6 6.Nc3 e6 7.e3 Bd6 8.Bg3 Ne7 9.Qb3 Bxg3 10.hxg3 Qc8 11.Rc1 c6 12.Qb4 Qd7 13.Bd3 Nc8 14.g4 g6 15.Ke2 Nd6 16.Rh2 h5 17.cxd5 exd5 18.Na4 Qe7 19.Nc5 O-O-O 20.g5 Nfe8 21.f4 Ng7 22.Rc3 Rhe8 23.Ra3 Kb8 24.Qa4 Nc8 25.Rb3 Nb6 26.Qb4 Ne6 27.Nxe6 Qxe6 28.Rh4 Rd7 29.a4 Nc4 30.Bxc4 dxc4 31.Rc3 Rxd4 32.f5 Qd5 33.Rxd4 Qxd4 34.Rxc4 Qd7 35.fxg6 fxg6 36.Rd4 Qf5 37.Qd6+ Kc8 38.a5 a6 39.Rf4 Qxg5 40.Rf7 Qg4+ 41.Kd2 1-0

>

Note the important tag:

<CG_id "39472.0.50585.24807">

For each game in the database, we maintain not one but two chunks of PGN. One in the format you know today, and the other with a group of new tags:

ChessgamesGameID
ChessgamesWhiteID
ChessgamesBlackID
ChessgamesEventID
ChessgamesGametype

(and maybe a few others i've forgotten.)

When you download/view PGN from Chessgames, whether you see the normal format or the expanded format would depend on a setting in your preferences.

It's rare when we provided annotated PGN, and when we do we want it to be on the lowest-common-denominator. Not all PGN viewers can handle RAV notation, etc. The notes usually come from a book where the {simple comments} is all that is required.

I envision it as integral to our new form of tournament building (also needing a writeup).

* * * * *

As for the stripped down PGN, I agree that is best for <CG>. Similar to <MillBase>'s approach in that matter.

I checked the abbrev RAV just to be sure it was what I thought -

RAV = Recursive Annotation Variations

which I found here:

http://portablegamenotation.com/Lit...

So given that even Tim Harding takes so much time and effort to write and talk about PGN, I don't feel so bad doing a little bit myself.

(1) Backlinks - CG_id = (tid, gid, w_pid, b_pid)
(2) Safeguarding tournaments via tid locking.
(3) Baselining tournaments upon creation.

* * * *

(1) Each game should include at least four id's which are a kind of back-link - (tid, gid, w_pid, b_pid).

When a game gets created in an unattached state, the tid is 0 or -1. Some indicator that the game is in full edit mode.

Any editor could change the PGN headers, provided the changes are logged somewhere (I would hope in the comment stream). That includes moves/etc.

Editors cannot directly change the id data. They can do so indirectly, e.g. by changing the name of a player. But software could intervene to ensure such changes are valid. Etc.

(2) Once a game is attached to a tournament, software ensures that it, and all other games belonging to the tournament, are PGN normalized.

At this point editors are no longer able to freely "hand-edit" the games. The rules are stricter, whatever they might be. This is to ensure the integrity of a tournament.

Hopefully, once a tournament is created, all edits to it are vetted through the Bistro. Perhaps a template for admin edits should require a url to a Bistro comment. <CG> should work out the details.

The main idea here is simple, once a game has a valid tid, only software that has the built-in checks necessary to ensure tournaments never become denormalized can change the data.

<No more hand editing, as a general rule, no matter the temptation of convenience.>

So, whatever the powers an admin may have, there are limits when it comes to tournament games (reasonable limits, much like those in the Magna Carta). Changes that affect a tournament game must only be done via update routines. Routines that understand the requirements for tournament integrity.

(3) As part of the recognition of how difficult it is to change a tournament - so there should be additional requirements before a tournament is created.

I feel that a redundant table of basic tournament data should be kept, recording certain basic facts (e.g. N_players, N_games, N_missing, xtabs(?), bracket dates, location).

This is a very important step, as I see it.

The handcrafted methods utilized previously just can't every meet the production demands.

Take a look at this graph:

https://zanchess.files.wordpress.co...

The blue shows the tournaments/year contained in <MillBase> vs. the red, which shows the same for <CG>.

And that gap doesn't even account for all the work currently being undertaken to repair the <CG> tournaments.

The full background can be read here:

https://zanchess.wordpress.com/2015...

Ultimately, we should be able to take a normalized PGN file containing all the games from a tournament, and just submit it to <CG> for promotion.

Of course, if games from the tournament already exist on <CG> this involves some difficult questions concerning merging data.

(1) Bulk correction of Round/Date tags for all games in a tournament.

(2) Simultaneous submission of missing games.

* * * * *

(A) Working assumption - all games have (tid,gid,w_pid,g_pid) in PGN download (however it gets encoded).

The idea is to allow those who wish to build a tournament via SCID or ChessBase.

<(I) Initial Collection Build.>

First step, build a collection of the pre-existing tournament games on <CG> - then download an incomplete tournament.

(I)(a) Of course, the user could also have used the ZIP downloads to find the incomplete tournament.

<(II) Missing games addition>

The user then adds the missing games from whatever source necessary. The tournament-to-be is now complete.

(II)(a) It is assumed the user has normalized the E/S/ED tags for all the games.

(II)(b) Perhaps there should be an option to do so when downloading a collection from <CG>. (Please note)

<(III) Round/Dating>

The biographer working at home now can hand-edit, or however, the game data to update the Round and Dates in the PGN.

This is usually the focus of much of the work of a biographer. It could still be done by hand, the old way, online. But now we also provide the biographer the option of using his/her own homebrew tools (like SCID, ChessBase or a python program, etc)

<IV. Crosstables Generation>

Actually, this step may involve something new - stubs, and so it might actually belong to step II. The idea is that a valid xtab must be able to be produced before a tournament is bulk submitted to <CG>.

Of course, we could consider this a null-step, if we disregard such a strict, but useful, step. I would not recommend such a policy in general.

But one must be practical.

Now the tournament PGN is submitted to <CG> and a complete tournament appears, only missing an intro.

(End of User Requirements/Design)

(<CG> Requires/Design in next post)

* * * * *

<Comments>

Nobody is forcing the adoption of the above. It is a alternative to the current methodology only. However, there is very little difference between working game by game on <CG> vs. game by game in your favorite database program. I would assume most biographers would eventually adapt to this style, no matter their feelings about it today.

<(1) Normalized data check>

This is a basic check, but the E/S/ED headers must be normalized. Else bounce the submission back to the user.

<(2) E/S/ED/Round/Data updating of existing <CG> games>

We are assuming that <CG_id> is contained in some of the games. <CG> could check the pgn to be exactly the same for all such games, else bounce the submission.

(This requirement should ultimately be relaxed to allow movelist corrections - but we are doing an interim design at the moment).

So, E/S/ED may be overwritten by the normalized submission. Also the Round and Date headers containing valuable biographical data.

<(3) Creation of new <CG> games>

Those games lacking a <CG_id>, or whose <CG_id> has (tid,gid) = (0,0), are created as new games on <CG>, with the other E/S/ED/Round/Date headers copied over.

* * * * *

This bulk submission can then be promoted to a tournament, with a tid assigned by <CG>.

Or... we could allow a submission, much like the above, where no gid=0 is allowed, but tid maps to an existing tid. Then the above could be used to supply either the Round/Date data to pre-existing <CG> tournament games. Or even E/S/ED/R/D data.

That's all really pretty straight-forward when you break it down.

* * * * *

Consider the advantage of this method to correct the Round/Dates of a pre-existing tournament. Just download the games, and use your database to update the PGN. Then you can check your results via a crosstable as you work. When the results look good, just upload the PGN.

Yes, chessbase has its own ("extended pgn" - maybe not "real" pgn - but there's also a few versions of any one pgn "standard" itself, usually explained in help files of different programs, etc) export format, which CVD doesn't exactly complain much about when using online. Some similar Java viewers would also allow a list of games to select from within the viewer window itself (maybe a good way of handling groupings tid games).

I liked the idea of using the comments nesting (can be created with chessbase 9 as one source) to avoid having to see a 10-fold rep (or 3-fold or 2-fold) of position ending not have to appear more than a couple of times on the pgn display viewer. I liked it even more for that case when the comment nesting was not used for <ending> positions in chessbase, but when strange things like colour-reversal or ECO code transpositions might've occurred in chessbase. Some of my own personal notes (unshareable) in Opening Explorer positions also include similar comments, one one jumps from one node to a next one (I can even think of QID positions where a transposition at move ply+2 does a "transposes" to a ply+0 position, simply because a B or R has moved twice along a diagonal instead of once to reach a transposed opening book position (I 1st saw this type of example in "chess, black & white", (or similar title) by Kaufman - a big example of an opening book debugger - so there's some homework for you). The only "trouble" when using a" transposes" word pointer is that it doesn't remember the move number increment after the transposition - but it sure is nice for merges with copying and pasting chessbase pgn into the same nesting scheme. Try it with some of your own annotating next time you add some personal notes with chessbase from a book covering 2 or more games all on the same opening.

I'd usually strip off: plycount, eventdate, result, and ECO (ECO can change from program to program!) headers when saving them to notepad, since (most) can be auto-regenerated through pgn upload utility. I'd rather rely on the good stuff at/after the final ply-count in the pgn's move list, like whether it's plycount matches anything in the header, whether it has a result or # or both following the move list, and whether it matches the header and the result makes sense (was it really a <black> mates in 2 puzzle, or is it a checkmate for <white>)? And only rely on the roster tag headers if the move list is empty/ambiguous to draw a conclusion from.

So to me, if a pgn tid set lets me create a crosstable primarily first, and displays a game or partial games move list result (perhaps empty unrecorded move list games included), and only then talks about things like consistency of dates or locales, then I am usually happy.

What's normally needed to build a (swiss - I already mentioned why one might want team "tags" or similar (or maybe better yet, find a way to <not> include them in the pgn source itself, while allowing an offline program to find out what is really going on) for some non-swisses...) is the player names (for each black and white player in a round), the round number (not necessarily the date), a result look-up, and to see which if any players had an empty game result to another player or had a bye.

I couldn't possibly imagine how various pairing software varieties handle such situations, but now that cg.com is in the business of trying to display crosstables that could be generated for display amongst other pairing software, it might be useful to know how display those irregularities (or when a tid is assigning things like 0 or 1 or 1/2 point byes, etc). If pairing is being done "live" in round k of an n round tournament (0<k<n), then the cg.com format might want the arbiter to get to pair to round k+1 via the format flexibility (some pairing rules rely on a group theory rule (or isomorphism to a rule by thinking of it in a math way) called associativity when trying to regenerate a table from one round to the next - can be a hard problem in general when ELO style FIDE algorithms aren't quite adhered to).

That's the main reason why I emphasize that if we treat pgn (or x-pgn styles, or RAV or whatever) as a <data structure>, that we primarily need to think about how it interacts with other <programs> first, and then maybe things like engines, or displaying moves or fen-string positions, and then worry about what the end user might see - like whether the game headers is human-readable and the moves can be read in his language notation of choice, and only then display the extra goodies afterwards, like the event-dates, plycount, ECO, annotations tags afterwards. It could also be able to provide navigation points into particular opening explorer positions (could you imagine a case of an "opening explorer notes" note where at an early ply like ply-7, a merge of all annotations of all 750,000 cg.com games are merged into that note for that move?!).

Maybe <most> tids would have a blank/no team header, if chosen as an extension for use to display team pairings like: http://www.olimpbase.org/articles/f... - where the author gives <opinions> about swisses.

[Event " Casino de Barcelona (2007) "] [Site "Barcelona ESP"]
[Date "2007.10.18"]
[EventDate "2007.10.18"]
[Round "1"]
[White " Leinier Dominguez Perez "] [Black " Marc Narciso Dublan "] [ECO " French (C11) "] [WhiteElo " http://ratings.fide.com/card.phtml?... "] [BlackElo " http://ratings.fide.com/card.phtml?... "] [WhiteTeam "-"]
[BlackTeam "-"]

1. e4 e6 2. d4 d5 3. c3 ♞f6 4. e5 ♘fd7 5. f4 c5 6. ♘f3 ♞c6 7. ♗e3 cxd4 .... etc 1-0

I'm not going to repeat the whole pgn of today's GOTD. But the Unicode and addition of the URLs to the pgn header might make for a good discussion of whether this has any benefits of a more universal display across languages, pending that we believe that most programs from 2010 can comprehend what the pgn standard couldn't quite handle a quarter century earlier. I'll leave that style of "pgn" open for discussion.

And some motivation/roadmap to what your primary goals are, at the onset.

I just tried to read through the posts, and as usual have to go back to the start to reread it all (maybe more than once), just to get oriented.

So adding in the names of all players (like cg.com uses a pid), I would even try to colour-pair an unpaired pid for a round to a fictional "ghost"-bye player in a header for a round, to have a crosstable generate correctly (so as to get a proper display sum in it for all but the ghost players). Chessbase's "T" command (its help error warned of "warning: experimental function"!) was very buggy doing it this way, when adding just the pids, 0-ply games, a round, result and event/section name - and fill in the other stuff later on.

2-5 days later I could add in the moves (a handheld was a handy way to capture game scoresheet results), and do a save, and hopefully watch the "T" command not crash. And later distribute the pgn to the TD afterwards. Much of the time it would actually work, but once in a while it would either look for an exit to a long-running loop. So it seems like chessbase was only expecting one of a few "valid" pairing options to display to crosstable via the T command.

Maybe in a newer chessbase release, there is support for doing or import/export of pairing rules from 3rd party pairings algorithms?! That was one of the biggest things I missed in seeking a 3-digit price-tag program, and would be great to see if pgn formats with 0-move lengths would be able to create tables or pairings that would allow a tournament creator to build on with - for doing things like displaying things like posting results to a newsgroup or place like cg.com in a real-time basis?!

If the moves of empty game results can be found later, then I'm all for adding them in - when the 2 scoresheets agree and are wanted to be published.

Doing things like spell-checking pid names or matching the correct rating might be something to be done later, within a future step of a tid update, after it gets finished?! Those were the types of problems I was encountering when I did some tournaments back then.

<SCID> doesn't do any pairing algorithms afaik.

It does offer several crosstable (xtab) formats -

RR which it calls "all-play-all"

Swiss where it gives color + results vs Round

and

Knockout (which understands compound Round notation).

The latter gets a little confused with 3rd & 4th place continuations, but displays all matches correctly, just not necessarily sorted properly.

It does not do Scheveningen or other Team match-ups.

1 d4 Nf6 2 c4 e6 3 Nf3 b6 4 g3 Bb4+ 5 Nc3 Ba6 6 e3 Be7 7 b3 Bb7

Of course RAV notation is very useful, not just for opening transpositions, but also for any analysis worth its salt.

But now I'm wondering what the real thrust of the discussion is?

Certainly, at some point <CG> should support detailed RAV's in user comments, much like <CT> already does. That is, if it's ever planned to allow user analysis to be easily played over online.

Users can just post valid RAV (PGN) excerpts already in a post, although it requires a cut-and-paste into SCID or similar.

Would having such an ability be useful for <CG> tournaments?

I'm not sure whether <pairing> programs are in general able to <just display> (maybe 0-move games) pgn's from a round if its output is being exported to a projector in the hall - but probably it's safe to assume they (ie swisssys, swissperfect - chess-results.com sometimes <does include> a pgn link on its output page) wouldn't have much for <pgn editor tools> like chessbase or SCID.

And I also <do> like the chessbase format of using a round.table compound format option of being able to export with, in the round header (or at least have an Xpgn header to carry a table/board header, along with the round - team pairings use both the table and board as well as the round, or specify the seat position, and the country team pairings for the round). Maybe this data could be carried as extra, optional drop-down fields to fill out on the gid/tid pages, instead of inserted as a compound round structure (or at least I'm not sure why we'd want to remove the extra data, once we know we had the seating positions for a round - usually generated by a rule related to FIDE ratings ranking).

A big fat "yes" vote from me on the importance of having a reference xtab, and being able to dynamically generate xtabs from tournament games.

Most people like leader-sorted RR's for the main display. But for purposes of crosschecking I think a Swiss is essential. No tournament should be accepted without a valid Swiss, showing color-pairings and rounds.

Also, I think name sorted versions of all the tables should be available somehow, again for checking. Since tiebreak rules can be so complicated, sorting on collation-names (i.e. surname, prenom) is important.

All xtab software should be able to produce name sorted tables, and then we have a canonical format to check.

Informant did that, having a column for placement.

How <CG> exposes these reference tables is a detail. Maybe by a dropdown selector, or a link to a page dedicated to the tables, etc.

Being able to dynamically generate xtabs from the actual games in a snapshot of the tournament is very valuable for checking for inconsistencies that can happen after the intro page is published.

The kind of errors that shouldn't happen but do.

And unless you're willing to write thousands of lines of code to find errors, the very best general-purpose tool is the Swiss xtab.

The problem is is that if we do allow 0-move game stubs for which to build a crosstable from strictly automatically, that we need to know how handle players paired to an unpaired bye (ghost pairing). If on the other hand, we do some of the editing of the table by hand, then we don't necessarily need all games present when voting in a tournament, but the scores of byes and unplayed games need to be inserted manually (especially when they are different from the FIDE usage recommendations of when to assign a bye of a certain score - pairings programs have about 5-10 other symbols other than win/loss/draw, and also carry the colour/floats in the pairing to the next round, and maybe it would be good for our tables to know how to handle these as well, for some not-quite-perfect-swiss algorithm - swisssys has a number of choices, too).

Scoring byes need a fictitious player or maybe more, let's name them BYEn, where n is used for enumeration purposes only.

No need for these players to have a bio, or to have their statistics tracked.

As far as the extra symbols used by pairing programs, I'd have to gain some familiarity to say anything further.

Just a quick word on the rationale for the terser <CG_id> tag.

Basically, this tag isn't intended for people to edit by hand. It's intended for transfer of information from one program to another in the PGN.

Therefore, it was designed for easy parsing, and to avoid taking up too much space in the PGN.

It is much easier to look for the one tag (<CG_id>), then to have to parse out <ChessgamesWhiteID>, <ChessgamesBlackID>, <ChessgamesTournamentID>, <ChessgamesGameID>.

The information is naturally a tuple, and all I have to do is process all the tags normally, then:

<

try:
g.id = tuple( [int(k) for k in g.CG_id.split(".")] ) except:
g.id = ( 0, 0, 0, 0 )

>

If people understand the tuple that's fine, but the tag is intended to mark off pre-existing <CG> games which a biographer downloads to begin the tournament build process.

These pre-existing games are only edited for Round/Date information. Missing games are added, but without the <CG_id> tag.

And it takes less file space because it is more compact as well. Additionally, if you do know the format you can do searches with an editor and have all the information right there, one line/game.