Ah, you've stumbled upon one an important nuance of Chessgames uploading. Time for your next lesson, grasshopper.

A program called "fixpgn" gets run once daily which scans all PGN files for various possible changes and modifies PGN to make this consistent. This does not happen instantly upon uploading (and that is intentional, motivated part by efficiency and part by the need to monitor uploads.)

Suppose the Librarian changes the name of "J Garcia" to "Juan Garcia" and the guy has 100 games. The software does not run through all of those games and make the necessary fixes to the PGN immediately, as you think it might.

Instead what happens is that this fixpgn program runs through the database and recreates a version of the PGN based on everything we know about the game (the players, the tournament, the moves, etc.) Then it compares the freshly produced PGN to what we have on file. If there's no difference it skips it, but if there is the slightest change the game gets updated. (The old version of the game is kept in a backup table for Librarian reference.)

So pgnfix not only corrects player names, but looks for mistakes in year formats, fixes changed ECO codes, reconciles PGN dates with database dates, and even word-wraps the moves. In short, it makes our PGN look nice, even if what we received looked crazy.

I'm a little surprised at how it works (since I naively expected fixpgn to run before posting a game to the public).

So, am I correct in assuming that <CG> processes generic games submitted by a user-at-large the same as those pulled from a tournament's official site?

E.g. you never explicitly make a crosstable to ensure all the games are present and accounted for, and that the results match those officially published?

It seems my initial observations of the Tromso OLM games are setting up lots of games which will need fixing by hand subsequently. At least that's how it seems to me.

For example, there's "shadow" players like this:

Evgenij Agrest (N=1) Evgeny Agrest (N=450)

Maybe using the FIDE id will fix a lot of this. Hopefully. But my program is beginning to spin out of control trying to patch all these special cases.

Comparing <CG> to <FIDE> is made much, much easier if each PGN file is normalized with itself.

Yes.

<E.g. you never explicitly make a crosstable to ensure all the games are present and accounted for, and that the results match those officially published?>

True, we don't attempt to do that, although it's worth mentioning that addressing this issue is exactly the point of the Biographer Bistro and the Game Collection Voting, and the new system of leaderboard editing. Of course that has nothing to do with new tournaments, at least not yet, but it's the same concept.

<It seems my initial observations of the Tromso OLM games are setting up lots of games which will need fixing by hand subsequently. At least that's how it seems to me.>

I'm sure you are right. I've already patched some errors simply by deleting the games. They weren't particularly important games so they had no kibitzing and were not in any collection. Sure enough, on the next data fetch the correct games replaced the initial bad PGN.

This is our version of the "roll back" that you mentioned. If necessary, we can delete bad games and then fetch the new data from the site. Of course it's always preferable to fix a game than delete a record for various reasons.

<For example, there's "shadow" players like this: Evgenij Agrest (N=1) Evgeny Agrest (N=450)>

A few days ago the software would have just assumed that must be the same player and matched them up correctly, but after Stonehenge's rightful complaint, we tightened the reins on speculative matching. The Librarian also catches a lot of things like that, albeit days after the game is uploaded.

<Maybe using the FIDE id will fix a lot of this.>

It surely will, and I suspect that the <Evgenij Agrest> shown above was inserted only before you pointed out that they are using the wrong FIDE tags in their PGN.

<BTW- is it possible to get a <CG> pgn download with <only> games that have been fixpgn processed?>

No, not really, but we run fixpgn all the time (I said midnight, but that's a bare minimum. We run it during the day very often, for example after user uploads.) So 99% of the time you won't see PGN that needs fixing, you just happened to stumble onto some recently.

It's possible to do a "mini-run" of fixpgn right after each data fetch to ameliorate this issue; then the odds of you finding a game in need of fixing would be very slim. Thanks for the idea.

About your comment <So I would use a 1-to-1 mapping of gid <-> pairing, and update accordingly> I have so many opinions on that I didn't respond. Not to say that it's a bad idea, but the devil's in the details.

I realize that all of these games were from Round 5 - and so were due to fixpgn not having run yet.

But I was thinking about this a little - and I don't understand why, if the games show up on a player's profile page they don't use the canonical name.

Whatever automated software that added the game knew the player's identity - how much harder would it be to tweak the White/Black player tags right then and there?

I can understand waiting to do the opening ECO, and maybe to scan the movelist for illegal moves and notation changes (like simplifying a move if one piece is absolutely pinned).

But not having the right player name in the PGN seems wrong to me, even now that I understand the process a little better.

All games in this database, as far as I know, are tagged based on type already (ie, classical, rapid, exhibition, etc). When you perform a head-to-head search (say, search "carlsen-caruana"), the box at the top can give you the results of the classical games only, which is great if those are the ones you're interested in. But when you scroll down and look at the list of games, there is no way to filter out the non-classical games. It would make things a lot cleaner and easier to browse.

I think this could be done easily and unobtrusively. At the bottom of all search pages, there's a "Refine search" box where you can filter by result. Adding one or two more similar options ("classical", "rapid/exhibition") would do the job nicely (as would adding it to the main search page, if that would be easier).

Please understand that until last year that field was so spotty and unreliable it couldn't be trusted, so our omission was intentional. At this point, after thousands of corrections and manual tournament sweeps by the Librarian, it has become very good and worthwhile data to search on.

A little curious that no one else has complained, but I can offer the following explanation.

Normally, you just wait and fixpgn does its job and all is fine a day later. And probably most people just want the moves of the game in the first place, and so all is fine there too.

But I'm trying to automate the checking of <CG> games against <FIDE>. This seems merited, if only for the benefit of the missing games found (e.g. Dubai 2014, and I think also for Tromso 2014).

Now, the problem is that I need <CG> to be consistent with itself to do that. Of course, I can wait a day for the pgn to be fixed, no problem.

Let's say I wait a day for round 5 to be fixed. The trouble is that by then round 6 will have been played, which again means my program will be confused (at least until I add special handling, or <CG> normalizes player names, or <CG> adds the <ChessgamesPlayerId> tag).

And so, the PGN will be troublesome the entire duration of the tournament - give or take a rest day or two.

https://www.youtube.com/watch?v=HZj...

* * * * *

Speaking of curiosities - when I do a search on <Alexander Ipatov>, <CG> delivers the two expected matches, but with the following complaint:

<ERROR: Over 32,000 games match your query.

Please select more restrictions on your search.>

And I don't even get my normal footers at the bottom of the page!

I think <CG> is using the wrong <Alexander Ipatov> as well, the player at Tromso is playing for board 4 for Turkey:

https://www.youtube.com/watch?v=nL4...

It's possible the Russian <Ipatov> is also playing, in which case <CG> is providing PGN with two distinct players with the same indistinguishable name.

So... what's the <Ipatov> story?

I don't have time to double check this - but I fairly sure it's a problem one way or the other.

<zanzibar> Repo Man is always intense :)

About the search, it's fairly easy to explain although the situation is pretty bad. Here's what's going on:

The search was modeled after the 'classic' search engine approach (Altavista, Yahoo, etc.) meaning if you search for "A B" there is an implicit OR, so you are really searching for "A or B". So that means when you search for "Alexander Ipatov" you really are searching for "Alexander OR Ipatov". Apparently we have over 32,000 games with Alexanders now.

I tried an experiment once to make it an implicit AND, so in that case it would work perfectly, but then the problem was that people would type "Alexandre" etc.

The practical solution is just to leave off names like "Alexander" and "John" (or for superusers, now it's possible to enter a PID.)

This is the technology that you see on typical Google searches. When you type a name, they know it, and they'll even correct your spelling for you.

Having that tool in the toolkit would open the doors to even smarter searches, like EZ Search could almost become natural langue. It would have a "thought" process like: "obviously you're talking about Fred Reinfeld playing the Ruy, not Ruy Reinfeld playing the Fred." Right now, if it ever seems that clever, it just got lucky.