<chessgames> please add these photos to the player bios...

http://zanchess.wordpress.com/2016/...

That is, when you get a chance...

This is hilarious, if you get it. ;>D

Well they only cost a tenner, Offy.

<tpstar> I've nothing personal against you but you really let the testosterone and masculine side of you show too much.

Chigorin vs Alapin, 1875 (kibitz #2)

An old comment got me wondering.

Seems that some occasional filtering looking for games with duplicate movelists should be part of regular maintenance.

<Seems that some occasional filtering looking for games with duplicate movelists should be part of regular maintenance.>

Of course.

As you know there are safeguards to prevent duplicate games from entering the database in the first place, but for various reasons they can end up in here in spite of such efforts.

And so a program called dupeblaster was written circa 2004 to weed out such problems. Here's a golden oldie post from the days when I discussed chess in the Café — Kibitzer's Café (kibitz #20610)

Dupeblaster helped at the time, but once it was run there were still problems that it couldn't recognize. Because dupeblaster proved to be insufficient, several other versions and renditions of the same idea have been written. Usually the problem is that the software doesn't see the duplicates as being duplicates, but at least one of them was overzealous, thereby creating some of the bugs that the current CG Editors have helped sort out.

The problem is that at this stage the duplicates which go undetected are of such a nature that our software can't seem to locate them. We've used various tricks, such as looking at FEN positions at various stages of the game and measuring the degree of similarity (e.g. 100% of it's the same position, 95% if only one pawn is different, etc.) It all got into very sophisticated tricks to find duplicate games, and in the end, you know happened? The darn thing didn't work any better than more primitive attempts.

(In case anybody is confused with this talk, understand that the term "duplicate" is used in a very unusual fashion here. Detecting two games with actually literally duplicated moves is quite easy, but our problems are never that easy. There's always a change in move order, or even an outright mistake. In short, when two games are in fact the same, but their moves or other information differs significantly enough to create an extra page, we call them "duplicates.")

So that's where it stands today. If you are champing at the bit for a chess programming project, write one that automatically detects likely duplicates and outputs a list in the form (a,b,p) where (a,b) are two GIDs and p is the "probability of certainty" that the games are duplicates. You are right, the editors would gobble it up.

Times have changed.

* * * * *

OK, one can imagine lots of levels of sophistication for this task.

But the 0th-level prototype would do the obvious - just compare the actual moveslists:

Chigorin vs Alapin, 1875 (likely wrong players):

1. e4 c5 2. Nc3 Nc6 3. Nf3 e6 4. d4 cxd4 5. Nxd4 d5 6. Bb5 Nge7 7. Bf4 a6 8. exd5 exd5 9. O-O Bd7 10. Re1 Nxd4 11. Qxd4 Bxb5 12. Nxd5 f6 13. Nc7+ Kf7 14. Qe4 g6 15. Qe6+ Kg7 16. Bh6+ Kxh6 17. Qh3+ Kg7 18. Ne6+ Kf7 19. Nxd8+ Rxd8 20. Qe6+ Kg7 21. Rad1 Rxd1 22. Rxd1 h5 23. Rd6 Ng8 24. Rd8 Rh7 25. Rc8 Ne7 26. Rc7 Kh8 27. Qxf6+ Kg8 28. Qe6+ Rf7 29. Rxb7 1-0

Chigorin vs A Ascharin, 1875 (more likely correct players)

1. e4 c5 2. Nc3 Nc6 3. Nf3 e6 4. d4 cxd4 5. Nxd4 d5 6. Bb5 Ne7 7. Bf4 a6 8. exd5 exd5 9. O-O Bd7 10. Re1 Nxd4 11. Qxd4 Bxb5 12. Nxd5 f6 13. Nc7+ Kf7 14. Qe4 g6 15. Qe6+ Kg7 16. Bh6+ Kxh6 17. Qh3+ Kg7 18. Ne6+ Kf7 19. Nxd8+ Rxd8 20. Qe6+ Kg7 21. Rad1 Rxd1 22. Rxd1 h5 23. Rd6 Ng8 24. Rd8 Rh7 25. Rc8 Ne7 26. Rc7 Kh8 27. Qxf6+ Kg8 28. Qe6+ Rf7 29. Rxb7 1-0

The movelists are identical. As pointed out back in

Chigorin vs Alapin, 1875 (kibitz #2) 2006 by <waddayaplay>

Chigorin vs A Ascharin, 1875 (kibitz #2) 2015 by <heuristic>

Akin to the idea bulk submission, there are many levels of sophistication which can be envisioned. But the basic versions are practical, desirable, fairly easily implementable, and needed.