True, I agree with you. I can add two more traits. Unusually high tolerance for the difficulty of positions; and indifference about how logical or normal moves appear to be.

It's a common knowledge that engines usually play relatively less accurately in 'simple' positions, and they excel in complications compared to humans.

An engine might want to play a rook lift to the third rank, or place a knight on the rim, while there are a lot of more 'normal' equally good moves present as well.

The most famous example I know is as follows:

C Allwermann vs S Kalinitschew, 1999

click for larger view

<I showed the position to a number of players in Wijk aan Zee, and all gave me simple wins – for instance 31.Rxb7, 31.Rd7 or even 31.Rxf6. Remember, the first time control is looming and tournament victory is in grasp.

So what does our hero play? 31.Qa7?!! “Fritzy!” squealed Anand and went into uncontrollable fits of laughter when he saw this and the following moves (I filmed his mirth and included it in my multimedia report in ChessBase Magazine 69). He and the other players immediately recognised the “hand” of the computer.>

http://www.youtube.com/watch?v=cx0n...

To the best of my (limited) understanding:

I think the idea is that agent A is a model for <what> a potential human player P does, namely selecting the best move of some available number of moves with some increasing probability the stronger the player is, but does not model <how> a human sometimes (or often) does, namely by following a retrospective plan, though often also just through prospective calculation of some number of moves ahead.

Consider a human being who has a memory span of a few minutes only, so that making plans is impossible, since they are forgotten. However, if that human being had exceptional calculating abilities and see many moves ahead, he could could play excellent chess (sadly, there are indeed such cases).

For example, I am playing a game and decide to go to the washroom to look at my pocket computer. I do not have time to let the software get to high ply and therefore play the move that comes in after a minute or so of calculating.

I am not sure how letting a computer get to high ply helps cheaters since it would seem impractical for the cheater's computer to ever get that high.

Or perhaps I am missing something.

I think you have the wrong sense of what it means to "motivate it" here. This is typical "academe speak", but it essentially means "justify", i.e. "our motive for using this methodology is so and so ..."

All they're doing is trying to find the best curve to fit the data, and there's lots of different ways of doing so as you can see below. Note that the data is fixed and unalterable; it's the best curve that's subject to change.

It would be a problem if it went the other around, namely that if they somehow forced the data into a preexisting curve they favored ahead of time. That wouldn't me a miracle ... it would just be cheating!

The idea is that calculating at high ply level will return a list of possible moves ordered according to valuation, from high to low. If the human player's move is the same as, say, the 3 highest scoring moves again and again and again as calculated by a very strong engine, that should raise suspicion that cheating may have occured.

Given that he beat several GM, that essentially rules out that IF a computer was used, that it was used on low-ply, since that would not likely beat even an average GM.

<IF a computer was used, that it was used on low-ply, since that would not likely beat even an average GM.>

(1) If you run engines in multi-thread mode, or with different (power-of-2) hash-table sizes or different multi-pv settings, or for different amounts of time, you will indeed get different values and often different "best moves".

(2) But if you run in single-thread mode to a fixed search depth, with the same hash and multi-pv and deterministic parameters (which Rybka and Houdini and Stockfish and Toga II etc. have ALWAYS had), you will generally get the same results. This helps for internal consistency and reproducibility. Of course cheaters' accomplices would be doing multi-thread, variable-time (including looking ahead while opponent is thinking), single-pv mode with I-don't-know-what hash size... But we cope with this difference because the approach is statistical to begin with.

(3) AylerKupp, I indeed have results on Sofia Polgar from 1989, and others being talked about, at http://www.cse.buffalo.edu/~regan/c... The difference from what one might regard as "proven" is clear!

(4) My theory has internal as well as external consistency checks, such as from multiple fitting procedures. My own first curve in 2008 was very bad, and Haworth previously had a similar one, but my own second curve was good(!), similar to how "equal temperament" makes all the world's pianos slightly out of tune but good enough.

(5) Rybka bunches the bottom 4 levels of the search into 1, so what it reports as depth 13 is more like 16 or 17 for comparison to other engines. In fixed-depth engine-match tests Rybka 3 is similar in strength to Houdini 3 depth 17 and Stockfish 2.3.1 depth 19-20 (I use 19 for speed and to have an odd search depth; I may go up to 21). Larry Kaufman estimated Rybka 3 depth 14 as Elo 2750; Rybka 4.1 reaches this depth faster but is a little weaker; I think it's really 2900+ in early middle game sliding down under 2500 in endgames where depth 13 happens really fast. Guid and Bratko did Rybka 3 only to depth 10 (http://en.chessbase.com/home/TabId/...) which is estimated over 200 Elo weaker. We have had no joint participation.

(6) The main flaw with the Guid-Bratko work---still in their latest paper---is that it does not adjust for the overall level of challenge in a game---or they try to with their "complexity" measure but it is artificial and weak. To do so, IMHO one needs to analyze all reasonable moves to equal thoroughness, such as in Many-PV mode. Nor does their work observe an important scaling effect with the overall evaluation, nor is it capable of producing internal confidence intervals ("error bars"). It is OK for relative ranks but not for an absolute quality measure. They do not have remotely near as much comparison data for scientific control.

They necessarily have the same assumption of independence of moves. I agree with AylerKupp's critique of that, but it should not cause bias. Rather the main effect should (only) be a lowering of the effective sample size (i.e., the number N of moves), which is reflected by an empirically-based adjustment I make to the internal error bars.

When will your research include taking into account changes in the accuracy of play due to different time controls and difficulty of positions? If possible, practical play (sacrificing the accuracy of play in favour of other goals, such as, for example, getting easier positions to play, or rejecting lines that lead to a drawish game in a must-win situation) should also be included, but since it's not easy, it's not a very big sin to omit that.

Nobody expects the Spanish Inquisition!

<KWRegan>

As far as the second game yes, I agree that pointing out a forced mate in 8 is not the type of thing that one thinks about when your opponent is resigning. But that doesn't mean that the forced mate was not calculated earlier and Allwermann was just pointing it out after his opponent resigned.

As far as the third game (the first cited Ivanov game) I don't agree that 22.Ra1 is "pretty similar" or "almost a carbon copy" to 31.Qa7 from Allwerman's first game. Not even close. Black's king is not trapped in the corner, the first rank is not vulnerable, and there are multiple pieces in the way. So it may perhaps look "pretty similar" to GM Gserper but I certainly wouldn't find 22.Ra1 in a million years. That's one reason, probably among many, that he is a GM and I am not.

All of this, of course, doesn't prove that Ivanov was not cheating. I don't know what country you are from but in mine (USA) a person is presumed innocent until proven guilty. And, in order to be proven guilty in US criminal law, the prosecution (accusers) must show that the accused had motive, opportunity, and means. That may not be consider right by all, particularly if the accuser is accused of a particularly bad crime and emotions are high, but it does make sense to me, even if it is different in other countries.

Ivanov certainly had motive (winning a tournament or at least doing better than expected) and he certainly had opportunity (playing in the tournament). But what was his means? If the accusers believe that Ivanov was cheating, how are they saying that he accomplished this? Without showing how he cheated, they have no proof. And without proof accusations of cheating are, at best, irresponsible and, at worst, subject to a libel suit.

I have found more articles and several videos that I need to watch and I haven't made up my mind with regards to Ivanov's cheating, but if "the question is, how does he do it" has not been answered, then as far as I'm concerned there is insufficient evidence to show that he was cheating. And I certainly wouldn't take what I consider flawed computer analysis as "proof".

In these cases the analyses have been done by comparing the computer's considered "best move" with the move that the player actually played, which presumably is the move that the player thought was the best move. This is not necessarily the case (Lasker comes immediately to mind) but I would think that it is the case the far majority of the time.

Now, the move that the computer considers best at low search ply may simply not be the objectively best move. Running it to a high search ply doesn't guarantee that it is the best move either, but I would say that my confidence that a computer's selected move is indeed the best in a given position is greater if the search is conducted at high ply than if it is stopped at low ply. Sure, there will always be exceptions but that is likely to be the case.

Now, if a player plays a move other than what the computer selects as the best move, the player is considered relatively weaker. But I am not ready to accept that the computer's move when limited to a low search ply is stronger than the move that the player selected. I am more willing to accept that situation at higher search plies and very often the computer's selected best move at a higher ply will be different than the computer's selected best move at lower plies. I have seen many instances in engine vs. engine tournaments when an engine's play, most often Rybka's, just falls apart when in time trouble (which seems to happen to Rybka a lot), its search ply is limited.

And, of course, in the case of dummies crashing there is no a priori information available to the dummies to help them reduce the amount of injury in a crash, so I don't think that is a good analogy in this case. Except, sadly, the description of "dummies crashing" applies to my chess game all too well.

Don't ask me why but on my way to dinner tonight I had two ideas that I wonder if you ever considered:

(1) You have indicated in your papers that you have run tests on 150,000+ games databases. Today multi-million game databases are not that uncommon; as of today ChessTempo's games database contains more than 2.5M games, and includes more than 800K games where both players are rated 2200+.

So my thought is: If instead of your 150,000+ games database you were to select N games from the 2.5M+ games database at random, how many games would need to be analyzed in order to have, say, + 5% confidence in the accuracy of the results? If the number were reasonably small (and I will let you decide what "reasonably small" means to you), would it be feasible to run your analyses at substantially higher fixed search depths (say d=24 for Rybka, d=28-29 for Houdini, and d=33 for Stockfish) to see if they match your results at lower search depths? That would seem to validate the concept that, within reason, the search depth is not a significant factor.

(2) Arranging a match between one or more master-level players (the more the better, of course) and Rybka 3 , with the master-level players restricted to classical time controls (either 40 moves in 2 hours or 40 moves in 90 minutes) and the Rybka 3 limited to a search depth = 13 (with PONDER=OFF, of course). Then, presumably, the "entity" that won the most games is playing the best moves. So, if the chess engine were to win a large majority of the games then it would indicate that, indeed, using a chess engine under those conditions is a suitable way of assessing a player's strength by comparing its moves against the engine's. If, on the other hand, the human player were to win the large majority of the games, then at least in my mind it would not be proper to use the engine's "best" moves as a benchmark for assessing the human player's overall strength. If the results of the match were evenly balanced, well, that would be the subject for another discussion.

Best of all, IMO, would be to arrange such a match between Rybka 3 and Mr. Ivanov and see if he could still beat Rybka 3 by a score of 10-0 (which I suspect was also said tongue-in-cheek).

http://en.wikipedia.org/wiki/Sample...