I'd bet on the engine winning without an opening book <and also> giving pawn odds.

<One of the things that most impresses me about AlphaZero and other neural network engines is that they don’t have opening books or tablebases.>

I think you are right. It seems AlphaZero performance actually decreases when forcing it to play openings it hasn't discovered and preferred on it's own. It can be seen by the fact the only games stockfish was able to win were the games played from a pre-arranged opening.

It seems to me with my basic chess understanding that the AlphaZero approach is most different from traditional engines at the opening phase mainly because most engines can usually avoid serious tactical blunders in middle-game and the endgame can sometimes be calculated to high degree of accuracy as less moves are possible at each step.

The opening and early middle-game is really where it's pattern recognition and positional superiority comes into play. It's ability to optimize the strategy is more important at the early stages when long term plans and positional advantages defeat the engine horizon of depth ply.

You still don't either understand what I'm saying or are deliberately ignoring it. No innovation in what? A system (hardware + software) in cat vs. dog image classification or in algorithms and/or larger training sets for convolutional neural networks (CNNs) used in image classification?

No doubt that there have been tremendous improvements in the accuracy of image classification of cats and dogs in the last 5 years. But what were the reason(s) for this improvement in accuracy? Were new concepts developed and applied? Were the existing algorithms enhanced in either minor or major ways? Was more powerful hardware used that allowed the improved classifications to be performed in an acceptable amount of time and/or allow deeper CNNs to be applied? Or all three?

If the improvements were due <mostly> by the development and application of new concepts or the application of old concepts in new ways then I would definitely say that innovation in this area has occurred in the last 5 years.

If the improvements were due <mostly> to hardware technological advances allowing the application of more powerful hardware or hardware architectures <and> if the software modifications were due mostly to the adaptation of the existing software to the new hardware architecture without modifying the existing algorithms, then I would definitely say that no <software> innovation in this area has occurred in the last 5 years (I'll assume that the reference to 25 years was a typo).

If the existing algorithms were enhanced then there may or may not have been innovation, depending on the magnitude of the enhancements. If they were minor without fundamentally altering the algorithms I would say that no significant innovation occurred. If they were major and implemented in a new ways then I would say that at least some <software> innovation took place.

So the answer as to whether innovation took place is not straightforward, it depends on the area that caused the majority of the increased accuracy to take place. And if there were enhancements in all 3 areas (new concepts, enhanced algorithms, new and more powerful hardware) then it would be instructive to determine what percentage of the accuracy improvement was due each of the 3 areas. You seem fixated in the idea that if substantial improvement in the accuracy of cat vs. dog classification has taken place then this <must> have been the result of innovation in the <software> and <algorithms>, nothing else.

You might find the following articles interesting, they address the current state of the art in cat vs. dog image classification.

1. Image Classifier – Cats vs. Dogs: https://towardsdatascience.com/imag...

2. Dogs vs. Cats: Image Classification with Deep Learning using TensorFlow in Python: https://www.datasciencecentral.com/...

I would be willing to bet that 5 years from now, if LeelaC0 and Stockfish played a 100-game match at classic time controls using computers with similar computational capability that Stockfish would win the match convincingly.

I would be willing to bet that 5 years from now, if LeelaC0 and Stockfish played a 100-game match at classic time controls using computers with similar computational capability that Stockfish would win the match convincingly. ...>

I don't know if they process information more intelligently but they certainly process it differently than other forms of AI. AI has many forms and there has been a lot of hype about it for decades, much of it unfortunately unfulfilled. You just have to take the long view.

Remember Expert Systems introduced around 1965? They process information about a particular domain based on a knowledge based consisting of facts and rules provided by experts in that domain. They have proved extremely useful in some narrow areas but not generally, perhaps because of the inefficiency of capturing the rules from human experts and the limited accessibility of the later to help in this task.

Remember the Fifth Generation Computer System project initiated by the Japanese Government around 1982? It was intended to use massively parallel computer systems programmed in the Prolog computer language which was declarative in nature (you told it <what> you wanted it to do rather than <how> to do it ) and, using rule-based AI techniques it would figure it out. It did not achieve the success expected of it possibly because it was ahead of its time in terms of the demands that it made on the hardware needed to implement it successfully. But now with relatively cheap and highly parallelized processors like GPUs and TPUs maybe the concept should be revisited.

Neural networks should not be considered hype. The fact is that they can implement solutions to problems in various domains and market sectors by use of appropriate training sets mean that they are very flexible. And if like AlphaZero they can teach themselves without relying on human experience as a basis means that they can potentially uncover new approaches in that domain without being constrained by previous human experiences and biases.

The reason for my offer of that bet (gee, I sound like David Levy!) is that, at this time and given similar computational capability, chess engines with handcrafted evaluation functions and using minimax will easily outperform neural network-based chess engines. And, in the same circumstance, alpha-beta search tree pruning assisted by heuristics has proven superior to MCTS as shown by the relative performance of Komodo 12.x and its MCTS option.

Will that situation be the same in 5 years? I don't really know, but it's my opinion that it will be. It depends on how the relative improvement in chess-playing neural networks and the success (if any) of implementing a much higher degree of parallelism in classic chess engine algorithms so that they would be amenable to efficient implementation on GPUs, TPUs, and whatever other highly parallel and relatively cheap processing hardware is developed.

Well, don't hold your breath. In order for classic engines to effectively use GPUs either their basic algorithms must be modified so that they are highly parallelizable or new algorithms developed. As I said to <john barleycorn> above, I'm sure that the recent successes of AlphaZero and LeelaC0 with stimulate this effort.

Attempts to use GPUs with traditional chess engine have not met with much success. The one I know the best is the effort by Srdjan Matovic with his Zeta classic approach-based chess engine with GPU support on and off since 2010 (see https://www.chessprogramming.org/Zeta).

It's success has been limited. While Zeta can play chess, it apparently can't do that very well, at least when it's not supported by a GPU. In the current CCRL engine vs. engine tournament (Apr-13-19) it's best version is rated at 2055 and ranked #3, compared to Leela Chess Zero's rating (also without GPU support) of 2650 and #127 ranking.

Srdjan Matovic has been developing the Zeta classic approach-based and GPU-supported chess engine on and off since 2010, so Leela Chess Zero is hardly the first chess engine to use GPU support. But "successfully" is a different story; while Zeta can play chess, it apparently can't do that very well. In the current CCRL engine vs. engine tournament it's rated at 2056 and ranked #344, compared to Leela Chess Zero's rating of 2650 and #124 ranking. And CCRL gives all engines with similar rating the same ranking, so there are actually 2279 engines considered to be better than Zeta and 1249 considered to be better than LeelaC0.

Neural networks should not be considered hype. The fact is that they can implement solutions to problems in various domains and market sectors by use of appropriate training sets mean that they are very flexible. And if like AlphaZero they can teach themselves without relying on human experience as a basis means that they can potentially uncover new approaches in that domain without being constrained by previous human experiences and biases. ...>

I would agree that this is mostly true. We usually know the problem that we want to solve, and using a neural network is one approach to solve the problem, specifically with supervised training (the neural network is provided with a learning set consisting of a set of input vectors and the correct output vectors resulting from the input vectors, and it figures out the best set of weights and biases to produce the best input output correspondence; i.e. solve the problem) and reinforced training (the neural network is provided with a similar set of input vector but is only provided a grade as to how well the network performed given the input vectors and its current set of weights and biases). In both cases the problem (e.g. playing a chess game) is known.

I wouldn't go that far. Given the apparent difficulty, at least at the moment, of effectively using GPUs and, probably, TPUs with classic chess engines, the ability of AlphaZero, LeelaC0, and other neural network-based engines to implement a superstring chess engine when supported by GPUs and TPUs has been a great success. So, as I've said before, it all depends on what you are interested in:

1. What is the superior system (hardware + software) to solve a problem (i.e. create a strong chess-playing engine)?

2. What is the superior approach (software only) to solve a problem (again, create a strong chess-playing engine) given that the hardware that the engine runs on has a given computational capability?

Why do I make a distinction? Because if and when (a big IF and a big WHEN) classic chess engines (consisting of a hand-crafted evaluation function and minimax search tree building with heuristics and alpha-beta pruning) are adapted to use highly parallel support devices (GPUs and TPUs) and achieve equivalent computational processing capability as current neural network-based engines (using reinforcement training and MCTS-based search trees), then the superior approach to playing chess will be decided, at least for the moment. And you know which side I'm betting on, particularly since I believe that classic chess engines proponents have been motivated by AlphaZero's success to look at ways to improve their classic engine performance.

I have to disagree with you there, JB. Humans learn to "solve problems" in many different ways and methods, sometimes they do it when they are as young as few months old.

For example, the ability to recognize a dog once you've seen a certain type of dog only once is a great example of "problem solving" where you are given a single example and able to deduce the characteristics of a prototype without defining it at all. Is it the ears and tail that makes a dog a dog? Not really, it is the "dogness" quality of the dog that one year olds are able to create a fairly accurate mental projection of and programmers have traditionally struggled to define accurately.

In that regards, computer neural networks solve problems in the same manner humans do, by learning from example sets and/or trial and error by that <learning> the features that describe the problem instead of trying to define them.

How do you define what hardware to use for your benchmark?

I mean, why not go with 1Hz 8bit CPU and 64KB memory?

Or in the same manner, why not go with a 64 core CPU equipped with two generation 2 TPUs?

What makes one hardware design more suitable to determine the "best software approach"?

My take is that there's only two ways to define machines of similar computational capability.

1. Money - they have to cost the same amount of money

2. Energy - they have to use the same amount of energy

Anything else is biased towards a specific approach, for example, your CPU flops method are biased towards CPU based algorithms such as stockfish.>

Energy costs money?>

Sigh ... I'll break my promise and address this fundamental question one more time since you still don't seem to get it, as though you were dense (which I don't think you are).

1. It <DOESN'T MATTER> what hardware you use for your benchmark nor whether the two hardware architectures are similar. If you want to determine the relative merits of the software to perform <any> task, you need to take the hardware out of the equation. No one hardware design is more or less suitable to determine "the best software approach", at least as long as the software will execute on that hardware.

2. In order to determine whether 2 (or more) sets of hardware have similar computational performance capabilities you need to look at some fundamental unit. Number of operations per second (or computations per second, if you prefer) is such a unit, and it doesn't matter whether you are using a CPU, a GPU, a TPU, or an abacus. An operation per second for one machine is roughly equivalent to an operation per second on another machine for the purpose of relative comparison, even though the operations may be different. They are not biased in any way towards a particular hardware architecture.

3. If you are not interested in determining which software approach (e.g. the classic chess engine or the neural network-based chess engine) is "better", then you don't need to bother with equivalent hardware computational performance capability.

3. Neither money nor energy are fundamental units in terms of performance capability although, of course, they will have an effect. If you want to make them your take, fine. You are, after all, in "love" with AlphaZero, and I'm certainly not about to interfere in matters of the heart.

<And yet, it is ridiculously unfit to perform Stockfish's (or AlphaZero's, for that matter) algorithm.>

Yeah, we must regret. The human brain even can't take the square root of 32419587 in less than a second. Mankind is doomed without A0. A miracle mankind still survived.

I don't understand what is it that you don't understand. You should probably phrase your question more precisely.

I understand your frustration from the going discussion. If we were corresponding through email then I wouldn't be replying but since this is a public discussion in an open forum I'm not going to stop pointing out what I perceive as the logical flaws in your arguments.

If you want to stop reading my posts then you can place me on your ignore list, but if you want me to stop responding to what I perceive as your erroneous claims you're gonna have to stop making them :)

<It <DOESN'T MATTER> what hardware you use for your benchmark> < No one hardware design is more or less suitable to determine "the best software approach">

This argument is logically flawed. If algorithm A1 outperforms algorithm A2 on hardware H1 but A2 outperforms A1 on hardware H2 then "the best software approach" is dependent on the available hardware.

As I've shown examples for such algorithms this constitute as a <logical proof>. No amount of <Sigh>s or excessive use of capital letters and brackets can dispute this.

It doesn't even matter that "best" and "approach" are subjective terms in that regards since the proof is about the dependency of the software and the hardware in determining algorithms qualities.

<In order to determine whether 2 (or more) sets of hardware have similar computational performance capabilities you need to look at some fundamental unit. Number of operations per second (or computations per second, if you prefer) is such a unit>

Number of operations per second is not a fundamental unit. Watts on the other hand are. AlphaZero utilize Watts better as it can achieve more operations per second with same amount of Watts. Watts as a fundamental unit simply don't fit your paradigm that people are "in love" with AlphaZero for no reason.

<If you are not interested in determining which software approach (e.g. the classic chess engine or the neural network-based chess engine) is "better", then you don't need to bother with equivalent hardware computational performance capability.>

I am interested in this question very much actually. I also agree that AlphaZero performs more bit operations per second than Stockfish. My argument is that AlphaZero's approach is more efficient, meaning, for the same amount of energy it can perform a lot more operations. Also for the same amount of budget it can achieve better chess prowess.

Those "efficiency" measures are more important to me in determining the "best software approach".

If you can solve 10 times more things than me with the same amount of effort, isn't it a consensus that your "approach" is "better"?

<Neither money nor energy are fundamental units in terms of performance capability although, of course, they will have an effect. If you want to make them your take, fine. You are, after all, in "love" with AlphaZero, and I'm certainly not about to interfere in matters of the heart.>