< Earlier Kibitzing · PAGE 19 OF 19 ·
|Jun-19-18|| ||Gypsy: <Tiggler: I am thinking about a problem that might be analogous, ...> I think you are correct. Your's is one of the 'deconvolution' type of problems. The go under many monikers: independent component analysis (different from principal component analysis), projection pursuit, and so on ...|
For these problems, some entropy based approach is usually a theoretically satisfying way to go. Sometimes, something like kurtosis (the 4th statistical cumulant, k4) is in fact used, but that is because it is a sort of poor man's entropy (in this case, it tends to be the first term of entropy cumulant expansion that matters).
Another place where entropy based methods seem to pop up a lot is when you deal with vectors in the first orthant and the world you are looking at is sort-of multiplicative, the quantities scale up or down a lot, and 'information' seems to be of importance (or at least of use).
|Jun-19-18|| ||Gypsy: I should clarify that <lastplanet> is not my own alter-ego but the alter-ego of my spousal unit. While we do most of things together, we each work on our own math problems. Fortunately, when she creates something nifty, like the <lastplanet> site, I am the person to to brag to. :-) Thus, I know the basics of the way computations work there.|
|Jun-19-18|| ||Gypsy: Here is a basic scheme for turning the moves (also a 'score') of a chess-game or chess-game fragment into a (soccer-like) score. It utilizes a chess-engine as an auxiliary oracle. |
X = [X0, X1, X2, ...]
be the sequence of position-evaluations by an engine, say, Stock-Fish.
x = diff(X) = [X1-X0, X2-X1, X3-X2, ....]
y = x^+ = max[x(i),0] ... for all i
z = z^- = max[-x(i),0] ... for all i
be the positive part and the negative part of x, respectively. (Note that both, x^+ & x^- are non-negative sequences, that x = (x^+) - (x^-), and that the indicated max operations has been taken taken element-wise over the entire sequence/vector.)
We define the game/game-segment (running) score by
Y = cumsum(y)
Z = cumsum(z)
and the overall score as
Y(end) : Z(end)
(Numerous variations of, as well as some objections to, the basic scheme will readily spring to mind; I suspect. We shall examine them later.)
|Jun-19-18|| ||Tiggler: OK, Gypsy, following with baited breath: don't let us down.|
I recall from a previous conversation a few years back that you mentioned a spousal unit who knew about social biology. Same one?
|Jun-20-18|| ||Gypsy: Marine biology (population dynamics). Same spousal unit; for life.|
We met in college: I was just finishing OR (and Stat) grad school; future <lastplanet> was just finishing undergrad degrees for a curious double mayor in mechanical engineering and biology. She then went to get her PhD in Ecology; now works on endangered marine species.
Amazingly creative with a great capacity for work, she picks up hobby projects. All in all, <lastplanet> is about an equivalent of a good half-dozen renaissance men.
|Jun-20-18|| ||Gypsy: < ... variations of, as well as some objections to, the basic scheme will readily spring to mind >|
Probably the most troubling concern I have with the basic scheme is what to do when people do not (in all human decency) resign their game when warranted, but play on and on their already ruined position.
Moreover, when an inevitable mate is already forming on the board, engines often switch from reporting evaluations of the position to reporting the least number of moves left to play to the unavoidable check-mate. And even if they not swith, engines return some absurdly large number, like +99, to indicate that is sees an inevitable mate. This clearly requires some kludge to get around the problem.
The workaround I am presently thinking of is to use the generic value of rook (5 pawns) as a cutoff point.
Thus, using the previous notation, rather than working with X directly, I think we should work with a truncated X', where values larger than 5 (or if 'White checkmating' is indicated) will be truncated to 5; and values lower than 5 (or if 'Black checkmating' is indicated) will be truncated to -5.
So, instead of scoring directly from X, I am thinking of scoring from X', which relates to X by
(1') X' = max(-5,min(5,X))
Again, the max and min are taken element-wise.
My reasoning for settling upon using the 'clear rook advantage' value of 5 as the best compromise goes follows:
In technical positions, I personally begin to suspect that a game is a win when the evaluations get to about 1.2 pawn. However, a lot of hard work still probably remains to bring the win home. The advantage of a clear-exchange, or clear 2 pawns, is often resigned and tends to fairly easy to win. The advantage of a minor piece (3 pawns) is totally resignable as long as there is enough pawns to bring of them to promotion. A rook advantage is a mating material even when the pawns are all gone. Two minors are also mating, except for the weird case of two knights. And if a pawn promotes I do not care whether one has one queen extra or two or three; it is just pointless running up of the material score.
Thus the value of clear rook seems to be a reasonable engineering compromise, given that some sort of kludge seems to be necessary.
I should note that when a position is not technical but combinative, position evaluations may easily jump all over the place, going above and below the threshold of 5 as players miss tactical nuances that computer readily sees. In such cases, the threshold at 5 may reduce both sides the overall score Y(end):Z(end), but both sides by the same amount. (Instead of final score being 15:12, for instance, it may get reduced to perhaps 11:8.) Not ideal, but an artifact that I hope to be able to ignore.
(The lastplanet site has had to deal with similar artifacts -- arising from schemes designed against teams running up goal scores, games decided by a shootout, and such)
|Jun-20-18|| ||Gypsy: Here is a bit more cheerful note:
I find it entertaining that the score, if done ply by ply, is essentially a blunder-counter -- my score is dominated by the errors and blunders that my opponent has made, and vice versa.
Since the engines' play is so much superior by now, our moves are essentially compared to a near optimal alpha-beta line. If thins move up for me on my move, the improvement comes from a lucky horizon effect. However, the possibilities for me to play second best moves, or to play outright blunders, are endless.
|Jun-20-18|| ||Tiggler: <Gypsy>:<R & S are diagonal matrices (of the right dimensions) ... these are the variables of the problem and it is from the optimal R & S that we extract the ratings at the end. (In the case of rating soccer teams, one of the diagonal matrices yields the ratings of offensive skills, the other yields the ratings of defensive skills.)>|
So how are these used to predict something?
|Jun-21-18|| ||Gypsy: <So how are these used to predict something?>|
Let's start with the basic stuff, and forgive me if I go pedantic and slow. For one thing, I want be to clear when I can. For another thing, we want to (need to, really) think through the possibilities that are opening up
<Poisson Model vs RAS>
In the soccer app model, the Poisson assumption is that the arrival rate(s) of goals between teams i & j is approximately Poisson, with parameters
lambda(i,j) = rho(i) - sig(j)
lambda(j,i) = rho(j) - sig(i)
Exponentiation relates us to the mean number of goals in a game
exp(lambda(i,j)) = exp(rho(i))*exp(-sig(j)) = r(i)/s(j)
exp(lambda(j,i)) = exp(rho(j))*exp(-sig(i)) = r(j)/s(i)
For us, the parameters r(i) & r(j) come from the diagonal of the matrix R; and 1/s(i) & 1/s(j) from the diagonal of the matrix S of the RAS computation:
<A numerical example:>
Therefore if, as in the LP blog, the rating of the team i comes from <r(i) = 24.2> (for offense) and <s(i)=10.00> (for defense), while the rating of the team j comes from <r(j) = 29> & <s(j) = 11>, the expected outcome of a game between i and j is
<2.1 : 2.9>;
and <2:3> when rounded to the nearest integer.
|Jun-21-18|| ||Gypsy: (I hope I got the arithmetic right)
At this point, we we may start moving beyond the predictions that elo can offer.
Using the Poisson distribution formulas, for instance, we can compute the probabilities of win/draw/loss for the two lambda.
(Even when soccer teams are evenly matched, probability of a draw goes down for high scoring games; thus if both lambdas are 1, the probability of a draw is larger than when both lambdas are 3.
|Jun-21-18|| ||Gypsy: When divining the probabilities via the Poisson model, we are helped by the fact that Poisson is a 1-parameter distribution family (lambda being that parameter). |
All of the usual parameters of Poisson can be readily expressed in terms of lambda. Most notably, the variance of Poisson is is also lambda, Var = lambda.
If the Poisson model is not really realistic, I think reasonable estimates of the variance can probably still be finessed from the RAS computations. But this feel as a bit too much of a tangent to follow at this point.
|Jun-21-18|| ||Gypsy: Instead, I hope that the following will be the low-hanging fruit of this approach to chess forecasting:|
The signature property of a true Poisson arrival process is that any particular time is equally likely for an arrival (an event). The process is memoryless (in the terminology of Markov processes).
In chess, that is seldom true. Instead, some people are specialists of the openings, some of middle game, and some are end-game specialists. Only a few of the best seem equally strong in all stages of the game (Caruana ?)
This thought begs a different one: Shouldn't we then rate the proves of players in the openings, middle game, and end-game separately? It is much easier to believe a constant error rate within each of these long established stage of the game than it is to believe a complete constancy of skill without.
Conveniently, scoring of the game fragments (and RAS computation) will work.
We just need chop each game into the appropriate pieces. And before trying something complicated, I'd be inclined to arbitrarily declare moves 1-15 as the opening, 15-30 as the middle-game, and 30-end as the endgame.
|Jun-21-18|| ||Tiggler: <Gypsy> OK thanks. All good as far a soccer is concerned, but I don't like the chess idea because of the reliance on engines.|
|Jun-22-18|| ||Gypsy: <... I don't like the chess idea because of the reliance on engines> Oh, you are talking to somebody who is a Ludite by temperament, Ludite by heart. I swore off engines for decades.|
It all reminds me a long ago talk by Bradley Efron (the first time I heard of bootstrap, other re-sampling techniques). He opened up approximately thus:
There is this tension between computing and mathematics. Computers are getting faster and faster. Suppose that somebody finally designe a machine capable of examining each countable number (aleph null) of objects. The first thing I would do, would be to run it on the Twin-Prime Conjecture. It would crank for a while and then it would return and say "True". Then I would let it loose on the Goldbach Conjecture. It would crank for a while then return and say "False". In a short order, I would thus dispose of most outstanding problems in number theory.
It would make number theory, as we know it, obsolete and I would become dejected for making myself obsolete before my time. But then, I would find out that new mathematical problems are emerging, derived from the use of the infernal machine. And those problems would allow me to be clever and inventive again and I would resume being a mathematician.
|Jun-22-18|| ||Gypsy: Speaking of Bootstrap. It is something that flies into the face of common sense and of mathematical rigor. (And nobody was more keenly aware of it than Efron himself) Yet, it is an absolutely indispensable tool in the applied work of a modern statistician. It allows one to empirically establish a fairly reasonable Null-Hypothesis where one has no prayer to establish one by classical methods.|
In a way, we are in a similar boat here. We would like to have some handle on the statistical distribution of moves; well, on the distribution of good moves perhaps, because most of the moves are often complete duds.
We are in a need of an oracle. When I was a kid, those oracles were called Grand Masters, and we didn't question much what they wrote. (Ludek Pachman was the oracle of that time and place for me. Then Russian tanks rolled their tanks into Prague and Pachman disappeared for many years.)
Today chess engines are generally more powerful oracles than even Ludek Pachman was in his best days. And, oh so convenient, they talk to us via the numeric tape of centi-pawns.
|Jun-24-18|| ||Tiggler: <Gypsy> I don't really like the idea of scores in chess, because there really are none apart from won or lost. Nearest thing in my mind is the question who is better? In any given chess position, this question arises, even to humans while playing. So if you wanted to use an engine to score a game, I would use the percentage of total moves when each side was rated as better.|
|Jul-06-18|| ||Gypsy: <Tiggler> Off at some "hard traveling" (ok, not all that 'hard'). Plan to come back to this.|
|Jul-07-18|| ||Tiggler: Iíve been doing some hard traveling, I thought you knowed|
Iíve been doing some hard traveling way down the road
Iíve been doing some hard traveling hard drinking hard gambling
Iíve been doing some hard traveling Lord
|Jul-25-18|| ||Gypsy: Bound for glory!|
|Jul-25-18|| ||Tiggler: This train is ...|
|Aug-11-18|| ||Boomie: <Tiggler: I think Eisenhauer was the only great President in my lifetime. He didn't have to worry about his legacy, he took us out of Korea, kept us out of all other foreign conflicts...>|
He did have one shameful action when the US overthrew the democratically elected president of Guatemala to protect United Fruit and other corporate interests.
Otherwise, I agree with your assessment. He knew how to get people who hated each other to work together. He possessed that rarest of qualities among politicians, common sense.
|Aug-11-18|| ||JimNorCal: "He possessed that rarest of qualities among politicians ..."|
Of course, it's worth noting he was not a politician. There was a bunch of dysfunction missing from his DNA, if you will.
|Aug-11-18|| ||Tiggler: Someone picked me up for misspelling his name. Sorry about that. His great-great grandfather, here during the founding of the USA, used the German spelling, I found that on some geneology site. My browser's spell checker does not recognize that word I just used. Perhaps I'll try the American version: <jeanieology>.|
|Aug-11-18|| ||JimNorCal: I read that his name demoralized the German propagandists in WWII. After crying up the strength of their "Iron Wall" of defense it was announced that the CinC of Western forces was Eisenhauer (iron hewer). "Hew" is an old fashioned verb meaning to cut through something.|
|Aug-12-18|| ||JimNorCal: Umm, yes. Eisenhower.|
< Earlier Kibitzing · PAGE 19 OF 19 ·