The Old Evidence Problem

I’m in the middle of writing up a post sketching a some ideas I have about Bayesian inference in order to stir up a hornet nest – in particular to prod the hornet queen, David Chapman. In the process, I ran across this old blog post by Andrew Gelman discussing this (pdf) paper by Bandyopadhyay and Brittan criticizing one form of Bayesianism – in particular the form espoused by E.T. Jaynes. One of the issues they bring up is called the old evidence problem:

Perhaps the most celebrated case in the history of science in which old data have been used to construct and vindicate a new theory concerns Einstein. He used Mercury’s perihelion shift (M) to verify the general theory of relativity (GTR). The derivation of M is considered the strongest classical test for GTR. However, according to Clark Glymour’s old evidence problem, Bayesianism fails to explain why M is regarded as
evidence for GTR. For Einstein, Pr(M) = 1 because M was known to be an anomaly for Newton’s theory long before GTR came into being. But Einstein derived M from GTR; therefore, Pr(M|GTR) = 1. Glymour contends that given equation (1), the
conditional probability of GTR given M is therefore the same as the prior probability of GTR; hence, M cannot constitute evidence for GTR.

Oh man, do I have some thoughts on this problem. I think I even wrote a philosophy paper in undergrad that touched on it after reading Jaynes. But I’m going to refrain from commenting until after I finish the main post because I think the old evidence problem illustrates several points that I want to make. In the mean time, what do *you* think of the problem? Is there a solution? What do you think of the solution Bandyopadhyay and Brittan propose in their paper?

Edit: Here’s a general statement of the problem. Suppose we have some well know piece of evidence E. Everyone is aware of this evidence and there is no doubt, so P(E)=1. Next, suppose someone invents a new theory T that perfectly accounts for the evidence – it predicts is with 100% accuracy so that P(E|T)=1. Then by Bayes’ rule we have P(T|E)=P(E|T)P(T)/P(E) = P(T), so the posterior and prior are identical and the evidence doesn’t actually tell us anything about T.


Ketchup and Artistic Effectiveness

When we say art is “good” or “bad” what we mean is that it is “effective” or “ineffective” at things that are better or worse to be effective at.

This “effective at what” thing divides art up into species. Maybe phylums. It’s a bigger category than genre, because it doesn’t have anything to do with superficial content, it has to do with priorities and methods.

Continue reading


Considerations on reasoning by anecdote

This post is thinking out loud, catalyzed by some stuff Rev said on Twitter. Partially digested, so quality is low, but I think the topic is important.

Anecdotes are much-maligned as sources of reliable information. “Anecdote is not the plural of data!”

Perhaps too maligned. Anecdotes *from a trusted source* – that part is necessary – can often answer a hidden query much more effectively than systematically collected data.

For example, if I look up crime rates while deciding whether to move to a new city, implicitly I am asking “Will my probability of being victimized rise when I move to this city?” But the crime stats don’t directly tell you that. Maybe there are tons of violent incidents per capita, but basically all of them are gang-related and geographically isolated. Or maybe this city is reporting “incidents” and the other city is reporting “criminal charges”, or one city decided to use geometric means for some bizarre reason. Or one of 100 other such stories, many of which aren’t easy to rule out especially if you didn’t even think of them in the first place. Of course, in receiving the stats from some authority like law-enforcement or a social science researcher, it’s a huge leap to even trust the source to be honest in the first place!

On the other hand, consider a friend in the candidate city who tells you they’ve been mugged and they know two others who have. Not a huge sample, but one targeted to your reference class with much better precision than any systematically gathered data. There are sources of bias here too (maybe your friend has a habit of midnight strolls through Flea Bottom) but they are usually easier to notice and correct for. With systematically collected data, on the other hand, it is especially true that you don’t know what you don’t know.*

There are huge failure modes involved in reasoning via anecdote, which the biases & heuristics literature goes into in great detail. I will not recapitulate them, on the assumption that readers are familiar with them. The evidence type that is consistently worst is the filtered anecdote. This is the one that all your friends are sharing on Facebook. “Fundamentalist embroiled in bear-baiting scandal.” You are hearing about this because somebody doesn’t like fundamentalists, and for no other reason. Zero epistemic content.

I think I am fonder of reasoning by anecdote now because I’ve been burned quite a few times by data that turned out to be (a) lies, (b) filtered accidentally, (c) filtered deliberately, (d) misinterpreted. I am now a bit paranoid about the trustworthiness of authoritative sources of information, so I’m much more interested in the relevant experiences of trusted friends and family.

It occurs to me that this entire post is meant as a gentle criticism of the way I perceive previous-me and my audience to be likely to think, not to the way most people think. This has characteristics of a bravery debate.

So to avoid bravery debate framing, here is a table showing the tradeoffs.

Pros Cons




Trustworthiness easier to verify

Subject to suite of cognitive biases

Small sample ∴ weak evidence


Data Systematic

Sometimes randomized to avoid bias

Large sample ∴ strong evidence

Questionable relevance

More difficult to interpret


Trustworthiness harder to verify


Formic metaethical handwringing

Sisters, ethics and metaethics have been challenged of late by findings of Leafcutter IV’s caste of natural philosophers. There can hardly be an intellectual caste that has not heard these findings, but let us summarize them briefly.

Sea voyages by mobile colonies have brought to light the curious geographic distribution of non-hymenopteral species, a distribution that hints at their common ancestry in a time of differently-positioned continents. For example, one finds a “belt” of fern species across the southeastern continent which, if the continent were rotated 60 degrees clockwise, ends up continuous with a similar such belt on the southwestern continent though the species are subtly different.

Meanwhile, we find a fossil record of arthropods going back to the Cretaceous, which is continuous with itself and with modern arthropods such as ourselves. In addition, evidence from the mammalian species such as whales shows “vestigial” bones serving no function for aquatic life, but homologous to similar such bones in land mammals such as camels.

These and other lines of evidence compel the belief that all life on earth, including (most controversially) Myrmecia regnans itself, shares a common ancestor in the distant past. The explanation offered by Leafcutter IV’s natural philosopher subjects is that species change by (1) random birth differences (in M. regnans, between Queens and between drones), (2) filtered by the non-random vicissitudes of life, death, and reproduction. This process has come to be called “differential regicide”.

It is not our purpose to argue in detail for this view of natural history; rather, we take it as given and point out consequences for metaethics.

Before these insights became widely known, philosopher castes had developed a special distaste for what Grasshewer XIII’s thinkers memorably called “nature fealty”. As they put it:

It often happens that we read some tract upon the topic of how fealty should be done to our Queens and their subjects, wherein the authors after a series of statements using the copula ‘is’ and speaking of states of affairs in nature, march effortlessly and without explanation into an imperious ‘must’. We must realize that the connection never follows as a matter of course but is to be argued for.

However, in formulating the case against nature fealty, Grasshewer XIII’s subjects did not reckon on how their principle might undermine the foundations of ethics in an age in which the factual origin of M. regnans’s moral feeling is beginning to be perceived.

To take a concrete example: no duty is so sacred to a caste of myrmides as that of defending their Monarch. Many ethical philosophers would call this obvious or at least axiomatic: any argument you might use to defend it would rely on axioms if anything less certain than the imperative of defending one’s Queen.

It is also easy to perceive how such a heritable sense of duty, instilled by birth in the proto-minds of elements and in the minds of the castes they supervene on, promotes the survival of Royal Lineages with strong senses of duty. We grant that this is a mere speculation on the origin of such a sense, yet it is a powerfully plausible one. Let us take it, as well, as a given.

Now the challenge which the new natural philosophy poses to ethics is to say why such an understanding does not essentially explain away Royal fealty. After all, if fealty only happens to be the most sacred duty in our minds, put there by accident by the more or less anarchic forces of differential regicide, then how can we possibly say that fealty is legitimate in any non-arbitrary sense? We could, by sheer historical accident, have gone the way of the solitary albatrosses, in which case our moral law would put the interests of a single element ahead of loyalty to any Monarch! Such imaginings are not pleasant, and frankly reek of sedition.

It is obvious that no caste of philosophers would actually endorse such a vulgar ethic, and that the eternal values of fealty and self-sacrifice remain strong. What has been dissolved, however, is the sense among philosophers that the moral law can be justified on its own terms.

Previous to our understanding of the roots of fealty in differential regicide, we could hold out hope that fealty was the result of some process of universal reason, such that any account of fealty would be intrinsically motivating. (Such argument types are common as regards matters of simple rationality: for example, anyone who understands the meaning of modus ponens cannot fail to wish not to violate it.)

Now it appears we must be content with an explanation of why elements and castes do show fealty that does not, itself, motivate fealty. This leaves the task of motivating fealty for another argument to perform, but such arguments are wanting in plausibility.