LobsTheorem

This document is ©2008 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/lobs-theorem/ .

This essay is meant for a reader who has attained a firm grasp of Bayes’ Theorem. An introduction to Bayes’ Theorem may be found at An Intuitive Explanation of Bayesian Reasoning . You should easily recognize, and intuitively understand, the concepts “prior probability”, “posterior probability”, “likelihood ratio”, and “odds ratio”. This essay is intended as a sequel to the Intuitive Explanation , but you might skip that introduction if you are already thoroughly Bayesian. Where the Intuitive Explanation focused on providing a firm grasp of Bayesian basics, the Technical Explanation builds, on a Bayesian foundation, theses about human rationality and philosophy of science.

The Intuitive Explanation of Bayesian Reasoning promised that mastery of addition, multiplication, and division would be sufficient background, with no subtraction required. To this the Technical Explanation of Technical Explanation adds logarithms. The math is simple, but necessary, and it appears first in the order of exposition. Some pictures may not be drawn with words alone.

As Jaynes (1996) emphasizes, the theorems of Bayesian probability theory are just that, mathematical theorems which follow inevitably from Bayesian axioms. One might naively think that there would be no controversy about mathematical theorems. But when do the theorems apply? How do we use the theorems in real-world problems? The Intuitive Explanation tries to avoid controversy, but the Technical Explanation willfully walks into the whirling helicopter blades. Bluntly, the reasoning in the Technical Explanation does not represent the unanimous consensus of Earth’s entire planetary community of Bayesian researchers. At least, not yet.

The Technical Explanation of Technical Explanation is so named because it begins with this question:

What is the difference between a technical understanding and a verbal understanding?

A fable:

Once upon a time, there was a teacher who cared for a group of physics students. One day she called them into her class, and showed them a wide, square plate of metal, next to a hot radiator. The students each put their hand on the plate, and found the side next to the radiator cool, and the distant side warm. And the teacher said, write down your guess why this happens. Some students guessed convection of air currents, and others guessed strange patterns of metals in the plate, and not one put down ‘This seems to me impossible’, and the answer was that before the students entered the room, the teacher turned the plate around.

(Taken from Verhagen 2001.)

There are many morals to this fable, and I have told it with different morals in different contexts. I usually take the moral that your strength as a rationalist is measured by your ability to be more confused by fiction than by reality. If you are equally good at explaining any story, you have zero knowledge. Occasionally I have heard a story that sounds confusing, and reflexively suppressed my feeling of confusion and accepted the story, and then later learned that the original story was untrue. Each time this happens to me, I vow anew to focus consciously on my fleeting feelings of bewilderment.

But in this case, the moral is that the apocryphal students failed to understand what constituted a scientific explanation. If the students measured the heat of the plate at different points and different times, they would soon see a pattern in the numbers. If the students knew the diffusion equation for heat, they might calculate that the plate equilibrated with the radiator and environment two minutes and fifteen seconds ago, turned around, and now approaches equilibrium again. Instead the students wrote down words on paper, and thought they were doing physics. I should rather compare it to the random guessing of Greek philosophers, such as Heraclitus who said “All is Fire”, and fancied it his theory of everything.

As a child I read books of popular physics, and fancied myself knowledgeable; I knew sound was waves of air, light was waves of electromagnetism, matter was waves of complex probability amplitudes. When I grew up I read the Feynman Lectures on Physics, and discovered a gem called ‘the wave equation’. I thought about that equation, on and off for three days, until I saw to my satisfaction it was dumbfoundingly simple. And when I understood, I realized that during all the time I had believed the honest assurance of physicists that sound and light and matter were waves, I had not the vaguest idea what ‘wave’ meant to a physicist.

So that is the difference between a technical understanding and a verbal understanding.

Do you believe that? If so, you should have applied the knowledge, and said: “But why didn’t you give a technical explanation instead of a verbal explanation?”

In “An Intuitive Explanation of Bayesian Reasoning” I tried to provide visual and physical metaphors for Bayesian probability; for example, evidence is a weight , a pressure upon belief, that slides prior probabilities to posterior probabilities.

Now we add a new metaphor, which is also the mathematical terminology: Visualize probability density or probability mass – probability as a lump of clay that you must distribute over possible outcomes.

Let’s say there’s a little light that can flash red, blue, or green each time you press a button. The light flashes one and only one color on each press of the button; the possibilities are mutually exclusive. You’re trying to predict the color of the next flash. On each try, you have a weight of clay, the probability mass, that you have to distribute over the possibilities red, green, and blue. You might put a fourth of your clay on the “green” possibility, a fourth of your clay on the “blue” possibility, and half your clay on the “red” possibility – like assigning a probability of green:25%, blue:25%, and red:50%. The metaphor is that probability is a conserved resource , to dole out sparingly. If you think that “blue” is more likely to flash on the next experiment, you can assign a higher probability to blue, but you have to take the probability mass from the other hypotheses – maybe steal some clay from red and add it to blue. You can never get any more clay. Your probabilities can’t sum to more than 1.0 (100%). You can’t predict a 75% chance of seeing red, and an 80% chance of seeing blue.

Why would you want to be careful with your probability mass, or dole it out sparingly? Why not slop probability all over the place? Let’s shift the metaphor from clay to money. You can bet up to a dollar of play money on each press of the button. An experimenter stands nearby, and pays you an amount of real money that depends on how much play money you bet on the winning light. We don’t care how you distributed your remaining play money over the losing lights. The only thing that matters is how much you bet on the light that actually won.

But we must carefully construct the scoring rule used to pay off the winners, if we want the players to be careful with their bets. Suppose the experimenter pays each player real money equal to the play money bet on the winning color. Under this scoring rule, if you observe that red comes up 6 times out of 10, your best strategy is to bet, not 60 cents on red, but the entire dollar on red, and you don’t care about the frequencies of blue and green. Why? Let’s say that blue and green each come up around 2 times out of 10. And suppose you bet 60 cents on red, 20 cents on blue, and 20 cents on green. In this case, 6 times out of 10 you would win 60 cents, and 4 times out of 10 you would win 20 cents, for an average payoff of 44 cents. Under that scoring rule, it makes more sense to allocate the entire dollar to red, and win an entire dollar 6 times out of 10. 4 times out of 10 you would win nothing. Your average payoff would be 60 cents.

If we wrote down the function for the payoff, it would be `Payoff = P(winner)`, where `P(winner)` is the amount of play money you bet on the winning color on that round. If we wrote down the function for the expected payoff given that Payoff rule, it would be `Expectation[Payoff] = (Sum[P(color)*F(color)]` for each color) . `P(color)` is the amount of play money you bet on a color, and `F(color)` is the frequency with which that color wins.

Suppose that the actual frequency of the lights is blue:30%, green:20%, and red:50%. And suppose that on each round I bet blue:\$0.40, green:\$0.50, and red:\$0.10. I would get \$0.40 30% of the time, \$0.50 20% of the time, and \$0.10 50% of the time, for an average payoff of \$0.12 + \$0.10 + \$0.05 or \$0.27.

That is:

```P(color) = play money assigned to that color
F(color) = frequency with which that color wins
Payoff = P(winner) = amount of play money allocated to winning color                 ```

Actual frequencies of winning:

`Blue: 30%   Green: 20%   Red: 50%`

In the long run, red wins 50% of the time, green wins 20% of the time, and blue wins 30% of the time. So our average payoff on each round is 50% of the payoff if red wins, plus 20% of the payoff if green wins, plus 30% of the payoff if blue wins.

The payoff is a function of the winning color and the betting scheme. We want to compute the average payoff, given a betting scheme and the frequencies at which each color wins. The mathematical term for this kind of computation, taking a function of each case and weighting it by the frequency of that case, is an expectation . Thus, to compute our expected payoff we would calculate:

`Expectation[Payoff] = Sum[P(color)*F(color)] for each color                    `
```P(color)*F(color) for blue  = \$0.40 * 30% = \$0.12
+ P(color)*F(color) for green = \$0.50 * 20% = \$0.10
+ P(color)*F(color) for red   = \$0.10 * 50% = \$0.05```
```= \$0.12 + \$0.10 + \$0.05
= \$0.27```

With this betting scheme I’ll win, on average, around 27 cents per round.

I allocated my play money in a grossly arbitrary way, and the question arises: Can I increase my expected payoff by allocating my play money more wisely? Given the scoring rule provided, I maximize my expected payoff by allocating my entire dollar to red. Despite my expected payoff of fifty cents per round, the light might actually flash green, blue, blue, green, green and I would receive an actual payoff of zero. However, the chance of the light coming up non-red on five successive rounds is approximately 3%.

Tversky and Edwards (1966) conducted an experiment. Subjects were shown a succession of cards, each card either red or blue. 70% of the cards were blue, and 30% red; the color sequence was random. The subjects, asked to guess each succeeding card, would guess blue around 70% of the time, and red about 30% of the time – as if they thought they had some way of predicting the random sequence! Even when the subjects were paid a nickel for each correct guess, they still only guessed blue about 76% of the time. Why is this odd? Because you do not need to bet on a guess to test it. You could just say “blue” each time, being paid a nickel about 70% of the time, accumulating thirty-five dollars over a thousand trials, while mentally noting your private guesses for any (imaginary) patterns you thought you spotted. If your predictions came out right, then you could switch to the newly discovered sequence. There was no need for the subjects to bet on any patterns they thought they saw; they could have simply bet on blue until some hypothesis was confirmed . But if human beings reasoned like that, people would not buy lottery tickets, but instead write down predictions in notebooks at home, and begin buying lottery tickets only when their predictions began succeeding.

The mistake revealed by the experiment was not that the subjects looked for patterns in a random-seeming sequence; that is curiosity, an admirable human trait. Dawes (1988) comments on this experiment: “Despite feedback through a thousand trials, subjects cannot bring themselves to believe that the situation is one in which they cannot predict.” But even if subjects refused to accept unpredictability and continued looking for patterns, they didn’t have to bet on their guesses. They just needed to make a mental note of the pattern’s prediction, then keep betting on blue while waiting for confirmation. My suspicion is that subjects just didn’t think of the winning strategy. They didn’t realize that their betting pattern did not have to resemble the observed sequence of cards. On each round, blue is the most likely next card. The best financial strategy is not betting a mostly-blue pattern resembling the mostly-blue sequence, but betting all blue, to win as many nickels as possible. If 70% of the time you predict blue and 30% of the time you predict red, and the cards do not correlate with your guesses, you shall predict correctly 0.70.7 + 0.30.3 = 58% of the time. If 100% of the time you predict blue, you’ll get a nickel 70% of the time.

Under conditions of uncertainty, your optimal betting pattern doesn’t resemble a typical sequence of cards. Similarly, I wonder how many betters on horse races realize that you don’t win by betting on the horse you think will win the race, but by betting on horses whose payoffs exceed what you think are the odds. But then, statistical thinkers that sophisticated would probably not bet on horse races

proper scoring rule (another standard math term) is a rule for scoring bets so that you maximize your expected payoff by betting play money that exactly equals the chance of that color flashing. We want a scoring rule so that if the lights actually flash at the frequency blue:30%, green:20%, and red:50%, you can maximize your average payoff only by betting 30 cents on blue, 20 cents on green, and 50 cents on red. A proper scoring rule is one that forces your optimal bet to exactly report your estimate of the probabilities. (This is also sometimes known as a “strictly proper scoring rule”.) As we’ve seen, not all scoring rules have this property; and if you invent a plausible-sounding scoring rule at random, it probably won’t have the property.

One rule with this proper property is to pay a dollar minus the squared error of the bet, rather than the bet itself – if you bet 0.3 on the winning light, your error would be 0.7, your squared error would be 0.49, and a dollar minus your squared error would be fifty-one cents. (Presumably your play money is denominated in the square root of cents, so that the squared error is a monetary sum.) (Readers with calculus may verify that in the simpler case of a light that has only two colors, with p being the bet on the first color and f the frequency of the first color, the expected payoff f*(1-((1-p)^2)) + (1-f)*(1-(p^2)) , with p variable and f constant, has its global maximum when we set p=f .)

We shall not use the squared-error rule. Ordinary statisticians take the squared error of everything in sight, but not Bayesian statisticians.

We add a new requirement: we require, not only a proper scoring rule, but that our proper scoring rule gives us the same answer whether we apply it to rounds individually or combined. This is what Bayesians do instead of taking the squared error of things; we require invariances.

Suppose I press the button twice in a row. There are nine possible outcomes: green-green, green-blue, green-red, blue-green, blue-blue, blue-red, red-green, red-blue, and red-red . Suppose that green wins, and then blue wins. The experimenter would assign the first score based on our probability assignments for p(green-1) and the second score based on p(blue-2|green-1) . We would make two predictions, and get two scores. Our first prediction was the probability we assigned to the color that won on the first round, green. Our second prediction was our probability that blue would win on the second round, given that green won on the first round. Why do we need to write p(blue-2|green-1) instead of just p(blue-2) ? Because you might have a hypothesis about the flashing light that says “blue never follows green”, or “blue always follows green” or “blue follows green with 70% probability”. If this is so, then after seeing green on the first round, you might want to revise your prediction – change your bets – for the second round. You can always revise your predictions right up to the moment the experimenter presses the button, using every scrap of information; but after the light flashes it is too late to change your bet.

(Don’t remember how to read P(A|B) ? See An Intuitive Explanation of Bayesian Reasoning .)

Suppose the actual outcome is green-1 followed by blue-2 . We require this invariance: I must get the same total score, regardless of whether:

• I am scored twice, first on my prediction for p(green-1) , and second on my prediction for p(blue-2|green-1) .
• I am scored once for my joint prediction p(blue-2 & green-1) .

Suppose I assign a 60% probability to green-1 , and then the green light flashes. I must now produce probabilities for the colors on the second round. I assess the possibility blue-2 , and allocate it 25% of my probability mass. Lo and behold, on the second round the light flashes blue. So on the first round my bet on the winning color was 60%, and on the second round my bet on the winning color was 25%. But I might also, at the start of the experiment and after assigning p(green-1) , imagine that the light first flashes green, imagine updating my theories based on that information, and then say what confidence I will give to blue on the next round if the first round is green. That is, I generate the probabilities p(green-1) and p(blue-2|green-1) . By multiplying these two probabilities together we would get the joint probability, p(green-1 & blue-2) = 15%.

A double experiment has nine possible outcomes. If I generate nine probabilities for p(green-1 & green-2), p(green-1 & blue-2), … , p(red-1 & blue-2), p(red-1 & red-2) , the probability mass must sum to no more than 1.0. I am giving predictions for nine mutually exclusive possibilities of a “double experiment”.

We require a scoring rule (and maybe it won’t look like anything an ordinary bookie would ever use) such that my score doesn’t change regardless of whether we consider the double result as two predictions or one prediction. I can treat the sequence of two results as a single experiment, “press the button twice”, and be scored on my prediction for p(blue-2 & green-1) = 15% . Or I can be scored once for my first prediction p(green-1) = 60% , then again on my prediction p(blue-2|green-1) = 25% . We require the same total score in either case, so that it doesn’t matter how we slice up the experiments and the predictions – the total score is always exactly the same. This is our invariance.

We have just required:

`Score(p(green-1 & blue-2)) = Score(p(green-1)) + Score(p(blue-2|green-1))                    `

`p(green-1 & blue-2) = p(green-1) * p(blue-2|green-1)`

The only possible scoring rule is:

`Score(p) = log(p)`

The new scoring rule is that your score is the logarithm of the probability you assigned to the winner.

The base of the logarithm is arbitrary – whether we use the logarithm base 10 or the logarithm base 2, the scoring rule has the desired invariance. But we must choose some actual base. A mathematician would choose base e; an engineer would choose base 10; a computer scientist would choose base 2. If we use base 10, we can convert to “decibels”, as in the Intuitive Explanation ; but sometimes bits are easier to manipulate.

The logarithm scoring rule is proper – it has its expected maximum when we say our exact expectations; it rewards honesty. If we think the blue light has a 60% probability of flashing, and we calculate our expected payoff for different betting schemas, we find that we maximize our expected payoff by telling the experimenter “60%”. (Readers with calculus can verify this.) The scoring rule also gives an invariant total, regardless of whether pressing the button twice counts as “one experiment” or “two experiments”. However, payoffs are now all negative , since we are taking the logarithm of the probability and the probability is between 0 and 1. The logarithm base 10 of 0.1 is -1; the logarithm base 10 of 1% is -2. That’s okay. We accepted that the scoring rule might not look like anything a real bookie would ever use. If you like, you can imagine that the experimenter has a pile of money, and at the end of the experiment he awards you some amount minus your large negative score. (Er, the amount plus your negative score.) Maybe the experimenter has a hundred dollars, and at the end of a hundred rounds you accumulated a score of -48, so you get fifty-two dollars.

A score of -48 in what base? We can eliminate the ambiguity in the score by specifying units. 10 decibels equals a factor of 10; -10 decibels equals a factor of 1/10. Assigning a probability of 0.01 to the actual outcome would score -20 decibels. A probability of 0.03 would score -15 decibels. Sometimes we may use bits: 1 bit is a factor of 2, -1 bit is a factor of 1/2. A probability of 0.25 would score -2 bits; a probability of 0.03 would score around -5 bits.

If you arrive at a probability assessment P for each color, with p(red), p(blue), p(green) , then your expected score is:

```Score = log(p)
Expectation[Score] = Sum[p*log(p)] for all outcomes p.
```

Suppose you had probabilities red:25%, blue:50%, green:25% . Let’s think in base 2 for a moment, to make things simpler. Your expected score is:

```red:    scores -2 bits, flashes 25% of the time
blue:   scores -1 bit,  flashes 50% of the time
green:  scores -2 bits, flashes 25% of the time

expected score: -1.50 bits```

Contrast our Bayesian scoring rule with the ordinary or colloquial way of speaking about degrees of belief, where someone might casually say, “I’m 98% certain that canola oil contains more omega-3 fats than olive oil.” What they really mean by this is that they feel 98% certain – there’s something like a little progress bar that measures the strength of the emotion of certainty, and this progress bar is 98% full. And the emotional progress bar probably wouldn’t be exactly 98% full, if we had some way to measure. The word “98%” is just a colloquial way of saying: “I’m almost but not entirely certain.” It doesn’t mean that you could get the highest expected payoff by betting exactly 98 cents of play money on that outcome. You should only assign a calibrated confidence of 98% if you’re confident enough that you think you could answer a hundred similar questions, of equal difficulty, one after the other, each independent from the others, and be wrong, on average, about twice. We’ll keep track of how often you’re right, over time, and if it turns out that when you say “90% sure” you’re right about 7 times out of 10, then we’ll say you’re poorly calibrated .

Remember Spock from Star Trek? Spock often says something along the lines of, “Captain, if you steer the Enterprise directly into a black hole, our probability of survival is only 2.837%.” Yet nine times out of ten the Enterprise is not destroyed. What kind of tragic fool gives a figure with four significant digits of precision that is wrong by two orders of magnitude?

The people who write this stuff have no idea what scientists mean by “probability”. They suppose that a probability of 99.9% is something like feeling really sure. They suppose that Spock’s statement expresses the challenge of successfully steering the Enterprise through a black hole, like a video game rated five stars for difficulty. What we mean by “probability” is that if you utter the words “two percent probability” on fifty independent occasions, it better not happen more than once.

If you say “98% probable” a thousand times, and you are surprised only five times, we still ding you for poor calibration. You’re allocating too much probability mass to the possibility that you’re wrong. You should say “99.5% probable” to maximize your score. The scoring rule rewards accurate calibration, encouraging neither humility nor arrogance.

At this point it may occur to some readers that there’s an obvious way to achieve perfect calibration – just flip a coin for every yes-or-no question, and assign your answer a confidence of 50%. You say 50% and you’re right half the time. Isn’t that perfect calibration? Yes. But calibration is only one component of our Bayesian score; the other component is discrimination .

Suppose I ask you ten yes-or-no questions. You know absolutely nothing about the subject, so on each question you divide your probability mass fifty-fifty between “Yes” and “No”. Congratulations, you’re perfectly calibrated – answers for which you said “50% probability” were true exactly half the time. This is true regardless of the sequence of correct answers or how many answers were Yes. In ten experiments you said “50%” on twenty occasions – you said “50%” to Yes-1, No-1; Yes-2, No-2; … . On ten of those occasions the answer was correct, the occasions: Yes-1; No-2; No-3; … . And on ten of those occasions the answer was incorrect: No-1; Yes-2; Yes-3; …

Now I give my own answers, putting more effort into it, trying to discriminate whether Yes or No is the correct answer. I assign 90% confidence to each of my favored answers, and my favored answer is wrong twice. I’m more poorly calibrated than you. I said “90%” on ten occasions and I was wrong two times. The next time someone listens to me, they may mentally translate “90%” into 80%, knowing that when I’m 90% sure I’m right about 80% of the time. But the probability you assigned to the final outcome is 1/2 to the tenth power, 0.001 or 1/1024. The probability I assigned to the final outcome is 90% to the eighth power times 10% to the second power, (0.9^8)*(0.1^2), which works out to 0.004 or 0.4%. Your calibration is perfect and mine isn’t, but my better discrimination between right and wrong answers more than makes up for it. My final score is higher – I assigned a greater joint probability to the final outcome of the entire experiment. If I’d been less overconfident and better calibrated, the probability I assigned to the final outcome would have been 0.8^8 * 0.2^2, 0.006.

Is it possible to do even better? Sure. You could have guessed every single answer correctly, and assigned a probability of 99% to each of your answers. Then the probability you assigned to the entire experimental outcome would be 0.99^10 ~ 90%.

Your score would be log(90%), -0.45 decibels or -0.15 bits. We need to take the logarithm so that if I try to maximize my expected score , Sum[p*log(p)] , I have no motive to cheat. Without the logarithm rule, I would maximize my expected score by assigning all my probability mass to the most probable outcome. Also, without the logarithm rule, my total score would be different depending on whether we counted several rounds as several experiments or as one experiment.

We thus dispose of another false stereotype of rationality, that rationality consists of being humble and modest and confessing helplessness in the face of the unknown. That’s just the cheater’s way out, assigning a 50% probability to all yes-or-no questions. Our scoring rule encourages you to do better if you can. If you are ignorant, confess your ignorance; if you are confident, confess your confidence. We penalize you for being confident and wrong, but we also reward you for being confident and right. That is the virtue of a proper scoring rule.

Suppose I flip a coin twenty times. If I believe the coin is fair, the best prediction I can make is to predict an even chance of heads or tails on each flip. If I believe the coin is fair, I assign the same probability to every possible sequence of twenty coinflips. There are roughly a million (1,048,576) possible sequences of twenty coinflips, and I have only 1.0 of probability mass to play with. So I assign to each individual possible sequence a probability of (1/2)^20 – odds of about a million to one; -20 bits or -60 decibels.

I made an experimental prediction and got a score of -60 decibels! Doesn’t this falsify the hypothesis? Intuitively, no. We do not flip a coin twenty times and see a random-looking result, then reel back and say, why, the odds of that are a million to one. But the odds are a million to one against seeing that exact sequence, as I would discover if I naively predicted the exact same outcome for the next sequence of twenty coinflips. It’s okay to have theories that assign tiny probabilities to outcomes, so long as no other theory does better. But if someone used an alternate hypothesis to write down the exact sequence in a sealed envelope in advance, and she assigned a probability of 99%, I would suspect the fairness of the coin. Provided that she only sealed one envelope, and not a million.

That tells us what we ought common-sensically to answer, but it doesn’t say how the common-sense answer arises from the math. To say why the common sense is correct, we need to integrate all that has been said so far into the framework of Bayesian revision of belief. When we’re done, we’ll have a technical understanding of the difference between a verbal understanding and a technical understanding.

Imagine an experiment which produces an integer result between 0 and 99. For example, the experiment might be a particle counter that tells us how many particles have passed through in a minute. Or the experiment might be to visit the supermarket on Wednesday, check the price of a 10-ounce bag of crushed walnuts, and write down the last two digits of the price.

We are testing several different hypotheses that try to predict the experimental result. Each hypothesis produces a probability distribution over all possible results; in this case, the integers between zero and ninety-nine. The possibilities are mutually exclusive, so the probability mass in the distribution must sum to 1.0 (or less); we cannot predict a 90% probability of seeing 42 and also a 90% probability of seeing 43.

Suppose there is a precise hypothesis, which predicts a 90% chance of seeing the result 51. (I.e., the hypothesis is that the supermarket usually prices walnuts with a price of “X dollars and 51 cents”.) The precise theory has staked 90% of its probability mass on the outcome 51. This leaves 10% probability mass remaining to spread over 99 other possible outcomes – all the numbers between 0 and 99 except 51. The theory makes no further specification, so we spread the remaining 10% probability mass evenly over 99 possibilities, assigning a probability of 1/990 to each non-51 result. For ease of writing, we’ll approximate 1/990 as 0.1%.

This probability distribution is analogous to the likelihood or conditional probability of the result given the hypothesis. Let us call it the likelihood distribution for the hypothesis, our chance of seeing each specified outcome if the hypothesis is true. The likelihood distribution for a hypothesis H is a function composed of all the conditional probabilities for p(0|H)=0.001, p(1|H)=0.001, …, p(51|H)=0.9, …, p(99|H)=0.001 . The probability mass contained in the likelihood distribution must sum to 1. It is a general rule that there is no way we can have a 90% chance of seeing 51 and also a 90% chance of seeing 52. Therefore, if we first assume the hypothesis H is true, there is still no way we can have a 90% chance of seeing 51 and also a 90% chance of seeing 52.

The precise theory predicts a 90% probability of seeing 51. Let there be also a vague theory, which predicts “a 90% probability of seeing a number in the 50s”.

Seeing the result 51, we do not say the outcome confirms both theories equally. Both theories made predictions, and both assigned probabilities of 90%, and the result 51 confirms both predictions. But the precise theory has an advantage because it concentrates its probability mass into a sharper point. If the vague theory makes no further specification, we count “a 90% probability of seeing a number in the 50s” as a 9% probability of seeing each number between 50 and 59.

Suppose we started with even odds in favor of the precise theory and the vague theory – odds of 1:1, or 50% probability for either hypothesis being true. After seeing the result 51, what are the posterior odds of the precise theory being true? (If you don’t remember how to work this problem, return to An Intuitive Explanation of Bayesian Reasoning .) The predictions of the two theories are analogous to their likelihood assignments – the conditional probability of seeing the result, given that the theory is true. What is the likelihood ratio between the two theories? The first theory allocated 90% probability mass to the exact outcome. The vague theory allocated 9% probability mass to the exact outcome. The likelihood ratio is 10:1. So if we started with even 1:1 odds, the posterior odds are 10:1 in favor of the precise theory. The differential pressure of the two conditional probabilities pushed our prior confidence of 50% to a posterior confidence of about 91% that the precise theory is correct. Assuming that these are the only hypotheses being tested, that this is the only evidence under consideration, and so on.

Why did the vague theory lose when both theories fit the evidence? The vague theory is timid; it makes a broad prediction, hedges its bets, allows many possibilities that would falsify the precise theory. This is not the virtue of a scientific theory. Philosophers of science tell us that theories should be bold, and subject themselves willingly to falsification if their prediction fails (Popper 1959). Now we see why. The precise theory concentrates its probability mass into a sharper point and thereby leaves itself vulnerable to falsification if the real outcome hits elsewhere; but if the predicted outcome is correct, precision has a tremendous likelihood advantage over vagueness.

The laws of probability theory provide no way to cheat, to make a vague hypothesis such that any result between 50 and 59 counts for as much favorable confirmation as the precise theory receives, for that would require probability mass summing to 900%. There is no way to cheat, providing you record your prediction in advance , so you cannot claim afterward that your theory assigns a probability of 90% to whichever result arrived. Humans are very fond of making their predictions afterward, so the social process of science requires an advance prediction before we say that a result confirms a theory. But how humans may move in harmony with the way of Bayes, and so wield the power, is a separate issue from whether the math works. When we’re doing the math, we just take for granted that likelihood density functions are fixed properties of a hypothesis and the probability mass sums to 1 and you’d never dream of doing it any other way.

You may want to take a moment to visualize that, if we define probability in terms of calibration, Bayes’ Theorem relates the calibrations. Suppose I guess that Theory 1 is 50% likely to be true, and I guess that Theory 2 is 50% likely to be true. Suppose I am well-calibrated; when I utter the words “fifty percent”, the event happens about half the time. And then I see a result R which would happen around nine-tenths of the time given Theory 1, and around nine-hundredths of the time given Theory 2, and I know this is so, and I apply Bayesian reasoning. If I was perfectly calibrated initially (despite the poor discrimination of saying 50/50), I will still be perfectly calibrated (and better discriminated) after I say that my confidence in Theory 1 is now 91%. If I repeated this kind of situation many times, I would be right around ten-elevenths of the time when I said “91%”. If I reason using Bayesian rules, and I start from well-calibrated priors, then my conclusions will also be well-calibrated. This only holds true if we define probability in terms of calibration! If “90% sure” is instead interpreted as, say, the strength of the emotion of surety, there is no reason to expect the posterior emotion to stand in an exact Bayesian relation to the prior emotion.

Let the prior odds be ten to one in favor of the vague theory. Why? Suppose our way of describing hypotheses allows us to either specify a precise number, or to just specify a first-digit; we can say “51”, “63”, “72”, or “in the fifties/sixties/seventies”. Suppose we think that the real answer is about equally liable to be an answer of the first kind or the second. However, given the problem, there are a hundred possible hypotheses of the first kind, and only ten hypotheses of the second kind. So if we think that either class of hypotheses has about an equal prior chance of being correct, we have to spread out the prior probability mass over ten times as many precise theories as vague theories. The precise theory that predicts exactly 51 would thus have one-tenth as much prior probability mass as the vague theory that predicts a number in the fifties. After seeing 51, the odds would go from 1:10 in favor of the vague theory to 1:1, even odds for the precise theory and the vague theory.

If you look at this carefully, it’s exactly what common sense would expect. You start out uncertain of whether a phenomenon is the kind of phenomenon that produces exactly the same result every time, or if it’s the kind of phenomenon that produces a result in the Xties every time. (Maybe the phenomenon is a price range at the supermarket, if you need some reason to suppose that 50..59 is an acceptable range but 49..58 isn’t.) You take a single measurement and the answer is 51. Well, that could be because the phenomenon is exactly 51, or because it’s in the fifties. So the remaining precise theory has the same odds as the remaining vague theory, which requires that the vague theory must have started out ten times as probable as that precise theory, since the precise theory has a sharper fit to the evidence.

If we just see one number, like 51, it doesn’t change the prior probability that the phenomenon itself was “precise” or “vague”. But, in effect, it concentrates all the probability mass of those two classes of hypothesis into a single surviving hypothesis of each class.

Of course, it is a severe error to say that a phenomenon is precise or vague, a case of what Jaynes calls the Mind Projection Fallacy (Jaynes 1996). Precision or vagueness is a property of maps, not territories. Rather we should ask if the price in the supermarket stays constant or shifts about. A hypothesis of the “vague” sort is a good description of a price that shifts about. A precise map will suit a constant territory.

Another example: You flip a coin ten times and see the sequence HHTTH:TTTTH. Maybe you started out thinking there was a 1% chance this coin was fixed. Doesn’t the hypothesis “This coin is fixed to produce HHTTH:TTTTH” assign a thousand times the likelihood mass to the observed outcome, compared to the fair coin hypothesis? Yes. Don’t the posterior odds that the coin is fixed go to 10:1? No. The 1% prior probability that “the coin is fixed” has to cover every possible kind of fixed coin – a coin fixed to produce HHTTH:TTTTH, a coin fixed to produce TTHHT:HHHHT, etc. The prior probability the coin is fixed to produce HHTTH:TTTTH is not 1%, but a thousandth of one percent. Afterward, the posterior probability the coin is fixed to produce HHTTH:TTTTH is one percent. Which is to say: You thought the coin was probably fair but had a one percent chance of being fixed to some random sequence; you flipped the coin; the coin produced a random-looking sequence; and that doesn’t tell you anything about whether the coin is fair or fixed. It does tell you, if the coin is fixed, which sequence it is fixed to.

This parable helps illustrate why Bayesians must think about prior probabilities. There is a branch of statistics, sometimes called “orthodox” or “classical” statistics, which insists on paying attention only to likelihoods. But if you only pay attention to likelihoods, then eventually some fixed-coin hypothesis will always defeat the fair coin hypothesis, a phenomenon known as “overfitting” the theory to the data. After thirty flips, the likelihood is a billion times as great for the fixed-coin hypothesis with that sequence, as for the fair coin hypothesis. Only if the fixed-coin hypothesis (or rather, that specific fixed-coin hypothesis) is a billion times less probable a priori , can the fixed-coin hypothesis possibly lose to the fair coin hypothesis.

If you shake the coin to reset it, and start flipping the coin again , and the coin produces HHTTH:TTTTH again , that is a different matter. That does raise the posterior odds of the fixed-coin hypothesis to 10:1, even if the starting probability was only 1%.

Similarly, if we perform two successive measurements of the particle counter (or the supermarket price on Wednesdays), and both measurements return 51, the precise theory wins by odds of 10 to 1.

So the precise theory wins, but the vague theory would still score better than no theory at all. Consider a third theory, the hypothesis of zero knowledge or maximum-entropy distribution , which makes equally probable any result between 0 and 99. Suppose we see the result 51. The vague theory produced a better prediction than the maximum-entropy distribution – assigned a greater likelihood to the outcome we observed. The vague theory is, literally, better than nothing. Suppose we started with odds of 1:20 in favor of the hypothesis of complete ignorance. (Why odds of 1:20? There is only one hypothesis of complete ignorance, and moreover, it’s a particularly simple and intuitive kind of hypothesis. Occam’s Razor.) After seeing the result of 51, predicted at 9% by the vague theory versus 1% by complete ignorance, the posterior odds go to 10:20 or 1:2. If we then see another result of 51, the posterior odds go to 10:2 or 83% probability for the vague theory, assuming there is no more precise theory under consideration.

Yet the timidity of the vague theory – its unwillingness to produce an exact prediction and accept falsification on any other result – renders it vulnerable to the bold, precise theory. (Providing, of course, that the bold theory correctly guesses the outcome!) Suppose the prior odds were 1:10:200 for the precise, vague, and ignorant theories – prior probabilities of 0.5%, 4.7%, and 94.8% for the precise, vague and ignorant theories. This figure reflects our prior probability distribution over classes of hypotheses, with the probability mass distributed over entire classes as follows: 50% that the phenomenon shifts across all digits, 25% that the phenomenon shifts around within some decimal bracket, and 25% that the phenomenon repeats the same number each time. 1 hypothesis of complete ignorance, 10 possible hypotheses for a decimal bracket, 100 possible hypotheses for a repeating number. Thus, prior odds of 1:10:200 for the precise hypothesis 51, the vague hypothesis “fifties”, and the hypothesis of complete ignorance.

After seeing a result of 51, with assigned probability of 90%, 9%, and 1%, the posterior odds go to 90:90:200 = 9:9:20. After seeing an additional result of 51, the posterior odds go to 810:81:20, or 89%, 9%, and 2%. The precise theory is now favored over the vague theory, which in turn is favored over the ignorant theory.

Now consider a stupid theory, which predicts a 90% probability of seeing a result between 0 and 9. The stupid theory assigns a probability of 0.1% to the actual outcome, 51. If the odds were initially 1:10:200:10 for the precise, vague, ignorant, and stupid theories, the posterior odds after seeing 51 once would be 90:90:200:1. The stupid theory has been falsified (posterior probability of 0.2%).

It is possible to have a model so bad that it is worse than nothing, if the model concentrates its probability mass away from the actual outcome, makes confident predictions of wrong answers. Such a hypothesis is so poor that it loses against the hypothesis of complete ignorance. Ignorance is better than anti-knowledge.Side note 1: In the field of Artificial Intelligence, there is a sometime fad that praises the glory of randomness. Occasionally an AI researcher discovers that if they add noise to one of their algorithms, the algorithm works better. This result is reported with great enthusiasm, followed by much fulsome praise of the creative powers of chaos, unpredictability, spontaneity, ignorance of what your own AI is doing, et cetera. (See Imagination Engines Inc. for an example; according to their sales literature they sell wounded and dying neural nets.) But how sad is an algorithm if you can increase its performance by injecting entropy into intermediate processing stages? The algorithm must be so deranged that some of its work goes into concentrating probability mass away from good solutions. If injecting randomness results in a reliable improvement, then some aspect of the algorithm must do reliably worse than random. Only in AI would people devise algorithms literally dumber than a bag of bricks , boost the results slightly back toward ignorance, and then argue for the healing power of noise.Side note 2: Robert Pirsig once said: “The world’s stupidest man may say the Sun is shining, but that doesn’t make it dark out.” (Pirsig 1974.) It is a classical logical fallacy to say, “Hitler believed in the Pythagorean Theorem. You don’t want to agree with Hitler, do you?” Consider that for someone to be reliably wrong on yes-or-no questions – say, to be wrong 90% of the time – that person would need to do all the hard work of discriminating truth from falsehood, just to be wrong so reliably. If someone is wrong on yes-or-no questions 99% of the time, we can get 99% accuracy just by inverting the responses. Anyone that stupid would be smarter than I am.

Suppose that in our experiment we see the results 52, 51, 58. The precise theory gives this conjunctive event a probability of a thousand to one times 90% times a thousand to one, while the vaguer theory gives this conjunctive event a probability of 9% cubed, which works out to… oh… um… let’s see… a million to one given the precise theory, versus a thousand to one given the vague theory. Or thereabouts; we are counting rough powers of ten. Versus a million to one given the zero-knowledge distribution that assigns an equal probability to all outcomes. Versus a billion to one given a model worse than nothing, the stupid hypothesis, which claims a 90% probability of seeing a number less than 10. Using these approximate numbers, the vague theory racks up a score of -30 decibels (a probability of 1/1000 for the whole experimental outcome), versus scores of -60 for the precise theory, -60 for the ignorant theory, and -90 for the stupid theory. It is not always true that the highest score wins, because we need to take into account our prior odds of 1:10:200:10, confidences of -23, -13, 0, and -13 decibels. The vague theory still comes in with the highest total score at -43 decibels. (If we ignored our prior probabilities, each new experiment would override the accumulated results of all the previous experiments; we could not accumulate knowledge. Furthermore, the fixed-coin hypothesis would always win.)

As always, we should not be alarmed that even the best theory still has a low score – recall the parable of the fair coin. Theories are approximations. In principle we might be able to predict the exact sequence of coinflips. But it would take better measurement and more computing power than we’re willing to expend. Maybe we could achieve 60/40 prediction of coinflips, with a good enough model…? We go with the best approximation we have, and try to achieve good calibration even if the discrimination isn’t perfect.

We’ve conducted our analysis so far under the rules of Bayesian probability theory, in which there’s no way to have more than 100% probability mass, and hence no way to cheat so that any outcome can count as “confirmation” of your theory. Under Bayesian law, play money may not be counterfeited; you only have so much clay. If you allocate more probability mass in one place, you have to take it from somewhere else; a coin cannot have a 90% chance of turning up heads and a 90% chance of turning up tails.

Unfortunately, human beings are not Bayesians. Human beings bizarrely attempt to defend hypotheses, making a deliberate effort to prove them or prevent disproof. This behavior has no analogue in the laws of probability theory or decision theory. In formal probability theory the hypothesis is , and the evidence is , and either the hypothesis is confirmed or it is not. In formal decision theory, an agent may make an effort to investigate some issue of which the agent is currently uncertain, not knowing whether the evidence shall go one way or the other. In neither case does one ever deliberately try to prove an idea, or try to avoid disproving it. One may test ideas of which one is genuinely uncertain, but not have a “preferred” outcome of the investigation. One may not try to prove hypotheses, nor prevent their proof. I cannot properly convey just how ridiculous the notion would be, to a true Bayesian; there are not even words in Bayes-language to describe the mistake…

One classic method for preventing a theory from disproof is arguing post facto that any observation presented proves the theory. Friedrich Spee von Langenfeld, a priest who heard the confessions of condemned witches, wrote in 1631 the Cautio Criminalis (‘prudence in criminal cases’) in which he bitingly described the decision tree for condemning accused witches. If the witch had led an evil and improper life, she was guilty; if she had led a good and proper life, this too was a proof, for witches dissemble and try to appear especially virtuous. After the woman was put in prison: if she was afraid, this proved her guilt; if she was not afraid, this proved her guilt, for witches characteristically pretend innocence and wear a bold front. Or on hearing of a denunciation of witchcraft against her, she might seek flight or remain; if she ran, that proved her guilt; if she remained, the devil had detained her so she could not get away. (Spee 1631.) Spee acted as confessor to many witches; he was thus in a position to observe every branch of the accusation tree, that no matter what the accused witch said or did, it was held a proof against her. In any individual case, you would only hear one branch of the dilemma.

It is for this reason that scientists write down their predictions in advance.

If you’ve read the Intuitive Explanation , you should recall the result I nicknamed the “Law of Conservation of Probability”, that for every expectation of evidence there is an equal and opposite expectation of counterevidence. If A is evidence in favor of B, not-A must be evidence in favor of not-B. The strengths of the evidences may not be equal; rare but strong evidence in one direction may be balanced by common but weak evidence in the other direction. But it is not possible for both A and not-A to be evidence in favor of B. That is, it’s not possible under the laws of probability theory. Humans often seem to want to have their cake and eat it too. Whichever result we witness is the one that proves our theory. As Spee put it, “The investigating committee would feel disgraced if it acquitted a woman; once arrested and in chains, she has to be guilty, by fair means or foul.”

The way human psychology seems to work is that first we see something happen, and then we try to argue that it matches whatever hypothesis we had in mind beforehand. Rather than conserved probability mass, to distribute over advance predictions , we have a feeling of compatibility – the degree to which the explanation and the event seem to ‘fit’. ‘Fit’ is not conserved. There is no equivalent of the rule that probability mass must sum to 1. A psychoanalyst may explain any possible behavior of a patient by constructing an appropriate structure of ‘rationalizations’ and ‘defenses’; it fits, therefore it must be true.

Now consider the fable told at the start of this essay – the students seeing a radiator, and a metal plate next to the radiator. The students would never predict in advance that the side of the plate near the radiator would be cooler. Yet, seeing the fact, they managed to make their explanations ‘fit’. They lost their precious chance at bewilderment, to realize that their models did not predict the phenomenon they observed. They sacrificed their ability to be more confused by fiction than by truth. And they did not realize “heat induction, blah blah, therefore the near side is cooler” is a vague and verbal prediction, spread across an enormously wide range of possible values for specific measured temperatures. Applying equations of diffusion and equilibrium would give a sharp prediction for possible joint values. It might not specify the first values you measured, but when you knew a few values you could generate a sharp prediction for the rest. The score for the entire experimental outcome would be far better than any less precise alternative, especially a vague and verbal prediction.

You now have a technical explanation of the difference between a verbal explanation and a technical explanation. It is a technical explanation because it enables you to calculate exactly how technical an explanation is. Vague hypotheses may be so vague that only a superhuman intelligence could calculate exactly how vague. Perhaps a sufficiently huge intelligence could extrapolate every possible experimental result, and extrapolate every possible verdict of the vague guesser for how well the vague hypothesis “fit”, and then renormalize the “fit” distribution into a likelihood distribution that summed to 1. But in principle one can still calculate exactly how vague is a vague hypothesis. The calculation is just not computationally tractable, the way that calculating airplane trajectories via quantum mechanics is not computationally tractable.

I hold that everyone needs to learn at least one technical subject. Physics; computer science; evolutionary biology; or Bayesian probability theory, but something . Someone with no technical subjects under their belt has no referent for what it means to “explain” something. They may think “All is Fire” is an explanation. Therefore do I advocate that Bayesian probability theory should be taught in high school. Bayesian probability theory is the sole piece of math I know that is accessible at the high school level, and that permits a technical understanding of a subject matter – the dynamics of belief – that is an everyday real-world domain and has emotionally meaningful consequences. Studying Bayesian probability would give students a referent for what it means to “explain” something.

Too many academics think that being “technical” means speaking in dry polysyllabisms. Here’s a “technical” explanation of technical explanation:The equations of probability theory favor hypotheses that strongly predict the exact observed data. Strong models boldly concentrate their probability density into precise outcomes, making them falsifiable if the data hits elsewhere, and giving them tremendous likelihood advantages over models less bold, less precise. Verbal explanation runs on psychological evaluation of unconserved post facto compatibility instead of conserved ante facto probability density. And verbal explanation does not paint sharply detailed pictures, implying a smooth likelihood distribution in the vicinity of the data.

Is this satisfactory? No. Hear the impressive and weighty sentences, resounding with the dull thud of expertise. See the hapless students, writing those sentences on a sheet of paper. Even after the listeners hear the ritual words, they can perform no calculations. You know the math, so the words are meaningful. You can perform the calculations after hearing the impressive words, just as you could have done before. But what of one who did not see any calculations performed? What new skills have they gained from that “technical” lecture, save the ability to recite fascinating words?

“Bayesian” sure is a fascinating word, isn’t it? Let’s get it out of our systems: Bayes Bayes Bayes Bayes Bayes Bayes Bayes Bayes Bayes…

The sacred syllable is meaningless, except insofar as it tells someone to apply math. Therefore the one who hears must already know the math.

Conversely, if you know the math, you can be as silly as you like, and still technical.

We thus dispose of yet another stereotype of rationality, that rationality consists of sere formality and humorless solemnity. What has that to do with the problem of distinguishing truth from falsehood? What has that to do with attaining the map that reflects the territory? A scientist worthy of a lab coat should be able to make original discoveries while wearing a clown suit, or give a lecture in a high squeaky voice from inhaling helium. It is written nowhere in the math of probability theory that one may have no fun. The blade that cuts through to the correct answer has no dignity or silliness of itself, though it may fit the hand of a silly wielder.

Our physics uses the same theory to describe an airplane, and collisions in a particle accelerator – particles and airplanes both obey special relativity and general relativity and quantum electrodynamics and quantum chromodynamics. But we use entirely different models to understand the aerodynamics of a 747 and a collision between gold nuclei. A computer modeling the aerodynamics of the 747 may not contain a single token representing an atom, even though no one denies that the 747 is made of atoms.

useful model isn’t just something you know, as you know that the airplane is made of atoms. A useful model is knowledge you can compute in reasonable time to predict real-world events you know how to observe. Physicists use different models to predict airplanes and particle collisions, not because the two events take place in different universes with different laws of physics, but because it would be too expensive to compute the airplane particle by particle.

As the saying goes: “The map is not the territory, but you can’t fold up the territory and put it in your glove compartment.” Sometimes you need a smaller map, to fit in a more cramped glove compartment. It doesn’t change the territory. The precision or vagueness of the map isn’t a fact about the territory, it’s a fact about the map.

Maybe someone will find that, using a model that violates conservation of momentum just a little, you can compute the aerodynamics of the 747 much more cheaply than if you insist that momentum is exactly conserved. So if you’ve got two computers competing to produce the best prediction, it might be that the best prediction comes from the model that violates conservation of momentum. This doesn’t mean that the 747 violates conservation of momentum in real life. Neither model uses individual atoms, but that doesn’t imply the 747 is not made of atoms. You would prove the 747 is made of atoms with experimental data that the aerodynamic models couldn’t handle; for example, you would train a scanning tunneling microscope on a section of wing and look at the atoms. Similarly, you could use a finer measuring instrument to discriminate between a 747 that really disobeyed conservation of momentum like the cheap approximation predicted, versus a 747 that obeyed conservation of momentum like underlying physics predicted. The winning theory is the one that best predicts all the experimental predictions together. Our Bayesian scoring rule gives us a way to combine the results of all our experiments, even experiments that use different methods.

Furthermore, the atomic theory allows, embraces, and in some sense mandates the aerodynamic model. By thinking abstractly about the assumptions of atomic theory, we realize that the aerodynamic model ought to be a good (and much cheaper) approximation of the atomic theory, and so the atomic theory supports the aerodynamic model, rather than competing with it. A successful theory can embrace many models for different domains, so long as the models are acknowledged as approximations, and in each case the model is compatible with (or ideally mandated by) the underlying theory.

Our fundamental physics – quantum mechanics, the standard family of particles, and relativity – is a theory that embraces an enormous family of models for macroscopic physical phenomena. There is the physics of liquids, and solids, and gases; yet this does not mean that there are fundamental things in the world that have the intrinsic property of liquidity.”Apparently there is colour, apparently sweetness, apparently bitterness, actually there are only atoms and the void.”– Democritus, 420 BC (from Robinson and Groves 1998).

In arguing that a “technical” theory should be defined as a theory that sharply concentrates probability into specific advance predictions, I am setting an extremely high standard of strictness. We have seen that a vague theory can be better than nothing. A vague theory can win out over the hypothesis of ignorance, if there are no precise theories to compete against it.

There is an enormous family of models belonging to the central underlying theory of life and biology; the underlying theory that is sometimes called neo-Darwinism, natural selection, or evolution. Some models in evolutionary theory are quantitative. The way in which DNA encodes proteins is redundant; two different DNA sequences can code for exactly the same protein. There are 4 DNA bases {ATCG} and 64 possible combinations of three DNA bases. But those 64 possible codons describe only 20 amino acids plus a stop code. Genetic drift ought therefore to produce non-functional changes in species genomes, through mutations which by chance become fixed in the gene pool. The accumulation rate of non-functional differences between the genomes of two species with a common ancestor, depends on such parameters as the number of generations elapsed and the intensity of selection at that genetic locus. That’s an example of a member of the family of evolutionary models that produces quantitative predictions. There are also disequilibrium allele frequencies under selection, stable equilibria for game-theoretical strategies, sex ratios, et cetera.

This all comes under the heading of “fascinating words”. Unfortunately, there are certain religious factions that spread gross disinformation about evolutionary theory. So I emphasize that many models within evolutionary theory make quantitative predictions that are experimentally confirmed, and that such models are far more than sufficient to demonstrate that, e.g., humans and chimpanzees are related by a common ancestor. If you’ve been victimized by creationist disinformation – that is, if you’ve heard any suggestion that evolutionary theory is controversial or untestable or “just a theory” or non-rigorous or non-technical or in any wise not confirmed by an unimaginably huge mound of experimental evidence – I recommend reading the Talk.Origins FAQ and studying evolutionary biology with math.

But imagine going back in time to the nineteenth century, when the theory of natural selection had only just been discovered by Charles Darwin and Alfred Russel Wallace. Imagine evolutionism just after its birth, when the theory had nothing remotely like the modern-day body of quantitative models and great heaping mountains of experimental evidence. There was no way of knowing that humans and chimpanzees would be discovered to have 95% shared genetic material. No one knew that DNA existed. Yet even so, scientists flocked to the new theory of natural selection. And later it turned out that there was a precisely copied genetic material with the potential to mutate, that humans and chimps were provably related, etc.

So the very strict, very high standard that I proposed for a “technical” theory is too strict. Historically, it has been possible to successfully discriminate true theories from false theories, based on predictions of the sort I called “vague”. Vague predictions of, say, 80% confidence, can build up a huge advantage over alternate hypotheses, given enough experiments. Perhaps a theory of this kind, producing predictions that are not precisely detailed but are nonetheless correct, could be called “semitechnical”?

But surely technical theories are more reliable than semitechnical theories? Surely technical theories should take precedence, command greater respect? Surely physics, which produces exceedingly exact predictions, is in some sense better confirmed than evolutionary theory? Not implying that evolutionary theory is wrong, of course; but however vast the mountains of evidence favoring evolution, does not physics go one better through vast mountains of precise experimental confirmation? Observations of neutron stars confirm the predictions of General Relativity to within one part in a hundred trillion (10^14). What does evolutionary theory have to match that?

Someone – I think either Roger Penrose or Richard Dawkins – said once that measured by the simplicity of the theory and the amount of complexity it explained, Darwin had the single greatest idea in the history of time.

Once there was a conflict between 19th-century physics and 19th-century evolutionism. According to the best physical models then in use, the Sun could not have been burning very long. 3000 years on chemical energy, or 40 million years on gravitational energy. There was no energy source known to 19th-century physics that would permit longer burning. 19th-century physics was not quite as powerful as modern physics – it did not have predictions accurate to within one part in 10^14. But 19th-century physics still had the mathematical character of modern physics; a discipline whose models produced detailed, precise, quantitative predictions. 19th-century evolutionary theory was wholly semitechnical, without a scrap of quantitative modeling. Not even Mendel’s experiments with peas were then known. And yet it did seem likely that evolution would require longer than a paltry 40 million years in which to operate – hundreds of millions, even billions of years. The antiquity of the Earth was a vague and semitechnical prediction, of a vague and semitechnical theory. In contrast, the 19th-century physicists had a precise and quantitative model, which through formal calculation produced the precise and quantitative dictum that the Sun simply could not have burned that long.”The limitations of geological periods, imposed by physical science, cannot, of course, disprove the hypothesis of transmutation of species; but it does seem sufficient to disprove the doctrine that transmutation has taken place through ‘descent with modification by natural selection.'”– Lord Kelvin, distinguished 19th-century physicist (from Zapato 1998).

History records who won.

The moral? If you can give 80% confident advance predictions on yes-or-no questions, it may be a “vague” theory, it may be wrong one time out of five, but you can still build up a heck of a huge scoring lead over the hypothesis of ignorance. Enough to confirm a theory, if there are no better competitors. Reality is consistent; every correct theory about the universe is compatible with every other correct theory. Imperfect maps can conflict, but there is only one territory. 19th-century evolutionism might have been a semitechnical discipline, but it was still correct (as we now know) and by far the best explanation (even in that day). Any conflict between evolutionism and another well-confirmed theory had to reflect some kind of anomaly, a mistake in the assertion that the two theories were incompatible. 19th-century physics couldn’t model the dynamics of the Sun – they didn’t know about nuclear reactions. They could not show that their understanding of the Sun was correct in technical detail , nor calculate from a confirmed model of the Sun to determine how long the Sun had existed. So in retrospect, we can say something like: “There was room for the possibility that 19th-century physics just didn’t understand the Sun.”

But that is hindsight. The real lesson is that, even though 19th-century physics was both precise and quantitative, it didn’t automatically dominate the semitechnical theory of 19th-century evolutionism. The theories were both well-supported. They were both correct in the domains over which they were generalized. The apparent conflict between them was an anomaly, and the anomaly turned out to stem from the incompleteness and incorrect application of 19th-century physics, not the incompleteness and incorrect application of 19th-century evolutionism. But it would be futile to compare the mountain of evidence supporting the one theory, versus the mountain of evidence supporting the other. Even in that day, both mountains were too large to suppose that either theory was simply mistaken. Mountains of evidence that large cannot be set to compete, as if one falsifies the other. You must be applying one theory incorrectly, or applying a model outside the domain it predicts well.

So you shouldn’t necessarily sneer at a theory just because it’s semitechnical. Semitechnical theories can build up high enough scores, compared to every available alternative, that you know the theory is at least approximately correct. Someday the semitechnical theory may be replaced or even falsified by a more precise competitor, but that’s true even of technical theories. Think of how Einstein’s General Relativity devoured Newton’s theory of gravitation.

But the correctness of a semitechnical theory – a theory that currently has no precise, computationally tractable models testable by feasible experiments – can be a lot less cut-and-dried than the correctness of a technical theory. It takes skill, patience, and examination to distinguish good semitechnical theories from theories that are just plain confused. This is not something that humans do well by instinct, which is why we have Science.

People eagerly jump the gun and seize on any available reason to reject a disliked theory. That is why I gave the example of 19th-century evolutionism, to show why one should not be too quick to reject a “non-technical” theory out of hand. By the moral customs of science, 19th-century evolutionism was guilty of more than one sin. 19th-century evolutionism made no quantitative predictions. It was not readily subject to falsification. It was largely an explanation of what had already been seen. It lacked an underlying mechanism, as no one then knew about DNA. It even contradicted the 19th-century laws of physics. Yet natural selection was such an amazingly good post-facto explanation that people flocked to it, and they turned out to be right. Science, as a human endeavor, requires advance prediction. Probability theory, as math, does not distinguish between post-facto and advance prediction, because probability theory assumes that probability distributions are fixed properties of a hypothesis.

The rule about advance prediction is a rule of the social process of science – a moral custom and not a theorem. The moral custom exists to prevent human beings from making human mistakes that are hard to even describe in the language of probability theory, like tinkering after the fact with what you claim your hypothesis predicts. People concluded that 19th-century evolutionism was an excellent explanation, even if it was post-facto. That reasoning was correct as probability theory , which is why it worked despite all scientific sins. Probability theory is math. The social process of science is a set of legal conventions to keep people from cheating on the math.

Yet it is also true that, compared to a modern-day evolutionary theorist, evolutionary theorists of the late 19th and early 20th century often went sadly astray. Darwin, who was bright enough to invent the theory, got an amazing amount right. But Darwin’s successors, who were only bright enough to accept the theory, misunderstood evolution frequently and seriously. The usual process of science was then required to correct their mistakes. It is incredible how few errors of reasoning Darwin made in The Origin of Species and The Descent of Man , compared to they who followed.

That is also a hazard of a semitechnical theory. Even after the flash of genius insight is confirmed, merely average scientists may fail to apply the insights properly in the absence of formal models. As late as the 1960s biologists spoke of evolution working “for the good of the species”, or suggested that individuals would restrain their reproduction to prevent species overpopulation of a habitat. The best evolutionary theorists knew better, but average theorists did not. (Williams 1966.)

So it is far better to have a technical theory than a semitechnical theory. Unfortunately, Nature is not always so kind as to render Herself describable by neat, formal, computationally tractable models, nor does She always provide Her students with measuring instruments that can directly probe Her phenomena. Sometimes it is only a matter of time. 19th-century evolutionism was semitechnical, but later came the math of population genetics, and eventually DNA sequencing. But Nature will not always give you a phenomenon that you can describe with technical models fifteen seconds after you have the basic insight.

Yet the cutting edge of science, the controversy , is most often about a semitechnical theory, or nonsense posing as a semitechnical theory. By the time a theory achieves technical status, it is usually no longer controversial (among scientists). So the question of how to distinguish good semitechnical theories from nonsense is very important to scientists, and it is not as easy as dismissing out of hand any theory that is not technical. To the end of distinguishing truth from falsehood exists the entire discipline of rationality. The art is not reducible to a checklist, or at least, no checklist that an average scientist can apply reliably after an hour of training. If it was that simple we wouldn’t need science.

Why do you care about scientific controversies?

No, seriously, why do you care about scientific controversies ?

The media thinks that only the cutting edge of science, the very latest controversies, are worth reporting on. How often do you see headlines like “General Relativity still governing planetary orbits” or “Phlogiston theory remains false”? By the time anything is solid science, it is no longer a breaking headline. “Newsworthy” science is based on the thinnest of evidence and wrong half the time. If it were not on the uttermost fringes of the scientific frontier, it would not be news. Scientific controversies are problems so difficult that even people who’ve spent years mastering the field can still fool themselves. That’s what makes the problem controversial and attracts all the media attention. So the reporters show up, and hear the scientists speak fascinating words. The reporters are told that “particles” are “waves”, but there is no understanding of math for the words to invoke. What the physicist means by “wave” is not what the reporters hear, even if the physicist’s math applies also to the structure of water as it crashes on the shore.

And then the reporters write stories, which are not worth the lives of the dead trees on which they are printed.

But what does it matter to you? Why should you pay attention to scientific controversies ? Why graze upon such sparse and rotten feed as the media offers, when there are so many solid meals to be found in textbooks? Nothing you’ll read as breaking news will ever hold a candle to the sheer beauty of settled science. Textbook science has carefully phrased explanations for new students, math derived step by step, plenty of experiments as illustration, and test problems.

And textbook science is beautiful! Textbook science is comprehensible , unlike mere fascinating words that can never be truly beautiful. Elementary science textbooks describe simple theories, and simplicity is the core of scientific beauty. Fascinating words have no power, nor yet any meaning, without the math. The fascinating words are not knowledge but the illusion of knowledge, which is why it brings so little satisfaction to know that “gravity results from the curvature of spacetime”. Science is not in the fascinating words, though it’s all the media will ever give you.

Is there ever justification for following a scientific controversy, while there remains any basic science you do not yet know? Yes. You could be an expert in that field, in which case that scientific controversy is your proper meat. Or the scientific controversy might be something you need to know now , because it affects your life. Maybe it’s the 19th century, and you’re gazing lustfully at a member of the appropriate sex wearing a 19th-century bathing suit, and you need to know whether your sexual desire comes from a psychology constructed by natural selection, or is a temptation placed in you by the Devil to lure you into hellfire.

It is not wholly impossible that we shall happen upon a scientific controversy that affects us, and find that we have a burning and urgent need for the correct answer. I shall therefore discuss some of the warning signs that historically distinguished vague hypotheses that later turned out to be unscientific gibberish, from vague hypotheses that later graduated to confirmed theories. Just remember the historical lesson of 19th-century evolutionism, and resist the temptation to fail every theory that misses a single item on your checklist. It is not my intention to give people another excuse to dismiss good science that discomforts them. If you apply stricter criteria to theories you dislike than theories you like (or vice versa!), then every additional nit you learn how to pick, every new logical flaw you learn how to detect, makes you that much stupider. Intelligence, to be useful, must be used for something other than defeating itself.

One of the classic signs of a poor hypothesis is that it must expend great effort in avoiding falsification – elaborating reasons why the hypothesis is compatible with the phenomenon, even though the phenomenon didn’t behave as expected.

Sagan (1995) gives the example of someone who claims that a dragon lives in their garage. Fascinated by this controversial question, we ignore all the textbooks providing total solutions to ancient mysteries on which alchemists spent their lives in vain… but never mind. We show up at the garage, look inside, and see: Nothing.

Ah, says the claimant, that’s because it’s an invisible dragon.

Now as Sagan says, this is an odd claim, but it doesn’t mean we can never know if the dragon is there. Maybe we hear heavy breathing, and discover that carbon dioxide and heat appears in the garage’s air. Clawed footprints stamp through the dust. Occasionally a great gout of fire bursts out from no visible source. If so, we conclude that the garage contains an invisible dragon, and the reporters depart, satisfied that the controversy is over. Once something is a fact, it’s no longer exciting; it’s no fun believing in things that any old fool can see are true. If the dragon were really there, it would be no more fun to believe in the dragon than to believe in zebras.

But now suppose instead that we bring up our measuring instruments to see if carbon dioxide is accumulating in the garage’s air, and the claimant at once says: “No, no, it’s an invisible non-breathing dragon!” Okay. We begin to examine the dirt, and the claimant says: “No, it’s a flying invisible non-breathing dragon, so it won’t leave footprints.” We start to unload audio equipment, and the claimant says it’s an inaudible dragon. We bring in a bag of flour, to throw into the air to outline the dragon’s form, and the claimant quickly says that this dragon is permeable to flour.

Carl Sagan originally drew the lesson that poor hypotheses need to do fast footwork to avoid falsification – to maintain an appearance of “fit”.

I would point out that the claimant obviously has a good model of the situation somewhere in his head, because he can predict, in advance, exactly which excuses he’s going to need. When we bring up our measuring instruments, he knows that he’ll have to excuse the lack of any carbon dioxide in the air. When we bring in a bag of flour, the claimant knows that he’ll need to excuse the lack of any dragon-shaped form in the floury air.

To a Bayesian, a hypothesis isn’t something you assert in a loud, emphatic voice. A hypothesis is something that controls your anticipations , the probabilities you assign to future experiences. That’s what a probability is , to a Bayesian – that’s what you score, that’s what you calibrate. So while our claimant may say loudly, emphatically, and honestly that he believes there’s an invisible dragon in the garage, he does not anticipate there’s an invisible dragon in the garage – he anticipates exactly the same experience as the skeptic.

When I judge the predictions of a hypothesis, I ask which experiences I would anticipate, not which facts I would believe.

The flip side:

I recently argued with a friend of mine over a question of evolutionary theory. My friend alleged that the clustering of changes in the fossil record (apparently, there are periods of comparative stasis followed by comparatively sharp changes; itself a controversial observation known as “punctuated equilibrium”) showed that there was something wrong with our understanding of speciation. My friend thought that there was some unknown force at work, not supernatural, but some natural consideration that standard evolutionary theory didn’t take into account. Since my friend didn’t give a specific competing hypothesis that produced better predictions, his thesis had to be that the standard evolutionary model was stupid with respect to the data – that the standard model made a specific prediction that was wrong; that the model did worse than complete ignorance or some other default competitor.

At first I fell into the trap; I accepted the implicit assumption that the standard model predicted smoothness, and based my argument on my recollection that the fossil record changes weren’t as sharp as he claimed. He challenged me to produce an evolutionary intermediate between Homo erectus and Homo sapiens ; I googled and found Homo heidelbergensis . He congratulated me and acknowledged that I had scored a major point, but still insisted that the changes were too sharp, and not steady enough. I started to explain why I thought a pattern of uneven change could arise from the standard model: environmental selection pressures might not be constant… “Aha!” my friend said, “you’re making your excuses in advance.”

But suppose that the fossil record instead showed a smooth and gradual set of changes. Might my friend have argued that the standard model of evolution as a chaotic and noisy process could not account for such smoothness? If it is a scientific sin to claim post facto that our beloved hypothesis predicts the data, should it not be equally a sin to claim post facto that the competing hypothesis is stupid on the data?

If a hypothesis has a purely technical model, there is no trouble; we can compute the prediction of the model formally, without informal variables to provide a handle for post facto meddling. But what of semitechnical theories? Obviously a semitechnical theory must produce some good advance predictions about something , or else why bother? But after the theory is semi-confirmed, can the detractors claim that the data show a problem with the semitechnical theory, when the “problem” is constructed post facto? At the least the detractors must be very specific about what data a confirmed model predicts stupidly, and why the confirmed model must make (post facto) that stupid prediction. How sharp a change is “too sharp”, quantitatively, for the standard model of evolution to permit? Exactly how much steadiness do you think the standard model of evolution predicts? How do you know? Is it too late to say that, after you’ve seen the data?

When my friend accused me of making excuses, I paused and asked myself which excuses I anticipated needing to make. I decided that my current grasp of evolutionary theory didn’t say anything about whether the rate of evolutionary change should be intermittent and jagged, or smooth and gradual. If I hadn’t seen the graph in advance, I could not have predicted it. (Unfortunately, I rendered even that verdict after seeing the data…) Maybe there are models in the evolutionary family that would make advance predictions of steadiness or variability, but if so, I don’t know about them. More to the point, my friend didn’t know either.

It is not always wise, to ask the opponents of a theory what their competitors predict. Get the theory’s predictions from the theory’s best advocates. Just make sure to write down their predictions in advance. Yes, sometimes a theory’s advocates try to make the theory “fit” evidence that plainly doesn’t fit. But if you find yourself wondering what a theory predicts, ask first among the theory’s advocates, and afterward ask the detractors to cross-examine.

Furthermore: Models may include noise. If we hypothesize that the data are trending slowly and steadily upward, but our measuring instrument has an error of 5%, then it does no good to point to a data point that dips below the previous data point, and shout triumphantly, “See! It went down! Down down down! And don’t tell me why your theory fits the dip; you’re just making excuses!” Formal, technical models often incorporate explicit error terms. The error term spreads out the likelihood density, decreases the model’s precision and reduces the theory’s score, but the Bayesian scoring rule still governs. A technical model can allow mistakes, and make mistakes, and still do better than ignorance. In our supermarket example, even the precise hypothesis of 51 still bets only 90% of its probability mass on 51; the precise hypothesis claims only that 51 happens nine times out of ten. Ignoring nine 51s, pointing at one case of 82, and crowing in triumph, does not a refutation make. That’s not an excuse, it’s an explicit advance prediction of a technical model.

The error term makes the “precise” theory vulnerable to a superprecise alternative that predicted the 82. The standard model would also be vulnerable to a precisely ignorant model that predicted a 60% chance of 51 on the round where we saw 82, spreading out the likelihood more entropically on that particular error. No matter how good the theory, science always has room for a higher-scoring competitor. But if you don’t present a better alternative, if you try only to show that an accepted theory is stupid with respect to the data, that scientific endeavor may be more demanding than just replacing the old theory with a new one.

Astronomers recorded the unexplained perihelion advance of Mercury, unaccounted for under Newtonian physics – or rather, Newtonian physics predicted 5557 seconds of arc per century, where the observed amount was 5600. (From Brown 1999.) But should the scientists of that day have junked Newtonian gravitation based on such small, unexplained counterevidence? What would they have used instead? Eventually, Newton’s theory of gravitation was set aside, after Einstein’s General Relativity precisely explained the orbital discrepancy of Mercury and also made successful advance predictions. But there was no way to know in advance that this was how things would turn out.

In the nineteenth century there was a persistent anomaly in the orbit of Uranus. People said, “Maybe Newton’s law starts to fail at long distances.” Eventually some bright fellows looked at the anomaly and said, “Could this be an unknown outer planet?” Urbain Le Verrier and John Couch Adams independently did some scribbling and figuring, using Newton’s standard theory – and predicted Neptune’s location to within one degree of arc, dramatically confirming Newtonian gravitation. (Brown 1999.)

Only after General Relativity precisely produced the perihelion advance of Mercury, did we know Newtonian gravitation would never explain it.

In the Intuitive Explanation we saw how Karl Popper’s insight that falsification is stronger than confirmation, translates into a Bayesian truth about likelihood ratios. Popper erred in thinking that falsification was qualitatively different from confirmation; both are governed by the same Bayesian rules. But Popper’s philosophy reflected an important truth about a quantitative difference between falsification and confirmation.”Popper was profoundly impressed by the differences between the allegedly ‘scientific’ theories of Freud and Adler and the revolution effected by Einstein’s theory of relativity in physics in the first two decades of this century. The main difference between them, as Popper saw it, was that while Einstein’s theory was highly ‘risky’, in the sense that it was possible to deduce consequences from it which were, in the light of the then dominant Newtonian physics, highly improbable (e.g. that light is deflected towards solid bodies – confirmed by Eddington’s experiments in 1919), and which would, if they turned out to be false, falsify the whole theory, nothing could, even in principle, falsify psychoanalytic theories. These latter, Popper came to feel, have more in common with primitive myths than with genuine science. That is to say, he saw that what is apparently the chief source of strength of psychoanalysis, and the principal basis on which its claim to scientific status is grounded, viz. its capability to accommodate, and explain, every possible form of human behaviour, is in fact a critical weakness, for it entails that it is not, and could not be, genuinely predictive. Psychoanalytic theories by their nature are insufficiently precise to have negative implications, and so are immunised from experiential falsification…”Popper, then, repudiates induction, and rejects the view that it is the characteristic method of scientific investigation and inference, and substitutes falsifiability in its place. It is easy, he argues, to obtain evidence in favour of virtually any theory, and he consequently holds that such ‘corroboration’, as he terms it, should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction, which might conceivably have been false. For Popper, a theory is scientific only if it is refutable by a conceivable event. Every genuine test of a scientific theory, then, is logically an attempt to refute or to falsify it…”Every genuine scientific theory then, in Popper’s view, is prohibitive, in the sense that it forbids, by implication, particular events or occurrences.”(Thornton 2002)

On Popper’s philosophy, the strength of a scientific theory is not how much it explains, but how much it doesn’t explain. The virtue of a scientific theory lies not in the outcomes it permits , but in the outcomes it prohibits . Freud’s theories, which seemed to explain everything, prohibited nothing.

Translating this into Bayesian terms, we find that the more outcomes a model prohibits , the more probability density the model concentrates in the remaining, permitted outcomes. The more outcomes a theory prohibits, the greater the knowledge-content of the theory. The more daringly a theory exposes itself to falsification, the more definitely it tells you which experiences to anticipate.

A theory that can explain any experience corresponds to a hypothesis of complete ignorance – a uniform distribution with probability density spread evenly over every possible outcome.

One of the most famous lessons of science is the case of the phlogiston theory of chemistry .

Phlogiston was the 18th century’s answer to the Elemental Fire of the Greek alchemists. Ignite wood, and let it burn. What is the orangey-bright “fire” stuff? Why does the wood transform into ash? To both questions, the 18th century chemists answered, “phlogiston”.

…and that was it, you see, that was their answer: “Phlogiston.”

Phlogiston escaped from burning substances as visible fire. As the phlogiston escaped, the burning substances lost phlogiston and so became ash, the “true material”. Flames extinguished in closed containers because the air became saturated with phlogiston. Charcoal left little residue upon burning because it was nearly pure phlogiston. (Moore 1961.)

This was a more primitive age of science, and so people did not notice and take offense that phlogiston theory made no advance predictions. Instead phlogiston theory just added on more and more independent clauses to explain more and more chemical observations. You couldn’t use phlogiston theory to predict the outcome of a chemical transformation – first you looked at the result, then you used phlogiston to explain it. It was not that, having never tried burning a flame in a closed container, phlogiston theorists predicted that the flame would go out when the air became “saturated” with phlogiston. Rather they lit a flame in a container, watched it go out, then said, “The air must have become saturated with phlogiston.”

You couldn’t even use phlogiston theory to constrain chemical transformations, to say what you did not expect to see. Phlogiston theory was infinitely flexible. In excusing everything, it explained nothing; a disguised hypothesis of zero knowledge.

The word phlogiston functioned not as an anticipation-controller but as a curiosity-stopper . You said “Why?” and the answer was “Phlogiston”.

Imagine looking at your hand, and knowing nothing of cells, nothing of biological chemistry, nothing of DNA. You know some anatomy, you know your hand contains muscles, but you don’t know why muscles move instead of lying there like clay. Your hand is just… stuff… and for some reason it moves under your direction. Is this not magic?”The animal body does not act as a thermodynamic engine … consciousness teaches every individual that they are, to some extent, subject to the direction of his will. It appears therefore that animated creatures have the power of immediately applying to certain moving particles of matter within their bodies, forces by which the motions of these particles are directed to produce derived mechanical effects… The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concurrence of atoms… Modern biologists were coming once more to the acceptance of something and that was a vital principle.”– Lord Kelvin (from Zapato 1998).

This was the theory of vitalism ; that the difference between living matter and non-living matter consisted of an elan vital or vis vitalis . Elan vital infused living matter and caused it to move as consciously directed. Elan vital participated in chemical transformations which no mere non-living particles could undergo. Wohler’s artificial synthesis of urea, a component of urine, was a major blow to the vitalistic theory because it showed that “mere chemistry” could duplicate a product of biology. (Moore 1961.)

Building on the previous lesson of phlogiston, we note at once that elan vital functions not as an anticipation-controller but as a curiosity-stopper. Vitalism doesn’t explain how the hand moves, nor tell you what transformations to expect from organic chemistry, and vitalism certainly permits no quantitative calculations. “Why? Elan vital!” And that was all there was to vitalism.

But the greater lesson lies in the vitalists’ reverence for the elan vital, their eagerness to pronounce it a mystery beyond all science. Meeting the great dragon Unknown, the vitalists did not draw their swords to do battle, but instead bowed their necks in submission. They took pride in their ignorance, made biology into a sacred mystery, and thereby became loath to relinquish their ignorance when evidence came knocking.

I quote Lord Kelvin to show that in every generation, there are scientific puzzles so wonderfully mys-TER-i-ous that they become sacred, making a solution sacrilege. Science is only good for explaining non-mysterious phenomena, like the course of planets, or the transformations of materials, or the biology of life; science can never answer questions about real mysteries like consciousness. Surely, if it were possible for science to explain consciousness, it would already have done so? As if all these other matters had not been mysteries for thousands of years and millions of years, from the dawn of intelligent thought right up until science solved them.

People have no sense of history. They learn about stars and chemistry and biology in school and it seems that these matters have always been the proper meat of science, that they have never been mysterious. Astrologers and alchemists and vitalists were merely fools, to make such big deals out of such simple questions. When science must deal with some new puzzling phenomenon, it is a great shock to the children of that generation, for they have never encountered something that feels mysterious before. Surely such a sacred mystery as consciousness is infinitely beyond the reach of dry scientific thinking; science is only suited to mundane questions such as biology.

Vitalism shared with phlogiston the error of encapsulating the mystery as a substance. Fire was mysterious, and the phlogiston theory encapsulated the mystery in a mysterious substance called “phlogiston”. Life was a sacred mystery, and vitalism encapsulated the sacred mystery in a mysterious substance called “elan vital”. Neither “explanation” helped concentrate the model’s probability density. The “explanation” just wrapped up the question as a small, hard, opaque black ball. In a play written by the author Moliere, a physician explains the power of a soporific by claiming that the soporific contains a “dormitive potency” – a fine parody of the art of fake explanation. (Cited in Kuhn 1962.)

It is a failure of human psychology that, faced with a mysterious phenomenon, we more readily postulate mysterious inherent substances than complex underlying processes.

But the deeper failure is supposing that an answer can be mysterious. Mystery is a property of questions, not answers. If a phenomenon feels mysterious, that is a fact about our state of knowledge, not a fact about the phenomenon itself. The vitalists saw a mysterious gap in their knowledge, and postulated a mysterious stuff that plugged the gap. They mixed up the map with the territory. All confusion and dismay exist in the mind, not in reality.

I call theories such as vitalism mysterious answers to mysterious questions . These are the signs of mysterious answers: First, the explanation acts as a curiosity-stopper rather than an anticipation-controller. Second, the hypothesis has no moving parts – the model is not a specific complex mechanism, but a blankly solid substance or force. The mysterious substance or mysterious force may be said to be here or there, to do this or that; but the reason why the mysterious force behaves thus is wrapped in a blank unity. Third, those who proffer the explanation cherish their ignorance; they speak proudly of how the phenomenon defeats ordinary science or is unlike merely mundane phenomena. Fourth, even after the answer is given, the phenomenon is still a mystery and possesses the same quality of sacred inexplicability that it had at the start.

The flip side:

Beware of checklist thinking: Having a sacred mystery, or a mysterious answer, is not the same as refusing to explain something. Some elements in our physics are taken as “fundamental”, not yet further reduced or explained. But these fundamental elements of our physics are governed by clearly defined, mathematically simple, formally computable causal rules.

Occasionally some crackpot objects to modern physics on the grounds that it does not provide an “underlying mechanism” for a mathematical law currently treated as fundamental. (Claiming that a mathematical law lacks an “underlying mechanism” is one of the entries on John Baez’s Crackpot Index; Baez 1998.) The “underlying mechanism” the crackpot proposes in answer is vague, verbal, and yields no increase in predictive power – otherwise we would not classify the claimant as a crackpot.

Our current physics makes the electromagnetic field fundamental, and refuses to explain it further. But the “electromagnetic field” is a fundamental governed by clear mathematical rules, with no properties outside the mathematical rules, subject to formal computation to describe its causal effect upon the world. Someday someone may suggest improved math that yields better predictions, but I would not indict the current model on grounds of mysteriousness. A theory that includes fundamental elements is not the same as a theory that contains mysterious elements .

Fundamentals should be simple. “Life” is not a good fundamental; “oxygen” is a good fundamental, and “electromagnetic field” is a better fundamental. Life might look simple to a vitalist – it’s the simple, magical ability of your muscles to move under your mental direction. Why shouldn’t life be explained by a simple, magical fundamental substance like elan vital ? But phenomena that seem psychologically very simple – little dots of light in the sky, orangey-bright hot flame, flesh moving under mental direction – often conceal vast depths of underlying complexity. The proposition that life is a complex phenomenon may seem incredible to the vitalist, staring at a blankly opaque mystery with no obvious handles; but yes, Virginia, there is underlying complexity. The criterion of simplicity that is relevant to Occam’s Razor is mathematical or computational simplicity. Once we render down our model into mathematically simple fundamental elements, not in themselves sharing the mysterious qualities of the mystery, interacting in clearly defined ways to produce the formerly mysterious phenomenon as a detailed prediction, that is as non-mysterious as humanity has ever figured out how to make anything.

The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb and name some current theory, not yet disproven, that I think is analogously flawed to vitalism and phlogiston? I shall dare, but don’t try this at home. I also warn my readers that they should not accept this opinion of mine with the same confidence that attaches to science’s dismissal of phlogiston.

I name the fad of emergence or emergent phenomena – systems which exhibit high-level behaviors that arise or “emerge” from the interaction of many low-level elements. Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem.

In decrying the emergence fad, I decry the use of “emergence” as an explanation in itself . It’s okay to have a completed model to which an emergence enthusiast could attach “emergent” as an adjective. One might legitimately have some specific model of how the behavior of an ant colony emerges from the behavior of the ants. A hypothesis like that can be formal and/or technical. The model of the ant colony has internal moving parts and produces specific predictions; it’s just that the model happens to fit the verbal term “emergent” – the behavior which emerges from modeling many interacting elements is different from the behavior of those elements considered in isolation. I do not consider it stupid to say that Phenomenon X emerges from Y, where Y is some specific model. The phrase “emerges from” is okay, if the phrase precedes some specific model to be judged on its own merits.

However, this is not the way “emergence” is commonly used. “Emergence” is commonly used as an explanation in its own right. I have lost track of how many times I have heard people say, “Intelligence is an emergent phenomenon!” as if that explained intelligence. This usage fits all the checklist items for a mysterious answer to a mysterious question. What do you know, after you have said that intelligence is “emergent”? You can make no new predictions. You do not know anything about the behavior of real-world minds that you did not know before. It feels like you believe a new fact, but you don’t anticipate any different outcomes. Your curiosity feels sated, but it has not been fed. The hypothesis has no moving parts – there’s no detailed internal model to manipulate. Those who proffer the hypothesis of “emergence” confess their ignorance of the internals, and take pride in it; they contrast the science of “emergence” to other sciences merely mundane. And even after the answer of “Why? Emergence!” is given, the phenomenon is still a mystery and possesses the same sacred impenetrability it had at the start.

To say that intelligence is an “emergent phenomenon” fits every possible behavior that intelligence could show, and therefore explains nothing. The model has no moving parts and does not concentrate its probability mass into specific outcomes. It is a disguised hypothesis of zero knowledge.

To see why I object to the academic fad in “emergence”, even though I have admitted the legitimacy of the phrase “emerges from”, consider that “arises from” is also a legitimate phrase. Gravity arises from the curvature of spacetime (according to a certain specific mathematical model, Einstein’s General Relativity). Chemistry arises from interactions between atoms (according to the specific model of quantum electrodynamics). Now suppose I should say that gravity is explained by “arisence” or that chemistry is an “arising phenomenon”, and claim that as my explanation.

A fun exercise is to eliminate the adjective “emergent” from any sentence in which it appears, and see if the sentence says anything different.Before: Human intelligence is an emergent product of neurons firing.After: Human intelligence is a product of neurons firing.Before: The behavior of the ant colony is the emergent outcome of the interactions of many individual ants.After: The behavior of the ant colony is the outcome of the interactions of many individual ants.Even better: A colony is made of ants. We can successfully predict some aspects of colony behavior using models that include only individual ants, without any global colony variables, showing that we understand how those colony behaviors arise from ant behaviors.

Another good exercise is to replace the word “emergent” with the old word, the explanation that people had to use before emergence was invented.Before: Life is an emergent phenomenon.After: Life is a magical phenomenon.Before: Human intelligence is an emergent product of neurons firing.After: Human intelligence is a magical product of neurons firing.

Does not each statement convey exactly the same amount of knowledge about the phenomenon’s behavior? Does not each hypothesis fit exactly the same set of outcomes?

Magic is unpopular nowadays, unfashionable, not something you could safely postulate in a peer-reviewed journal. Why? Once upon a time, a few exceptionally wise scientists noticed that explanations which invoked “magic” just didn’t work as a way of understanding the world. By dint of strenuous evangelism, these wise scientists managed to make magical explanations unfashionable within a small academic community. But humans are still humans, and they have the same emotional needs and intellectual vulnerabilities. So later academics invented a new word, “emergence”, that carried exactly the same information content as “magic”, but had not yet become unfashionable. “Emergence” became very popular, just as saying “magic” used to be very popular. “Emergence” has the same deep appeal to human psychology, for the same reason. “Emergence” is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Emergence is a popular fad because it is the junk food of curiosity. You can explain anything using emergence, and so people do just that; for it feels so wonderful to explain things. Humans are still humans, even if they’ve taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed in different clothes but still the same species psychology.

Many people in this world believe that after dying they will face a stern-eyed fellow named St. Peter, who will examine their actions in life and accumulate a score for morality. Presumably St. Peter’s scoring rule is unique and invariant under trivial changes of perspective. Unfortunately, believers cannot obtain a quantitative, precisely computable specification of the scoring rule, which seems rather unfair.

The religion of Bayesianity holds that your eternal fate depends on the probability judgments you made in life. Unlike lesser faiths, Bayesianity can give a quantitative, precisely computable specification of how your eternal fate is determined.

Our proper Bayesian scoring rule provides a way to accumulate scores across experiments, and the score is invariant regardless of how we slice up the “experiments” or in what order we accumulate the results. We add up the logarithms of the probabilities. This corresponds to multiplying together the probability assigned to the outcome in each experiment, to find the joint probability of all the experiments together. We take the logarithm to simplify our intuitive understanding of the accumulated score, to maintain our grip on the tiny fractions involved, and to ensure we maximize our expected score by stating our honest probabilities rather than placing all our play money on the most probable bet.

Bayesianity states that, when you die, Pierre-Simon Laplace examines every single event in your life, from finding your shoes next to your bed in the morning, to finding your workplace in its accustomed spot. Every losing lottery ticket means you cared enough to play. Laplace assesses the advance probability you assigned to each event. Where you did not assign a precise numerical probability in advance, Laplace examines your degree of anticipation or surprise, extrapolates other possible outcomes and your extrapolated reactions, and renormalizes your extrapolated emotions to a likelihood distribution over possible outcomes. (Hence the phrase “Laplacian superintelligence”.)

Then Laplace takes every event in your life, and every probability you assigned to each event, and multiplies all the probabilities together. This is your Final Judgment – the probability you assigned to your life.

Those who follow Bayesianity strive all their lives to maximize their Final Judgment. This is the sole virtue of Bayesianity. The rest is just math.

Mark you: the path of Bayesianity is strict. What probability shall you assign each morning, to the proposition, “The sun shall rise?” (We shall discount such quibbles as cloudy days, and that the Earth orbits the Sun.) Perhaps one who did not follow Bayesianity would be humble, and give a probability of 99.9%. But we who follow Bayesianity shall discard all considerations of modesty and arrogance, and scheme only to maximize our Final Judgment. Like an obsessive video-game player, we care only about this numerical score. We’re going to face this Sun-shall-rise issue 365 times per year, so we might be able to improve our Final Judgment considerably by tweaking our probability assignment.

As it stands, even if the Sun rises every morning, every year our Final Judgment will decrease by a factor of 0.7 (.999^365), roughly -0.52 bits. Every two years, our Final Judgment will decrease more than if we found ourselves ignorant of a coinflip’s outcome! Intolerable. If we increase our daily probability of sunrise to 99.99%, then each year our Final Judgment will decrease only by a factor of 0.964. Better. Still, in the unlikely event that we live exactly 70 years and then die, our Final Judgment will only be 7.75% of what it might have been. What if we assign a 99.999% probability to the sunrise? Then after 70 years, our Final Judgment will be multiplied by 77.4%.

Why not assign a probability of 1.0?

One who follows Bayesianity will never assign a probability of 1.0 to anything . Assigning a probability of 1.0 to some outcome uses up all your probability mass. If you assign a probability of 1.0 to some outcome, and reality delivers a different answer, you must have assigned the actual outcome a probability of 0 . This is Bayesianity’s sole mortal sin. Zero times anything is zero. When Laplace multiplies together all the probabilities of your life, the combined probability will be zero. Your Final Judgment will be doodly-squat, zilch, nada, nil. No matter how rational your guesses during the rest of your life, you’ll spend eternity next to some guy who believed in flying saucers and got all his information from the Weekly World News. Again we find it helpful to take the logarithm, revealing the innocent-sounding “zero” in its true form. Risking an outcome probability of zero is like accepting a bet with a payoff of negative infinity.

What if humanity decides to take apart the Sun for mass (stellar engineering), or to switch off the Sun because it’s wasting entropy? Well, you say, you’ll see that coming, you’ll have a chance to alter your probability assignment before the actual event. What if an Artificial Intelligence in someone’s basement recursively self-improves to superintelligence, stealthily develops nanotechnology, and one morning it takes apart the Sun? If on the last night of the world you assign a probability of 99.999% to tomorrow’s sunrise, your Final Judgment will go down by a factor of 100,000. Minus 50 decibels! Awful, isn’t it?

So what is your best strategy? Well, suppose you 50% anticipate that a basement-spawned AI superintelligence will disassemble the Sun sometime in the next 10 years, and you figure there’s about an equal chance of this happening on any given day between now and then. On any given night, you would 99.98% anticipate the sun rising tomorrow. If this is really what you anticipate, then you have no motive to say anything except 99.98% as your probability. If you feel nervous that this anticipation is too low, or too high, it must not be what you anticipate after your nervousness is taken into account.

But the deeper truth of Bayesianity is this: you cannot game the system. You cannot give a humble answer, nor a confident one. You must figure out exactly how much you anticipate the Sun rising tomorrow, and say that number. You must shave away every hair of modesty or arrogance, and ask whether you expect to end up being scored on the Sun rising, or failing to rise. Look not to your excuses, but ask which excuses you expect to need. After you arrive at your exact degree of anticipation, the only way to further improve your Final Judgment is to improve the accuracy, calibration, and discrimination of your anticipation. You cannot do better except by guessing better and anticipating more precisely.

Er, well, except that you could commit suicide when you turned five, thereby preventing your Final Judgment from decreasing any further. Or if we patch a new sin onto the utility function, enjoining against suicide, you could flee from mystery, avoiding all situations in which you thought you might not know everything. So much for that religion.

Ideally, we predict the outcome of the experiment in advance, using our model, and then we perform the experiment to see if the outcome accords with our model. Unfortunately, we can’t always control the information stream. Sometimes Nature throws experiences at us, and by the time we think of an explanation, we’ve already seen the data we’re supposed to explain. This was one of the scientific sins committed by 19th-century evolutionism; Darwin observed the similarity of many species, and their adaptation to particular local environments, before the hypothesis of natural selection occurred to him. 19th-century evolutionism began life as a post facto explanation, not an advance prediction.

Nor is this a trouble only of semitechnical theories. In 1846, the successful deduction of Neptune’s existence from gravitational perturbations in the orbit of Uranus was considered a grand triumph for Newton’s theory of gravitation. Why? Because Neptune’s existence was the first observation that confirmed an advance prediction of Newtonian gravitation. All the other phenomena that Newton explained, such as orbits and orbital perturbations and tides, had been observed in great detail before Newton explained them. No one seriously doubted that Newton’s theory was correct. Newton’s theory explained too much too precisely, and it replaced a collection of ad-hoc models with a single unified mathematical law. Even so, the advance prediction of Neptune’s existence, followed by the observation of Neptune at almost exactly the predicted location, was considered the first grand triumph of Newton’s theory at predicting what no previous model could predict. Considerable time elapsed between widespread acceptance of Newton’s theory and the first impressive advance prediction of Newtonian gravitation. By the time Newton came up with his theory, scientists had already observed, in great detail, most of the phenomena that Newtonian gravitation predicted.

But the rule of advance prediction is a morality of science, not a law of probability theory. If you have already seen the data you must explain, then Science may darn you to heck, but your predicament doesn’t collapse the laws of probability theory. What does happen is that it becomes much more difficult for a hapless human to obey the laws of probability theory. When you’re deciding how to rate a hypothesis according to the Bayesian scoring rule, you need to figure out how much probability mass that hypothesis assigns to the observed outcome. If we must make our predictions in advance, then it’s easier to notice when someone is trying to claim every possible outcome as an advance prediction, using too much probability mass, being deliberately vague to avoid falsification, and so on.

No numerologist can predict next week’s winning lottery numbers, but they will be happy to explain the mystical significance of last week’s winning lottery numbers. Say the winning Mega Ball was 7 in last week’s lottery, out of 52 possible outcomes. Obviously this happened because 7 is the lucky number. So will the Mega Ball in next week’s lottery also come up 7? We understand that it’s not certain, of course, but if it’s the lucky number, you ought to assign a probability of higher than 1/52… and then we’ll score your guesses over the course of a few years, and if your score is too low we’ll have you flogged… what’s that you say? You want to assign a probability of exactly 1/52? But that’s the same probability as every other number; what happened to 7 being lucky? No, sorry, you can’t assign a 90% probability to 7 and also a 90% probability to 11. We understand they’re both lucky numbers. Yes, we understand that they’re very lucky numbers. But that’s not how it works.

Even if the listener does not know the way of Bayes and does not ask for formal probabilities, they will probably become suspicious if you try to cover too many bases. Suppose they ask you to predict next week’s winning Mega Ball, and you use numerology to explain why the 1 ball would fit your theory very well, and why the 2 ball would fit your theory very well, and why the 3 ball would fit your theory very well… even the most credulous listener might begin to ask questions by the time you got to 12. Maybe you could tell us which numbers are unlucky and definitely won’t win the lottery? Well, 13 is unlucky, but it’s not absolutely impossible (you hedge, anticipating in advance which excuse you might need).

But if we ask you to explain last week’s lottery numbers, why, the 7 was practically inevitable. That 7 should definitely count as a major success for the “lucky numbers” model of the lottery. And it couldn’t possibly have been 13; luck theory rules that straight out.

Imagine that you wake up one morning and your left arm has been replaced by a blue tentacle. The blue tentacle obeys your motor commands – you can use it to pick up glasses, drive a car, etc. How would you explain this hypothetical scenario? Take a moment to ponder this puzzle before continuing.

spoiler space

spoiler space

spoiler space

How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen.

It would be easy enough to produce a verbal explanation that “fit” the hypothetical. There are many explanations that can “fit” anything, including (as a special case of “anything”) my arm being replaced by a blue tentacle. Divine intervention is a good all-purpose explanation. Or aliens with arbitrary motives and capabilities. Or I could be mad, hallucinating, dreaming my life away in a hospital. Such explanations “fit” all outcomes equally well, and equally poorly, equating to hypotheses of complete ignorance.

The test of whether a model of reality “explains” my arm turning into a blue tentacle, is whether the model concentrates significant probability mass into that particular outcome. Why that dream, in the hospital? Why would aliens do that particular thing to me, as opposed to the other billion things they might do? Why would my arm turn into a tentacle on that morning, after remaining an arm through every other morning of my life? And in all cases I must look for an argument compelling enough to make that particular prediction in advance , not mere compatibility. Once I already knew the outcome, it would become far more difficult to sift through hypotheses to find good explanations. Whatever hypothesis I tried, I would be hard-pressed not to allocate more probability mass to yesterday’s blue-tentacle outcome than if I extrapolated blindly, seeking the model’s most likely prediction for tomorrow.

A model does not always predict all the features of the data. Nature has no privileged tendency to present me with solvable challenges. Perhaps a deity toys with me, and the deity’s mind is computationally intractable. If I flip a fair coin there is no way to further explain the outcome, no model that makes a better prediction than the maximum-entropy hypothesis. But if I guess a model with no internal detail or a model that makes no further predictions, I not only have no reason to believe that guess, I have no reason to care. Last night my arm was replaced with a blue tentacle. Why? Aliens! So what will they do tomorrow? Similarly, if I attribute the blue tentacle to a hallucination as I dream my life away in a coma, I still don’t know any more about what I’ll hallucinate tomorrow. So why do I care whether it was aliens or hallucination?

What might be a good explanation, then, if I woke up one morning and found my arm transformed into a blue tentacle? To claim a “good explanation” for this hypothetical experience would require an argument such that, contemplating the hypothetical argument now , before my arm has transformed into a blue tentacle, I would go to sleep worrying that my arm really would transform into a tentacle.

People play games with plausibility, explaining events they expect to never actually encounter, yet this necessarily violates the laws of probability theory. How many people who thought they could ‘explain’ the hypothetical experience of waking up with their arm replaced by a tentacle, would go to sleep wondering if it might really happen to them? Had they the courage of their convictions, they would say: I do not expect to ever encounter this hypothetical experience, and therefore I cannot explain, nor have I a motive to try. Such things only happen in webcomics, and I need not prepare explanations, for in real life I shall never have a chance to use them. If I ever find myself in this impossible situation, let me miss no jot or tittle of my valuable bewilderment.

To a Bayesian, probabilities are anticipations, not mere beliefs to proclaim from the rooftops. If I have a model that assigns probability mass to waking up with a blue tentacle, then I am nervous about waking up with a blue tentacle. What if the model is a fanciful one, like a witch casting a spell that transports me into a randomly selected webcomic? Then the prior probability of webcomic witchery is so low that my real-world understanding doesn’t assign any significant weight to that hypothesis. The witchcraft hypothesis, if taken as a given, might assign non-insignificant likelihood to waking up with a blue tentacle. But my anticipation of that hypothesis is so low that I don’t anticipate any of the predictions of that hypothesis. That I can conceive of a witchcraft hypothesis should in no wise diminish my stark bewilderment if I actually wake up with a tentacle, because the real-world probability I assign to the witchcraft hypothesis is effectively zero. My zero-probability hypothesis wouldn’t help me explain waking up with a tentacle, because the argument isn’t good enough to make me anticipate waking up with a tentacle.

In the laws of probability theory, likelihood distributions are fixed properties of a hypothesis. In the art of rationality, to explain is to anticipate . To anticipate is to explain . Suppose I am a medical researcher, and in the ordinary course of pursuing my research, I notice that my clever new theory of anatomy seems to permit a small and vague possibility that my arm will transform into a blue tentacle. “Ha ha!”, I say, “how remarkable and silly!”, and feel ever so slightly nervous. That would be a good explanation for waking up with a tentacle, if it ever happened.

If a chain of reasoning doesn’t make me nervous, in advance, about waking up with a tentacle, then that reasoning would be a poor explanation if the event did happen, because the combination of prior probability and likelihood was too low to make me allocate any significant real-world probability mass to that outcome.

If you start from well-calibrated priors, and you apply Bayesian reasoning, you’ll end up with well-calibrated conclusions. Imagine that two million entities, scattered across different planets in the universe, have the opportunity to encounter something so strange as waking up with a tentacle (or – gasp! – ten fingers). One million of these entities say “one in a thousand” for the prior probability of some hypothesis X, and each hypothesis X says “one in a hundred” for the likelihood of waking up with a tentacle. And one million of these entities say “one in a hundred” for the prior probability of some hypothesis Y, and each hypothesis Y says “one in ten” for the likelihood of waking up with a tentacle. If we suppose that all entities are well-calibrated, then we shall look across the universe and find ten entities who wound up with a tentacle because of hypotheses of plausibility class X, and a thousand entities who wound up with tentacles because of hypotheses of plausibility class Y. So if you find yourself with a tentacle, and if your probabilities are well-calibrated, then the tentacle is more likely to stem from a hypothesis you would class as probable than a hypothesis you would class as improbable. (What if your probabilities are poorly calibrated, so that when you say “million-to-one” it happens one time out of twenty? Then you’re grossly overconfident, and we adjust your probabilities in the direction of less discrimination / greater entropy.)

The hypothesis of being transported into a webcomic, even if it “explains” the scenario of waking up with a blue tentacle, is a poor explanation because of its low prior probability. The webcomic hypothesis doesn’t contribute to explaining the tentacle, because it doesn’t make you anticipate waking up with a tentacle.

If we start with a quadrillion sentient minds scattered across the universe, quite a lot of entities will encounter events that are very likely, only about a mere million entities will experience events with lifetime likelihoods of a billion-to-one (as we would anticipate, surveying with infinite eyes and perfect calibration), and not a single entity will experience the impossible.

If, somehow, you really did wake up with a tentacle, it would likely be because of something much more probable than “being transported into a webcomic”, some perfectly normal reason to wake up with a tentacle which you just didn’t see coming. A reason like what? I don’t know. Nothing. I don’t anticipate waking up with a tentacle, so I can’t give any good explanation for it. Why should I bother crafting excuses that I don’t expect to use? If I was worried I might someday need a clever excuse for waking up with a tentacle, the reason I was nervous about the possibility would be my explanation.

Reality dishes out experiences using probability, not plausibility. If you find out that your laptop doesn’t obey conservation of momentum, then reality must think that a perfectly normal thing to do to you. How could violating conservation of momentum possibly be perfectly normal? I anticipate that question has no answer and will never need answering. Similarly, people do not wake up with tentacles, so apparently it is not perfectly normal.

There is a shattering truth, so surprising and terrifying that people resist the implications with all their strength. Yet there are a lonely few with the courage to accept this satori. Here is wisdom, if you would be wise:Since the beginningNot one unusual thingHas ever happened.

Alas for those who turn their eyes from zebras and dream of dragons! If we cannot learn to take joy in the merely real, our lives shall be empty indeed.

This document is ©2005 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/technical/ .

If you enjoyed A Technical Explanation of Technical Explanation , you may enjoy the earlier works in the series, the Twelve Virtues of Rationality , The Simple Truth , and An Intuitive Explanation of Bayesian Reasoning .

If you’ve already read all that, you can move on to Overcoming Bias .

References:

John Baez (1998). “ The Crackpot Index. ”

Kevin Brown (1999). “ Anomalous Precessions ” In mathpages.com: Postings to sci.math collected by Kevin Brown.

Robyn M. Dawes (1988). “Rational Choice in an Uncertain World”. Harcourt Brace Jovanovich, Inc.

E. T. Jaynes (1996). “ Probability Theory: The Logic of Science ” Published posthumously by Cambridge University Press, ed. G. Larry Bretthorst. (2003).

T. S. Kuhn (1962). “The Structure of Scientific Revolutions.” The Chicago University Press.

Friedrich Spee von Langenfeld (1631). “Cautio Criminalis, Or, a Book on Witch Trials.” Translated: Marcus Hellyer, 2003. University Press of Virginia.

Ruth Moore (1961). “The Coil of Life.” London Constable. See also Phlogiston Theory , Demise of Phlogiston , and Friedrich W�hler .

Robert Pirsig (1974). “Zen and the Art of Motorcycle Maintenance.” New York: Bantam Books.

Karl Popper (1959). “The Logic of Scientific Discovery”. Hutchinson, London.

D. Robinson and J. Groves (1998). “Philosophy for Beginners.” Cambridge: Icon Books.

Carl Sagan (1995). “The Demon-Haunted World: Science as a Candle in the Dark.” Random House, New York, NY.

Stephen Thornton (2002). “ Karl Popper ”. In Edward N. Zalta (ed.), “The Stanford Encyclopedia of Philosophy” (Winter 2002 Edition).

A. Tversky and W. Edwards (1966). “Information versus reward in binary choice.” Journal of Experimental Psychology, 71, 680-683. See also Y. Schul and R. Mayo 2003, “ Searching for certainty in an uncertain world. ” In Journal of Behavioral Decision Making, 16:2, 93-106.

Joachim Verhagen (2001). From the “ Canonical List of Science Jokes ”, version 7.27, collected by Joachim Verhagen.

George Williams (1966). “Adaptation and Natural Selection: A Critique of Some Current Evolutionary Thought.” Princeton, NJ: Princeton University Press.

J.F. Yates, J.W. Lee, W.R. Sieck, I. Choi,&P.C. Price (2002). “Probability judgment across cultures.” In T. Gilovich, D. Griffin,&D. Kahneman (Eds.), “Heuristics and Biases.” New York: Cambridge.

Lyle Zapato (1998). “ Lord Kelvin Quotations ”

Bayes’ Theorem
for the curious and bewildered;
an excruciatingly gentle introduction.

This page has now been obsoleted by a vastly improved guide to Bayes’s Theorem, the Arbital Guide to Bayes’s Rule . Please read that instead. Seriously. I mean it. The current version is also plagued by a number of technical problems, with various applets no longer working. A mostly functional archived version of this essay can be found here.

Your friends and colleagues are talking about something called “Bayes’ Theorem” or “Bayes’ Rule”, or something called Bayesian reasoning.  They sound really enthusiastic about it, too, so you google and find a webpage about Bayes’ Theorem and…

It’s this equation.  That’s all.  Just one equation.  The page you found gives a definition of it, but it doesn’t say what it is, or why it’s useful, or why your friends would be interested in it.  It looks like this random statistics thing.

So you came here.  Maybe you don’t understand what the equation says.  Maybe you understand it in theory, but every time you try to apply it in practice you get mixed up trying to remember the difference between p(a|x) and p(x|a) , and whether p(a)*p(x|a) belongs in the numerator or the denominator.  Maybe you see the theorem, and you understand the theorem, and you can use the theorem, but you can’t understand why your friends and/or research colleagues seem to think it’s the secret of the universe.  Maybe your friends are all wearing Bayes’ Theorem T-shirts, and you’re feeling left out.  Maybe you’re a girl looking for a boyfriend, but the boy you’re interested in refuses to date anyone who “isn’t Bayesian”.  What matters is that Bayes is cool, and if you don’t know Bayes, you aren’t cool.

Why does a mathematical concept generate this strange enthusiasm in its students?  What is the so-called Bayesian Revolution now sweeping through the sciences, which claims to subsume even the experimental method itself as a special case?  What is the secret that the adherents of Bayes know?  What is the light that they have seen?

Soon you will know.  Soon you will be one of us.

While there are a few existing online explanations of Bayes’ Theorem, my experience with trying to introduce people to Bayesian reasoning is that the existing online explanations are too abstract.  Bayesian reasoning is very counterintuitive.   People do not employ Bayesian reasoning intuitively, find it very difficult to learn Bayesian reasoning when tutored, and rapidly forget Bayesian methods once the tutoring is over.  This holds equally true for novice students and highly trained professionals in a field.  Bayesian reasoning is apparently one of those things which, like quantum mechanics or the Wason Selection Test, is inherently difficult for humans to grasp with our built-in mental faculties.

Or so they claim.  Here you will find an attempt to offer an intuitive explanation of Bayesian reasoning – an excruciatingly gentle introduction that invokes all the human ways of grasping numbers, from natural frequencies to spatial visualization.  The intent is to convey, not abstract rules for manipulating numbers, but what the numbers mean, and why the rules are what they are (and cannot possibly be anything else).  When you are finished reading this page, you will see Bayesian problems in your dreams.

And let’s begin.

Here’s a story problem about a situation that doctors often encounter:

1% of women at age forty who participate in routine screening have breast cancer.  80% of women with breast cancer will get positive mammographies.  9.6% of women without breast cancer will also get positive mammographies.  A woman in this age group had a positive mammography in a routine screening.  What is the probability that she actually has breast cancer?

What do you think the answer is?  If you haven’t encountered this kind of problem before, please take a moment to come up with your own answer before continuing.

Next, suppose I told you that most doctors get the same wrong answer on this problem – usually, only around 15% of doctors get it right.  (“Really?  15%?  Is that a real number, or an urban legend based on an Internet poll?”  It’s a real number.  See Casscells, Schoenberger, and Grayboys 1978; Eddy 1982; Gigerenzer and Hoffrage 1995; and many other studies.  It’s a surprising result which is easy to replicate, so it’s been extensively replicated.)

Do you want to think about your answer again?  Here’s a Javascript calculator if you need one.  This calculator has the usual precedence rules; multiplication before addition and so on.  If you’re not sure, I suggest using parentheses.

Calculator:  Result:

On the story problem above, most doctors estimate the probability to be between 70% and 80%, which is wildly incorrect.

Here’s an alternate version of the problem on which doctors fare somewhat better:

10 out of 1000 women at age forty who participate in routine screening have breast cancer.  800 out of 1000 women with breast cancer will get positive mammographies.  96 out of 1000 women without breast cancer will also get positive mammographies.  If 1000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?

Calculator:  Result:

And finally, here’s the problem on which doctors fare best of all, with 46% – nearly half – arriving at the correct answer:

100 out of 10,000 women at age forty who participate in routine screening have breast cancer.  80 of every 100 women with breast cancer will get a positive mammography.  950 out of  9,900 women without breast cancer will also get a positive mammography.  If 10,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?

Calculator:  Result:

The correct answer is 7.8%, obtained as follows:  Out of 10,000 women, 100 have breast cancer; 80 of those 100 have positive mammographies.  From the same 10,000 women, 9,900 will not have breast cancer and of those 9,900 women, 950 will also get positive mammographies.  This makes the total number of women with positive mammographies 950+80 or 1,030.  Of those 1,030 women with positive mammographies, 80 will have cancer.  Expressed as a proportion, this is 80/1,030 or 0.07767 or 7.8%.

To put it another way, before the mammography screening, the 10,000 women can be divided into two groups:

• Group 1:  100 women with breast cancer.
• Group 2:  9,900 women without breast cancer.

Summing these two groups gives a total of 10,000 patients, confirming that none have been lost in the math.  After the mammography, the women can be divided into four groups:

• Group A:  80 women with breast cancer, and a positive mammography.
• Group B:  20 women with breast cancer, and a negative mammography.
• Group C:  950 women without   breast cancer, and a positive mammography.
• Group D:  8,950 women without breast cancer, and a negative mammography.

Calculator:  Result:  As you can check, the sum of all four groups is still 10,000.  The sum of groups A and B, the groups with breast cancer, corresponds to group 1; and the sum of groups C and D, the groups without breast cancer, corresponds to group 2; so administering a mammography does not actually change the number of women with breast cancer.  The proportion of the cancer patients (A + B) within the complete set of patients (A + B + C + D) is the same as the 1% prior chance that a woman has cancer: (80 + 20) / (80 + 20 + 950 + 8950) = 100 / 10000 = 1%.

The proportion of cancer patients with positive results, within the group of all patients with positive results, is the proportion of (A) within (A + C):   80 / (80 + 950) = 80 / 1030 = 7.8%.  If you administer a mammography to 10,000 patients, then out of the 1030 with positive mammographies, 80 of those positive-mammography patients will have cancer.  This is the correct answer, the answer a doctor should give a positive-mammography patient if she asks about the chance she has breast cancer; if thirteen patients ask this question, roughly 1 out of those 13 will have cancer.

The most common mistake is to ignore the original fraction of women with breast cancer, and the fraction of women without breast cancer who receive false positives, and focus only on the fraction of women with breast cancer who get positive results.  For example, the vast majority of doctors in these studies seem to have thought that if around 80% of women with breast cancer have positive mammographies, then the probability of a women with a positive mammography having breast cancer must be around 80%.

Figuring out the final answer always requires all three pieces of information – the percentage of women with breast cancer, the percentage of women without breast cancer who receive false positives, and the percentage of women with breast cancer who receive (correct) positives.

To see that the final answer always depends on the original fraction of women with breast cancer, consider an alternate universe in which only one woman out of a million has breast cancer.  Even if mammography in this world  detects breast cancer in 8 out of 10 cases, while returning a false positive on a woman without breast cancer in only 1 out of 10 cases, there will still be a hundred thousand false positives for every real case of cancer detected.  The original probability that a woman has cancer is so extremely low that, although a positive result on the mammography does increase the estimated probability, the probability isn’t increased to certainty or even “a noticeable chance”; the probability goes from 1:1,000,000 to 1:100,000.

Similarly, in an alternate universe where only one out of a million women does not have breast cancer, a positive result on the patient’s mammography obviously doesn’t mean that she has an 80% chance of having breast cancer!  If this were the case her estimated probability of having cancer would have been revised drastically downward after she got a positive result on her mammography – an 80% chance of having cancer is a lot less than 99.9999%!  If you administer mammographies to ten million women in this world, around eight million women with breast cancer will get correct positive results, while one woman without breast cancer will get false positive results.  Thus, if you got a positive mammography in this alternate universe, your chance of having cancer would go from 99.9999% up to 99.999987%.  That is, your chance of being healthy would go from 1:1,000,000 down to 1:8,000,000.

These two extreme examples help demonstrate that the mammography result doesn’t replace your old information about the patient’s chance of having cancer; the mammography slides the estimated probability in the direction of the result.  A positive result slides the original probability upward; a negative result slides the probability downward.  For example, in the original problem where 1% of the women have cancer, 80% of women with cancer get positive mammographies, and 9.6% of women without cancer get positive mammographies, a positive result on the mammography slides the 1% chance upward to 7.8%.

Most people encountering problems of this type for the first time carry out the mental operation of replacing the original 1% probability with the 80% probability that a woman with cancer gets a positive mammography.  It may seem like a good idea, but it just doesn’t work.  “The probability that a woman with a positive mammography has breast cancer” is not at all the same thing as “the probability that a woman with breast cancer has a positive mammography”; they are as unlike as apples and cheese.  Finding the final answer, “the probability that a woman with a positive mammography has breast cancer”, uses all three pieces of problem information – “the prior probability that a woman has breast cancer”, “the probability that a woman with breast cancer gets a positive mammography”, and “the probability that a woman without breast cancer gets a positive mammography”.

To see that the final answer always depends on the chance that a woman without breast cancer gets a positive mammography, consider an alternate test, mammography+.  Like the original test, mammography+ returns positive for 80% of women with breast cancer.  However, mammography+ returns a positive result for only one out of a million women without breast cancer – mammography+ has the same rate of false negatives, but a vastly lower rate of false positives.  Suppose a patient receives a positive mammography+.  What is the chance that this patient has breast cancer?  Under the new test, it is a virtual certainty – 99.988%, i.e., a 1 in 8082 chance of being healthy.

Calculator:  Result:
Remember, at this point, that neither mammography nor mammography+ actually change the number of women who have breast cancer.  It may seem like “There is a virtual certainty you have breast cancer” is a terrible thing to say, causing much distress and despair; that the more hopeful verdict of the previous mammography test – a 7.8% chance of having breast cancer – was much to be preferred.  This comes under the heading of “Don’t shoot the messenger”.  The number of women who really do have cancer stays exactly the same between the two cases.  Only the accuracy with which we detect cancer changes.  Under the previous mammography test, 80 women with cancer (who already had cancer, before the mammography) are first told that they have a 7.8% chance of having cancer, creating X amount of uncertainty and fear, after which more detailed tests will inform them that they definitely do have breast cancer.  The old mammography test also involves informing 950 women without breast cancer that they have a 7.8% chance of having cancer, thus creating twelve times as much additional fear and uncertainty.  The new test, mammography+, does not give 950 women false positives, and the 80 women with cancer are told the same facts they would have learned eventually, only earlier and without an intervening period of uncertainty.  Mammography+ is thus a better test in terms of its total emotional impact on patients, as well as being more accurate.  Regardless of its emotional impact, it remains a fact that a patient with positive mammography+ has a 99.988% chance of having breast cancer.

Of course, that mammography+ does not give 950 healthy women false positives means that all 80 of the patients with positive mammography+ will be patients with breast cancer.  Thus, if you have a positive mammography+, your chance of having cancer is a virtual certainty.  It is because mammography+ does not generate as many false positives (and needless emotional stress), that the (much smaller) group of patients who do get positive results will be composed almost entirely of genuine cancer patients (who have bad news coming to them regardless of when it arrives).

Similarly, let’s suppose that we have a less discriminating test, mammography*, that still has a 20% rate of false negatives, as in the original case.  However, mammography* has an 80% rate of false positives.  In other words, a patient without breast cancer has an 80% chance of getting a false positive result on her mammography* test.  If we suppose the same 1% prior probability that a patient presenting herself for screening has breast cancer, what is the chance that a patient with positive mammography* has cancer?

• Group 1:  100 patients with breast cancer.
• Group 2:  9,900 patients without breast cancer.

After mammography* screening:

• Group A:  80 patients with breast cancer and a “positive” mammography*.
• Group B:  20 patients with breast cancer and a “negative” mammography*.
• Group C:  7920 patients without breast cancer and a “positive” mammography*.
• Group D:  1980 patients without breast cancer and a “negative” mammography*.

Calculator:  Result:
The result works out to 80 / 8,000, or 0.01.  This is exactly the same as the 1% prior probability that a patient has breast cancer!  A “positive” result on mammography* doesn’t change the probability that a woman has breast cancer at all.  You can similarly verify that a “negative” mammography* also counts for nothing.  And in fact it must be this way, because if mammography* has an 80% hit rate for patients with breast cancer, and also an 80% rate of false positives for patients without breast cancer, then mammography* is completely uncorrelated with breast cancer.  There’s no reason to call one result “positive” and one result “negative”; in fact, there’s no reason to call the test a “mammography”.  You can throw away your expensive mammography* equipment and replace it with a random number generator that outputs a red light 80% of the time and a green light 20% of the time; the results will be the same.  Furthermore, there’s no reason to call the red light a “positive” result or the green light a “negative” result.  You could have a green light 80% of the time and a red light 20% of the time, or a blue light 80% of the time and a purple light 20% of the time, and it would all have the same bearing on whether the patient has breast cancer: i.e., no bearing whatsoever.

We can show algebraically that this must hold for any case where the chance of a true positive and the chance of a false positive are the same, i.e:

• Group 1:  100 patients with breast cancer.
• Group 2:  9,900 patients without breast cancer.

Now consider a test where the probability of a true positive and the probability of a false positive are the same number M (in the example above, M=80% or M = 0.8):

• Group A:  100*M patients with breast cancer and a “positive” result.
• Group B:  100*(1 – M) patients with breast cancer and a “negative” result.
• Group C:  9,900*M patients without breast cancer and a “positive” result.
• Group D:  9,900*(1 – M) patients without breast cancer and a “negative” result.

The proportion of patients with breast cancer, within the group of patients with a “positive” result, then equals 100*M / (100*M + 9900*M) = 100 / (100 + 9900) = 1%.  This holds true regardless of whether M is 80%, 30%, 50%, or 100%.  If we have a mammography* test that returns “positive” results for 90% of patients with breast cancer and returns “positive” results for 90% of patients without breast cancer, the proportion of “positive”-testing patients who have breast cancer will still equal the original proportion of patients with breast cancer, i.e., 1%.

You can run through the same algebra, replacing the prior proportion of patients with breast cancer with an arbitrary percentage P:

• Group 1:  Within some number of patients, a fraction P have breast cancer.
• Group 2:  Within some number of patients, a fraction (1 – P) do not have breast cancer.

After a “cancer test” that returns “positive” for a fraction M of patients with breast cancer, and also returns “positive” for the same fraction M of patients without cancer:

• Group A:  P*M patients have breast cancer and a “positive” result.
• Group B:  P*(1 – M) patients have breast cancer and a “negative” result.
• Group C:  (1 – P)*M patients have no breast cancer and a “positive” result.
• Group D:  (1 – P)*(1 – M) patients have no breast cancer and a “negative” result.

The chance that a patient with a “positive” result has breast cancer is then the proportion of group A within the combined group A + C, or P*M / [P*M + (1 – P)*M], which, cancelling the common factor M from the numerator and denominator, is P / [P + (1 – P)] or P / 1 or just P.  If the rate of false positives is the same as the rate of true positives, you always have the same probability after the test as when you started.

The original proportion of patients with breast cancer is known as the prior probability.   The chance that a patient with breast cancer gets a positive mammography, and the chance that a patient without breast cancer gets a positive mammography, are known as the two conditional probabilities.   Collectively, this initial information is known as the priors.   The final answer – the estimated probability that a patient has breast cancer, given that we know she has a positive result on her mammography – is known as the revised probability or the posterior probability.   What we’ve just shown is that if the two conditional probabilities are equal, the posterior probability equals the prior probability.

Actually, priors are true or false just like the final answer – they reflect reality and can be judged by comparing them against reality.  For example, if you think that 920 out of 10,000 women in a sample have breast cancer, and the actual number is 100 out of 10,000, then your priors are wrong.  For our particular problem, the priors might have been established by three studies – a study on the case histories of women with breast cancer to see how many of them tested positive on a mammography, a study on women without breast cancer to see how many of them test positive on a mammography, and an epidemiological study on the prevalence of breast cancer in some specific demographic.

Suppose that a barrel contains many small plastic eggs.  Some eggs are painted red and some are painted blue.  40% of the eggs in the bin contain pearls, and 60% contain nothing.   30% of eggs containing pearls are painted blue, and 10% of eggs containing nothing are painted blue.  What is the probability that a blue egg contains a pearl?  For this example the arithmetic is simple enough that you may be able to do it in your head, and I would suggest trying to do so.

But just in case…  Result:  A more compact way of specifying the problem:

• p(pearl) = 40%
• p(blue|pearl) = 30%
• p(blue|~pearl) = 10%
• p(pearl|blue) = ?

“~” is shorthand for “not”, so ~pearl reads “not pearl”.

blue|pearl is shorthand for “blue given pearl” or “the probability that an egg is painted blue, given that the egg contains a pearl”.  One thing that’s confusing about this notation is that the order of implication is read right-to-left, as in Hebrew or Arabic.  blue|pearl means “blue <- pearl”, the degree to which pearl-ness implies blue-ness, not the degree to which blue-ness implies pearl-ness.  This is confusing, but it’s unfortunately the standard notation in probability theory.

The item on the right side is what you already know or the premise, and the item on the left side is the implication or conclusion.   If we have p(blue|pearl) = 30% , and we already know that some egg contains a pearl, then we can conclude there is a 30% chance that the egg is painted blue.  Thus, the final fact we’re looking for – “the chance that a blue egg contains a pearl” or “the probability that an egg contains a pearl, if we know the egg is painted blue” – reads p(pearl|blue) .

Let’s return to the problem.  We have that 40% of the eggs contain pearls, and 60% of the eggs contain nothing.  30% of the eggs containing pearls are painted blue, so 12% of the eggs altogether contain pearls and are painted blue.  10% of the eggs containing nothing are painted blue, so altogether 6% of the eggs contain nothing and are painted blue.  A total of 18% of the eggs are painted blue, and a total of 12% of the eggs are painted blue and contain pearls, so the chance a blue egg contains a pearl is 12/18 or 2/3 or around 67%.

The applet below, courtesy of Christian Rovner, shows a graphic representation of this problem:
(Are you having trouble seeing this applet?  Do you see an image of the applet rather than the applet itself?  Try downloading an updated Java .)

Looking at this applet, it’s easier to see why the final answer depends on all three probabilities; it’s the differential pressure between the two conditional probabilities,  p(blue|pearl) and p(blue|~pearl) , that slides the prior probability p(pearl) to the posterior probability p(pearl|blue) .

As before, we can see the necessity of all three pieces of information by considering extreme cases (feel free to type them into the applet).  In a (large) barrel in which only one egg out of a thousand contains a pearl, knowing that an egg is painted blue slides the probability from 0.1% to 0.3% (instead of sliding the probability from 40% to 67%).  Similarly, if 999 out of 1000 eggs contain pearls, knowing that an egg is blue slides the probability from 99.9% to 99.966%; the probability that the egg does not contain a pearl goes from 1/1000 to around 1/3000.  Even when the prior probability changes, the differential pressure of the two conditional probabilities always slides the probability in the same direction.   If you learn the egg is painted blue, the probability the egg contains a pearl always goes up – but it goes up from the prior probability, so you need to know the prior probability in order to calculate the final answer.  0.1% goes up to 0.3%, 10% goes up to 25%, 40% goes up to 67%, 80% goes up to 92%, and 99.9% goes up to 99.966%.  If you’re interested in knowing how any other probabilities slide, you can type your own prior probability into the Java applet.  You can also click and drag the dividing line between pearl and ~pearl in the upper bar, and watch the posterior probability change in the bottom bar.

Studies of clinical reasoning show that most doctors carry out the mental operation of replacing the original 1% probability with the 80% probability that a woman with cancer would get a positive mammography.  Similarly, on the pearl-egg problem, most respondents unfamiliar with Bayesian reasoning would probably respond that the probability a blue egg contains a pearl is 30%, or perhaps 20% (the 30% chance of a true positive minus the 10% chance of a false positive).  Even if this mental operation seems like a good idea at the time, it makes no sense in terms of the question asked.  It’s like the experiment in which you ask a second-grader:  “If eighteen people get on a bus, and then seven more people get on the bus, how old is the bus driver?”  Many second-graders will respond:  “Twenty-five.”  They understand when they’re being prompted to carry out a particular mental procedure, but they haven’t quite connected the procedure to reality.  Similarly, to find the probability that a woman with a positive mammography has breast cancer, it makes no sense whatsoever to replace the original probability that the woman has cancer with the probability that a woman with breast cancer gets a positive mammography.  Neither can you subtract the probability of a false positive from the probability of the true positive.  These operations are as wildly irrelevant as adding the number of people on the bus to find the age of the bus driver.

I keep emphasizing the idea that evidence slides probability because of research that shows people tend to use spatial intutions to grasp numbers.  In particular, there’s interesting evidence that we have an innate sense of quantity that’s localized to left inferior parietal cortex – patients with damage to this area can selectively lose their sense of whether 5 is less than 8, while retaining their ability to read, write, and so on.  (Yes, really!)  The parietal cortex processes our sense of where things are in space (roughly speaking), so an innate “number line”, or rather “quantity line”, may be responsible for the human sense of numbers.  This is why I suggest visualizing Bayesian evidence as sliding the probability along the number line; my hope is that this will translate Bayesian reasoning into something that makes sense to innate human brainware.  (That, really, is what an “intuitive explanation” is. )  For more information, see Stanislas Dehaene’s The Number Sense.

A study by Gigerenzer and Hoffrage in 1995 showed that some ways of phrasing story problems are much more evocative of correct Bayesian reasoning.  The least evocative phrasing used probabilities.  A slightly more evocative phrasing used frequencies instead of probabilities; the problem remained the same, but instead of saying that 1% of women had breast cancer, one would say that 1 out of 100 women had breast cancer, that 80 out of 100 women with breast cancer would get a positive mammography, and so on.  Why did a higher proportion of subjects display Bayesian reasoning on this problem?  Probably because saying “1 out of 100 women” encourages you to concretely visualize X women with cancer, leading you to visualize X women with cancer and a positive mammography, etc.

The most effective presentation found so far is what’s known as natural frequencies – saying that 40 out of 100 eggs contain pearls, 12 out of 40 eggs containing pearls are painted blue, and 6 out of 60 eggs containing nothing are painted blue.  A natural frequencies presentation is one in which the information about the prior probability is included in presenting the conditional probabilities.  If you were just learning about the eggs’ conditional probabilities through natural experimentation, you would – in the course of cracking open a hundred eggs – crack open around 40 eggs containing pearls, of which 12 eggs would be painted blue, while cracking open 60 eggs containing nothing, of which about 6 would be painted blue.  In the course of learning the conditional probabilities, you’d see examples of blue eggs containing pearls about twice as often as you saw examples of blue eggs containing nothing.

It may seem like presenting the problem in this way is “cheating”, and indeed if it were a story problem in a math book, it probably would be cheating.  However, if you’re talking about real doctors, you want to cheat; you want the doctors to draw the right conclusions as easily as possible.  The obvious next move would be to present all medical statistics in terms of natural frequencies.  Unfortunately, while natural frequencies are a step in the right direction, it probably won’t be enough.  When problems are presented in natural frequences, the proportion of people using Bayesian reasoning rises to around half.  A big improvement, but not big enough when you’re talking about real doctors and real patients.

A presentation of the problem in natural frequencies might be visualized like this:

In the frequency visualization, the selective attrition of the two conditional probabilities changes the proportion of eggs that contain pearls.  The bottom bar is shorter than the top bar, just as the number of eggs painted blue is less than the total number of eggs.  The probability graph shown earlier is really just the frequency graph with the bottom bar “renormalized”, stretched out to the same length as the top bar.  In the frequency applet you can change the conditional probabilities by clicking and dragging the left and right edges of the graph.  (For example, to change the conditional probability blue|pearl , click and drag the line on the left that stretches from the left edge of the top bar to the left edge of the bottom bar.)

In the probability applet, you can see that when the conditional probabilities are equal, there’s no differential pressure – the arrows are the same size – so the prior probability doesn’t slide between the top bar and the bottom bar.  But the bottom bar in the probability applet is just a renormalized (stretched out) version of the bottom bar in the frequency applet, and the frequency applet shows why the probability doesn’t slide if the two conditional probabilities are equal.  Here’s a case where the prior proportion of pearls remains 40%, and the proportion of pearl eggs painted blue remains 30%, but the number of empty eggs painted blue is also 30%:

If you diminish two shapes by the same factor, their relative proportion will be the same as before.  If you diminish the left section of the top bar by the same factor as the right section, then the bottom bar will have the same proportions as the top bar – it’ll just be smaller.  If the two conditional probabilities are equal, learning that the egg is blue doesn’t change the probability that the egg contains a pearl – for the same reason that similar triangles have identical angles; geometric figures don’t change shape when you shrink them by a constant factor.

In this case, you might as well just say that 30% of eggs are painted blue, since the probability of an egg being painted blue is independent of whether the egg contains a pearl.  Applying a “test” that is statistically independent of its condition just shrinks the sample size.  In this case, requiring that the egg be painted blue doesn’t shrink the group of eggs with pearls any more or less than it shrinks the group of eggs without pearls.  It just shrinks the total number of eggs in the sample.

Here’s what the original medical problem looks like when graphed.  1% of women have breast cancer, 80% of those women test positive on a mammography, and 9.6% of women without breast cancer also receive positive mammographies.

As is now clearly visible, the mammography doesn’t increase the probability a positive-testing woman has breast cancer by increasing the number of women with breast cancer – of course not; if mammography increased the number of women with breast cancer, no one would ever take the test!  However, requiring a positive mammography is a membership test that eliminates many more women without breast cancer than women with cancer.  The number of women without breast cancer diminishes by a factor of more than ten, from 9,900 to 950, while the number of women with breast cancer is diminished only from 100 to 80.  Thus, the proportion of 80 within 1,030 is much larger than the proportion of 100 within 10,000.  In the graph, the left sector (representing women with breast cancer) is small, but the mammography test projects almost all of this sector into the bottom bar.  The right sector (representing women without breast cancer) is large, but the mammography test projects a much smaller fraction of this sector into the bottom bar.  There are, indeed, fewer women with breast cancer and positive mammographies than there are women with breast cancer – obeying the law of probabilities which requires that p(A) >= p(A&B) .  But even though the left sector in the bottom bar is actually slightly smaller, the proportion of the left sector within the bottom bar is greater – though still not very great.  If the bottom bar were renormalized to the same length as the top bar, it would look like the left sector had expanded.  This is why the proportion of “women with breast cancer” in the group “women with positive mammographies” is higher than the proportion of “women with breast cancer” in the general population – although the proportion is still not very high.  The evidence of the positive mammography slides the prior probability of 1% to the posterior probability of 7.8%.

Suppose there’s yet another variant of the mammography test, mammography@, which behaves as follows.  1% of women in a certain demographic have breast cancer.  Like ordinary mammography, mammography@ returns positive 9.6% of the time for women without breast cancer.  However, mammography@ returns positive 0% of the time (say, once in a billion) for women with breast cancer.  The graph for this scenario looks like this:

What is it that this test actually does?  If a patient comes to you with a positive result on her mammography@, what do you say?

“Congratulations, you’re among the rare 9.5% of the population whose health is definitely established by this test.”

Mammography@ isn’t a cancer test; it’s a health test!  Few women without breast cancer get positive results on mammography@, but only women without breast cancer ever get positive results at all.  Not much of the right sector of the top bar projects into the bottom bar, but none of the left sector projects into the bottom bar.  So a positive result on mammography@ means you definitely don’t have breast cancer.

What makes ordinary mammography a positive indicator for breast cancer is not that someone named the result “positive”, but rather that the test result stands in a specific Bayesian relation to the condition of breast cancer.  You could call the same result “positive” or “negative” or “blue” or “red” or “James Rutherford”, or give it no name at all, and the test result would still slide the probability in exactly the same way.  To minimize confusion, a test result which slides the probability of breast cancer upward should be called “positive”.  A test result which slides the probability of breast cancer downward should be called “negative”.  If the test result is statistically unrelated to the presence or absence of breast cancer – if the two conditional probabilities are equal – then we shouldn’t call the procedure a “cancer test”!  The meaning of the test is determined by the two conditional probabilities; any names attached to the results are simply convenient labels.

The bottom bar for the graph of mammography@ is small; mammography@ is a test that’s only rarely useful.  Or rather, the test only rarely gives strong evidence, and most of the time gives weak evidence.  A negative result on mammography@ does slide probability – it just doesn’t slide it very far.  Click the “Result” switch at the bottom left corner of the applet to see what a negative result on mammography@ would imply.  You might intuit that since the test could have returned positive for health, but didn’t, then the failure of the test to return positive must mean that the woman has a higher chance of having breast cancer – that her probability of having breast cancer must be slid upward by the negative result on her health test.

This intuition is correct!  The sum of the groups with negative results and positive results must always equal the group of all women.  If the positive-testing group has “more than its fair share” of women without breast cancer, there must be an at least slightly higher proportion of women with cancer in the negative-testing group.  A positive result is rare but very strong evidence in one direction, while a negative result is common but very weak evidence in the opposite direction.  You might call this the Law of Conservation of Probability – not a standard term, but the conservation rule is exact.  If you take the revised probability of breast cancer after a positive result, times the probability of a positive result, and add that to the revised probability of breast cancer after a negative result, times the probability of a negative result, then you must always arrive at the prior probability.  If you don’t yet know what the test result is, the expected revised probability after the test result arrives – taking both possible results into account – should always equal the prior probability.

On ordinary mammography, the test is expected to return “positive” 10.3% of the time – 80 positive women with cancer plus 950 positive women without cancer equals 1030 women with positive results.  Conversely, the mammography should return negative 89.7% of the time:  100% – 10.3% = 89.7%.  A positive result slides the revised probability from 1% to 7.8%, while a negative result slides the revised probability from 1% to 0.22%.  So p(cancer|positive)*p(positive) + p(cancer|negative)*p(negative) = 7.8%*10.3% + 0.22%*89.7% = 1% = p(cancer) , as expected.

Calculator:  Result:

Why “as expected”?  Let’s take a look at the quantities involved:

One of the common confusions in using Bayesian reasoning is to mix up some or all of these quantities – which, as you can see, are all numerically different and have different meanings.  p(A&B) is the same as p(B&A) , but p(A|B) is not the same thing as p(B|A) , and p(A&B) is completely different from p(A|B) .  (I don’t know who chose the symmetrical “|” symbol to mean “implies”, and then made the direction of implication right-to-left, but it was probably a bad idea.)

To get acquainted with all these quantities and the relationships between them, we’ll play “follow the degrees of freedom”.  For example, the two quantities p(cancer) and p(~cancer) have 1 degree of freedom between them, because of the general law p(A) + p(~A) = 1 .  If you know that p(~cancer) = .99 , you can obtain p(cancer) = 1 – p(~cancer) = .01 .  There’s no room to say that p(~cancer) = .99 and then also specify p(cancer) = .25 ; it would violate the rule p(A) + p(~A) = 1 .

p(positive|cancer) and p(~positive|cancer) also have only one degree of freedom between them; either a woman with breast cancer gets a positive mammography or she doesn’t.  On the other hand, p(positive|cancer) and p(positive|~cancer) have two degrees of freedom.  You can have a mammography test that returns positive for 80% of cancerous patients and 9.6% of healthy patients, or that returns positive for 70% of cancerous patients and 2% of healthy patients, or even a health test that returns “positive” for 30% of cancerous patients and 92% of healthy patients.  The two quantities, the output of the mammography test for cancerous patients and the output of the mammography test for healthy patients, are in mathematical terms independent; one cannot be obtained from the other in any way, and so they have two degrees of freedom between them.

What about p(positive&cancer) , p(positive|cancer) , and p(cancer) ?  Here we have three quantities; how many degrees of freedom are there?  In this case the equation that must hold is p(positive&cancer) = p(positive|cancer) * p(cancer) .  This equality reduces the degrees of freedom by one.  If we know the fraction of patients with cancer, and chance that a cancerous patient has a positive mammography, we can deduce the fraction of patients who have breast cancer and a positive mammography by multiplying.  You should recognize this operation from the graph; it’s the projection of the top bar into the bottom bar.  p(cancer) is the left sector of the top bar, and p(positive|cancer) determines how much of that sector projects into the bottom bar, and the left sector of the bottom bar is p(positive&cancer) .

Similarly, if we know the number of patients with breast cancer and positive mammographies, and also the number of patients with breast cancer, we can estimate the chance that a woman with breast cancer gets a positive mammography by dividing: p(positive|cancer) = p(positive&cancer) / p(cancer) .  In fact, this is exactly how such medical diagnostic tests are calibrated; you do a study on 8,520 women with breast cancer and see that there are 6,816 (or thereabouts) women with breast cancer and positive mammographies, then divide 6,816 by 8520 to find that 80% of women with breast cancer had positive mammographies.  (Incidentally, if you accidentally divide 8520 by 6,816 instead of the other way around, your calculations will start doing strange things, such as insisting that 125% of women with breast cancer and positive mammographies have breast cancer.  This is a common mistake in carrying out Bayesian arithmetic, in my experience.)  And finally, if you know p(positive&cancer) and p(positive|cancer) , you can deduce how many cancer patients there must have been originally.  There are two degrees of freedom shared out among the three quantities; if we know any two, we can deduce the third.

How about p(positive) , p(positive&cancer) , and p(positive&~cancer) ?  Again there are only two degrees of freedom among these three variables.  The equation occupying the extra degree of freedom is p(positive) = p(positive&cancer) + p(positive&~cancer) .  This is how p(positive) is computed to begin with; we figure out the number of women with breast cancer who have positive mammographies, and the number of women without breast cancer who have positive mammographies, then add them together to get the total number of women with positive mammographies.  It would be very strange to go out and conduct a study to determine the number of women with positive mammographies – just that one number and nothing else – but in theory you could do so.  And if you then conducted another study and found the number of those women who had positive mammographies and breast cancer, you would also know the number of women with positive mammographies and no breast cancer – either a woman with a positive mammography has breast cancer or she doesn’t.  In general, p(A&B) + p(A&~B) = p(A) .  Symmetrically, p(A&B) + p(~A&B) = p(B) .

What about p(positive&cancer) , p(positive&~cancer) , p(~positive&cancer) , and p(~positive&~cancer) ?  You might at first be tempted to think that there are only two degrees of freedom for these four quantities – that you can, for example, get p(positive&~cancer) by multiplying p(positive) * p(~cancer) , and thus that all four quantities can be found given only the two quantities p(positive) and p(cancer) .  This is not the case!  p(positive&~cancer) = p(positive) * p(~cancer) only if the two probabilities are statistically independent – if the chance that a woman has breast cancer has no bearing on whether she has a positive mammography.  As you’ll recall, this amounts to requiring that the two conditional probabilities be equal to each other – a requirement which would eliminate one degree of freedom.  If you remember that these four quantities are the groups A, B, C, and D, you can look over those four groups and realize that, in theory, you can put any number of people into the four groups.  If you start with a group of 80 women with breast cancer and positive mammographies, there’s no reason why you can’t add another group of 500 women with breast cancer and negative mammographies, followed by a group of 3 women without breast cancer and negative mammographies, and so on.  So now it seems like the four quantities have four degrees of freedom.  And they would, except that in expressing them as probabilities, we need to normalize them to fractions of the complete group, which adds the constraint that p(positive&cancer) + p(positive&~cancer) + p(~positive&cancer) + p(~positive&~cancer) = 1 .  This equation takes up one degree of freedom, leaving three degrees of freedom among the four quantities.  If you specify the fractions of women in groups A, B, and D, you can deduce the fraction of women in group C.

Given the four groups A, B, C, and D, it is very straightforward to compute everything else:  p(cancer) = A + B , p(~positive|cancer) = B / (A + B) , and so on.  Since ABCD contains three degrees of freedom, it follows that the entire set of 16 probabilities contains only three degrees of freedom.  Remember that in our problems we always needed three pieces of information – the prior probability and the two conditional probabilities – which, indeed, have three degrees of freedom among them.  Actually, for Bayesian problems, any three quantities with three degrees of freedom between them should logically specify the entire problem.  For example, let’s take a barrel of eggs with p(blue) = 0.40 ,  p(blue|pearl) = 5/13 , and p(~blue&~pearl) = 0.20 .  Given this information, you can compute p(pearl|blue) .

As a story problem:
Suppose you have a large barrel containing a number of plastic eggs.  Some eggs contain pearls, the rest contain nothing.  Some eggs are painted blue, the rest are painted red.  Suppose that 40% of the eggs are painted blue, 5/13 of the eggs containing pearls are painted blue, and 20% of the eggs are both empty and painted red.  What is the probability that an egg painted blue contains a pearl?

Try it – I assure you it is possible.

Calculator:  Result:  You probably shouldn’t try to solve this with just a Javascript calculator, though.  I used a Python console.  (In theory, pencil and paper should also work, but I don’t know anyone who owns a pencil so I couldn’t try it personally.)

As a check on your calculations, does the (meaningless) quantity p(~pearl|~blue)/p(pearl) roughly equal .51?  (In story problem terms:  The likelihood that a red egg is empty, divided by the likelihood that an egg contains a pearl, equals approximately .51.)  Of course, using this information in the problem would be cheating.

If you can solve that problem, then when we revisit Conservation of Probability, it seems perfectly straightforward.  Of course the mean revised probability, after administering the test, must be the same as the prior probability.  Of course strong but rare evidence in one direction must be counterbalanced by common but weak evidence in the other direction.

Because:

p(cancer|positive)*p(positive)
+ p(cancer|~positive)*p(~positive)
= p(cancer)

In terms of the four groups:

p(cancer|positive)  = A / (A + C)
p(positive)         = A + C
p(cancer&positive)  = A
p(cancer|~positive) = B / (B + D)
p(~positive)        = B + D
p(cancer&~positive) = B
p(cancer)           = A + B

Let’s return to the original barrel of eggs – 40% of the eggs containing pearls, 30% of the pearl eggs painted blue, 10% of the empty eggs painted blue.  The graph for this problem is:

What happens to the revised probability, p(pearl|blue) , if the proportion of eggs containing pearls is kept constant, but 60% of the eggs with pearls are painted blue (instead of 30%), and 20% of the empty eggs are painted blue (instead of 10%)?  You could type 60% and 20% into the inputs for the two conditional probabilities, and see how the graph changes – but can you figure out in advance what the change will look like?

If you guessed that the revised probability remains the same, because the bottom bar grows by a factor of 2 but retains the same proportions, congratulations!  Take a moment to think about how far you’ve come.  Looking at a problem like

1% of women have breast cancer.  80% of women with breast cancer get positive mammographies.  9.6% of women without breast cancer get positive mammographies.  If a woman has a positive mammography, what is the probability she has breast cancer?

the vast majority of respondents intuit that around 70-80% of women with positive mammographies have breast cancer.  Now, looking at a problem like

Suppose there are two barrels containing many small plastic eggs.  In both barrels, some eggs are painted blue and the rest are painted red.  In both barrels, 40% of the eggs contain pearls and the rest are empty.  In the first barrel, 30% of the pearl eggs are painted blue, and 10% of the empty eggs are painted blue.  In the second barrel, 60% of the pearl eggs are painted blue, and 20% of the empty eggs are painted blue.  Would you rather have a blue egg from the first or second barrel?

you can see it’s intuitively obvious that the probability of a blue egg containing a pearl is the same for either barrel.  Imagine how hard it would be to see that using the old way of thinking!

It’s intuitively obvious, but how to prove it?  Suppose that we call P the prior probability that an egg contains a pearl, that we call M the first conditional probability (that a pearl egg is painted blue), and N the second conditional probability (that an empty egg is painted blue).  Suppose that M and N are both increased or diminished by an arbitrary factor X – for example, in the problem above, they are both increased by a factor of 2.  Does the revised probability that an egg contains a pearl, given that we know the egg is blue, stay the same?

• p(pearl) = P
• p(blue|pearl) = M*X
• p(blue|~pearl) = N*X
• p(pearl|blue) = ?

From these quantities, we get the four groups:

• Group A:  p(pearl&blue)   = P*M*X
• Group B:  p(pearl&~blue)  = P*(1 – (M*X))
• Group C:  p(~pearl&blue)  = (1 – P)*N*X
• Group D:  p(~pearl&~blue) = (1 – P)*(1 – (N*X))

The proportion of eggs that contain pearls and are blue, within the group of all blue eggs, is then the proportion of group (A) within the group (A + C), equalling P*M*X / (P*M*X + (1 – P)*N*X) .  The factor X in the numerator and denominator cancels out, so increasing or diminishing both conditional probabilities by a constant factor doesn’t change the revised probability.

The probability that a test gives a true positive divided by the probability that a test gives a false positive is known as the likelihood ratio of that test.   Does the likelihood ratio of a medical test sum up everything there is to know about the usefulness of the test?

No, it does not!  The likelihood ratio sums up everything there is to know about the meaning of a positive result on the medical test, but the meaning of a negative result on the test is not specified, nor is the frequency with which the test is useful.  If we examine the algebra above, while p(pearl|blue) remains constant, p(pearl|~blue) may change – the X does not cancel out.  As a story problem, this strange fact would look something like this:

Suppose that there are two barrels, each containing a number of plastic eggs.  In both barrels, 40% of the eggs contain pearls and the rest contain nothing.  In both barrels, some eggs are painted blue and the rest are painted red.  In the first barrel, 30% of the eggs with pearls are painted blue, and 10% of the empty eggs are painted blue.  In the second barrel, 90% of the eggs with pearls are painted blue, and 30% of the empty eggs are painted blue.  Would you rather have a blue egg from the first or second barrel?  Would you rather have a red egg from the first or second barrel?

For the first question, the answer is that we don’t care whether we get the blue egg from the first or second barrel.  For the second question, however, the probabilities do change – in the first barrel, 34% of the red eggs contain pearls, while in the second barrel 8.7% of the red eggs contain pearls!  Thus, we should prefer to get a red egg from the first barrel.  In the first barrel, 70% of the pearl eggs are painted red, and 90% of the empty eggs are painted red.  In the second barrel, 10% of the pearl eggs are painted red, and 70% of the empty eggs are painted red.

Calculator:  Result:  What goes on here?  We start out by noting that, counter to intuition, p(pearl|blue) and p(pearl|~blue) have two degrees of freedom among them even when p(pearl) is fixed – so there’s no reason why one quantity shouldn’t change while the other remains constant.  But we didn’t we just get through establishing a law for “Conservation of Probability”, which says that p(pearl|blue)*p(blue) + p(pearl|~blue)*p(~blue) = p(pearl) ?  Doesn’t this equation take up one degree of freedom?  No, because p(blue) isn’t fixed between the two problems.  In the second barrel, the proportion of blue eggs containing pearls is the same as in the first barrel, but a much larger fraction of eggs are painted blue!  This alters the set of red eggs in such a way that the proportions do change.  Here’s a graph for the red eggs in the second barrel:

Let’s return to the example of a medical test.  The likelihood ratio of a medical test – the number of true positives divided by the number of false positives – tells us everything there is to know about the meaning of a positive result.  But it doesn’t tell us the meaning of a negative result, and it doesn’t tell us how often the test is useful.  For example, a mammography with a hit rate of 80% for patients with breast cancer and a false positive rate of 9.6% for healthy patients has the same likelihood ratio as a test with an 8% hit rate and a false positive rate of 0.96%.  Although these two tests have the same likelihood ratio, the first test is more useful in every way – it detects disease more often, and a negative result is stronger evidence of health.

The likelihood ratio for a positive result summarizes the differential pressure of the two conditional probabilities for a positive result, and thus summarizes how much a positive result will slide the prior probability.  Take a probability graph, like this one:

The likelihood ratio of the mammography is what determines the slant of the line.  If the prior probability is 1%, then knowing only the likelihood ratio is enough to determine the posterior probability after a positive result.

But, as you can see from the frequency graph, the likelihood ratio doesn’t tell the whole story – in the frequency graph, the proportions of the bottom bar can stay fixed while the size of the bottom bar changes.   p(blue) increases but p(pearl|blue) doesn’t change, because p(pearl&blue) and p(~pearl&blue) increase by the same factor.  But when you flip the graph to look at p(~blue) , the proportions of p(pearl&~blue) and p(~pearl&~blue) do not remain constant.

Of course the likelihood ratio can’t tell the whole story; the likelihood ratio and the prior probability together are only two numbers, while the problem has three degrees of freedom.

Suppose that you apply two tests for breast cancer in succession – say, a standard mammography and also some other test which is independent of mammography.  Since I don’t know of any such test which is independent of mammography, I’ll invent one for the purpose of this problem, and call it the Tams-Braylor Division Test, which checks to see if any cells are dividing more rapidly than other cells.  We’ll suppose that the Tams-Braylor gives a true positive for 90% of patients with breast cancer, and gives a false positive for 5% of patients without cancer.  Let’s say the prior prevalence of breast cancer is 1%.  If a patient gets a positive result on her mammography and her Tams-Braylor, what is the revised probability she has breast cancer?

One way to solve this problem would be to take the revised probability for a positive mammography, which we already calculated as 7.8%, and plug that into the Tams-Braylor test as the new prior probability.  If we do this, we find that the result comes out to 60%.

Calculator:  Result:  But this assumes that first we see the positive mammography result, and then the positive result on the Tams-Braylor.  What if first the woman gets a positive result on the Tams-Braylor, followed by a positive result on her mammography.  Intuitively, it seems like it shouldn’t matter.  Does the math check out?

First we’ll administer the Tams-Braylor to a woman with a 1% prior probability of breast cancer.

Calculator:  Result:  Then we administer a mammography, which gives 80% true positives and 9.6% false positives, and it also comes out positive.

Calculator:  Result:  Lo and behold, the answer is again 60%.  (If it’s not exactly the same, it’s due to rounding error – you can get a more precise calculator, or work out the fractions by hand, and the numbers will be exactly equal.)

An algebraic proof that both strategies are equivalent is left to the reader.  To visualize, imagine that the lower bar of the frequency applet for mammography projects an even lower bar using the probabilities of the Tams-Braylor Test, and that the final lowest bar is the same regardless of the order in which the conditional probabilities are projected.

We might also reason that since the two tests are independent, the probability a woman with breast cancer gets a positive mammography and a positive Tams-Braylor is 90% * 80% = 72%.  And the probability that a woman without breast cancer gets false positives on mammography and Tams-Braylor is 5% * 9.6% = 0.48%.  So if we wrap it all up as a single test with a likelihood ratio of 72%/0.48%, and apply it to a woman with a 1% prior probability of breast cancer:

Calculator:  Result:  …we find once again that the answer is 60%.

Suppose that the prior prevalence of breast cancer in a demographic is 1%.  Suppose that we, as doctors, have a repertoire of three independent tests for breast cancer.  Our first test, test A, a mammography, has a likelihood ratio of 80%/9.6% = 8.33.  The second test, test B, has a likelihood ratio of 18.0 (for example, from 90% versus 5%); and the third test, test C, has a likelihood ratio of 3.5 (which could be from 70% versus 20%, or from 35% versus 10%; it makes no difference).  Suppose a patient gets a positive result on all three tests.  What is the probability the patient has breast cancer?

Here’s a fun trick for simplifying the bookkeeping.  If the prior prevalence of breast cancer in a demographic is 1%, then 1 out of 100 women have breast cancer, and 99 out of 100 women do not have breast cancer.  So if we rewrite the probability of 1% as an odds ratio, the odds are:

1:99

And the likelihood ratios of the three tests A, B, and C are:

8.33:1 = 25:3
18.0:1 = 18:1
3.5:1 =  7:2

The odds for women with breast cancer who score positive on all three tests, versus women without breast cancer who score positive on all three tests, will equal:

1*25*18*7:99*3*1*2 =
3,150:594

To recover the probability from the odds, we just write:
3,150 / (3,150 + 594) = 84%

This always works regardless of how the odds ratios are written; i.e., 8.33:1 is just the same as 25:3 or 75:9.  It doesn’t matter in what order the tests are administered, or in what order the results are computed.  The proof is left as an exercise for the reader.

E. T. Jaynes, in “Probability Theory With Applications in Science and Engineering”, suggests that credibility and evidence should be measured in decibels.

Decibels?

Decibels are used for measuring exponential differences of intensity.  For example, if the sound from an automobile horn carries 10,000 times as much energy (per square meter per second) as the sound from an alarm clock, the automobile horn would be 40 decibels louder.  The sound of a bird singing might carry 1,000 times less energy than an alarm clock, and hence would be 30 decibels softer.  To get the number of decibels, you take the logarithm base 10 and multiply by 10.

decibels = 10 log 10 (intensity)
or
intensity = 10 (decibels/10)

Suppose we start with a prior probability of 1% that a woman has breast cancer, corresponding to an odds ratio of 1:99.  And then we administer three tests of likelihood ratios 25:3, 18:1, and 7:2.  You could multiply those numbers… or you could just add their logarithms:

10 log 10 (1/99) = -20
10 log 10 (25/3) = 9
10 log 10 (18/1) = 13
10 log 10 (7/2)  = 5

It starts out as fairly unlikely that a woman has breast cancer – our credibility level is at -20 decibels.  Then three test results come in, corresponding to 9, 13, and 5 decibels of evidence.  This raises the credibility level by a total of 27 decibels, meaning that the prior credibility of -20 decibels goes to a posterior credibility of 7 decibels.  So the odds go from 1:99 to 5:1, and the probability goes from 1% to around 83%.

In front of you is a bookbag containing 1,000 poker chips.  I started out with two such bookbags, one containing 700 red and 300 blue chips, the other containing 300 red and 700 blue.  I flipped a fair coin to determine which bookbag to use, so your prior probability that the bookbag in front of you is the red bookbag is 50%.  Now, you sample randomly, with replacement after each chip.  In 12 samples, you get 8 reds and 4 blues.  What is the probability that this is the predominantly red bag?

Just for fun, try and work this one out in your head.  You don’t need to be exact – a rough estimate is good enough.  When you’re ready, continue onward.

According to a study performed by Lawrence Phillips and Ward Edwards in 1966, most people, faced with this problem, give an answer in the range 70% to 80%.  Did you give a substantially higher probability than that?  If you did, congratulations – Ward Edwards wrote that very seldom does a person answer this question properly, even if the person is relatively familiar with Bayesian reasoning.  The correct answer is 97%.

The likelihood ratio for the test result “red chip” is 7/3, while the likelihood ratio for the test result “blue chip” is 3/7.  Therefore a blue chip is exactly the same amount of evidence as a red chip, just in the other direction – a red chip is 3.6 decibels of evidence for the red bag, and a blue chip is -3.6 decibels of evidence.  If you draw one blue chip and one red chip, they cancel out.  So the ratio of red chips to blue chips does not matter; only the excess of red chips over blue chips matters.  There were eight red chips and four blue chips in twelve samples; therefore, four more red chips than blue chips.  Thus the posterior odds will be:

4 :3 4 = 2401:81
which is around 30:1, i.e., around 97%.

The prior credibility starts at 0 decibels and there’s a total of around 14 decibels of evidence, and indeed this corresponds to odds of around 25:1 or around 96%.  Again, there’s some rounding error, but if you performed the operations using exact arithmetic, the results would be identical.

We can now see intuitively that the bookbag problem would have exactly the same answer, obtained in just the same way, if sixteen chips were sampled and we found ten red chips and six blue chips.

You are a mechanic for gizmos.  When a gizmo stops working, it is due to a blocked hose 30% of the time.  If a gizmo’s hose is blocked, there is a 45% probability that prodding the gizmo will produce sparks.  If a gizmo’s hose is unblocked, there is only a 5% chance that prodding the gizmo will produce sparks.  A customer brings you a malfunctioning gizmo.  You prod the gizmo and find that it produces sparks.  What is the probability that a spark-producing gizmo has a blocked hose?

Calculator:  Result:  What is the sequence of arithmetical operations that you performed to solve this problem?

(45%*30%) / (45%*30% + 5%*70%)

Similarly, to find the chance that a woman with positive mammography has breast cancer, we computed:

p(positive|cancer)*p(cancer)
_______________________________________________
p(positive|cancer)*p(cancer) + p(positive|~cancer)*p(~cancer)

which is
p(positive&cancer) / [p(positive&cancer) + p(positive&~cancer)]
which is
p(positive&cancer) / p(positive)
which is
p(cancer|positive)

The fully general form of this calculation is known as Bayes’ Theorem or Bayes’ Rule:

Given some phenomenon A that we want to investigate, and an observation X that is evidence about A – for example, in the previous example, A is breast cancer and X is a positive mammography – Bayes’ Theorem tells us how we should update our probability of A, given the new evidence X.

By this point, Bayes’ Theorem may seem blatantly obvious or even tautological, rather than exciting and new.  If so, this introduction has entirely succeeded in its purpose.

So why is it that some people are so excited about Bayes’ Theorem?

“Do you believe that a nuclear war will occur in the next 20 years?  If no, why not?”  Since I wanted to use some common answers to this question to make a point about rationality, I went ahead and asked the above question in an IRC channel, #philosophy on EFNet.

One EFNetter who answered replied “No” to the above question, but added that he believed biological warfare would wipe out “99.4%” of humanity within the next ten years.  I then asked whether he believed 100% was a possibility.  “No,” he said.  “Why not?”, I asked.  “Because I’m an optimist,” he said.  (Roanoke of #philosophy on EFNet wishes to be credited with this statement, even having been warned that it will not be cast in a complimentary light.  Good for him!)  Another person who answered the above question said that he didn’t expect a nuclear war for 100 years, because “All of the players involved in decisions regarding nuclear war are not interested right now.”  “But why extend that out for 100 years?”, I asked.  “Pure hope,” was his reply.

What is it exactly that makes these thoughts “irrational” – a poor way of arriving at truth?  There are a number of intuitive replies that can be given to this; for example:  “It is not rational to believe things only because they are comforting.”  Of course it is equally irrational to believe things only because they are discomforting; the second error is less common, but equally irrational.  Other intuitive arguments include the idea that “Whether or not you happen to be an optimist has nothing to do with whether biological warfare wipes out the human species”, or “Pure hope is not evidence about nuclear war because it is not an observation about nuclear war.”

There is also a mathematical reply that is precise, exact, and contains all the intuitions as special cases.  This mathematical reply is known as Bayes’ Theorem.

For example, the reply “Whether or not you happen to be an optimist has nothing to do with whether biological warfare wipes out the human species” can be translated into the statement:

p(you are currently an optimist | biological war occurs within ten years and wipes out humanity) =
p(you are currently an optimist | biological war occurs within ten years and does not wipe out humanity)

Since the two probabilities for p(X|A) and p(X|~A) are equal, Bayes’ Theorem says that p(A|X) = p(A) ; as we have earlier seen, when the two conditional probabilities are equal, the revised probability equals the prior probability.  If X and A are unconnected – statistically independent – then finding that X is true cannot be evidence that A is true; observing X does not update our probability for A; saying “X” is not an argument for A.

But suppose you are arguing with someone who is verbally clever and who says something like, “Ah, but since I’m an optimist, I’ll have renewed hope for tomorrow, work a little harder at my dead-end job, pump up the global economy a little, eventually, through the trickle-down effect, sending a few dollars into the pocket of the researcher who ultimately finds a way to stop biological warfare – so you see, the two events are related after all, and I can use one as valid evidence about the other.”  In one sense, this is correct – any correlation, no matter how weak, is fair prey for Bayes’ Theorem; but Bayes’ Theorem distinguishes between weak and strong evidence.  That is, Bayes’ Theorem not only tells us what is and isn’t evidence, it also describes the strength of evidence.  Bayes’ Theorem not only tells us when to revise our probabilities, but how much to revise our probabilities.  A correlation between hope and biological warfare may exist, but it’s a lot weaker than the speaker wants it to be; he is revising his probabilities much too far.

Let’s say you’re a woman who’s just undergone a mammography.  Previously, you figured that you had a very small chance of having breast cancer; we’ll suppose that you read the statistics somewhere and so you know the chance is 1%.  When the positive mammography comes in, your estimated chance should now shift to 7.8%.  There is no room to say something like, “Oh, well, a positive mammography isn’t definite evidence, some healthy women get positive mammographies too.  I don’t want to despair too early, and I’m not going to revise my probability until more evidence comes in.  Why?  Because I’m a optimist.”  And there is similarly no room for saying, “Well, a positive mammography may not be definite evidence, but I’m going to assume the worst until I find otherwise.  Why?  Because I’m a pessimist.”  Your revised probability should go to 7.8%, no more, no less.

Bayes’ Theorem describes what makes something “evidence” and how much evidence it is.  Statistical models are judged by comparison to the Bayesian method because, in statistics, the Bayesian method is as good as it gets – the Bayesian method defines the maximum amount of mileage you can get out of a given piece of evidence, in the same way that thermodynamics defines the maximum amount of work you can get out of a temperature differential.  This is why you hear cognitive scientists talking about Bayesian reasoners .  In cognitive science, Bayesian reasoner is the technically precise codeword that we use to mean rational mind.

There are also a number of general heuristics about human reasoning that you can learn from looking at Bayes’ Theorem.

For example, in many discussions of Bayes’ Theorem, you may hear cognitive psychologists saying that people do not take prior frequencies sufficiently into account, meaning that when people approach a problem where there’s some evidence X indicating that condition A might hold true, they tend to judge A’s likelihood solely by how well the evidence X seems to match A, without taking into account the prior frequency of A.  If you think, for example, that under the mammography example, the woman’s chance of having breast cancer is in the range of 70%-80%, then this kind of reasoning is insensitive to the prior frequency given in the problem; it doesn’t notice whether 1% of women or 10% of women start out having breast cancer.  “Pay more attention to the prior frequency!” is one of the many things that humans need to bear in mind to partially compensate for our built-in inadequacies.

A related error is to pay too much attention to p(X|A) and not enough to p(X|~A) when determining how much evidence X is for A.  The degree to which a result X is evidence for A depends, not only on the strength of the statement we’d expect to see result X if A were true, but also on the strength of the statement we wouldn’t expect to see result X if A weren’t true.   For example, if it is raining, this very strongly implies the grass is wet – p(wetgrass|rain) ~ 1 – but seeing that the grass is wet doesn’t necessarily mean that it has just rained; perhaps the sprinkler was turned on, or you’re looking at the early morning dew.  Since p(wetgrass|~rain) is substantially greater than zero, p(rain|wetgrass) is substantially less than one.  On the other hand, if the grass was never wet when it wasn’t raining, then knowing that the grass was wet would always show that it was raining, p(rain|wetgrass) ~ 1 , even if p(wetgrass|rain) = 50% ; that is, even if the grass only got wet 50% of the times it rained.  Evidence is always the result of the differential between the two conditional probabilities.  Strong evidence is not the product of a very high probability that A leads to X, but the product of a very low probability that not-A could have led to X.

The Bayesian revolution in the sciences is fueled, not only by more and more cognitive scientists suddenly noticing that mental phenomena have Bayesian structure in them; not only by scientists in every field learning to judge their statistical methods by comparison with the Bayesian method; but also by the idea that science itself is a special case of Bayes’ Theorem; experimental evidence is Bayesian evidence.   The Bayesian revolutionaries hold that when you perform an experiment and get evidence that “confirms” or “disconfirms” your theory, this confirmation and disconfirmation is governed by the Bayesian rules.  For example, you have to take into account, not only whether your theory predicts the phenomenon, but whether other possible explanations also predict the phenomenon.  Previously, the most popular philosophy of science was probably Karl Popper’s falsificationism – this is the old philosophy that the Bayesian revolution is currently dethroning.  Karl Popper’s idea that theories can be definitely falsified, but never definitely confirmed, is yet another special case of the Bayesian rules; if p(X|A) ~ 1 – if the theory makes a definite prediction – then observing ~X very strongly falsifies A.  On the other hand, if p(X|A) ~ 1 ,  and we observe X, this doesn’t definitely confirm the theory; there might be some other condition B such that p(X|B) ~ 1 , in which case observing X doesn’t favor A over B.  For observing X to definitely confirm A, we would have to know, not that p(X|A) ~ 1 , but that p(X|~A) ~ 0 , which is something that we can’t know because we can’t range over all possible alternative explanations.  For example, when Einstein’s theory of General Relativity toppled Newton’s incredibly well-confirmed theory of gravity, it turned out that all of Newton’s predictions were just a special case of Einstein’s predictions.

You can even formalize Popper’s philosophy mathematically.  The likelihood ratio for X, p(X|A)/p(X|~A) , determines how much observing X slides the probability for A; the likelihood ratio is what says how strong X is as evidence.  Well, in your theory A, you can predict X with probability 1, if you like; but you can’t control the denominator of the likelihood ratio, p(X|~A) – there will always be some alternative theories that also predict X, and while we go with the simplest theory that fits the current evidence, you may someday encounter some evidence that an alternative theory predicts but your theory does not.  That’s the hidden gotcha that toppled Newton’s theory of gravity.  So there’s a limit on how much mileage you can get from successful predictions; there’s a limit on how high the likelihood ratio goes for confirmatory evidence.

On the other hand, if you encounter some piece of evidence Y that is definitely not predicted by your theory, this is enormously strong evidence against your theory.  If p(Y|A) is infinitesimal, then the likelihood ratio will also be infinitesimal.  For example, if p(Y|A) is 0.0001%, and p(Y|~A) is 1%, then the likelihood ratio p(Y|A)/p(Y|~A) will be 1:10000.  -40 decibels of evidence!  Or flipping the likelihood ratio, if p(Y|A) is very small, then p(Y|~A)/p(Y|A) will be very large, meaning that observing Y greatly favors ~A over A.  Falsification is much stronger than confirmation.  This is a consequence of the earlier point that very strong evidence is not the product of a very high probability that A leads to X, but the product of a very low probability that not-A could have led to X.  This is the precise Bayesian rule that underlies the heuristic value of Popper’s falsificationism.

Similarly, Popper’s dictum that an idea must be falsifiable can be interpreted as a manifestation of the Bayesian conservation-of-probability rule; if a result X is positive evidence for the theory, then the result ~X would have disconfirmed the theory to some extent.  If you try to interpret both X and ~X as “confirming” the theory, the Bayesian rules say this is impossible!  To increase the probability of a theory you must expose it to tests that can potentially decrease its probability; this is not just a rule for detecting would-be cheaters in the social process of science, but a consequence of Bayesian probability theory.  On the other hand, Popper’s idea that there is only falsification and no such thing as confirmation turns out to be incorrect.  Bayes’ Theorem shows that falsification is very strong evidence compared to confirmation, but falsification is still probabilistic in nature; it is not governed by fundamentally different rules from confirmation, as Popper argued.

So we find that many phenomena in the cognitive sciences, plus the statistical methods used by scientists, plus the scientific method itself, are all turning out to be special cases of Bayes’ Theorem.  Hence the Bayesian revolution.

Why wait so long to introduce Bayes’ Theorem, instead of just showing it at the beginning?  Well… because I’ve tried that before; and what happens, in my experience, is that people get all tangled up in trying to apply Bayes’ Theorem as a set of poorly grounded mental rules; instead of the Theorem helping, it becomes one more thing to juggle mentally, so that in addition to trying to remember how many women with breast cancer have positive mammographies, the reader is also trying to remember whether it’s p(X|A) in the numerator or p(A|X) , and whether a positive mammography result corresponds to A or X, and which side of p(X|A) is the implication, and what the terms are in the denominator, and so on.  In this excruciatingly gentle introduction, I tried to show all the workings of Bayesian reasoning without ever introducing the explicit Theorem as something extra to memorize, hopefully reducing the number of factors the reader needed to mentally juggle.

Even if you happen to be one of the fortunate people who can easily grasp and apply abstract theorems, the mental-juggling problem is still something to bear in mind if you ever need to explain Bayesian reasoning to someone else.

If you do find yourself losing track, my advice is to forget Bayes’ Theorem as an equation and think about the graph.   p(A) and p(~A) are at the top.  p(X|A) and p(X|~A) are the projection factors.  p(X&A) and p(X&~A) are at the bottom.  And p(A|X) equals the proportion of p(X&A) within p(X&A)+p(X&~A).  The graph isn’t shown here – but can you see it in your mind?

And if thinking about the graph doesn’t work, I suggest forgetting about Bayes’ Theorem entirely – just try to work out the specific problem in gizmos, hoses, and sparks, or whatever it is.

Having introduced Bayes’ Theorem explicitly, we can explicitly discuss its components.

We’ll start with p(A|X).  If you ever find yourself getting confused about what’s A and what’s X in Bayes’ Theorem, start with p(A|X) on the left side of the equation; that’s the simplest part to interpret.  A is the thing we want to know about.  X is how we’re observing it; X is the evidence we’re using to make inferences about A.  Remember that for every expression p(Q|P), we want to know about the probability for Q given P, the degree to which P implies Q – a more sensible notation, which it is now too late to adopt, would be p(Q<-P) .

p(Q|P) is closely related to p(Q&P), but they are not identical.  Expressed as a probability or a fraction, p(Q&P) is the proportion of things that have property Q and property P within all things; i.e., the proportion of “women with breast cancer and a positive mammography” within the group of all women.   If the total number of women is 10,000, and 80 women have breast cancer and a positive mammography, then p(Q&P) is 80/10,000 = 0.8%.  You might say that the absolute quantity, 80, is being normalized to a probability relative to the group of all women.   Or to make it clearer, suppose that there’s a group of 641 women with breast cancer and a positive mammography within a total sample group of 89,031 women.  641 is the absolute quantity.  If you pick out a random woman from the entire sample, then the probability you’ll pick a woman with breast cancer and a positive mammography is p(Q&P), or 0.72% (in this example).

On the other hand, p(Q|P) is the proportion of things that have property Q and property P within all things that have P; i.e., the proportion of women with breast cancer and a positive mammography within the group of all women with positive mammographies.   If there are 641 women with breast cancer and positive mammographies, 7915 women with positive mammographies, and 89,031 women, then p(Q&P) is the probability of getting one of those 641 women if you’re picking at random from the entire group of 89,031, while p(Q|P) is the probability of getting one of those 641 women if you’re picking at random from the smaller group of 7915.

In a sense, p(Q|P) really means p(Q&P|P) , but specifying the extra P all the time would be redundant.  You already know it has property P, so the property you’re investigating is Q – even though you’re looking at the size of group Q&P within group P, not the size of group Q within group P (which would be nonsense).  This is what it means to take the property on the right-hand side as given; it means you know you’re working only within the group of things that have property P.  When you constrict your focus of attention to see only this smaller group, many other probabilities change.  If you’re taking P as given, then p(Q&P) equals just p(Q) – at least, relative to the group P.   The old p(Q), the frequency of “things that have property Q within the entire sample”, is revised to the new frequency of “things that have property Q within the subsample of things that have property P”.  If P is given, if P is our entire world, then looking for Q&P is the same as looking for just Q.

If you constrict your focus of attention to only the population of eggs that are painted blue, then suddenly “the probability that an egg contains a pearl” becomes a different number; this proportion is different for the population of blue eggs than the population of all eggs.  The given, the property that constricts our focus of attention, is always on the right side of p(Q|P); the P becomes our world, the entire thing we see, and on the other side of the “given”  P always has probability 1 – that is what it means to take P as given.  So p(Q|P) means “If P has probability 1, what is the probability of Q?” or “If we constrict our attention to only things or events where P is true, what is the probability of Q?”  Q, on the other side of the given, is not certain – its probability may be 10% or 90% or any other number.  So when you use Bayes’ Theorem, and you write the part on the left side as p(A|X) – how to update the probability of A after seeing X, the new probability of A given that we know X, the degree to which X implies A – you can tell that X is always the observation or the evidence, and A is the property being investigated, the thing you want to know about.

The right side of Bayes’ Theorem is derived from the left side through these steps:

The first step, p(A|X) to p(X&A)/p(X) , may look like a tautology.  The actual math performed is different, though.  p(A|X) is a single number, the normalized probability or frequency of A within the subgroup X.  p(X&A)/p(X) are usually the percentage frequencies of X&A and X within the entire sample, but the calculation also works if X&A and X are absolute numbers of people, events, or things.  p(cancer|positive) is a single percentage/frequency/probability, always between 0 and 1.  (positive&cancer)/(positive) can be measured either in probabilities, such as 0.008/0.103, or it might be expressed in groups of women, for example 194/2494.  As long as both the numerator and denominator are measured in the same units, it should make no difference.

Going from p(X) in the denominator to p(X&A)+p(X&~A) is a very straightforward step whose main purpose is as a stepping stone to the last equation.  However, one common arithmetical mistake in Bayesian calculations is to divide p(X&A) by p(X&~A) , instead of dividing p(X&A) by [p(X&A) + p(X&~A)] .  For example, someone doing the breast cancer calculation tries to get the posterior probability by performing the math operation 80 / 950, instead of 80 / (80 + 950).  I like to think of this as a rose-flowers error.  Sometimes if you show young children a picture with eight roses and two tulips, they’ll say that the picture contains more roses than flowers.  (Technically, this would be called a class inclusion error.)  You have to add the roses and the tulips to get the number of flowers , which you need to find the proportion of roses within the flowers.  You can’t find the proportion of roses in the tulips, or the proportion of tulips in the roses.  When you look at the graph, the bottom bar consists of all the patients with positive results.  That’s what the doctor sees – a patient with a positive result.  The question then becomes whether this is a healthy patient with a positive result, or a cancerous patient with a positive result.  To figure the odds of that, you have to look at the proportion of cancerous patients with positive results within all patients who have positive results, because again, “a patient with a positive result” is what you actually see.  You can’t divide 80 by 950 because that would mean you were trying to find the proportion of cancerous patients with positive results within the group of healthy patients with positive results; it’s like asking how many of the tulips are roses, instead of asking how many of the flowers are roses.  Imagine using the same method to find the proportion of healthy patients.  You would divide 950 by 80 and find that 1,187% of the patients were healthy.  Or to be exact, you would find that 1,187% of cancerous patients with positive results were healthy patients with positive results.

The last step in deriving Bayes’ Theorem is going from p(X&A) to p(X|A)*p(A) , in both the numerator and the denominator, and from p(X&~A) to p(X|~A)*p(~A) , in the denominator.

Why?  Well, one answer is because p(X|A), p(X|~A), and p(A) correspond to the initial information given in all the story problems.  But why were the story problems written that way?

Because in many cases, p(X|A), p(X|~A), and p(A) are what we actually know; and this in turn happens because p(X|A) and p(X|~A) are often the quantities that directly describe causal relations, with the other quantities derived from them and p(A) as statistical relations.   For example, p(X|A), the implication from A to X, where A is what we want to know and X is our way of observing it, corresponds to the implication from a woman having breast cancer to a positive mammography.  This is not just a statistical implication but a directcausal relation; a woman gets a positive mammography because she has breast cancer.  The mammography is designed to detect breast cancer, and it is a fact about the physical process of the mammography exam that it has an 80% probability of detecting breast cancer.  As long as the design of the mammography machine stays constant, p(X|A) will stay at 80%, even if p(A) changes – for example, if we screen a group of woman with other risk factors, so that the prior frequency of women with breast cancer is 10% instead of 1%.  In this case, p(X&A) will change along with p(A), and so will p(X), p(A|X), and so on; but p(X|A) stays at 80%, because that’s a fact about the mammography exam itself.  (Though you do need to test this statement before relying on it; it’s possible that the mammography exam might work better on some forms of breast cancer than others.)  p(X|A) is one of the simple facts from which complex facts like p(X&A) are constructed; p(X|A) is an elementary causal relation within a complex system, and it has a direct physical interpretation.  This is why Bayes’ Theorem has the form it does; it’s not for solving math brainteasers, but for reasoning about the physical universe.

Once the derivation is finished, all the implications on the right side of the equation are of the form p(X|A) or p(X|~A) , while the implication on the left side is p(A|X) .  As long as you remember this and you get the rest of the equation right, it shouldn’t matter whether you happened to start out with p(A|X) or p(X|A) on the left side of the equation, as long as the rules are applied consistently – if you started out with the direction of implication p(X|A) on the left side of the equation, you would need to end up with the direction p(A|X) on the right side of the equation.  This, of course, is just changing the variable labels; the point is to remember the symmetry, in order to remember the structure of Bayes’ Theorem.

The symmetry arises because the elementary causal relations are generally implications from facts to observations, i.e., from breast cancer to positive mammography.  The elementary steps in reasoning are generally implications from observations to facts, i.e., from a positive mammography to breast cancer.  The left side of Bayes’ Theorem is an elementary inferential step from the observation of positive mammography to the conclusion of an increased probability of breast cancer.  Implication is written right-to-left, so we write p(cancer|positive) on the left side of the equation.  The right side of Bayes’ Theorem describes the elementary causal steps – for example, from breast cancer to a positive mammography – and so the implications on the right side of Bayes’ Theorem take the form p(positive|cancer) or p(positive|~cancer) .

And that’s Bayes’ Theorem.  Rational inference on the left end, physical causality on the right end; an equation with mind on one side and reality on the other.  Remember how the scientific method turned out to be a special case of Bayes’ Theorem?  If you wanted to put it poetically, you could say that Bayes’ Theorem binds reasoning into the physical universe.

Okay, we’re done.

If you liked An Intuitive Explanation of Bayesian Reasoning , you may also wish to read A Technical Explanation of Technical Explanation by the same author, which goes into greater detail on the application of Bayescraft to human rationality and the philosophy of science. You may also enjoy the Twelve Virtues of Rationality and The Simple Truth .

Other authors:

E. T. Jaynes:  Probability Theory With Applications in Science and Engineering (full text online).  Theory and applications for Bayes’ Theorem and Bayesian reasoning. See also Jaynes’s magnum opus, Probability Theory: The Logic of Science .

D. Kahneman, P. Slovic and A. Tversky, eds, Judgment under uncertainty:  Heuristics and biases .   If it seems to you like human thinking often isn’t Bayesian… you’re not wrong.  This terrifying volume catalogues some of the blatant searing hideous gaping errors that pop up in human cognition. See also this forthcoming book chapter for a summary of some better-known biases.

Bellhouse, D.R.:  The Reverend Thomas Bayes FRS: a Biography to Celebrate the Tercentenary of his Birth .  A more “traditional” account of Bayes’s life.

Google Directory for Bayesian analysis (courtesy of the Open Directory Project).

An Intuitive Explanation of Bayesian Reasoning is ©2003 by Eliezer S. Yudkowsky .
BayesApplet is ©2003 by Christian Rovner.  (Email address:  Append “tutopia.com” to “cro1@”).

Last updated: 2006.06.04

Yudkowsky’s “Intuitive Explanation of Bayesian Reasoning” and Rovner’s “BayesApplet” may both be freely used by any nonprofit organization or educational institution.  No royalties or per-page charges are necessary to reproduce this document as course materials, either in printed form or online.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/bayes/ .

Thanks to Eric Mitchell, Chris Rovner, Vlad Tarko, Gordon Worley, and Gregg Young for catching errors in the text.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute . If you’ve found Yudkowsky’s pages on rationality useful, please consider donating to the Machine Intelligence Research Institute.

### Bibliography:

Bayes, Thomas (1763):  “An essay towards solving a problem in the doctrine of chances.”  Philosophical Transactions of the Royal Society.  53 : 370-418.

Casscells, W., Schoenberger, A., and Grayboys, T. (1978):  “Interpretation by physicians of clinical laboratory results.” N Engl J Med.299 :999-1001.

Dehaene, Stanislas (1997):  The Number Sense : How the Mind Creates Mathematics.   Oxford University Press.

Eddy, David M. (1982):  “Probabilistic reasoning in clinical medicine:  Problems and opportunities.”  In D. Kahneman, P. Slovic, and A. Tversky, eds, Judgement under uncertainty: Heuristics and biases . Cambridge University Press, Cambridge, UK.

Edwards, Ward (1982):  “Conservatism in human information processing.”  In D. Kahneman, P. Slovic, and A. Tversky, eds, Judgement under uncertainty: Heuristics and biases . Cambridge University Press, Cambridge, UK.

Gigerenzer, Gerd and Hoffrage, Ulrich (1995):  “How to improve Bayesian reasoning without instruction: Frequency formats.”  Psychological Review.102 : 684-704.

Jaynes, E. T. (1996):  Probability Theory With Applications in Science and Engineering.   Posthumous manuscript, placed online.  http://bayes.wustl.edu/etj/science.pdf.html

“I remember this paper I wrote on existentialism. My teacher gave it back with an F. She’d underlined true and truth wherever it appeared in the essay, probably about twenty times, with a question mark beside each. She wanted to know what I meant by truth.”

— Danielle Egan (journalist)

Author’s Foreword:

This essay is meant to restore a naive view of truth.

Someone says to you: “My miracle snake oil can rid you of lung cancer in just three weeks.” You reply: “Didn’t a clinical study show this claim to be untrue?” The one returns: “This notion of ‘truth’ is quite naive; what do you mean by ‘true’?”

Many people, so questioned, don’t know how to answer in exquisitely rigorous detail. Nonetheless they would not be wise to abandon the concept of ‘truth’. There was a time when no one knew the equations of gravity in exquisitely rigorous detail, yet if you walked off a cliff, you would fall.

Often I have seen – especially on Internet mailing lists – that amidst other conversation, someone says “X is true”, and then an argument breaks out over the use of the word ‘true’. This essay is not meant as an encyclopedic reference for that argument. Rather, I hope the arguers will read this essay, and then go back to whatever they were discussing before someone questioned the nature of truth.

In this essay I pose questions. If you see what seems like a really obvious answer, it’s probably the answer I intend. The obvious choice isn’t always the best choice, but sometimes, by golly, it is . I don’t stop looking as soon I find an obvious answer, but if I go on looking, and the obvious-seeming answer still seems obvious, I don’t feel guilty about keeping it. Oh, sure, everyone thinks two plus two is four, everyone says two plus two is four, and in the mere mundane drudgery of everyday life everyone behaves as if two plus two is four, but what does two plus two really, ultimately equal? As near as I can figure, four. It’s still four even if I intone the question in a solemn, portentous tone of voice. Too simple, you say? Maybe, on this occasion, life doesn’t need to be complicated. Wouldn’t that be refreshing?

If you are one of those fortunate folk to whom the question seems trivial at the outset, I hope it still seems trivial at the finish. If you find yourself stumped by deep and meaningful questions, remember that if you know exactly how a system works, and could build one yourself out of buckets and pebbles, it should not be a mystery to you.

If confusion threatens when you interpret a metaphor as a metaphor, try taking everything completely literally.

Imagine that in an era before recorded history or formal mathematics, I am a shepherd and I have trouble tracking my sheep. My sheep sleep in an enclosure, a fold; and the enclosure is high enough to guard my sheep from wolves that roam by night. Each day I must release my sheep from the fold to pasture and graze; each night I must find my sheep and return them to the fold. If a sheep is left outside, I will find its body the next morning, killed and half-eaten by wolves. But it is so discouraging, to scour the fields for hours, looking for one last sheep, when I know that probably all the sheep are in the fold. Sometimes I give up early, and usually I get away with it; but around a tenth of the time there is a dead sheep the next morning.

If only there were some way to divine whether sheep are still grazing, without the inconvenience of looking! I try several methods: I toss the divination sticks of my tribe; I train my psychic powers to locate sheep through clairvoyance; I search carefully for reasons to believe all the sheep are in the fold. It makes no difference. Around a tenth of the times I turn in early, I find a dead sheep the next morning. Perhaps I realize that my methods aren’t working, and perhaps I carefully excuse each failure; but my dilemma is still the same. I can spend an hour searching every possible nook and cranny, when most of the time there are no remaining sheep; or I can go to sleep early and lose, on the average, one-tenth of a sheep.

Late one afternoon I feel especially tired. I toss the divination sticks and the divination sticks say that all the sheep have returned. I visualize each nook and cranny, and I don’t imagine scrying any sheep. I’m still not confident enough, so I look inside the fold and it seems like there are a lot of sheep, and I review my earlier efforts and decide that I was especially diligent. This dissipates my anxiety, and I go to sleep. The next morning I discover two dead sheep. Something inside me snaps, and I begin thinking creatively.

That day, loud hammering noises come from the gate of the sheepfold’s enclosure.

The next morning, I open the gate of the enclosure only a little way, and as each sheep passes out of the enclosure, I drop a pebble into a bucket nailed up next to the door. In the afternoon, as each returning sheep passes by, I take one pebble out of the bucket. When there are no pebbles left in the bucket, I can stop searching and turn in for the night. It is a brilliant notion. It will revolutionize shepherding.

That was the theory. In practice, it took considerable refinement before the method worked reliably. Several times I searched for hours and didn’t find any sheep, and the next morning there were no stragglers. On each of these occasions it required deep thought to figure out where my bucket system had failed. On returning from one fruitless search, I thought back and realized that the bucket already contained pebbles when I started; this, it turned out, was a bad idea. Another time I randomly tossed pebbles into the bucket, to amuse myself, between the morning and the afternoon; this too was a bad idea, as I realized after searching for a few hours. But I practiced my pebblecraft, and became a reasonably proficient pebblecrafter.

One afternoon, a man richly attired in white robes, leafy laurels, sandals, and business suit trudges in along the sandy trail that leads to my pastures.

The man takes a badge from his coat and flips it open, proving beyond the shadow of a doubt that he is Markos Sophisticus Maximus, a delegate from the Senate of Rum. (One might wonder whether another could steal the badge; but so great is the power of these badges that if any other were to use them, they would in that instant be transformed into Markos.)

“Call me Mark,” he says. “I’m here to confiscate the magic pebbles, in the name of the Senate; artifacts of such great power must not fall into ignorant hands.”

“That bleedin’ apprentice,” I grouse under my breath, “he’s been yakkin’ to the villagers again.” Then I look at Mark’s stern face, and sigh. “They aren’t magic pebbles,” I say aloud. “Just ordinary stones I picked up from the ground.”

A flicker of confusion crosses Mark’s face, then he brightens again. “I’m here for the magic bucket!” he declares.

“It’s not a magic bucket,” I say wearily. “I used to keep dirty socks in it.”

Mark’s face is puzzled. “Then where is the magic?” he demands.

An interesting question. “It’s hard to explain,” I say.

My current apprentice, Autrey, attracted by the commotion, wanders over and volunteers his explanation: “It’s the level of pebbles in the bucket,” Autrey says. “There’s a magic level of pebbles, and you have to get the level just right, or it doesn’t work. If you throw in more pebbles, or take some out, the bucket won’t be at the magic level anymore. Right now, the magic level is,” Autrey peers into the bucket, “about one-third full.”

“I see!” Mark says excitedly. From his back pocket Mark takes out his own bucket, and a heap of pebbles. Then he grabs a few handfuls of pebbles, and stuffs them into the bucket. Then Mark looks into the bucket, noting how many pebbles are there. “There we go,” Mark says, “the magic level of this bucket is half full. Like that?”

“No!” Autrey says sharply. “Half full is not the magic level. The magic level is about one-third. Half full is definitely unmagic. Furthermore, you’re using the wrong bucket.”

Mark turns to me, puzzled. “I thought you said the bucket wasn’t magic?”

“It’s not,” I say. A sheep passes out through the gate, and I toss another pebble into the bucket. “Besides, I’m watching the sheep. Talk to Autrey.”

Mark dubiously eyes the pebble I tossed in, but decides to temporarily shelve the question. Mark turns to Autrey and draws himself up haughtily. “It’s a free country,” Mark says, “under the benevolent dictatorship of the Senate, of course. I can drop whichever pebbles I like into whatever bucket I like.”

Autrey considers this. “No you can’t,” he says finally, “there won’t be any magic.”

“Look,” says Mark patiently, “I watched you carefully. You looked in your bucket, checked the level of pebbles, and called that the magic level. I did exactly the same thing.”

“That’s not how it works,” says Autrey.

“Oh, I see,” says Mark, “It’s not the level of pebbles in my bucket that’s magic, it’s the level of pebbles in your bucket. Is that what you claim? What makes your bucket so much better than mine, huh?”

“Well,” says Autrey, “if we were to empty your bucket, and then pour all the pebbles from my bucket into your bucket, then your bucket would have the magic level. There’s also a procedure we can use to check if your bucket has the magic level, if we know that my bucket has the magic level; we call that a bucket compare operation.”

Another sheep passes, and I toss in another pebble.

“He just tossed in another pebble!” Mark says. “And I suppose you claim the new level is also magic? I could toss pebbles into your bucket until the level was the same as mine, and then our buckets would agree. You’re just comparing my bucket to your bucket to determine whether you think the level is ‘magic’ or not. Well, I think your bucket isn’t magic, because it doesn’t have the same level of pebbles as mine. So there!”

“Wait,” says Autrey, “you don’t understand -”

“By ‘magic level’, you mean simply the level of pebbles in your own bucket. And when I say ‘magic level’, I mean the level of pebbles in my bucket. Thus you look at my bucket and say it ’isn’t magic’, but the word ‘magic’ means different things to different people. You need to specify whose magic it is. You should say that my bucket doesn’t have ’Autrey’s magic level’, and I say that your bucket doesn’t have ’Mark’s magic level’. That way, the apparent contradiction goes away.”

“But -” says Autrey helplessly.

“Different people can have different buckets with different levels of pebbles, which proves this business about ‘magic’ is completely arbitrary and subjective.”

“Mark,” I say, “did anyone tell you what these pebbles do? ”

“ Do? ” says Mark. “I thought they were just magic.”

“If the pebbles didn’t do anything,” says Autrey, “our ISO 9000 process efficiency auditor would eliminate the procedure from our daily work.”

“Darwin,” says Autrey.

“Hm,” says Mark. “Charles does have a reputation as a strict auditor. So do the pebbles bless the flocks, and cause the increase of sheep?”

“No,” I say. “The virtue of the pebbles is this; if we look into the bucket and see the bucket is empty of pebbles, we know the pastures are likewise empty of sheep. If we do not use the bucket, we must search and search until dark, lest one last sheep remain. Or if we stop our work early, then sometimes the next morning we find a dead sheep, for the wolves savage any sheep left outside. If we look in the bucket, we know when all the sheep are home, and we can retire without fear.”

Mark considers this. “That sounds rather implausible,” he says eventually. “Did you consider using divination sticks? Divination sticks are infallible, or at least, anyone who says they are fallible is burned at the stake. This is an extremely painful way to die; it follows that divination sticks are infallible.”

“You’re welcome to use divination sticks if you like,” I say.

“Oh, good heavens, of course not,” says Mark. “They work infallibly, with absolute perfection on every occasion, as befits such blessed instruments; but what if there were a dead sheep the next morning? I only use the divination sticks when there is no possibility of their being proven wrong. Otherwise I might be burned alive. So how does your magic bucket work?”

How does the bucket work…? I’d better start with the simplest possible case. “Well,” I say, “suppose the pastures are empty, and the bucket isn’t empty. Then we’ll waste hours looking for a sheep that isn’t there. And if there are sheep in the pastures, but the bucket is empty, then Autrey and I will turn in too early, and we’ll find dead sheep the next morning. So an empty bucket is magical if and only if the pastures are empty -”

“Hold on,” says Autrey. “That sounds like a vacuous tautology to me. Aren’t an empty bucket and empty pastures obviously the same thing?”

“It’s not vacuous,” I say. “Here’s an analogy: The logician Alfred Tarski once said that the assertion ‘Snow is white’ is true if and only if snow is white. If you can understand that, you should be able to see why an empty bucket is magical if and only if the pastures are empty of sheep.”

“Hold on,” says Mark. “These are buckets . They don’t have anything to do with sheep . Buckets and sheep are obviously completely different. There’s no way the sheep can ever interact with the bucket.”

“Then where do you think the magic comes from?” inquires Autrey.

Mark considers. “You said you could compare two buckets to check if they had the same level… I can see how buckets can interact with buckets. Maybe when you get a large collection of buckets, and they all have the same level, that’s what generates the magic. I’ll call that the coherentist theory of magic buckets.”

“Interesting,” says Autrey. “I know that my master is working on a system with multiple buckets – he says it might work better because of ‘redundancy’ and ‘error correction’. That sounds like coherentism to me.”

“They’re not quite the same -” I start to say.

“Let’s test the coherentism theory of magic,” says Autrey. “I can see you’ve got five more buckets in your back pocket. I’ll hand you the bucket we’re using, and then you can fill up your other buckets to the same level -”

Mark recoils in horror. “Stop! These buckets have been passed down in my family for generations, and they’ve always had the same level! If I accept your bucket, my bucket collection will become less coherent, and the magic will go away!”

“But your current buckets don’t have anything to do with the sheep!” protests Autrey.

Mark looks exasperated. “Look, I’ve explained before, there’s obviously no way that sheep can interact with buckets. Buckets can only interact with other buckets.”

“I toss in a pebble whenever a sheep passes,” I point out.

“When a sheep passes, you toss in a pebble?” Mark says. “What does that have to do with anything?”

“It’s an interaction between the sheep and the pebbles,” I reply.

“No, it’s an interaction between the pebbles and you ,” Mark says. “The magic doesn’t come from the sheep, it comes from you . Mere sheep are obviously nonmagical. The magic has to come from somewhere , on the way to the bucket.”

I point at a wooden mechanism perched on the gate. “Do you see that flap of cloth hanging down from that wooden contraption? We’re still fiddling with that – it doesn’t work reliably – but when sheep pass through, they disturb the cloth. When the cloth moves aside, a pebble drops out of a reservoir and falls into the bucket. That way, Autrey and I won’t have to toss in the pebbles ourselves.”

Mark furrows his brow. “I don’t quite follow you… is the cloth magical?”

I shrug. “I ordered it online from a company called Natural Selections. The fabric is called Sensory Modality.” I pause, seeing the incredulous expressions of Mark and Autrey. “I admit the names are a bit New Agey. The point is that a passing sheep triggers a chain of cause and effect that ends with a pebble in the bucket. Afterward you can compare the bucket to other buckets, and so on.”

“I still don’t get it,” Mark says. “You can’t fit a sheep into a bucket. Only pebbles go in buckets, and it’s obvious that pebbles only interact with other pebbles.”

“The sheep interact with things that interact with pebbles…” I search for an analogy. “Suppose you look down at your shoelaces. A photon leaves the Sun; then travels down through Earth’s atmosphere; then bounces off your shoelaces; then passes through the pupil of your eye; then strikes the retina; then is absorbed by a rod or a cone. The photon’s energy makes the attached neuron fire, which causes other neurons to fire. A neural activation pattern in your visual cortex can interact with your beliefs about your shoelaces, since beliefs about shoelaces also exist in neural substrate. If you can understand that, you should be able to see how a passing sheep causes a pebble to enter the bucket.”

“At exactly which point in the process does the pebble become magic?” says Mark.

“It… um…” Now I’m starting to get confused. I shake my head to clear away cobwebs. This all seemed simple enough when I woke up this morning, and the pebble-and-bucket system hasn’t gotten any more complicated since then. “This is a lot easier to understand if you remember that the point of the system is to keep track of sheep.”

Mark sighs sadly. “Never mind… it’s obvious you don’t know. Maybe all pebbles are magical to start with, even before they enter the bucket. We could call that position panpebblism.”

“Ha!” Autrey says, scorn rich in his voice. “Mere wishful thinking! Not all pebbles are created equal. The pebbles in your bucket are not magical. They’re only lumps of stone!”

Mark’s face turns stern. “Now,” he cries, “now you see the danger of the road you walk! Once you say that some people’s pebbles are magical and some are not, your pride will consume you! You will think yourself superior to all others, and so fall! Many throughout history have tortured and murdered because they thought their own pebbles supreme!” A tinge of condescension enters Mark’s voice. “Worshipping a level of pebbles as ‘magical’ implies that there’s an absolute pebble level in a Supreme Bucket. Nobody believes in a Supreme Bucket these days.”

“One,” I say. “Sheep are not absolute pebbles. Two, I don’t think my bucket actually contains the sheep. Three, I don’t worship my bucket level as perfect – I adjust it sometimes – and I do that because I care about the sheep.”

“Besides,” says Autrey, “someone who believes that possessing absolute pebbles would license torture and murder, is making a mistake that has nothing to do with buckets. You’re solving the wrong problem.”

Mark calms himself down. “I suppose I can’t expect any better from mere shepherds. You probably believe that snow is white, don’t you.”

“Um… yes?” says Autrey.

“It doesn’t bother you that Joseph Stalin believed that snow is white?”

“Um… no?” says Autrey.

Mark gazes incredulously at Autrey, and finally shrugs. “Let’s suppose, purely for the sake of argument, that your pebbles are magical and mine aren’t. Can you tell me what the difference is?”

“My pebbles represent the sheep!” Autrey says triumphantly. “ Your pebbles don’t have the representativeness property, so they won’t work. They are empty of meaning. Just look at them. There’s no aura of semantic content; they are merely pebbles. You need a bucket with special causal powers.”

“Ah!” Mark says. “Special causal powers, instead of magic.”

“Exactly,” says Autrey. “I’m not superstitious. Postulating magic, in this day and age, would be unacceptable to the international shepherding community. We have found that postulating magic simply doesn’t work as an explanation for shepherding phenomena. So when I see something I don’t understand, and I want to explain it using a model with no internal detail that makes no predictions even in retrospect, I postulate special causal powers. If that doesn’t work, I’ll move on to calling it an emergent phenomenon.”

“What kind of special powers does the bucket have?” asks Mark.

“Hm,” says Autrey. “Maybe this bucket is imbued with an about-ness relation to the pastures. That would explain why it worked – when the bucket is empty, it means the pastures are empty.”

“Where did you find this bucket?” says Mark. “And how did you realize it had an about-ness relation to the pastures?”

“It’s an ordinary bucket ,” I say. “I used to climb trees with it… I don’t think this question needs to be difficult.”

“I’m talking to Autrey,” says Mark.

“You have to bind the bucket to the pastures, and the pebbles to the sheep, using a magical ritual – pardon me, an emergent process with special causal powers – that my master discovered,” Autrey explains.

Autrey then attempts to describe the ritual, with Mark nodding along in sage comprehension.

“You have to throw in a pebble every time a sheep leaves through the gate?” says Mark. “Take out a pebble every time a sheep returns?”

Autrey nods. “Yeah.”

“That must be really hard,” Mark says sympathetically.

Autrey brightens, soaking up Mark’s sympathy like rain. “Exactly!” says Autrey. “It’s extremely hard on your emotions. When the bucket has held its level for a while, you… tend to get attached to that level.”

A sheep passes then, leaving through the gate. Autrey sees; he stoops, picks up a pebble, holds it aloft in the air. “Behold!” Autrey proclaims. “A sheep has passed! I must now toss a pebble into this bucket, my dear bucket, and destroy that fond level which has held for so long – ” Another sheep passes. Autrey, caught up in his drama, misses it; so I plunk a pebble into the bucket. Autrey is still speaking: ” – for that is the supreme test of the shepherd, to throw in the pebble, be it ever so agonizing, be the old level ever so precious. Indeed, only the best of shepherds can meet a requirement so stern -“

“Autrey,” I say, “if you want to be a great shepherd someday, learn to shut up and throw in the pebble. No fuss. No drama. Just do it.”

“And this ritual,” says Mark, “it binds the pebbles to the sheep by the magical laws of Sympathy and Contagion, like a voodoo doll.”

Autrey winces and looks around. “Please! Don’t call it Sympathy and Contagion. We shepherds are an anti-superstitious folk. Use the word ‘intentionality’, or something like that.”

“Can I look at a pebble?” says Mark.

“Sure,” I say. I take one of the pebbles out of the bucket, and toss it to Mark. Then I reach to the ground, pick up another pebble, and drop it into the bucket.

Autrey looks at me, puzzled. “Didn’t you just mess it up?”

I shrug. “I don’t think so. We’ll know I messed it up if there’s a dead sheep next morning, or if we search for a few hours and don’t find any sheep.”

“But -” Autrey says.

“I taught you everything you know, but I haven’t taught you everything I know,” I say.

Mark is examining the pebble, staring at it intently. He holds his hand over the pebble and mutters a few words, then shakes his head. “I don’t sense any magical power,” he says. “Pardon me. I don’t sense any intentionality.”

“A pebble only has intentionality if it’s inside a ma- an emergent bucket,” says Autrey. “Otherwise it’s just a mere pebble.”

“Not a problem,” I say. I take a pebble out of the bucket, and toss it away. Then I walk over to where Mark stands, tap his hand holding a pebble, and say: “I declare this hand to be part of the magic bucket!” Then I resume my post at the gates.

Autrey laughs. “Now you’re just being gratuitously evil.”

I nod, for this is indeed the case.

“Is that really going to work, though?” says Autrey.

I nod again, hoping that I’m right. I’ve done this before with two buckets, and in principle, there should be no difference between Mark’s hand and a bucket. Even if Mark’s hand is imbued with the elan vital that distinguishes live matter from dead matter, the trick should work as well as if Mark were a marble statue.

Mark is looking at his hand, a bit unnerved. “So… the pebble has intentionality again, now?”

“Yep,” I say. “Don’t add any more pebbles to your hand, or throw away the one you have, or you’ll break the ritual.”

Mark nods solemnly. Then he resumes inspecting the pebble. “I understand now how your flocks grew so great,” Mark says. “With the power of this bucket, you could keep in tossing pebbles, and the sheep would keep returning from the fields. You could start with just a few sheep, let them leave, then fill the bucket to the brim before they returned. And if tending so many sheep grew tedious, you could let them all leave, then empty almost all the pebbles from the bucket, so that only a few returned… increasing the flocks again when it came time for shearing… dear heavens, man! Do you realize the sheer power of this ritual you’ve discovered? I can only imagine the implications; humankind might leap ahead a decade – no, a century!”

“It doesn’t work that way,” I say. “If you add a pebble when a sheep hasn’t left, or remove a pebble when a sheep hasn’t come in, that breaks the ritual. The power does not linger in the pebbles, but vanishes all at once, like a soap bubble popping.”

Mark’s face is terribly disappointed. “Are you sure?”

I nod. “I tried that and it didn’t work.”

Mark sighs heavily. “And this… math … seemed so powerful and useful until then… Oh, well. So much for human progress.”

“Mark, it was a brilliant idea,” Autrey says encouragingly. “The notion didn’t occur to me, and yet it’s so obvious… it would save an enormous amount of effort… there must be a way to salvage your plan! We could try different buckets, looking for one that would keep the magical pow- the intentionality in the pebbles, even without the ritual. Or try other pebbles. Maybe our pebbles just have the wrong properties to have inherent intentionality. What if we tried it using stones carved to resemble tiny sheep? Or just write ‘sheep’ on the pebbles; that might be enough.”

“Not going to work,” I predict dryly.

Autrey continues. “Maybe we need organic pebbles, instead of silicon pebbles… or maybe we need to use expensive gemstones. The price of gemstones doubles every eighteen months, so you could buy a handful of cheap gemstones now, and wait, and in twenty years they’d be really expensive.”

“You tried adding pebbles to create more sheep, and it didn’t work?” Mark asks me. “What exactly did you do?”

“I took a handful of dollar bills. Then I hid the dollar bills under a fold of my blanket, one by one; each time I hid another bill, I took another paperclip from a box, making a small heap. I was careful not to keep track in my head, so that all I knew was that there were ‘many’ dollar bills, and ‘many’ paperclips. Then when all the bills were hidden under my blanket, I added a single additional paperclip to the heap, the equivalent of tossing an extra pebble into the bucket. Then I started taking dollar bills from under the fold, and putting the paperclips back into the box. When I finished, a single paperclip was left over.”

“What does that result mean?” asks Autrey.

“It means the trick didn’t work. Once I broke ritual by that single misstep, the power did not linger, but vanished instantly; the heap of paperclips and the pile of dollar bills no longer went empty at the same time.”

“You actually tried this?” asks Mark.

“Yes,” I say, “I actually performed the experiment, to verify that the outcome matched my theoretical prediction. I have a sentimental fondness for the scientific method, even when it seems absurd. Besides, what if I’d been wrong?”

“If it had worked,” says Mark, “you would have been guilty of counterfeiting! Imagine if everyone did that; the economy would collapse! Everyone would have billions of dollars of currency, yet there would be nothing for money to buy!”

“Not at all,” I reply. “By that same logic whereby adding another paperclip to the heap creates another dollar bill, creating another dollar bill would create an additional dollar’s worth of goods and services.”

Mark shakes his head. “Counterfeiting is still a crime… You should not have tried.”

“I was reasonably confident I would fail.”

“Aha!” says Mark. “You expected to fail! You didn’t believe you could do it!”

“Indeed,” I admit. “You have guessed my expectations with stunning accuracy.”

“Well, that’s the problem,” Mark says briskly. “Magic is fueled by belief and willpower. If you don’t believe you can do it, you can’t. You need to change your belief about the experimental result; that will change the result itself.”

“Funny,” I say nostalgically, “that’s what Autrey said when I told him about the pebble-and-bucket method. That it was too ridiculous for him to believe, so it wouldn’t work for him.”

“How did you persuade him?” inquires Mark.

“I told him to shut up and follow instructions,” I say, “and when the method worked, Autrey started believing in it.”

Mark frowns, puzzled. “That makes no sense. It doesn’t resolve the essential chicken-and-egg dilemma.”

“Sure it does. The bucket method works whether or not you believe in it.”

“That’s absurd! ” sputters Mark. “I don’t believe in magic that works whether or not you believe in it!”

“I said that too,” chimes in Autrey. “Apparently I was wrong.”

Mark screws up his face in concentration. “But… if you didn’t believe in magic that works whether or not you believe in it, then why did the bucket method work when you didn’t believe in it? Did you believe in magic that works whether or not you believe in it whether or not you believe in magic that works whether or not you believe in it?”

“I don’t… think so…” says Autrey doubtfully.

“Then if you didn’t believe in magic that works whether or not you… hold on a second, I need to work this out on paper and pencil -” Mark scribbles frantically, looks skeptically at the result, turns the piece of paper upside down, then gives up. “Never mind,” says Mark. “Magic is difficult enough for me to comprehend; metamagic is out of my depth.”

“Mark, I don’t think you understand the art of bucketcraft,” I say. “It’s not about using pebbles to control sheep. It’s about making sheep control pebbles. In this art, it is not necessary to begin by believing the art will work. Rather, first the art works, then one comes to believe that it works.”

“Or so you believe,” says Mark.

“So I believe,” I reply, “ because it happens to be a fact. The correspondence between reality and my beliefs comes from reality controlling my beliefs, not the other way around.”

Another sheep passes, causing me to toss in another pebble.

“Ah! Now we come to the root of the problem,” says Mark. “What’s this so-called ‘reality’ business? I understand what it means for a hypothesis to be elegant, or falsifiable, or compatible with the evidence. It sounds to me like calling a belief ‘true’ or ‘real’ or ‘actual’ is merely the difference between saying you believe something, and saying you really really believe something.”

I pause. “Well…” I say slowly. “Frankly, I’m not entirely sure myself where this ‘reality’ business comes from. I can’t create my own reality in the lab, so I must not understand it yet. But occasionally I believe strongly that something is going to happen, and then something else happens instead. I need a name for whatever-it-is that determines my experimental results, so I call it ‘reality’. This ‘reality’ is somehow separate from even my very best hypotheses. Even when I have a simple hypothesis, strongly supported by all the evidence I know, sometimes I’m still surprised. So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies ‘belief’, and the latter thingy ‘reality’.”

Mark snorts. “I don’t even know why I bother listening to this obvious nonsense. Whatever you say about this so-called ‘reality’, it is merely another belief. Even your belief that reality precedes your beliefs is a belief. It follows, as a logical inevitability, that reality does not exist; only beliefs exist.”

“Hold on,” says Autrey, “could you repeat that last part? You lost me with that sharp swerve there in the middle.”

“No matter what you say about reality, it’s just another belief,” explains Mark. “It follows with crushing necessity that there is no reality, only beliefs.”

“I see,” I say. “The same way that no matter what you eat, you need to eat it with your mouth. It follows that there is no food, only mouths.”

“Precisely,” says Mark. “Everything that you eat has to be in your mouth. How can there be food that exists outside your mouth? The thought is nonsense, proving that ‘food’ is an incoherent notion. That’s why we’re all starving to death; there’s no food.”

Autrey looks down at his stomach. “But I’m not starving to death.”

“ Aha! ” shouts Mark triumphantly. “And how did you utter that very objection? With your mouth , my friend! With your mouth ! What better demonstration could you ask that there is no food?”

“ What’s this about starvation? ” demands a harsh, rasping voice from directly behind us. Autrey and I stay calm, having gone through this before. Mark leaps a foot in the air, startled almost out of his wits.

Inspector Darwin smiles tightly, pleased at achieving surprise, and makes a small tick on his clipboard.

“Just a metaphor!” Mark says quickly. “You don’t need to take away my mouth, or anything like that -”

“ Why do you need a mouth if there is no food ?” demands Darwin angrily. “ Never mind. I have no time for this foolishness . I am here to inspect the sheep. ”

“Flocks thriving, sir,” I say. “No dead sheep since January.”

“ Excellent. I award you 0.12 units of fitness . Now what is this person doing here? Is he a necessary part of the operations? ”

“As far as I can see, he would be of more use to the human species if hung off a hot-air balloon as ballast,” I say.

“Ouch,” says Autrey mildly.

“I do not care about the human species . Let him speak for himself .”

Mark draws himself up haughtily. “This mere shepherd ,” he says, gesturing at me, “has claimed that there is such a thing as reality. This offends me, for I know with deep and abiding certainty that there is no truth. The concept of ‘truth’ is merely a stratagem for people to impose their own beliefs on others. Every culture has a different ‘truth’, and no culture’s ‘truth’ is superior to any other. This that I have said holds at all times in all places, and I insist that you agree.”

“Hold on a second,” says Autrey. “If nothing is true, why should I believe you when you say that nothing is true?”

“I didn’t say that nothing is true -” says Mark.

“Yes, you did,” interjects Autrey, “I heard you.”

“- I said that ‘truth’ is an excuse used by some cultures to enforce their beliefs on others. So when you say something is ‘true’, you mean only that it would be advantageous to your own social group to have it believed.”

“And this that you have said,” I say, “is it true?”

“Absolutely, positively true!” says Mark emphatically. “People create their own realities.”

“Hold on,” says Autrey, sounding puzzled again, “saying that people create their own realities is, logically, a completely separate issue from saying that there is no truth, a state of affairs I cannot even imagine coherently, perhaps because you still have not explained how exactly it is supposed to work -”

“There you go again,” says Mark exasperatedly, “trying to apply your Western concepts of logic, rationality, reason, coherence, and self-consistency.”

“Great,” mutters Autrey, “now I need to add a third subject heading, to keep track of this entirely separate and distinct claim -”

“It’s not separate,” says Mark. “Look, you’re taking the wrong attitude by treating my statements as hypotheses, and carefully deriving their consequences. You need to think of them as fully general excuses, which I apply when anyone says something I don’t like. It’s not so much a model of how the universe works, as a “Get Out of Jail Free” card. The key is to apply the excuse selectively . When I say that there is no such thing as truth, that applies only to your claim that the magic bucket works whether or not I believe in it. It does not apply to my claim that there is no such thing as truth.”

“Um… why not?” inquires Autrey.

Mark heaves a patient sigh. “Autrey, do you think you’re the first person to think of that question? To ask us how our own beliefs can be meaningful if all beliefs are meaningless? That’s the same thing many students say when they encounter this philosophy, which, I’ll have you know, has many adherents and an extensive literature.”

“So what’s the answer?” says Autrey.

“We named it the ‘reflexivity problem’,” explains Mark.

“But what’s the answer ?” persists Autrey.

Mark smiles condescendingly. “Believe me, Autrey, you’re not the first person to think of such a simple question. There’s no point in presenting it to us as a triumphant refutation.”

“But what’s the actual answer? ”

“Now, I’d like to move on to the issue of how logic kills cute baby seals -”

“ You are wasting time ,” snaps Inspector Darwin.

“Not to mention, losing track of sheep,” I say, tossing in another pebble.

Inspector Darwin looks at the two arguers, both apparently unwilling to give up their positions. “Listen,” Darwin says, more kindly now, “I have a simple notion for resolving your dispute. You say,” says Darwin, pointing to Mark, “that people’s beliefs alter their personal realities. And you fervently believe,” his finger swivels to point at Autrey, “that Mark’s beliefs can’t alter reality. So let Mark believe really hard that he can fly, and then step off a cliff. Mark shall see himself fly away like a bird, and Autrey shall see him plummet down and go splat, and you shall both be happy.”

We all pause, considering this.

“It sounds reasonable…” Mark says finally.

“There’s a cliff right there,” observes Inspector Darwin.

Autrey is wearing a look of intense concentration. Finally he shouts: “Wait! If that were true, we would all have long since departed into our own private universes, in which case the other people here are only figments of your imagination – there’s no point in trying to prove anything to us -”

A long dwindling scream comes from the nearby cliff, followed by a dull and lonely splat. Inspector Darwin flips his clipboard to the page that shows the current gene pool and pencils in a slightly lower frequency for Mark’s alleles.

Autrey looks slightly sick. “Was that really necessary?”

“ Necessary? ” says Inspector Darwin, sounding puzzled. “It just happened … I don’t quite understand your question.”

Autrey and I turn back to our bucket. It’s time to bring in the sheep. You wouldn’t want to forget about that part. Otherwise what would be the point?

This document is ©2008 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/the-simple-truth/ .

If you enjoyed this writing, let your journey continue with An Intuitive Explanation of Bayesian Reasoning . You may also enjoy The Twelve Virtues of Rationality and A Technical Explanation of Technical Explanation

CognitiveBiases-1

This document is ©2007 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/cognitive-biases/ .

From August 2007 through May 2009, I blogged daily on the topic of human rationality at the econblog Overcoming Bias by Robin Hanson, getting around a quarter-million monthly pageviews. This then forked off the community blog Less Wrong , and I moved my old posts there as well for seed content (with URL forwarding, so don’t worry if links are to overcomingbias.com).

I suspected I could write faster by requiring myself to publish daily. The experiment was a smashing success.

Currently the majority of all my writing is on Less Wrong. To be notified when and if this material is compacted into e-books (or even physical books), subscribe to this announcement list .

The material is heavily interdependent, and reading in chronological order may prove helpful:

To see how interdependent it is, try looking over this graph of the dependency structure:

To read organized collections of posts, use the Sequences on the Less Wrong wiki.

This document is ©2008 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/overcoming-bias/ .