Bayes' theorem

Bayes' theorem is a rule for changing your mind when new evidence turns up. It sounds abstract, but it fixes a mistake almost everyone makes.

Here is the mistake. A test being accurate is not the same as a positive result being trustworthy. Those feel like the same thing, and they are not. Whether you should believe a positive result depends on how rare the thing is that you are testing for.

Say a disease is very rare, so only 1 person in 10,000 has it. You take a good test and it comes back positive. Should you panic? Probably not yet. Because the disease is so rare, almost everyone taking the test is healthy, and even a small slip-up rate among all those healthy people produces a lot of false alarms. Those false alarms can easily outnumber the true cases, so most positives are wrong.

Bayes' theorem is the recipe for combining the two things that matter: how good the test is, and how common the disease is. Get both into the sum and you find the real chance that a positive result means what you fear. Play with the sliders above and watch the answer swing.

The theorem itself is one short line:

\[ P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}. \]

Read it as: the probability of \(A\) given that you saw \(B\) equals how likely \(B\) was if \(A\) were true, times how likely \(A\) was to begin with, divided by how likely \(B\) was overall. In the working language of the subject, that is posterior = (likelihood × prior) / evidence.

The medical test, worked through. Take a disease with 1% prevalence, a test with 99% sensitivity (it catches 99% of sick people) and 95% specificity (it clears 95% of healthy people, so it falsely flags 5%). Now picture 10,000 people. About 100 are sick, and the test finds 99 of them. The other 9,900 are healthy, and 5% of those, which is 495 people, test positive by mistake. So a positive result lands on 99 + 495 = 594 people, but only 99 of them are truly sick.

That makes the chance a positive is real about \(99/594 \approx 0.167\), only about 17%. The test is genuinely good, yet a positive is wrong five times out of six, purely because the disease is rare and the healthy majority manufactures so many false positives. Counting in whole people out of 10,000, rather than juggling percentages, is what makes this land. This blind spot has a name: the base-rate fallacy.

The same three words, prior, likelihood, posterior, show up far beyond medicine. Spam filters weigh how often a word appears in junk mail against how common junk is. Courts wrestle with the same trap under the name the prosecutor's fallacy. And most of modern machine learning, from a simple naive Bayes classifier upward, is Bayes' theorem doing bookkeeping at scale.

Where it comes from. The theorem is a two-line consequence of the product rule. The probability that both \(A\) and \(B\) happen can be sliced two ways, \(P(A\cap B) = P(A\mid B)\,P(B) = P(B\mid A)\,P(A)\). Set the two right-hand sides equal and divide by \(P(B)\) and Bayes' theorem drops out. The denominator is usually expanded with the law of total probability, \(P(B) = P(B\mid A)\,P(A) + P(B\mid \lnot A)\,P(\lnot A)\), which is exactly the "true positives plus false positives" sum from the worked example.

The odds form is cleaner. Divide the theorem for \(A\) by the theorem for \(\lnot A\) and the awkward denominator cancels, leaving posterior odds = prior odds × likelihood ratio. The likelihood ratio for a positive test is sensitivity divided by the false-positive rate, here \(0.99/0.05 \approx 19.8\). Prior odds of 1 to 99 times 19.8 give posterior odds near 1 to 5, the same 17% in disguise. This form makes sequential updating trivial: each new independent piece of evidence just multiplies in its own likelihood ratio, so today's posterior is tomorrow's prior.

What the probabilities mean. There is a genuine split here. A frequentist reads \(P\) as a long-run frequency, so a fixed unknown (does this patient have the disease?) either is or is not true and has no probability. A Bayesian reads \(P\) as a degree of belief, which can attach to a one-off fact and be updated by evidence. That reading is what lets Bayes' theorem run as an engine of inference. The cost is the prior: you must state your belief before the data, and for well-behaved cases a conjugate prior keeps the posterior in the same family and the algebra closed. Where the prior is weak, conclusions can be sensitive to it, which is a feature to be examined, not hidden.

Two cautions worth keeping. Cromwell's rule says never assign a prior of exactly 0 or 1 to anything you could conceivably be wrong about, because no evidence can ever move a probability off those pegs. And the prosecutor's fallacy is the base-rate trap in a courtroom: the chance of the evidence given innocence, \(P(E\mid H)\), is not the chance of innocence given the evidence, \(P(H\mid E)\). Swapping them, and quietly dropping the base rate of guilt in the relevant population, has convicted innocent people. The same machinery scales up into naive Bayes classifiers and Bayesian networks that reason over many linked variables at once.