The discrete case

We introduce Bayes' theorem within the context of Classification. Suppose you have to assign observations to classes on the basis of the value measured on one attribute x. This attribute is first assumed to be discrete, and we will denote its values x^(l).

We define the prior probability of class C_k as the fraction of all observations that belong to that class (irrespective of the value of x). We denote it P{C_k}.
In a similar way, we define the unconditional probability P{x^(l)} as the fraction of all observations for which x = x^(l), irrespective of the observations' class labels.

_____

Now consider only those observations that are in a given C_k . The class-conditional probability P{x^(l)| C_k}of x^(l) is the proportion of all observations in class C_k for which the measured value of x is x^(l). For a given class, it is a function of l.
In a similar way, consider only those observations with x = x^(l) . The posterior probability P{C_k | x^(l)} of class C_k is the fraction of these observations that are in C_k. For a given x^(l), it is a function of k.
Before any measurement on x was performed on the new observation, our best guess the an observation to belong to C_k was P{C_k}. After the measurement is made, more information on the observation is available, and we can "tune" our estimate from P{C_k} to P{C_k | x^(l)}. This explains the terms "prior" and "posterior".

Bayes' theorem establishes the connection between these quantities. It states that:

So what Bayes' theorem says is:

"You want to know what class a new observation belongs to by measuring the value of attribute x? You'll never know for sure, but you can improve on your initial guesses P{C_k} by calculating the posterior probability P{C_k | x^(l)} of each class, as they now take into account the result of the measurement on x.

For this purpose, for each class:

* Take the prior probability P{C_k} of the class. It depends only on the class label.

* Multiply it by P{x^(l) | C_k}, the class-conditional probability of the measured value x^(l) of the attribute,

* and divide the result by P{x^(l)}, the unconditional probability of the the measured value of the attribute. This last number is the same for all classes, and depends only on l".

__________________

The denominator P{x^(l)} plays the role of a normalization factor. To see that, sum all the posterior probabilities across the classes:

as an observation certainly belongs to one class (even if you don't know which one), and one only (exhaustive and mutually exclusive events).

Replace each term in the sum by its value as given by Bayes' formula to obtain:

Now Bayes' formula can be written:

where the normalizing role of the denominator becomes clear.

The continuous case

If the attribute is continuous, Bayes' formula still holds with few changes. The definition of the class priors remains the same. But:

* Unconditional probabilities P{x^(l)} have to be replaced by the unconditional probability density p(x), which is the probability density function of x if class labels are ignored.

* Class-conditional probabilities P{x^(l) | C_k} have to be replaced by class-conditional probability densities p(x | C_k). Each one is the p.d.f. of x within one class when all other classes are ignored.

and Bayes' formula now reads:

The general formulation

Bayes' theorem is not limited to Classification problems. In fact, its most general formulation is to be found in the basis of Probability Theory, and does not explicitly refer to Classification or any other application. It states that:

* If A is an event,

* and {B₁, B₂, ..., B_n} are events that are both exhaustive and mutually exclusive,

then, for any i:

One can think of P{B_i} as our initial guess about the likelihood of the event B_i. Now if event A occurs, then we should modify our guess from P{B_i} to

P{B_i | A }.

________________________________________________

Download this Glossary