Multinomial distribution

Definition of the multinomial distribution

The binomial distribution B(n, p) is obtained by considering n tosses of a coin (Heads and Tails), with p being the probability that the coin will land on a Head.

The multinomial distribution is a generalization of the binomial distribution : each "toss" can now produce more than just two outcomes. For example, on may imagine rolling a "die" with k faces, with p_i being the probability for the die to land on face r_i.

A multinomial distribution is entirely characterized by :

    * n, the number of times the die is rolled.

    * The set of probabilities {p₁, p₂ , ..., p_k}       with    p₁ + p₂ + ...,  p_k = 1

-----

After having rolled the die n times, we denote by n_i the number of times the die landed face r_i up. We therefore have n₁ + n₂ + ...,  n_k = n. Because rolling a die is a random activity, the n_is are the realizations of k random variables that we denote N_i (i = 1, 2, ..., k). These variables are not independent, as they are linked by the relation S_i N_i  = n.

The multinomial distribution Mult(n, p₁, p₂ , ..., p_k ) is the joint distribution of the k random variables N_i. It is therefore a multivariate, discrete distribution. Its support is the set of k-uplets of non negative integers {n₁, n₂, ...,  n_k} such that n₁ + n₂ + ...+ n_k = n.

The multinomial probability distribution

The distribution Mult(n, p₁, p₂ , ..., p_k ) is determined by the values of the probabilities of each of the possible k-uplets. We denote these probabilities by P{N₁ = n₁, ..., N_k = n_k }.

We'll show that :

for all the k-uplets within the support of the distribution (and 0 otherwise).

-----

The binomial distribution is a special case with k = 2.

Multinomial coefficient

The term n! / (n₁!.n₂!...n_k!)  is called the multinomial coefficient. It is the number of different "words" that can be written with an alphabet containing k letters by using the first letter n₁ times, the second letter n₂ times etc...

We'll justify this result two different ways.

-----

Note that the multinomial coefficient is equal to the coefficient of the monom xⁿ¹xⁿ² ...x^nk in the development of  (x₁ + x₂ + ...+ x_k )ⁿ , hence the name of the distribution.

Marginal distributions of the multinomial distribution

Recall that N_i is the number of observations in the modality r_i.

Distribution

Consider N_i. After each draw :

    * The result is r_i with probability p_i.

    * And therefore the result is "Not  r_i" (that is, any other modality) with probability (1 - p_i).

Therefore N_i is binomial B(n, p_i).

Mean

N_i being B(n, p_i), we have :

Variance

N_i being B(n, p_i), we have :

Covariances

We'll give two demonstrations of the following result :

All the covariances are negative : for a given number of draws n, any increase of n_i will be conducive, on the average, to a reduction of the number of observations in any other modality.

Recall that we gave here a third demonstration of this result by calling on the Theorem of iterated expectation.

-----

Because of the constraint :

S_i N_i  = n

the covariance matrix of the multinomial distribution is not full rank : its rank is k - 1.

Correlation coefficient

By reporting to the definition of the correlation coefficient between two variables, we have :

Note that this expression does not contain n.

Conditional distributions of the multinomial distribution

Let us partition the set of variables {N_i} into two groups. For ease of exposition, let k' be an integer smaller than k, and consider :

    * A first group containing the first k' variables {N₁, N₂ , ..., N_k'}.

    * A second group containing the last k' - k variables {N_k'+1, N_k'+2 , ..., N_k}.

We now impose the condition  N₁+ N₂ + ...+ N_k' = z (and therefore N_k'+1+ N_k'+2 + ...+ N_k = n - z). In other words, we consider only these series of n draws that lead to N₁+ N₂ + ...+ N_k' = z, and ignore the others.

Then it can be shown that the joint distribution of {N₁, N₂ , ..., N_k'} conditionally to N₁+ N₂ + ...+ N_k' = z  in the multinomial distribution Mult(z, p'₁, p'₂ , ..., p'_k') with :

p'_i = p_i /(p₁+ p₂ + ... + p_k')

with a similar result for the other group.

Moment generating function of the multinomial distribution

The moment generating function of the multinomial distribution Mult(n, p₁, p₂ , ..., p_k ) is :

Fusing modalities

Let X = {X₁, X₂, ..., X_k} be a set of k random variables distributed as Mult(n, p₁, p₂ , ..., p_k ).

Then X^* = {X₁ + X₂, ..., X_k} is a set of (k - 1) r.v. distributed as Mult(n, p₁+ p₂ , ..., p_k ).

Why ?

This result generalizes easily to any partitioning of the set of variables into groups of variables.

Link with the Poisson distribution

There is an intimate link between the multinomial distribution and the Poisson distribution. We showed here that if {N₁, N₂, ..., N_k} are k independent (but not necessarily identically distributed) Poisson variables, then the joint distribution of {N₁, N₂, ..., N_k} conditionally to their sum is a multinomial distribution.

___________________________________________________
 

We calculate the probability mass function of the multinomial distribution. The key part is establishing the expression of the multinomial coefficient, which we do two by different methods.

PROBABILITY MASS FUNCTION OF THE MULTINOMIAL DISTRIBUTION

________________________________________________________

We now use two different methods for calculating Cov(N_i, N_j ), the covariance of the numbers of observations in two modalities of the multinomial distribution.

The second method represents these (random) numbers as sums of auxiliary Bournoulli variables. This approach is often quite effective with discrete probability distributions problems (see for example the calculation of the mean of the hypergeometric distribution).

COVARIANCES OF THE MULTINOMIAL DISTRIBUTION

 ____________________________________________

Related readings