Multinomial distribution

Definition of the multinomial distribution

The binomial distribution B(n, p) is obtained by considering n tosses of a coin (Heads and Tails), with p being the probability that the coin will land on a Head.

The multinomial distribution is a generalization of the binomial distribution : each "toss" can now produce more than just two outcomes. For example, one may imagine rolling a "die" with k faces, with p_i being the probability for the die to land on face A_i.

A multinomial distribution is entirely characterized by :

    * n, the number of times the die is rolled.

    * The set of probabilities {p₁, p₂ , ..., p_k}       with    p₁ + p₂ + ...+ p_k = 1

-----

After rolling the die n times, we denote n_i the number of times the die landed face A_i up. We therefore have n₁ + n₂ + ...+  n_k = n. Because rolling a die is a random activity, the n_is are the realizations of k random variables that we denote X_i (i = 1, 2, ..., k). These variables are not independent, as they are linked by the relation Σ_i X_i  = n.

The multinomial distribution Mult(n, p₁, p₂ , ..., p_k ) is the joint distribution of the k random variables X_i. It is therefore a multivariate, discrete distribution. Its support is the set of k-uples of non negative integers {n₁, n₂, ...,  n_k} such that n₁ + n₂ + ...+ n_k = n.

The multinomial probability distribution

The distribution Mult(n; p₁, p₂ , ..., p_k ) is determined by the values of the probabilities of each of the possible k-uples. We denote these probabilities by P{X₁ = n₁, ..., X_k = n_k }.

We'll show that :

for all the k-uples within the support n₁ + n₂ + ... + n_k  = n of the distribution (and 0 otherwise).

-----

The binomial distribution is a special case with k = 2.

Multinomial coefficient

The term n! / (n₁!.n₂!...n_k!)  is called the multinomial coefficient. It is the number of different "words" that can be written with an alphabet containing k letters by using the first letter n₁ times, the second letter n₂ times etc...

We'll justify this result two different ways.

-----

Note that the multinomial coefficient is equal to the coefficient of the monom xⁿ¹xⁿ² ...x^nk in the development of  (x₁ + x₂ + ...+ x_k )ⁿ , hence the name of the distribution.

Moment generating function of the multinomial distribution

The moment generating function of the multinomial distribution Mult(n, p₁, p₂ , ..., p_k ) is :

Mean

We show below that for any i, X_i follows the binomial distribution B(n, p_i).

It follows immediately that

Covariance matrix of the multinomial distribution

The multinomial distribution is the distribution of a random vector X = {X₁, X₂ , ..., X_k}. It is therefore appropriate to calculate its covariance matrix.

Variances

We show below that for any i, X_i follows the binomial distribution B(n, p_i ).

Hence

Covariances

We'll give two demonstrations of the following result :

All the covariances are negative : for a given number of draws n, any increase of n_i will be conducive, on the average, to a reduction of the number of observations in any other category.

We gave here a third demonstration of this result by calling on the Theorem of iterated expectation.

-----

Because of the constraint Σ_i X_i  = n, the covariance matrix of the multinomial distribution is not full rank, a fact that we'll establish directly.

Correlation coefficients

By reporting to the definition of the correlation coefficient between two variables, we have :

Note that this expression does not contain n.

Merging categories

Suppose that the first r variables of the random vector X = {X₁, X₂ , ..., X_k} are replaced by the single variable Y defined as the sum of these variables :

Y = X₁ + X₂ + ... + X_r

The first r categories {A₁, ..., A_r } are said to be merged.

Because the X_is are linked by X₁ + X₂ + ... + X_k = n, we also have Y = n - (X_{r + 1} + X_{r + 2} + ... + X_k ).

We'll show that

In other words :

    1) Merge categories,

    2) Then assign to this new category a probability equal to the sum of the probabilities of the merged cargories,

and you have the multinomial distribution of the new category plus the remaining categories.

Marginal distributions of the multinomial distribution

Recall that X_i is the number of observations in the category A_i.

Single component

Consider category A_i. The outcome of each draw :

    * Is A_i with probability p_i.

    * And therefore is "Not  A_i" (that is, any other category) with probability (1 - p_i).

Hence X_i is binomial B(n, p_i).

Several components

Things are a bit more complicated when more than one component are considered.

Given the multinomial distribution Mult(n; p₁, p₂ , ..., p_k ), we'll calculate the joint distribution of {X₁, X₂, ..., X_r}, the set of the r first components and discover that this distribution is not multinomial.

Because this result is a bit awkward, litterature sometimes defines the marginal distributions of the multinomial distribution as the distribution of the augmented vector {X₁, X₂, ..., X_r, n - (X₁ + X₂ + ... + X_r)} which is multinomial (see preceding paragraph).

-----

This result generalizes straightforwardly to any group of components.

Conditional distributions of the multinomial distribution

Let the vector {X₁, X₂ , ..., X_k} follow the multinomial distribution Mult(n; p₁, p₂ , ..., p_k ).

We consider the distribution of the r first components {X₁, X₂ , ..., X_r} of the vector conditionally to the values of the last (k - r) components. In other words, we want the distribution of the variable :

{X₁, X₂ , ..., X_r | X_{r + 1} = n_{r + 1}, X_{r + 2} = n_{r + 2}, ..., X_k = n_{k + 2} } 

We'll show that this distribution is multinomial Mult(m; p'₁, p'₂ , ..., p'_r ) with :

    * m = n - (n_{r + 1} + n_{r + 2} + ... + n_{k + 2} )

    * p'_i = p_i /(p₁+ p₂ + ... + p_r )     

-----

This result generalizes straightforwardly to any group of components. 

Estimation of the parameters p_i

We'll show that the Maximum Likelihood Estimator (MLE) p_i^* of the parameter p_i is :

Goodness-of-fit test

The Chi-square tests (goodness-of-fit, identity, independence, ...) are very important non parametric tests. They are all basically the same test, which is a goodness-of-fit test for the multinomial distribution. The test statistic is the so-called "Pearson's Chi-square", which we demonstrate below to be asymptotically χ² distributed. This result is fundamental.

-----

Note that the basic Chi-square test is not the only goodness-of-fit test for the multinomial distribution. We build here the Likelihood Ratio Test which serves exactly the same purpose.

Link with the Poisson distribution

There is an intimate link between the multinomial distribution and the Poisson distribution. We show here that if {X₁, X₂, ..., X_k} are k independent (but not necessarily identically distributed) Poisson variables, then the joint distribution of {X₁, X₂, ..., X_k} conditionally to their sum is a multinomial distribution.

___________________________________________________
 

We calculate the probability mass function of the multinomial distribution. The key part is establishing the expression of the multinomial coefficient, which we do two by different methods.

We then calculate the Maximum Likelihood Estimator (MLE) p_i^* of the parameter p_i. This calculation is a simple exercise in constrained optimization by the method of Lagrangian multipliers.

PROBABILITY MASS FUNCTION OF THE MULTINOMIAL DISTRIBUTION

________________________________________________________

We now use two different methods for calculating Cov(X_i, X_j ), the covariance of the numbers of observations in two categories of the multinomial distribution.

The second method represents these (random) numbers as sums of auxiliary Bournoulli variables. This approach is often quite effective with discrete probability distributions problems (see for example the calculation of the mean of the hypergeometric distribution).

COVARIANCE MATRIX OF THE MULTINOMIAL DISTRIBUTION

_________________________________________________________

In this Tutorial :

    1) We first establish the distribution of the vector {Y, X_{r + 1}, X_{r + 2}, ..., X_k ) where Y is the sum of the r first components of X :

Y = X₁ + X₂ + ..., X_r 

We'll show that this distribution is Mult(n; p, p_{r +1}, ..., p_k ) with p = p₁ + p₂ + ..., p_r.

    2) We then calculate the joint distribution of {X₁, X₂, ..., X_r }, the genuine marginal distribution of the multinomial distribution. This distribution will turn out not to be multinomial, so the distribution of the "augmented" vector {X₁, X₂, ..., X_r, n - (X₁ + X₂ + ..., X_r)}, whose distribution is multinomial (see preceding paragraph) is sometimes presented as the marginal distribution of X.

    3) We finally calculate the joint distribution of a group of categories conditionally to the values of the other categories, and show that this distribution is multinomial (see here).

MERGED CATEGORIES

MARGINAL AND CONDITIONAL DISTRIBUTIONS

_________________________________

In this Tutorial, we demonstrate the fundamental Pearson's theorem which is the base on which all Chi-square tests are built. The theorem states that the so-called "Pearson's Chi-square statistic", which is the statistic common to all Chi-square tests, is asymptotically χ² distributed.

Although Linear Algebra can wrap-up the issue in a few compact lines, we'll develop a longer but elementary method which permits following the proof step by step.

An alternative to the Chi-square statistic is Wilk's G² statistic, whose behavior is compared here to that of the Chi-square statistic.

PEARSON'S THEOREM

______________________________________________________

Related readings :