Likelihood Ratio Test  (LRT)

A general and powerful method for building tests.

-----

Likelihood was invented for the purpose of quantifying the fit between a probability distribution and a sample : the larger the likelihood, the better the fit.

On the other side, tests are meant to discriminate between two nonoverlapping groups of distributions that both claim to contain the distribution that generated the sample.

One may therefore anticipate the possibility to use the likelihood of the sample for the various competing distributions to settle the issue of which group is more likely to contain the distribution that generated the sample.

Vocabulary and notations

Let's first clarify some notations commonly used when dealing with Likelihood Ratio tests.

    * First, all the distributions we'll consider belong to the same family, the various distributions in the family differing only though the value of a parameter θ (which may be a vector parameter, i.e. a set of several scalar parameters). For example, we may consider the family of normal distributions N(µ, σ²), of which each member is fully characterized by the values of µ and σ².

    * The two groups of distributions are then defined respectively by the null hypothesis and the alternative hypothesis. For example, we might want to test :

        - H₀ : µ = µ₀

against

        - H₁ : µ ≠ µ₀

The hypothesis we'll consider can be indifferently simple or composite (whereas the Neyman-Pearson lemma applies only to two simple hypothesis).

In what follows, it will be convenient to consider that :

    * H₀ does not just denote the null hypothesis, but the set of values of the parameter θ defined by H₀ as well and, by extension, the set of distributions defined by this set of values of the parameter.

    * H₁ does not just denote the alternative hypothesis, but the set of values of the parameter θ defined by H₁ as well and, by extension, the set of distributions defined by this set of values of the parameter.

So the notation H₀ U H₁ will designate the set of the values of the parameter θ defined by either H₀ or H₁ and, by extension, the set of distributions defined by this set of values of the parameter.

Principle of Likelihood Ratio tests

The Likelihood Ratio Test (LRT) approach reasons as follows.

    1) Suppose that H₀ is true : the distribution that generated the sample belongs indeed to H₀. We certainly expect the sample to exhibit a large likelihood for the distribution that generated it, and consequently we expect this likelihood to be close to the largest likelihood encountered when scanning through all the distributions in H₀.

Considering the distributions in H₁ will probably change nothing : none of these distributions generated the sample, so none of these distributions is expected to display a large likelihood for the sample.

Consequently the largest likelihood in H₀ is not anticipated to be substantially smaller than the largest likelihood observed over the complete set of distributions H₀ U H₁.

This can be formalized as follows. Denote :

        - max_{H 0} L(x; θ) the largest likelihood observed over H₀.

        - max_{H 0 U H 1} L(x; θ) the largest likelihood observed over H₀ U H₁.

Then the ratio

λ = max_{H 0} L(x; θ) / max_{H 0 U H 1} L(x; θ)  

which is certainly smaller than 1, is expected to be close to 1.

    2) Conversely, suppose that H₀ is false (and therefore that H₁ is true). The distribution that generated the sample belongs to H₁, and not to H₀. None of the distributions in H₀ is anticipated to exhibit a large likelihood. The largest likelihood of all is anticipated to be found for a distribution in H₁ because the distribution that generated the sample is in H₁.

Consequently, the ratio

λ = max_{H 0} L(x; θ) / max_{H 0 U H 1} L(x; θ)  

although always positive, is expected to have a small value (close to 0).

The test

So it appears that the statistic :

is a plausible choice for testing H₀ against H₁ : small values (close to 0) of Λ favor H₁, while large values (close to 1) of Λ favor H₀.

Suppose that if H₀ is true, the probability distribution g(λ) of the random variable Λ is known and that the integral

can be calculated (or at least tabulated) and the result inverted so that for a given α, c can be determined explicitely .

We then have all the ingredients needed for testing H₀ against H₁ at the α significance level (upper and lower images of this illustration) :

The decision made by the test is then :

    * If λ > c then accept H₀ and reject H₁.

    * If λ ≤ c then accept H₁ and reject H₀.

Distribution of the test statistic

Unfortunately, the test statistic Λ and its distribution g(λ) are usually very complicated (see for example Bartlett's test).

Functions of the statistic

It is sometimes possible to identify a function f(.) such that f(Λ) is more tractable. One can then carry out a test equivalent to the original test by using

f(Λ) instead of Λ as a test statistic.

The transformation may be :

    * Reasonably intuitive, as in here,

    * Or require a good deal of effort to be identified, as in here.

Asymptotic distribution

It can be shown that, under some regularity conditions, the r.v. -2.logΛ is asymptotically χ² distributed (i.e. for large samples). The number of degrees of freedom of the distribution is equal to the difference between :

        - The number of free parameters in H₀ U H₁,

        - And the number of free parameters in H₀.

This deserves a little explanation. Suppose that an LRT is designed for the purpose of testing :

        - H₀ : µ = µ₀

against

        - H₁ : µ ≠ µ₀

for normal distributions where the variance σ² assumed to be known.

    * In H₀, the likelihood has a certain value determined by the sample, and no parameter is available to make this likelihood change. So H₀ has no (i.e. 0) free parameter.

    * In H₀ U H₁ the likelihood will be maximized over the range of µ, which is unconstrained over ]-∞, +∞[. Therefore, H₀ U H₁ has one free parameter (that is, µ).

The difference between the numbers of free parameters of H₀ U H₁ and H₀ is therefore 1 - 0 = 1.

Similarly, if the same hypothesis are tested in a setting where the variance is not assumed to be known :

        - H₀ has one free parameter, namely σ²,

        - While H₀ U H₁ has two free parameters, namely µ and σ²,

and the difference between the numbers of free parameters of H₀ U H₁ and H₀ is therefore 2 - 1 = 1.

-----

The term "free" makes reference to the fact that the effective number of parameters (or "number of free parameters") may be smaller than the actual number of parameters. For example, suppose that we consider the sub-family of the family of normal distributions N(µ, σ²) defined by µ² = σ². Although both the mean and the variance of the distributions are free to vary across their entire respective ranges, the sub-family is described by a single "free parameter".

The number of free parameters may be regarded as the intrinsic dimension of the manifold bearing all the allowed values of the parameters linked by constraining relationships.

The "regularity conditions" refered to earlier in this paragraph say that the manifold should have no singularity (like self-crossing) and no boundary (like a half plane). We'll see an example where this second condition is not met, with, as a result, the asymptotic distribution of -2.logΛ being clearly not χ².

-----

The quality of the fit between the distribution of -2.logΛ for finite samples and the corresponding χ² limit distribution is of course of great concern to the analyst. A first experimental approach to this important question is given as an interactive animation in one of the Tutorials below.

-----

Note that once it is known that the asymptotic distribution of -2.logΛ is χ², the asymptotic distribution of Λ can easily be infered (see animation). But because tables of quantiles exist for the χ² distribution only, it is customary to base a Likelihood Ratio Test on the value of -2.logΛ and not on that of Λ.

___________________

Results concerning the χ² nature of the asymptotic distribution of -2.logΛ are difficult and are stated without proof. We'll only demonstrate the simplest of them, which is as follows :

    * For the pair of hypothesis H₀ : θ = θ₀ against H₁ : θ ≠ θ₀ 

    * -2logΛ is asymptotically distributed as χ²₁,

and that alone will require some effort on our part.

Likelihood Ratio Test and Maximum Likelihood Estimation

There is a clear connection between LRTs and Maximum Likelihood Estimation (MLE).

    * The denominator of the test statistic is

max_{H 0 U H 1} L(x; θ)

We are looking for the value of θ that will make the likelihood of the sample largest over the complete domain of θ. This value is, by definition, the Maximum Likelihood Estimate of θ. Denote θ^* the Maximum Likelihood Estimator of θ : the denominator now can be written as L(x; θ^*).

    * Similarly, the numerator of the test statistic is

max_{H 0} L(x; θ)

Again, we are looking for the Maximum Likelihood Estimation of θ over the sub-domain of θ defined by H₀. Denote θ^*₀  the corresponding estimator. The numerator is now L(x; θ^*₀).

-----

The LRT test statistic Λ is therefore given by

Likelihood Ratio Test and Sufficiency

Since a statistic that is sufficient for θ contains all the information in the sample pertaining to the estimation of θ, it should be considered as a distinct possibility that a LRT can be expressed not just in terms of the sample X, but also in the seemingly more restrictive terms of a sufficient statistic T(X).

It is indeed the case and we'll show that, given any sufficient statistic T(X), the Λ(X) statistic of any LRT can be expressed as a function of T only. In other words, there is always a function Λ^*(.) such that

Λ(X) = Λ^*(T(X))

for any sample X.

As a consequence, we'll see that when a sufficient statistic for θ is available, LRTs can be built using this sufficient statistic only.

Likelihood Ratio Test and Neyman-Pearson lemma

We defined a LRT by considering the ratio of the maximum likelihood over H₀ to the maximum likelihood over the complete parameter space H₀ U H₁. We might as well (and perhaps more naturally) have considered the ratio of the maximum likelihood over H₀ to the maximum likelihood over H₁ only, with a similar line of reasoning and similar results (except, of course, the asymptotic distribution of the test statistic).

This alternative presentation of Likelihood Ratio Tests has the advantage of making obvious the connection between LRTs and the Neyman-Pearson lemma. For suppose that H₀ is the simple hypothesis θ = θ₀ and that H₁ is the simple hypothesis θ = θ₁. The "sets" of distributions H₀ and H₁ then both contain a single distribution, the "maximization" procedures at the numerator and denominator of the test statistic are trivial, and we are now within the framework of the Neyman-Pearson lemma. Recall that the best critical region of a N-P test is defined by

L(x; θ₁ ) / L(x; θ₀ )  >  k_α

which is exactly the expression of the special case of a LRT when both the null and the alternative hypothesis are simple.

So it appears that Likelihood Ratio Tests may be regarded as generalizations of the Neyman-Pearson test to situations where at least one hypothesis is composite.

-----

Note that whereas a Neyman-Pearson test is always a Uniformly Most Powerful (UMP) test, it is in general not the case of Likelihood Ratio Tests.

Likelihood Ratio Tests and nested models

A model M₁ is said to be nested inside another model M₂ if the set of parameters of M₁ is a subset of the set of parameters of M₂ :

    * M₂ may be regarded as M₁ to which some additional parameters have been added for the purpose of attempting to improve its predictive capacity.

    * Conversely, M₁ may be regarded as M₂ from which some parameters have been deleted for the purpose of making it easier to interpret and more stable (see bias-variance tradeoff).

The question is then : "Are the two models significantly different ?". The Likelihood Ratio paradigm is a natural approach for devising a test meant to answer the question.

Denote (β₁, β₂, ..., β_p) the set of parameters of M₁, and (β₁, β₂, ..., β_p, β_{p + 1}, ..., β_{p + q} ) the set of parameters of M₂. Model M₁ may be regarded as model M₂ whose last q parameters are constrained to be equal to 0.

Using the same convention as before, we have :

    * H₀ : β_{p + 1} = ... = β_{p + q} = 0    (M₁ is nested in M₂)

    * H₁ : At least one of these parameters is different from 0 (M₁ is not nested in M₂).

The test statistic Λ will be the ratio of the likelihood :

    * Of model M₁,

    * And of model M₂.

Since M₂ will always have a larger likelihood than M₁, the value of this ratio is always between 0 and 1. A value of Λ close to 0 means that the likelihood of M₂ is substantially larger than than that of M₁, and therefore that the models are significantly different.

-----

The simplest example of a Likelihood Ratio Test on nested models is to be found in Multiple Linear Regression within the context of variable selection. But the reader is warned that we balk at the complexity of the calculations, and elaborate the test from simpler (but logically equivalent) geometric considerations instead.

Caveat

 Likelihood Ratio Tests are popular because :

    * They are intuitively appealing,

    * They often coincide with classical tests that were originally designed by more direct methods (see Tutorial).

    * The standard asymptotic probability distribution of the test statistic Λ is a bonus as the probability distribution of a LRT statistic is often intractable..

Yet, none of these reasons is compelling. Further investigations of the properties of LRTs show that a LRT may behave poorly (low power) compared to a "hand-crafted" test specially designed for the problem at hand. This point is well illustrated by the comparative study of :

    * The goodness-of-fit Likelihood Ratio Test for the multinomial distribution,

    * And the more conventional Chi-square test serving the same purpose.

So, just as Maximum Likelihood Estimation should not systematically be considered as the default method for parameter estimation, Likelihood Ratio Tests are only to be considered as a convenient but not universal method for building a test.

________________________________________________________________________

In this Tutorial, we use the Likelihood Ratio paradigm to :

    * First build two tests about the value of the mean of the normal distribution. The first test assumes that the variance is known, while the second test will not assume the variance to be known.

    * Then build a test about the value of the variance of the normal distribution with unknown mean.

The test statistics obtained by respecting a strict compliance to the LRT methodology are usually a bit complicated and have messy distributions. But we'll be able to transform these statistics into new random variables whose distributions could more easily be calculated.

But we won't even have to calculate these distributions for some reasoning about these new variables will allow us to establish that these three tests are in fact equivalent respectively to :

    * The one sample t-test when the variance is known (also known as "z-test"),

    * The one sample t-test when the variance is unknown.

    * The F-test when the mean is unknown.

EXAMPLES OF LIKELIHOOD RATIO TESTS

_________________________________________

The results of the preceding Tutorial are both encouraging and disappointing :

    * Using only the basic principles of the Likelihood Ratio Test, we were able to build important tests in a somewhat mechanical way which did not require a deep understanding of the problem at hand.

    * But these tests turned out to be equivalent to very classical tests (z, t, F), so that the LRTs seemed in the final analysis to bring nothing new except cumbersome calculations.

So in order to confirm the usefulness of the concept of LRT, it remains to build a LRT that has no classical equivalent. This is what we do now.

In this Tutorial, we again consider the normal distribution N(µ, σ²) with µ and σ² both unknown, and we want to test :

    * H₀ : µ ≤ µ₀ 

against

    * H₁ : µ > µ₀

No modification of a classical test can handle this pair of hypothesis, and we therefore devise a LRT from the ground up. This endeavor is difficult enough to justify a Tutorial of its own. In particular, Λ will at first seem to be hopelessly beyond tractability, and only a very smart transformation of this statistic will allow us to successfully conclude the building of the test.

-----

We'll also see that we are in a situation where the asymptotic distribution of -2logΛ is definitely not a χ² distribution, but we'll only touch upon the reasons why it is so.

AN ORIGINAL LIKELIHOOD RATIO TEST

_________________________________________

We now move on to some more LRTs.

    * First, we elaborate a test meant to test the null hypothesis θ = θ₀ vs the alternative hypothesis θ ≠ θ₀ for the uniform distribution U[0, θ].

    * We then address a somewhat similar problem pertaining to the value of the parameter θ of the shifted exponential distribution exp[-(x - θ)].

In both problems the likelihood function is discontinuous, and maximization of the likelihood cannot be achieved by differentiation.

________________

We finally develop three versions of the LRT identity test for the exponential distribution Exp(λ) : samples are drawn from several different exponential populations, and the question is whether these populations are identically distributed (same value of the parameter λ).

    * We first treat the basic problem as we just stated it. We take this opportunity to explore experimentally the quality of the fit between a

-2.logΛ statistic and its asymptotic χ² distribution with an interactive animation.

The animation uses 2, 3 or 4 samples (whose sizes are adjustable) so that the three basic shapes of the χ² distribution are examined in turn :

        - χ²₁ with a vertical asymptote,

        - χ²₂ which is just an exponential distribution,

        - χ²₃ with its well known asymetric bell-shaped curve (see here).

We'll see that in all three cases, the basic geometry of the limit distribution is respected by the empirical distribution of the statistic, and that the quality of the fit visibly improves when more and more observations are drawn from the exponential distribution.

-----

    * We then introduce a realistic difficulty : observations smaller than a certain threshold γ cannot be measured. The distributions are therefore "truncated" and the data "censored".

We'll therefore have to first calculate the distribution of the censored data. Only then shall we be able to generalize the preceding identity test to truncated exponential distributions.

-----

    * In fact, we'll show that a (weaker) identity test can still be devised if γ is not known provided that it is the same for all the distributions.

MORE EXAMPLES OF LIKELIHOOD RATIO TESTS

__________________________________________________

Most Likelihood Ratio Tests are salvaged from intractability only by the fact that -2logΛ is asymptotically χ² distributed. This life-saver is rather weak, though, as is any asymptotic result used as an approximation of an exact but unknown result for finite samples (see animation in the preceding Tutorial). Besides, establishing this asymptotic behavior is no easy matter.

We'll first give an example where it can be shown directly that -2logΛ is asymptotically χ²₁ distributed. We'll build the LRT statistic Λ of H₀ : λ = λ₀ against H₁ : λ ≠ λ₀ for the Poisson(λ) distribution, then calculate the asymptotic distribution of -2logΛ and show that this distribution is indeed χ²₁.

We'll then generalize this result and show that for any distribution p(x; θ), if :

    * H₀ : θ = θ₀ 

    * H₁ : θ ≠ θ₀ 

then -2logΛ is asymptotically χ²₁ distributed.

It can be shown that the result extends to any compatible pair of hypothesis (H₀, H₁), provided that the regularity conditions mentioned earlier are respected.

The proof will be given for a single scalar parameter θ only. It can be shown (difficult and not treated) that a similar result also holds for vector parameters θ. The important and difficult part is to establish that the number of degrees of freedom of the limit χ² distribution is equal to the difference between :

    * The number of free parameters of the denominator of the test statistic Λ,

    * And the number of free parameters of the numerator of this statistic.

____________________

We conclude this Tutorial by showing that the statistic Λ of a Likelihood Ratio Test can always be expressed as a function of any sufficient statistic T(X). This allows building a LRT by using only a sufficient statistic T(X) when one is available. The calculations are then usually simpler than that of the standard LRT, as we'll show by revisiting two examples of LRTs developed in the preceding Tutorials.

ASYMPTOTIC RESULTS
LRT AND SUFFICIENCY

__________________________________________________

We conclude with yet another Likelihood Ratio Test : the goodness-of-fit test for a multinomial distribution. This test is not the most complicated or most important we've built along these Tutorials, but it enjoys a somewhat special status.

Recall that the so-called "Chi-square tests" (goodness-of-fit, identity, independence) are very important non parametric tests. They all rely on the "Pearson's Chi-square" statistic whose exact distribution under H₀ is usually unknown, but whose asymptotic distribution is χ², hence the generic name of the tests. The statistic also finds its origin in the basic goodness-of-fit problem for the multinomial distribution, but, contrary to the LRT statistic described in this Tutorial, it was specifically designed for this purpose. The -2logΛ statistic of the multinomial goodness-of-fit LRT is sometimes called "Wilks' G²".

From a theoretical standpoint, the two tests are asymptotically equivalent : the general results about the χ² asymptotic distribution of the -2logΛ statistic of a LRT will show us that this distribution is the same as the asymptotic distribution of Pearson's Chi-square. In addition, we'll even show directly that

-2logΛ converges in distribution to the Pearson's Chi-square statistic.

But the analyst is more interested in another question : "For finite samples, which one of the two tests is more powerful ?". To our knowledge, this question is intractable, but it can be experimentally adressed by a simulation that we present at the end of this Tutorial. This simulation compares the behaviors and performances of the Chi-2 and G² statistics under various conditions.

MULTINOMIAL GOODNESS-OF-FIT LRT

______________________________________________________

Related readings :