Confidence interval

We suggest that you first read the entry on "Interval Estimation".

-----

Reminder on Interval Estimation

Let E = (x₁, ..., x_n) be a sample from an unknown distribution. Let also q be a parameter of this distribution, and q_* an estimate of the value of this parameter. It is sometimes possible to identify an interval such that one can assert that this interval "covers" the (unknown) true value q₀ of the parameter with a certain given probability P.

This interval is then called a confidence interval for the estimate q_*. Its endpoints are random variables that depend only on the sample (and are therefore "statistics").

The width of the confidence interval is a measure of the uncertainty about the the position of the true value q₀ of the estimated parameter.

Confidence level

The probability P is arbitrarily chosen by the analyst. It is called the confidence level for the confidence interval, and is denoted by (1 - a). The most frequently chosen values for a are 0.05 and 0.01, corresponding to 95% and 99% confidence level.

So, if one chooses a = 0.05, the corresponding confidence interval has a 0.95 probability to cover the true value q₀ of the parameter.

-----

For a given sample, the width of the confidence interval depends on the chosen confidence level.

• If one wants to increase the probability for the interval to cover the true value q₀ of the parameter, then the width of the interval will increase. This is because the only way to increase our belief that q₀ is in a certain region is to make the region larger.

• Conversely, if one wants a shorter confidence interval (for a given sample), then the confidence level will decrease, meaning that as we consider smaller regions, our belief that q₀ lies in this regions decreases.

Sample size

For a given confidence level, the endpoints of the confidence interval depend on the sample size : the larger the sample, the shorter the confidence interval. This is not surprising : the larger the sample, the more information we have about the distribution, and therefore about the parameter q, and the smaller is the uncertainty about its true value q₀.

Given a sample, the only leeway available to the analyst is the tradeoff between confidence level and confidence interval width. If he tries to impose a higher confidence level, the confidence interval will become wider : the only way to increase the probability for q₀ to be inside the interval is then to make the interval wider.

But if observations can be obtained at a reasonable cost, then it is possible to:

• Choose a confidence level 1 - a,
• Choose a confidence interval width,

and then determine the minimal sample size needed to meet both constraints.

This question is addressed in the first Tutorial below.

Shortest confidence interval

For a given confidence level, there is an infinity of confidence intervals that share this confidence level (lower image of the illustration below).

Which one should we choose ?

Clearly, the best choice is that of the shortest interval, that is, the interval that most narrows down the region where the true value q₀ is expected to lie. For the most classical pivots, (standard normal and t distributions), it can be shown that the shortest confidence intervals are symmetric with respect to the estimated value. So, the systematic (and often unexplained) use of these intervals is fully justified.

Asymmetric intervals

Pivots are not necessarily symmetric. For example, the pivot relative to the variance of a normal distribution is a  variable, whose distribution is not symmetric. It is then customary to build "equal-tailed" confidence intervals, that is, such that a/2 areas are defined on either end of the distribution curve of the pivot. The confidence interval is then asymetric (lower image of the illustration below).

There is no reason why these equal-tailed intervals should be the shortest confidence intervals, and in fact, they are not. Identifying the shortest confidence interval can then only be done by iterative optimization techniques.

Approximate confidence intervals

Few exact confidence intervals are known. So, most of the time, it is necessary to seek formulas that yield approximate confidence intervals with a reasonable level of accuracy. There are two main ways of obtaining approximate confidence intervals :

Asymptotic intervals

            Even when no exact confidence interval is known, it is often possible to identify an asymptotic confidence interval, that is, an approximate confidence interval whose accuracy gets better and better when sample size gets larger and larger. For very large samples an asymptotic interval is almost an exact confidence interval.

This approach relies on identifying a quantity that is not a pivot for finite size samples, but whose distribution tends towards a limit distribution that does not depend on the value of the estimated parameter as the sample size grows without limit.

This limit distribution is then used for building a confidence interval, even (and incorrectly so) for finite samples.

Approximate formulas

            For moderate size samples, an asymptotic confidence intervals may be grossly wrong. A better solution is then to forego the limit pivotal quantity, and rather to find a genuine pivot whose distribution will be a good approximation of the distribution of the statistic used for defining the asymptotic pivot as in the previous paragraph.

The most classical example of this approach is Welch's approximation. It is used for building approximate confidence intervals on the difference of the means of two independent normal distributions of unknown and different variances.

Welch's approximation is addressed below.

Multiple confidence intervals

It is sometimes possible to define not just one confidence interval for one parameter, but a 2-D confidence region pertaining to a couple of parameters considered simultaneously.

For exemple, one can define a confidence region for the pair (m, s²) of sample drawn from a normal distribution N(µ, s²).

Parameters of a model

A major application of interval estimation is to calculate confidence intervals for the estimated values of the parameters of a model (for example, see here).

A large confidence interval  on the estimated value of a parameter means that a there is a large uncertainty about the true value of this parameter. The two main causes for this large uncertainty are :

• Model overparametrization, that renders parameters values and model predictions very sensitive to the particular sample used for building the model.
• Coupling (usually, collinearity) between some of the model independent variables. This coupling  does not affect model predictions much, but makes interpretation of the numerical values of the parameters impossible.

Confidence strip

The concept of confidence interval may be extended to modeling : it is sometimes possible to associate a confidence interval (for a given confidence level) to each and every prediction of the model. For example, in Simple Linear Regression, to each value of the independent variable corresponds :

    * A prediction of the value of the response variable, and

    * A confidence interval covering this prediction  .

This exceptionally favorable situation is made possible by the linearity of the model and the simplistic assumptions about the measurement errors. In general, no exact confidence interval can be found for model predictions, but it is sometimes possible to find approximate confidence intervals by linearizing the model in the parameter space.

The endpoints of the confidence interval then define a "confidence strip" (lower image in the illustration below). The model predictions are more reliable in regions where the strip is narrow than in regions where the strip is wide.

Local data density plays a major role in defining the strip width : the uncertainty about the model predictions is larger in low density areas than in of high density areas.

This notion generalizes to models with more than one independent variable, but there is then no graphic representation.

Confidence intervals and tests

When a confidence interval is available for the estimated value of a parameter, it is possible to devise a test about the value of this parameter. A confidence interval materializes a formula like :

Pr{a < q₀ < b}= 1 - a

where a and b are the endpoints of the interval.

Let's now consider the null hypothesis H₀ : q = q₀ that we wish to test at the a significance level.

Recall that such a tests relies on our ability to identify a statistic S whose value on the sample will be S₀, and such that, if H₀ is true, we can find an interval such that :

• The probability for to S₀ be outside the interval is equal to a,   or, equivalently,
• The probability for to S₀ be inside the interval is equal to 1 - a.

But a confidence interval has all the ingredients we need for the test :

• The statistic S is the estimator q^* of the parameter q used for building the confidence interval.
• The critical region of the test is defined by the endpoints a and b of the confidence interval.
• The significance level of the test is equal to the confidence level of the confidence interval.

Conversely, an by the same argument taken the other way around, a test on the value of a distribution parameter leads to a confidence interval on the estimated value of the parameter.

So, as long as we consider parameter estimation, "confidence interval" and "test" are identical concepts.

But the domain of tests extends far beyond that of confidence intervals. For example, a normality test bears on the hypothesis :

 H₀ : "The sample was generated by a normal distribution"

and cannot be reduced to building a confidence interval.

_____________________________________________________________________

 The first Tutorial describes the methods used to obtain exact confidence intervals for the means of normal distributions under various circumstances.

EXACT CONFIDENCE INTERVALS

FOR THE MEANS OF NORMAL DISTRIBUTIONS

_________________________________________________________________

 In the general case (variances unknown and not assumed to be equal), no exact confidence interval is known for the difference of the means of two independent normal distributions. But is is possible to calculate two types of approximate intervals :

• An asymptotic interval, that is almost an exact interval for large samples.
• An approximate interval based on "Welch's approximation", that delivers reasonably accurate intervals even for moderately sized samples.

ASYMPTOTIC CONFIDENCE INTERVALS

AND WELCH'S APPROXIMATION

This table of contents is deceivingly short :

• The asymptotic confidence interval is simple enough, but demonstrating its validity is difficult, and is not done in the Tutorial, which states the result without proof.
• Welch's approximation is also simple, but needs some (very instructive) calculations, that are given completely.

 
Warning : these Tutorials are the same as those mentioned on the "Interval estimation" page.

_________________________________________________

Related readings