Interval estimation

Point estimation

If you're not familiar with estimation, we suggest that you first read the entry on "Point estimation".

Interval estimation

Given a n-sample drawn from a distribution, a point estimator q^* delivers a number q_* , the point estimate of the value of the parameter q of the distribution. If this estimator has the nice properties expected from a good estimator, the estimate q_* is our best bet about the true value q₀ of q. Yet, q_* is a raw number that carries no information about how close it may be to the true value q₀. All we know is that, in vague terms, q_* is "probably" not very far from q₀, but we are unable to quantify the terms "probably" or "far".

What we would like is a procedure that gives us a feel, in probabilistic terms, for "how close to q₀" the estimate q_* actually is. This is the objective pursued by interval estimation. More precisely, given a sample, we want to identify a segment :

• Whose left and right ends are determined by the sample only (and are therefore random variables, and more precisely, statistics),
• and such that we can say with certainty :

"The probability for this segment to cover the true value q₀ is known, and is equal to P."

This last sentence deserves some explaining. Suppose we know how to build the segment. We then imagine that a large number of n-samples are drawn from the distribution. For each new sample, a new segment is constructed with our procedure. This (random) segment will then cover q₀ 100.P% of the time. Then, for any given sample, we have identified a limited region that covers the true value q₀ with probability P. If P is large, we have then found a limited region that is very likely to cover  q₀.

Note that we resisted the temptation to say "The probability for q₀ to be inside the segment is P", for q₀ is unknown but  is not a random variable.

At first sight the situation looks hopeless, for the segment clearly has to be close to q₀  most of the time, but we don't know where q₀ is (if we knew, the whole idea of estimation, whether "point" or "interval", would be pointless). More precisely, if the ends of the segment are denoted by L and R, it is clear that the probability for the segment to cover q₀ :

Pr{L  q₀ R} = P

depends on q₀, which is unknown. Alternatively, if we want this probability not to depend on q₀, then L and R have to depend on q₀, which is equally useless.

Interval estimation of the mean of a normal distribution (variance is known)

Let us now give an example where this small miracle occurs.

Let N(µ, s²) be a normal distribution (red curve in the illustration below) from which a n-sample is drawn. The variance s² is supposed to be known, but not the mean µ. We use the sample mean m as as estimate of the distribution mean µ. It is distributed as N(µ, s²/n) (blue curve).

The distribution of m can be turned into a standard normal distribution (mean equal 0, unit variance) by the simple transformation T :

Note that :
   * µ is the expectation of m.
   * s / n^1/2 is the standard deviation of m.

So m' is of the form :
               {Point estimate - E[Point estimate]}/ Standard deviation of Point estimate

a kind of standardization found in all areas of statistics.

We have m'~N(0, 1) (black curve).

Now place two cut-off points L' and R' on this standard normal distribution in such a way that the area under each wing of the standard gaussian is, say, 0.025 (red areas). These points are changed into L and R by the inverse of transformation T. The transform of the mean (i.e. m') has a 0.95 probability of being outside the red area, and so has m.

More generally, if we denote the probability P (arbitrarily chosen by the analyst) by 1 - a, then :

where z_{a /2}  denotes the number such that the area under the standard gaussian curve to the right of z_{a /2}  is a /2. 

The inverse of transformation T gives :

 This expression is equivalent to :

-----
These two expressions have different interpretations :

• The first expression is the probability for the random variable m to be inside a fixed segment centered on µ and of length 2(.z_{a /2} .s.n^-1/2 ).
• The second expression if the probability for a certain random segment centered on m, of the same fixed length  2(.z_{a /2} .s.n^-1/2 ), to cover µ.

 So, indeed, we have found a (random) segment :

• Whose ends are completely determined by the sample,
• And that covers the distribution mean µ with a given, arbitrary probability P = 1 - a,

which is exactly what we were looking for.

This segment is called the confidence interval of m (green segment in the illustration above) for the confidence level 1 - a.

Note that 2(.z_{a /2} .s.n^-1/2 ), the length of the confidence interval, is proportional to the standard deviation of m; the point estimate of µ. So it can be said that :

• Point estimation tells us about the central tendency of the distribution of the parameter,
• Interval estimation tells us about both the central tendency and the spread of the distribution of the parameter.

Pivot

Why does something that seemed impossible turns out to be very easy ? We feared that any definition we could give of the confidence interval would be useless because the ends of this interval would have to depend on µ. But the ends of the interval we just found do not depend on µ, and this is because we identified a quantity :

whose distribution does not depend on µ.

More generally, the very possibility of building confidence intervals for an estimate of a parameter q depends on identifying a quantity whose distribution does not depend on the true value q₀ of this parameter. Such a quantity, when it exists, is called a pivotal quantity or pivot.

Note that a pivot is not a statistic because its definition involves not just the observations in the sample, but also
the true value q₀ of the parameter.

Conversely, any pivotal quantity :

• involving only one parameter,  and
• whose distribution is known,

can be used to build confidence intervals for this parameter. Any pair of numbers L' and R' define two "exclusion" regions (the red regions in the above example) under the probability distribution curve of the pivot. The sum of the areas of these two regions is a number a, and the same argument we developed above shows that the back-transformed of L' and R', call them L and R, define a confidence interval with confidence level a.

Equal-tailed intervals

So it appears that, given a pivotal quantity and a confidence level a, there is some arbitrariness in the way L and R are defined. In the above example, we might have chosen L' and R' in many different ways such the sum of the red areas still be equal to a. This would have defined asymetrical confidence intervals, all with the same confidence level a (lower image in the illustration below).
 

A natural requirement is then that, for a given confidence level a, the confidence interval be as short as possible, as we want to narrow down the region in which we expect q₀ to lie with a given probability.

Identifying shortest confidence intervals may prove difficult. For most commonly encountered distributions, one will have to be satisfied with equal-tailed intervals when the distributions are symmetrical and bell-shaped. Fortunately, these equal-tailed intervals are also the shortest confidence intervals for these distributions (this is in particular the case for the normal and for the t distribution).

Sample size

For a given confidence level, the ends of the confidence interval depend on the sample size (see here) : the larger the sample, the narrower the confidence interval. This is not surprising as, the larger the sample, the more information we have on the distribution, hence on its parameter q, and the smaller is the uncertainty about its true value q₀.

For a given sample size, the analyst can only trade confidence level against confidence interval width. A larger imposed confidence level will cause the confidence interval to become larger : the only way to be more confident that the interval covers q₀ is to make the interval wider.

But if observations can be collected at a reasonable cost, it is possible to :

• Choose a confidence level 1 - a,
• Choose a confidence interval width,

and determine how many observations are needed to satisfy both requirements.

Let us get back to our example, where the variance is known to be equal to, say, 4. We want to find the number of observations needed to have both :

• A 95% confidence level,  and
• A 2.5 confidence interval width (1.25 on each side of m).

The width of the confidence interval is :

Width = 2(.z_{a /2} .s.n^-1/2 )

 with a = 0.05. Tables of the fractiles of the standard normal distribution tell us then that z_{a /2}  = 1.960. The sample size must therefore verify :

2.5 = 2.(1.960).2.n^-1/2  = 7.84.n^-1/2 

or

n^1/2  ~ 3.1

So, for the foregoing conditions to be satisfied, n must be at least equal to (3.1)², that is, the sample must have at least 10 observations.

Confidence intervals on the parameters of a model

One of the main applications of interval estimation is to calculate confidence intervals for the parameters of a model. The values of these parameters are point estimates of the true values of these parameters. Confidence intervals for these estimates are very important, as they indicate how trustworthy these estimates are.

See for example the calculation of confidence intervals for the parameters of a Multiple Linear Regression model.

____________________________________________________________

 The first Tutorial describes the methods used to obtain exact confidence intervals for the means of normal distributions under various circumstances.

EXACT CONFIDENCE INTERVALS

FOR THE MEANS OF NORMAL DISTRIBUTIONS

_________________________________________________________________

In the general case (variances unknown and not assumed to be equal), no exact confidence interval is known for the difference of the means of two independent normal distributions. But is is possible to calculate two types of approximate intervals :

• An asymptotic interval, that is almost an exact interval for large samples.
• An approximate interval based on "Welch's approximation", that delivers reasonably accurate intervals even for moderately sized samples.

ASYMPTOTIC CONFIDENCE INTERVALS

AND WELCH'S APPROXIMATION

This table of contents is deceivingly short :

• The asymptotic confidence interval is simple enough, but demonstrating its validity is difficult, and is not done in the tutorial.
• Welch's approximation is also simple, but needs some (very instructive) calculations, that are given completely.

 __________________________________

Related readings