Interactive animation

Probability integral transformation (PIT)

# The Probability Integral Transformation

Let X be any continuous random variable with a probability density p(x), and let F(x) be its cumulative distribution function (cdf).

Let's use the function F to transform X into the new rv Y = F(X).

Then the Probability Integral Transform (theorem) states that :

 Y = F(X) ~ Uniform[0, 1]

-----

The P.I.T. may also be formulated the other way around.

Let :

* F be the cdf of the continuous rv X.

* Z be a rv uniformly distibuted in [0, 1] :

Z ~ Uniform[0, 1]

Then F -1(Z) is distributed as X.

 F -1(Z) ~ X        with Z ~ Uniform[0, 1]

This last result leads to a powerful method of simulation of a random variable X, provided that :

* The inverse of its distribution function be known in a closed form,

* It is possible to simulate a random variable uniformly distributed in [0, 1], which any computer can do up to a high level of accuracy.

Many animations in this site are based on the P.I.T. for simulating draws from a given distribution.

-----

The P.I.T. is demonstrated here and illustrated by the animation below.

# Rationale of the Probability Integral Transformation

Consider a region where the probability density p(x) is low. Observations are few and far between in this region. But because p(x) is the derivative of F(x), this region is also a region where the derivative (the slope) of F is low. Consequently, F will tend to pack the images of the observations into a small region, thereby compensating for the low initial density.

Conversely, in a high density region, the derivative of F is high, and F will tend to pull the observations apart, thereby compensating for the high initial density (lower image of the above illustration).

-----

So the distribution of F(X) can be anticipated to be "smoother" than that of X. It is indeed the case as this distribution is in fact the perfectly flat Uniform[0, 1] distribution.

# Animation

This animation illustrates the Probability Integral Transform.

 The "Book of Animations" on your computer

 * Create a density function p(x) to your liking by clicking repetitively anywhere in the gray background frame. * When you're satisfied with the density, click on "Go" and watch the build-up of the histogram of your density. * Click on "Pause", then repetitively on "Next". Each time you click, a new observation is drawn from Uniform[0, 1]. The distribution of the images of these observations through F -1 is the density you created.

# Probability integral transformation and bootstrap

The probability integral transformation can be used for simulating a distribution and thus for estimating one of its related quantities (e.g. a moment) by the Monte-Carlo method. Yet, for this purpose, the distribution function must be known analytically, or at least tabulated. In practice, it is often the case that the distribution is only known through a sample drawn from the population. The empirical distribution function may then be used as a substitute to the true distribution function.

This idea is the starting point of the method of estimation known as "bootstrap".

# Probability integral transformation and goodness-of-fit tests

An important family of goodness-of-fit tests uses the Probability Integral Transformation to first transform the observations into independent and uniformly distributed variables. As a consequence, the distribution of the test statistic for any reference distribution is the same as the distribution of this statistic for the uniform distribution. The distribution of the test statistic is therefore "distribution free".

____________________________________________________________

 Uniform distribution Probability density Cumulative distribution function EDF-based goodness-of-fit tests Bootstrap