Moment generating function

The probability density function (continuous variable) or the probability mass function (discrete variable) of a random variable X contains all the information you'll ever need about this variable. Therefore, it seems that it should always be possible:

    * to calculate the mean, variance and higher order  moments of X  from its p.d.f. (or its discrete p.m.f.).

    * to calculate the distribution of, say, the sum of two independent random variables X and Y whose distributions are known.

Yet, in practice, it turns out that these calculations are often intractable. The Moment Generating Function (m.g.f.) may then come to our help.

Before we describe the m.g.f., a little digression is in order. The difficulty we just mentioned is not specific to Probability Theory. In fact, just about any disciplin in Physics or Mathematics will run into the same problem sooner or later:  a certain property of a function f is needed, the equations are there, but cannot be solved in a closed form.

A very powerful and general idea, that exists under many guises, consists then in transforming the original function f into a new and appropriately chosen function g, such that comparatively simple calculations on g will provide the desired results.

The obstacle is therefore circumvented, at the expense of appropriately transforming the original function f. This ploy is depicted in the following illustration :

The lower image makes explicit the principle of "transformation of a function" :

    * First calculate g, the "image" of  f under the transformation,

    * Then conduct the appropriate calculations on g to obtain the desired quantity Q.

The m.g.f. results from a transformation of the p.d.f. (or the p.m.f.). This transformation is defined as follows :

    1) A new parameter t is introduced,

    2) Then the random variable e^tX  is created,

    3) E[e^tX], the expectation of e^tX is calculated. For example, if X is continuous :

where p(x) is the p.d.f. of X.

The integration is over x, therefore x is not to be found in the final result, which is then a function of t only. This function is, by definition, the Moment Generating Function of the variable X (or of its p.d.f. p(x)), and will be denoted M_X (t), or M(t) when there is no risk of confusion.

 ________________________________________

Here are some important properties of the Moment Generating Function :

Existence

Not all distributions have a moment generating function. For example, we'll show that the Cauchy distribution has no m.g.f.. (but they all have the complex numbers version of the m.g.f., the characteristic function, that is not addressed in this Glossary).

Moments and moment generating function

If r.v. X has a m.g.f., then the n^th order moment of its distribution is equal to the value of the n^th order derivative of its m.g.f. for t = 0.

Of course, the moment generating function gets its name from this property.

Using this property is often more convenient than direct calculation for calculating the moments of a r.v. (except possibly for the uniform distribution), as will be seen throughout this site.

Sum of independent r.v.

This result is used for calculating the distributions of many classical r.v. that are defined, or can be interpreted as the sum of iid r.v. :

    * Chi-square,

    * Negative binomial,

The important question of the distribution of the empirical mean of a r.v. is often succesfully addressed by refering to this property (see for example the case of the normal distribution).

Uniqueness of the moment generating fuction

This result is both very useful and very difficult to demonstrate (we won't). We use it many times throughout this site, for example :

    * Calculation of the Chi-square distribution.

    * Additivity property of the binomial, negative binomial, Gamma and Poisson distributions.

    * Linear transform of a normal r.v., linear combination of independent normal r.v. (see here)

    * Identification of the distribution of the sum of iid exponential variables (see here).

    * Identification of the distribution of the sum of a random number of i.i.d. exponential variables (see here).

    * etc...

Convergence property of the moment generating function

Let {X_i} be a series of r.v. with respective moment generating functions M_i(t). If the series {M_i(t)} converges towards a limit M(t), and if this limit is the m.g.f. of a r.v. X , then the series {X_i} converges in distribution towards X.

This fundamental result is very difficult, and will not be demonstrated. Yet, we'll use it in the course of demonstrating the Central Limit Theorem.

__________________________________________________________________________

The above mentioned properties are detailed and some of them are demonstrated in the following Tutorial.

PROPERTIES OF THE MOMENT GENERATING FUNCTION

______________________________________________________

Related readings :