Moment generating function

The probability density function (continuous variable) or the probability mass function (discrete variable) of a random variable X contains all the information you'll ever need about this variable. Therefore, it seems that it should always be possible:

    * to calculate the mean, variance and higher order  moments of X  from its p.d.f. (or its discrete p.m.f.).

    * to calculate the distribution of, say, the sum of two independent random variables X and Y whose distributions are known.


Yet, in practice, it turns out that these calculations are often intractable. The Moment Generating Function (m.g.f.) may then come to our help.

 

Before we describe the m.g.f., a little digression is in order. The difficulty we just mentioned is not specific to Probability Theory. In fact, just about any disciplin in Physics or Mathematics will run into the same problem sooner or later:  a certain property of a function f is needed, the equations are there, but cannot be solved in a closed form.

A very powerful and general idea, that exists under many guises, consists then in transforming the original function f into a new and appropriately chosen function g, such that comparatively simple calculations on g will provide the desired results.

The obstacle is therefore circumvented, at the expense of appropriately transforming the original function f. This ploy is depicted in the following illustration :

 

 

 

 

The lower image makes explicit the principle of "transformation of a function" :

    * First calculate g, the "image" of  f under the transformation,

    * Then conduct the appropriate calculations on g to obtain the desired quantity Q.

 

The m.g.f. results from a transformation of the p.d.f. (or the p.m.f.). This transformation is defined as follows :

    1) A new parameter t is introduced,

    2) Then the random variable etX  is created,

    3) E[etX], the expectation of etX is calculated. For example, if X is continuous :

 

where p(x) is the p.d.f. of X.

 

The integration is over x, therefore x is not to be found in the final result, which is then a function of t only. This function is, by definition, the Moment Generating Function of the variable X (or of its p.d.f. p(x)), and will be denoted MX (t), or M(t) when there is no risk of confusion.

 ________________________________________

 

Here are some important properties of the Moment Generating Function :

Existence

Not all distributions have a moment generating function. For example, we'll show that the Cauchy distribution has no m.g.f.. (but they all have the complex numbers version of the m.g.f., the characteristic function, that is not addressed in this Glossary).

Moments and moment generating function

If r.v. X has a m.g.f., then the nth order moment of its distribution is equal to the value of the nth order derivative of its m.g.f. for t = 0.

 

 

 

 

Of course, the moment generating function gets its name from this property.

 

Using this property is often more convenient than direct calculation for calculating the moments of a r.v. (except possibly for the uniform distribution), as will be seen throughout this site.

Sum of independent r.v.

 

If the r.v. X1 and X2  both have a m.g.f.
then
X = X1 + X2  has a m.g.f,. which is the product of the m.g.f. of X1 and of X2

 

 

This result is used for calculating the distributions of many classical r.v. that are defined, or can be interpreted as the sum of iid r.v. :

    * Chi-square,

    * Negative binomial,

 

The important question of the distribution of the empirical mean of a r.v. is often succesfully addressed by refering to this property (see for example the case of the normal distribution).

Uniqueness of the moment generating fuction

 

 

If the two r.v. X et Y have identical moment generating functions,
then
their distributions are also identical.

 

 

This result is both very useful and very difficult to demonstrate (we won't). We use it many times throughout this site, for example :

    * Calculation of the Chi-square distribution.

    * Additivity property of the binomial, negative binomialGamma and Poisson distributions.

    * Linear transform of a normal r.v., linear combination of independent normal r.v. (see here)

    * Identification of the distribution of the sum of iid exponential variables (see here).

    * Identification of the distribution of the sum of a random number of i.i.d. exponential variables (see here).

    * etc...

Convergence property of the moment generating function

Let {Xi} be a series of r.v. with respective moment generating functions Mi(t). If the series {Mi(t)} converges towards a limit M(t), and if this limit is the m.g.f. of a r.v. X , then the series {Xi} converges in distribution towards X.

 

This fundamental result is very difficult, and will not be demonstrated. Yet, we'll use it in the course of demonstrating the Central Limit Theorem.

__________________________________________________________________________


 

Tutorial

 

The above mentioned properties are detailed and some of them are demonstrated in the following Tutorial.

 

 

PROPERTIES OF THE MOMENT GENERATING FUNCTION

The m.g.f. generates moments

Mean

Second order moment

All moments

M.g.f. of a linear transform of a r.v.

M.g.f. of the sum of independent r.v.

Uniqueness (without proof)

Convergence of the m.g.f. (without proof)

Existence of the m.g.f., characteristic function

TUTORIAL

  

______________________________________________________

 

Related readings :

Central Limit Theorem

Download this Glossary

 

Want to contribute to this site ?