Expectation  (Conditional)

Joint probability distribution

Let X and Y be two random variables.

Discrete case

If X and Y are discrete, the pair (X, Y) is entirely described by the joint probability function p(x, y) :

p(x_i , y_j ) = P{X = x_i , Y = y_j }

with the normalization relation :

S_i S_j p(x_i , y_j ) = 1  

Continuous case

When both variables are continuous, the pair (X, Y) is entirely described by the joint probability density function p(x, y) defined by :

p(x, y)dxdy = P{x X x + dx, y Y y + dy}

with the normalization relation :

Conditional probability distributions

General case

Consider the discrete case. Select one particular value of X, say x. Keep drawing realizations of (X, Y) but consider only those draws for which X = x and ignore all other draws. Under this setting, Y follows a probability distribution called the probability distribution of Y conditionally to X = x. This distribution is denoted p_{Y  | X} {Y = y | X = x}, or  p_{Y  | X} {y | x} for short.

We have :

where :

    * p(x, y) is the joint probability function of (X, Y).

    * p_X (x) is the (marginal) probability function of X.

-----

A similar definition applies to the continuous case.

Special case : X and Y are independent

If X and Y are independent, the probability distribution of Y conditionally to X = x is always the same for any x, and is just equal to the (marginal) probability distribution of Y. This leads to the following expression :

that we demonstrate below.

Conditional expectation

General case

Under the condition X = x, Y  has a certain expectation which is called the expectation of Y conditionally to X = x. It is a number, not a random variable.

It is denoted E [Y | X = x]. It is equal, by definition, to :

with a similar expression for continuous variables :

Special case : X and Y are independent

If X and Y are independent, then the expectation of Y conditionally to X = x is equal to the (unconditional) expectation of Y :

a result that we demonstrate below.

Iterated expectation

If now X is not held fixed, the expectation of Y conditionally to X becomes a random variable. It therefore has an expectation, which is given by :

This result is called "The Theorem of Iterated Expectation", of which we give two demonstrations here.

-----

Because of its great practical effectiveness for solving many different types of problems, a special set of Tutorials is dedicated to the Theorem of Iterated Expectation.

Expectation of the product of two random variables

General case

Let X and Y be two random variables. What is the expectation E[XY] of their product ?

We'll show that :

You'll find here an example of application of this result when the two variables are not independent.

Special case : X and Y are independent

If X and Y are independent, we'll show that the above result simplifies to :

or in words :

The expectation of the product of two independent r.v. is equal to the product of their expectations.

Converse

Conversely, if two r.v. are such that the expectation of their product is equal to the product of their expectations, can we conclude that the two variables are independent ? The answer is definitely "No". We'll easily conclude that under this condition, the two variables are indeed uncorrelated, but genuine independence will require the following stronger condition :

X and Y are independent if and only if, for any pair of functions {f(.), g(.)} we have E[f(X).g(Y)] = E[f(X)].E[g(Y)]

that we'll demonstrate.

Conditional expectation and Regression

Recall that Regression consists in predicting as accurately as possible the values of a function y = f(x) corrupted by some random noise e(x) for any value of x when the function is known only through a sample of pairs (x_i , y_i). Because of the noise, the observed values are random, and the predictions of the regression model are random variables.

The central result about Regression is that, for any value x of the regressor, the best prediction y for Y (in the Mean Square Error sense) is given by :

This result is developed and demonstrated here.

_________________________________________________

We first give three examples of calculation of a conditional expectation :

   * The first one bears on discrete variables, and the joint probabilities are given explicitely, as is often the case.

   * The second one calculates the expectation of the number of Heads in the first m tosses of a coin conditionally to the total number of Heads in n tosses to be equal to a given value x.

   * The third one bears on continuous variables. The joint probability density function is given analytically, and the calculated conditional expectation of Y will appear as a function of x.

-----

After demonstrating some elementary facts about independent r.v., we address the important question of the expectation of the product of two r.v.. We demonstrate the two results presented above, and then demonstrate the more difficult necessary and sufficient condition for two r.v. to be independent.

CONDITIONAL EXPECTATION

____________________________________________

Related readings :