Tutorials

1

OVERVIEW :
   Least Squares Line. Coefficient of determination.
   Geometric interpretation. Statistical properties of estimators.
   Normality assumption. Probability distributions, confidence    intervals and tests.
   Leverage points. Influential points.

2

Least Squares Line (LSL)
   Residuals
   Special case : slope is equal to 0.

3

Coefficient of determination, R².
   Sums of Squares : total, explained, residual.
   Coefficient of determination and correlation coefficient

4

Space of variables    Linear Regression is a projection.
   Geometric interpretation of the adjusted values, of the    residuals and of the Sums of Squares.

Tutorials

5

Assumptions on errors.
   Statistical properties of estimators.

 6

Unbiased estimation of error variance.

7

Statistical properties of adjusted values, residuals and    predictions.

8

The normality assumption.
    Probability distributions of :
      * Slope and intercept.
      * Estimated error variance.
      * Predictions.

9

Testing for no slope.
    Testing for R² = 0.
    Equivalence of the two tests.

10

Leverage point.
   Standardized and studentized residuals.
   DFFITS, Cook's distance.
   Influential points.DFBeta, covariance ratio.

Tutorials

1

Assumptions about the data matrix.
   The space of variables, geometric representation.
   Linear Regression as a projection.
   Least Squares (LS) estimation of the parameters.
   Hat matrix and leverages. 

2

Assumptions about the errors.
   Mean and covariance matrix of LS estimators.
   Gauss-Markov theorem

3

Residuals : définition and geometric properties.
   Statistical properties of the residuals.
   Statistical properties of adjusted values.
   Covariance of residuals of adjusted values.
   Statistical properties of prediction errors.    
   Estimation of the variance of the errors and of the parameters.

4

Sums of Squares : Total, Explained and Residual.
    Not centered and centered coefficient of determination R².
   Geometric interpretations.
   Adjusted R².

5

The normality (or "gaussian") assumption.
   Parameter estimation by Maximum Likelihood.
   Distribution of estimators (variance known and unknown).
   Distribution of prediction errors.

Tutorials

6

Confidence intervals on parameters and estimated variance of    errors.    
   Confidence intervals on model predictions.

7

F test on nested models.
   F test or Student's test on a single variable. Equivalence of the    two tests.
   Fisher's global test of significance of the regression ("R² test").

8

Three definitions of the quality of a MLR model.
    The "test set" method". Cross-validation. Adjusted R².
    Test between nested models.
    Mallows' C_p.
    Penalized likelihood : AIC, BIC.

9

Variable selection as an optimization problem.
    Using tests or quality criteria.
    Forward selection, backward selection, stepwise slection.
    Caveat.

10

Checking the validity of the MLR assumptions :
       * Linearity
       * Independence of the errors.
       * Normality of the errors.
       * Homoskedasticity.

Tutorial

1

Collinearity.
    Three definitions of Ridge Regression.
    Statistical properties of the ridge estimators.
    Choosing the value of the ridge parameter.

Tutorial

2

Singular Value Decomposition ().
    Ridge Regression in the singular form.
    Ridge Regression and Principal Components Analysis.
    MSE of the parameters.
    Effective number of parameters.