|
|
|
|
|
|
Discriminant Analysis (DA)
Strictly speaking, Discriminant Analysis is synonymous with Classification. The term therefore refers to many different techniques such as Logistic Regression, Decision Trees or Neural networks.
But common use of the expression refers more specifically
to two different approaches to classification.
We suggest that you first read the entry on Fisher's linear discriminant.
The goal is to identify new variables that are particularly effective at separating classes. These new variables are linear combinations of the original variables, and are called or "Discriminant Axes" or "Discriminant Functions".
In the top illustration below, neither x1 nor x2 can be of much use, individually, for separating class C1 from class C2 , because projections of theses two classes on the axes severely overlap. Now if you consider the oblique line whose equation is y = x1 + x2 , (bottom illustration), then it is clearly a perfectly discriminating axis because class projections on this axis are completely separated.
a 
It can be shown that the Factors are the Principal Components of the set of class barycenters weighted by the corresponding class populations (see Tutorial below).
________________________________
Discriminant Analysis generates one "Classification Function" per class, and an observation x is assigned to the class with the largest value for its Classification Function. These functions are linear, or occasionally quadratic in the variables. They rely on the severe assumption that classes have normal distributions. When these conditions are met, Discriminant Analysis generates not only class assignments, but also the probabilities for an observation to belong to any of the classes (so-called "posterior probabilities").
Discriminant Analysis is a venerable technique, whose intricacies have now been explored in every detail. At this time, it is probably the most widely used classification technique. Yet, Data Modeling has made more recent and more powerful techniques popular. Among them, the best known are :
1) Logistic Regression,
2) Decision Trees,
3) So-called "Support Vector Machines", or SVM, which are very robust classifiers,
3) Neural Networks, which are probably the most powerful classification technique at this time.
_____________________________________________________________
|
Tutorial 1 |
Owing th the complexity of the subject, the first Tutorial is just an overviw of Discriminant Analysis which expounds the basic principales withoug going into the mathematics.
OVERVIEW OF DISCRIMINANT ANALYSIS
|
Discriminant Factor Analysis (DFA) Building a classifier "Geometrical" Classification Functions "Probabilistic" Classification Functions Costs Assumption on classes Choosing the "best" set of predictors |
||
|
TUTORIAL |
|
|
____________________________________________
|
Tutorial 2 |
In this Tutorial, we examine the "descriptive part" of Discriminant Analysis. It identifies directions in the space of observations on which the classes are particularly well separated. Just as in Principal Components Analysis, data may then be projected on planes defined by pairs of such directions for visual interpretation.
DISCRIMINANT FACTOR ANALYSIS
|
Which variables are useful for discriminating between classes ? Geometric criterion Statistical criterion Creating Discriminant Functions The simplest case What when class populations are not identical ? Discriminant Functions as Principal Components How many Discriminant Functions ? What when classes are not spherical ? The Mahalanobis distance Re-defining the "distance" What if classes have different covariance matrices ? The complete DFA scheme Is the model trustworthy ? Theory Validation techniques Linear or quadratic ? Predictor selection Why select predictors ? Selecting predictors in practice "Global quality" criterion, Wilks' Lambda "F-to-enter" "F-to-remove" The search strategy Categorical variables Class indicators Multiple Correspondance Analysis |
||
|
TUTORIAL |
|
|
_____________________________________________________________
|
Tutorial 3 |
We now want to build a real classifier, that is a predictive model whose input will be the attributes of the observations (or "predictors"), and whose output will be :
* Either a "hard decision", that is a class label.
* Or, better, a probability for each class that the observation belongs to the class.
We'll do that with the concept of "Discriminant Functions", that we first describe in general terms.
BUILDING THE DISCRIMINANT ANALYSIS CLASSIFIER
|
The general concept of "Discriminant Function" The geometric discriminant functions The Mahalanobis distance criterion Two classes only : Multiple Linear Regression Numerical coding of the classes Is the result the same as that of DA ? A slight improvement over standard DA MLR : a caveat Logistic Regression The probabilistic discriminant functions The Bayes' theorem Linear and quadratic discriminant functions Class boundaries Actually building the classifier The parameters of the model Performance of the model on the sample The classification matrix Costs Is the model good ? Is the model trustworthy ? Theory Validation techniques Linear or quadratic ? Predictor selection Why select predictors ? Selecting predictors in practice "Global quality" criterion, Wilks' Lambda "F-to-enter" "F-to-remove" The search strategy Categorical variables Class indicators Multiple Correspondance Analysis |
||
|
TUTORIAL |
|
|
______________________________________________________
|
Tutorial 4 |
We give here some mathematical complements to substantiate the results of the previous Tutorials.
COMPLEMENTS ON DISCRIMINANT ANALYSIS
|
The concept of "metrics", the Mahalanobis distance Decomposition of the total covariance matrix The Mahalanobis distance criterion The exact probabilistic discriminant functions Unequal covariance matrices Equal covariance matrices Score Variance of a projection on a line Discriminant Analysis "at large" |
||
|
TUTORIAL |
|
|
______________________________________________
Related readings
|
|
|
|
|
|
|
|
