Validation

Validating a model is trying to estimate its generalization capacity.

Now recall that the true quality of a model cannot be infered from its performance on data that was used to build it in the first place (except for exceptional cases, like Linear Regression in ideal conditions, see Tutorial on Linear Regression). As a matter of fact, it is quite common to come across models whose performance on "building data" is good (but illusory), and that behave poorly once put in the field in real world conditions.

Validating a model is a delicate and time consuming endeavor. Classical approaches to validation include :

• Theory. In rare and somewhat irrealistic cases, theory can be pushed up to estimating the generalization capacity of the model (see Tutorial on Linear Regression).
• Use of the model in application conditions may be simulated with nothing but data that is available at model construction time. Let's mention :
         - Use of a single or multiple validation set.
         - Use re-sampling methods ("Cross-validation", "Leave-One-Out", and more recently, "Bootstrap"), that create several validation sets according to specific schemes. Note that these validation techniques estimate the generalization capacity of a model architecture, rather than that of a particular model.

              - Some mathematical theories, that we can't describe here, proceed with a linear approximation (in parameter space) of the model, from which an estimation of performance of the model can be derived. These new techniques are beginning to be implemented in the context of supervised Neural Networks. 

 ________________________________________________________

Related readings :