Model Selection and Evaluation

ACTIONS TO IMPROVE THE MODEL

1. Metrics and Scoring :

Let’s say we have two hypothesis for a task, h(x) and h’(x). How would we know which one is better. Well from a high level perspective, we might take the following steps:

• Measure the accuracy of both hypotheses.
• Determine whether there is any statistical significance between the two results. If there is, select the better performing hypothesis. If not, we cannot say with any statistical certainty that either h(x) or h’(x) is better.

When we have a classification task, we will consider the accuracy of our model by its ability to assign an instance to its correct class. Consider this on a binary level. We have two classes, 1 and 0. We would classify a correct prediction therefore as being when the model classifies a class 1 instance as class 1, or a class 0 instance as class 0. Assuming our 1 class as being the ‘Positive class’ and the 0 class being the ‘Negative class’, we can build a table that outlines all the possibilities our model might produce.

2. Overfitting :

Overfitting is a key consideration when looking at the quality of a model. Overfitting occurs when we have a hypothesis that performs significantly better on its training data examples than it does on the test data examples. This is an extremely important concept in machine learning, because overfitting is one of the main features we want to avoid. For a machine learning model to be robust and effective in the ‘real world’, it needs to be able to predict unseen data well, it needs to be able to generalise, as I defined in the first article. Overfitting essentially prevents generalisation and presents us with model that initially looks great because it fits to our training data very well, but what we ultimately find is that the model will have fit to the training data too well. The model has essentially not identified general relationships in the data, but instead has focused on figuring out exactly how to predict this one sample set. This can happen for a number of reasons:

• The learning process is allowed to continue for too long.
• The examples int he training set are not an accurate representation of the test set and therefore also of the wider picture.
• Some features in the dataset are uninformative and so the model is distracted, assigning weights to these values that don’t realise any positive value.

3. Cross Validation :

Cross Validation can be considered under the model improvement section. It is a particularly useful method for smaller datasets. Splitting our data into training and test data will reduce the number of samples that our model receives to train on. When we have a relatively small number of instances, this can have a large impact on the performance of our model because it does not have enough data points to study relationships and build reliable and robust coefficients. In addition to this, we make an assumption here that our training data has a similar relationship between the variables that our testing data does, because only in this situation will we actually be able to generate a reasonable model that can predict with a good level of accuracy on unseen data.

The solution that we can turn to here is a method called cross validation. In cross validation, we run our modelling process on different subsets of the data to get multiple measures of model quality. Consider the following dataset, split into 5 sections, called folds.

Cross validation can give us a more accurate measure of model quality, however it can take a long time because it estimates multiple models. For small datasets, where the extra computational burden isn’t a large issue, we can run cross validation.