Fitting Problems in Machine Learning Model Training

September 15, 2024

What Are Fitting Problems

Fitting problems boil down to how well a model learns the patterns in the data. A model that learns well is said to have a good fit; one that learns poorly falls into one of two categories: overfitting or underfitting.

Fitting means approximating the relationships in the data with a model function, and the degree of fit reflects how closely the model function matches the actual data.

Overfitting

Overfitting means the model performs very well on the training data but poorly on test data. The usual cause is fitting the training set too closely, to the point of learning its noise, which leads to large errors when generalizing to new data.

An Overfitting Example

Here is an example using a training case on BigML. The model predicts whether a hotel customer who makes a booking will later cancel it.

The dataset is shown below.

Overfitting training dataset

The prediction target is the value of reservation_status.

Prediction target

With the dataset ready, we can start training. The model here is a decision tree. Below is the evaluation score from an 8:2 train/validation split.

Evaluation with full accuracy

The prediction accuracy reaches an astonishing 100%, which is clearly implausible, so overfitting is likely: the model has absorbed noise from the dataset.

Now look at its metrics and curves.

Evaluation metrics with full accuracy

These are the evaluation metrics, and the accuracy here is only 42%, indicating that the dataset is flawed and the model cannot make useful predictions.

Fixing Overfitting

This case uses a decision tree, so open the tree and inspect its branches.

Decision tree branches

The problem is immediately visible in the labeled fields: reservation_status and is_canceled are both output labels, but training treated is_canceled as an input. As a result, the other fields didn't take part in the computation at all.

To fix this, remove is_canceled from the training fields in the dataset, retrain, and re-evaluate. The result is:

Evaluation after fix

The accuracy now looks normal and fairly good. Let's check the evaluation metrics.

Evaluation metrics after fix

The accuracy in the evaluation metrics now exceeds 80%, which is a very good result.

We can also look at the shape of the decision tree:

Decision tree after fix

Everything looks good: the overfitting problem has been eliminated.

Underfitting

Underfitting means the model performs poorly on both the training and test sets, because the fit is too weak to capture the patterns in the data.

An Underfitting Example

Again using BigML, I found an underfitting dataset, built the dataset on BigML, and trained a model, still a decision tree.

Underfitting training dataset

The evaluation scores are shown below.

Underfitting evaluation

The accuracy doesn't look especially low, but the Phi coefficient is only 0.3, meaning predictions correlate very weakly with actual values. This model is also flawed and unfit for prediction.

What the Phi coefficient means

The Phi coefficient measures the correlation between predictions and actual outcomes: +1 means perfect prediction, 0 means completely random, and -1 means completely inverted. It generally ranges from -1 to 1, and the closer to 1, the better.

Fixing Underfitting

There are several common ways to address underfitting. The first is to add or remove features (add features to datasets with too few, remove them from datasets with too many), adjusting the feature count so the model can learn the patterns during training. The second is to try a different model, because different models suit different datasets. For example, if the data has simple linear relationships, linear models such as linear or logistic regression fit better; if there are many features with non-linear relationships, a decision tree is more appropriate. The third is to increase the sample size if there are too few samples, which also helps the model learn from the dataset.

Summary

In machine learning, both overfitting and underfitting can be spotted by training the model and evaluating the results. Overfitting is usually a labeling problem in the dataset and can be fixed with appropriate corrections. Underfitting needs closer observation to decide whether the model choice is wrong, the features are insufficient, or the sample space is too small.