Train, Validation and Test Sets: Understanding Overfitting

A greenhouse manager wants a model that says whether a bed needs watering from two probe readings. One version scores 100% on the readings it was built from, and the manager is delighted until it is used on the next morning's readings and gets one in five wrong. The model was never smart; it had simply memorised the answers. Detecting that difference requires keeping some data hidden, which is why every serious ML workflow starts by splitting.

Three sets, three jobs

The validation set exists because every time you tweak something and look at a score, you are subtly fitting to that data. After dozens of tweaks the validation score is optimistic. The test set is the one you have not touched, so it still tells the truth, but only until you start tuning against it too. Once you do, it has become a second validation set and you need fresh data.

Watching overfitting happen

Our model is k-nearest neighbours: to label a new bed, find the k most similar beds in the training data and take a majority vote. The number k is a hyperparameter, a setting you choose rather than something fitted. With k = 1, the model copies the single closest example, noise and all. A larger k smooths that out. The data below is synthetic, with a rule (moisture plus a little heat) and 15% of labels flipped on purpose to mimic faulty probes, produced by a tiny seeded generator so every reader gets identical numbers.

# Greenhouse sensors: does this bed need watering? (1 = yes)
class Lcg:
    """Tiny deterministic random source so everyone gets the same numbers."""
    def __init__(self, seed):
        self.state = seed
    def next(self):
        self.state = (1103515245 * self.state + 12345) % (2 ** 31)
        return self.state / 2 ** 31

rng = Lcg(2024)

def make_row():
    moisture = rng.next()                 # 0 = bone dry, 1 = soaked
    heat = rng.next()                     # 0 = cool, 1 = hot
    needs_water = 1 if moisture + 0.3 * heat < 0.55 else 0
    if rng.next() < 0.15:                 # 15% of labels are noisy (miscalibrated probes)
        needs_water = 1 - needs_water
    return ([moisture, heat], needs_water)

data = [make_row() for _ in range(300)]

# shuffle with Fisher-Yates, then cut 60 / 20 / 20
for i in range(len(data) - 1, 0, -1):
    j = int(rng.next() * (i + 1))
    data[i], data[j] = data[j], data[i]
train, val, test = data[:180], data[180:240], data[240:]
print("sizes:", len(train), len(val), len(test))

def knn_predict(train_rows, x, k):
    ranked = sorted(train_rows, key=lambda r: (r[0][0] - x[0]) ** 2 + (r[0][1] - x[1]) ** 2)
    votes = sum(label for _, label in ranked[:k])
    return 1 if votes * 2 > k else 0

def accuracy(train_rows, eval_rows, k):
    hits = sum(1 for x, y in eval_rows if knn_predict(train_rows, x, k) == y)
    return hits / len(eval_rows)

print(" k   train   val")
results = {}
for k in (1, 3, 5, 9, 15, 25, 45):
    tr, va = accuracy(train, train, k), accuracy(train, val, k)
    results[k] = va
    print(f"{k:>2}   {tr:.3f}  {va:.3f}")

best_k = max(results, key=results.get)
print("best k on validation:", best_k)
print("test accuracy, used once:", round(accuracy(train, test, best_k), 3))

sizes: 180 60 60
 k   train   val
 1   1.000  0.800
 3   0.872  0.817
 5   0.872  0.817
 9   0.867  0.800
15   0.878  0.833
25   0.856  0.833
45   0.856  0.817
best k on validation: 15
test accuracy, used once: 0.833

Read the table row by row. With k = 1 the training accuracy is a perfect 1.000, but that is not an achievement: each training bed is its own nearest neighbour at distance zero. On validation data the same model gets 0.800, tied for the lowest of the lot. As k grows the training score falls (the model stops copying noise) while validation rises slightly, then flattens. That widening gap between a high training score and a lower validation score is the signature of overfitting.

Be careful about over-reading small differences. The validation set has 60 rows, so one row is worth 0.017. The gap between k = 9 (0.800) and k = 15 (0.833) is two rows, which is well within the luck of the draw. The honest conclusion is that every value of k from 1 to 45 lands within a few rows of the others on validation data; what separates k = 1 is only its suspicious perfect training score. With the noise we injected, roughly 85% is the ceiling, and the run lands close to it.

Rules for splitting honestly

• Shuffle first unless the data has a time order. The split above uses a seeded shuffle so that results are reproducible.
• Split by time for anything that predicts the future. If you randomly mix last month and next month, the model gets to see tomorrow while training. Train on earlier dates and test on later ones.
• Keep related rows together. If one patient or one greenhouse supplies many rows, put all of that entity on one side, or the model memorises it and the test score is flattered.
• Do all learned preprocessing after the split, as shown in the data preparation lesson.
• Look at the test set once. Pick a model using validation data, then report the test score and stop.

Common mistakes and the symptoms they cause

• Reporting training accuracy. The symptom is near-perfect numbers for memorising models such as k = 1 that fall apart on the first real batch.
• Tuning on the test set. Test accuracy looks great, then live performance disappoints, because the settings were chosen to suit that particular sample.
• Random split on time-ordered data. Excellent offline results and a model that fails on next week's data.
• Drawing conclusions from a tiny validation set. Two models differ by one example and you declare a winner; rerunning with a different shuffle reverses the order.

How this shows up in real projects and interviews

Real projects often use cross-validation, which repeats the train-and-validate cycle over several folds so that one lucky split cannot mislead you; it is introduced in Supervised Learning. In interviews, the question "how do you know your model is not overfitting?" is nearly universal. A solid answer names the three sets, mentions the training/validation gap, notes that a time-ordered problem needs a time-based split, and says that the test set is touched once. Adding that you would also check variability, for example by re-splitting with a few different seeds, shows you understand that one number is not the whole story.