Machine Learning Metrics and Your First Model: Accuracy, Precision, Recall

An online shop wants to flag orders likely to be returned, so a human can check the address and the size before shipping. Two models are tested on 100 orders. Model A scores 90% accuracy and model B scores 91%. The team picks B and moves on, and a week later someone points out that model A never flags anything at all. Accuracy alone cannot tell "correctly quiet" from "does nothing", and learning to see that is the purpose of this lesson.

The confusion matrix: four kinds of outcome

For a yes/no question there are four ways a prediction can land. Call "returned" the positive class. A true positive is a flagged order that really was returned. A false positive is a flagged order that was fine (a wasted check). A false negative is a return the model missed. A true negative is a correctly ignored good order. Every classification metric is arithmetic on those four counts.

# Online shop: will this order be returned? (1 = returned)
actual =    [0]*90 + [1]*10                    # 10% of orders come back
lazy =      [0]*100                            # model A: never predicts a return
attentive = [0]*84 + [1]*6 + [1]*7 + [0]*3     # model B: flags 13 orders

def confusion(actual, predicted):
    tp = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 1)
    fp = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 1)
    fn = sum(1 for a, p in zip(actual, predicted) if a == 1 and p == 0)
    tn = sum(1 for a, p in zip(actual, predicted) if a == 0 and p == 0)
    return tp, fp, fn, tn

def report(name, actual, predicted):
    tp, fp, fn, tn = confusion(actual, predicted)
    accuracy = (tp + tn) / len(actual)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    print(f"{name:<10} TP={tp:>2} FP={fp:>2} FN={fn:>2} TN={tn:>2} | "
          f"acc={accuracy:.2f} prec={precision:.2f} rec={recall:.2f} f1={f1:.2f}")

report("lazy", actual, lazy)
report("attentive", actual, attentive)

lazy       TP= 0 FP= 0 FN=10 TN=90 | acc=0.90 prec=0.00 rec=0.00 f1=0.00
attentive  TP= 7 FP= 6 FN= 3 TN=84 | acc=0.91 prec=0.54 rec=0.70 f1=0.61

The lazy model is right 90 times out of 100 because 90 orders were fine, yet it catches 0 of the 10 returns. The attentive model catches 7 of 10 (recall 0.70) at the price of 6 false alarms (precision 0.54). The accuracy gap is a single percentage point while the practical gap is enormous. Notice the guards in the code (if tp + fp else 0.0): a model that flags nothing has an undefined precision, and an unguarded division raises ZeroDivisionError.

Choosing a metric from the cost of mistakes

Precision and recall pull against each other, and which matters more is a business decision, not a statistical one. If a flagged order costs a ten-minute manual check, a few false alarms are cheap and you should push for high recall. If a flag triggers an expensive action, such as cancelling a customer's order, false positives are costly and precision matters more. The framing sheet from the first lesson asks for exactly this cost, so the metric is chosen before the model is trained.

A first regression model and its error metrics

When the target is a number instead of a category, you are doing regression and the metrics change. A bakery wants to predict loaves sold from the forecast temperature. The simplest model is a straight line, and for one input its best-fit parameters have a closed form (least squares): the slope is the covariance of x and y divided by the variance of x. We fit on the first six days and score on the last two, always against a baseline that predicts the training average.

# Bakery: predict loaves sold from the day's forecast temperature (deg C)
temps = [8, 11, 13, 15, 18, 20, 22, 25]
sold =  [96, 90, 84, 82, 70, 66, 61, 52]

def fit_line(xs, ys):
    """Least-squares line y = slope * x + intercept (closed form)."""
    mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
    slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
    return slope, my - slope * mx

def mae(actual, predicted):
    return sum(abs(a - p) for a, p in zip(actual, predicted)) / len(actual)

def rmse(actual, predicted):
    return (sum((a - p) ** 2 for a, p in zip(actual, predicted)) / len(actual)) ** 0.5

# fit on the first 6 days, evaluate on the last 2
train_x, train_y = temps[:6], sold[:6]
test_x, test_y = temps[6:], sold[6:]
slope, intercept = fit_line(train_x, train_y)
print(f"model: sold = {slope:.2f} * temp + {intercept:.1f}")

baseline = sum(train_y) / len(train_y)          # always predict the average
print("baseline prediction:", round(baseline, 1))
print("baseline MAE:", round(mae(test_y, [baseline] * len(test_y)), 2))
preds = [slope * t + intercept for t in test_x]
print("model preds:", [round(p, 1) for p in preds])
print("model MAE   :", round(mae(test_y, preds), 2))
print("model RMSE  :", round(rmse(test_y, preds), 2))

model: sold = -2.56 * temp + 117.6
baseline prediction: 81.3
baseline MAE: 24.83
model preds: [61.3, 53.6]
model MAE   : 0.91
model RMSE  : 1.12

MAE (mean absolute error) is the average size of the miss, in loaves: the model is off by about 0.9 loaves per day against 24.8 for the baseline. RMSE squares the errors before averaging, so one large miss counts for more; when RMSE is much larger than MAE, a few big mistakes are hiding in the average. Two caveats keep this honest. Only two test days is far too few to trust any of these numbers, and the line was fitted on temperatures from 8 to 20 degrees, so predicting for a 40-degree day would be extrapolation the model has no evidence for.

Common mistakes and the symptoms they cause

• Using accuracy on imbalanced data. A 90% accurate model that has recall 0.00. Look at the confusion matrix whenever one class is rare.
• Optimising the wrong metric. A team maximises F1 when the business only cares about catching at least 80% of fraud; the model is "better" on paper and worse for the decision.
• Unguarded divisions. Precision or recall computed when a model predicts no positives raises ZeroDivisionError in a test run, or returns nan in a library.
• Judging regression by one tiny test set. Two good days prove nothing, and an error metric with no baseline is hard to read.

How this shows up in real projects and interviews

In practice you will rarely hand-code these metrics, because libraries provide them, but you need to know exactly what each one measures so you can choose and explain it. Interviewers like the question "your model has 99% accuracy; is that good?". The strong answer asks about class balance, asks for the baseline, asks for the confusion matrix and then asks which error costs more. Once you are comfortable here, the next step is learning specific algorithms and how to select between them with cross-validation in Supervised Learning.