Classification: Logistic Regression and Decision Trees Explained

A gym wants to know which members will renew their membership. For each of 24 past members it has the average visits per week, the number of months they have belonged, and what they did at renewal time. The answer is a category, renew or lapse, not a number, so this is classification. Two models solve it in opposite ways: one draws a single smooth boundary, the other asks a chain of yes/no questions. Building each from scratch shows why they behave so differently.

Logistic regression: a line squeezed into a probability

A straight-line score can be any number, but a probability must sit between 0 and 1. Logistic regression computes a weighted score z = w1*visits + w2*months + b and passes it through the sigmoid, 1 / (1 + e^-z), which maps large negative scores to near 0, zero to exactly 0.5 and large positive scores to near 1. The weights are trained by gradient descent exactly as in the previous lesson, but with a different loss, log-loss, which punishes confident wrong answers heavily. A pleasant result of the maths is that the gradient is still just (prediction - truth) * feature.

import math

# Gym memberships: (visits_per_week, months_as_member) -> renewed? (1 = yes)
members = [
    (0.5, 2, 0), (1.0, 3, 0), (0.0, 5, 0), (1.5, 2, 0), (2.0, 4, 0), (0.5, 9, 0),
    (1.0, 14, 0), (2.5, 3, 0), (1.5, 8, 0), (3.0, 2, 0), (3.5, 5, 1), (2.5, 12, 1),
    (4.0, 7, 0), (3.0, 18, 1), (4.5, 3, 1), (2.0, 20, 1), (5.0, 10, 1), (3.5, 24, 1),
    (1.0, 30, 1), (4.0, 16, 1), (2.5, 6, 0), (3.0, 9, 1), (1.5, 15, 1), (5.5, 4, 1),
]

# --- standardise both features ---
cols = list(zip(*[(v, m) for v, m, _ in members]))
means = [sum(c) / len(c) for c in cols]
stds = [(sum((v - mu) ** 2 for v in c) / len(c)) ** 0.5 for c, mu in zip(cols, means)]
X = [[(v - means[0]) / stds[0], (m - means[1]) / stds[1]] for v, m, _ in members]
y = [r[2] for r in members]

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# --- logistic regression by gradient descent on log-loss ---
w, b, lr = [0.0, 0.0], 0.0, 0.5
for step in range(1, 401):
    gw, gb = [0.0, 0.0], 0.0
    for xi, yi in zip(X, y):
        err = sigmoid(w[0] * xi[0] + w[1] * xi[1] + b) - yi
        gw[0] += err * xi[0]
        gw[1] += err * xi[1]
        gb += err
    w = [w[0] - lr * gw[0] / len(X), w[1] - lr * gw[1] / len(X)]
    b -= lr * gb / len(X)

def p_renew(visits, months_):
    x0 = (visits - means[0]) / stds[0]
    x1 = (months_ - means[1]) / stds[1]
    return sigmoid(w[0] * x0 + w[1] * x1 + b)

hits = sum(1 for xi, yi in zip(X, y) if (sigmoid(w[0] * xi[0] + w[1] * xi[1] + b) >= 0.5) == bool(yi))
print("weights (visits, months):", [round(v, 2) for v in w], "bias:", round(b, 2))
print("training accuracy:", round(hits / len(y), 3))
print("P(renew) for 4 visits/week, 6 months:", round(p_renew(4, 6), 3))
print("P(renew) for 1 visit/week, 6 months :", round(p_renew(1, 6), 3))

weights (visits, months): [3.15, 3.82] bias: 0.11
training accuracy: 0.917
P(renew) for 4 visits/week, 6 months: 0.83
P(renew) for 1 visit/week, 6 months : 0.007

Both weights are positive, so more visits and a longer membership raise the probability of renewing; because the inputs were standardised, the weights are comparable, and months is slightly stronger here. The output is a probability, not just a label, and that matters: you can choose the threshold (0.5 is only a default) and rank members by risk. Note that the model gets 22 of 24 training members right, and as always that number comes from the data it learned from.

Decision trees: questions instead of weights

A decision tree learns a flowchart. At each node it picks one feature and one cut-off that best separates the classes, sends rows left or right, and repeats on each side. "Best" is measured by Gini impurity: 2 * p * (1 - p) where p is the share of positives in a group. It is 0 for a pure group and 0.5 for a 50/50 mix. The split chosen is the one with the lowest weighted impurity of the two children. The code below tries every cut-off on both features at every node, then prints the tree as readable rules.

members = [
    (0.5, 2, 0), (1.0, 3, 0), (0.0, 5, 0), (1.5, 2, 0), (2.0, 4, 0), (0.5, 9, 0),
    (1.0, 14, 0), (2.5, 3, 0), (1.5, 8, 0), (3.0, 2, 0), (3.5, 5, 1), (2.5, 12, 1),
    (4.0, 7, 0), (3.0, 18, 1), (4.5, 3, 1), (2.0, 20, 1), (5.0, 10, 1), (3.5, 24, 1),
    (1.0, 30, 1), (4.0, 16, 1), (2.5, 6, 0), (3.0, 9, 1), (1.5, 15, 1), (5.5, 4, 1),
]
FEATURES = ["visits_per_week", "months_as_member"]

def gini(rows):
    if not rows:
        return 0.0
    p = sum(r[-1] for r in rows) / len(rows)
    return 2 * p * (1 - p)

def best_split(rows):
    best = None
    for f in range(2):
        for t in sorted({r[f] for r in rows}):
            left = [r for r in rows if r[f] <= t]
            right = [r for r in rows if r[f] > t]
            if not left or not right:
                continue
            score = (len(left) * gini(left) + len(right) * gini(right)) / len(rows)
            if best is None or score < best[0]:
                best = (score, f, t)
    return best

def build(rows, depth, max_depth):
    labels = [r[-1] for r in rows]
    if depth == max_depth or gini(rows) == 0 or best_split(rows) is None:
        return round(sum(labels) / len(labels))
    _, f, t = best_split(rows)
    return (f, t,
            build([r for r in rows if r[f] <= t], depth + 1, max_depth),
            build([r for r in rows if r[f] > t], depth + 1, max_depth))

def show(node, indent=""):
    if not isinstance(node, tuple):
        print(indent + ("-> renews" if node else "-> lapses"))
        return
    f, t, left, right = node
    print(f"{indent}if {FEATURES[f]} <= {t}:")
    show(left, indent + "    ")
    print(f"{indent}else:")
    show(right, indent + "    ")

def predict(node, row):
    while isinstance(node, tuple):
        f, t, left, right = node
        node = left if row[f] <= t else right
    return node

print("root impurity:", round(gini(members), 3))
for depth in (1, 2, 3):
    tree = build(members, 0, depth)
    acc = sum(predict(tree, r) == r[-1] for r in members) / len(members)
    print(f"max_depth={depth}: training accuracy {acc:.3f}")
show(build(members, 0, 2))

root impurity: 0.5
max_depth=1: training accuracy 0.792
max_depth=2: training accuracy 0.875
max_depth=3: training accuracy 0.958
if months_as_member <= 9:
    if visits_per_week <= 2.5:
        -> lapses
    else:
        -> renews
else:
    if visits_per_week <= 1.0:
        -> lapses
    else:
        -> renews

The printed tree is something you could hand to a membership manager: newer members need about three visits a week to be likely to renew, while long-standing members only lapse if they almost never visit. Look at the depth sweep too. Training accuracy climbs from 0.792 to 0.958 as the tree gets deeper, and it would reach 1.0 if allowed to keep splitting until every leaf holds a single member. That perfect score would be memorisation, so the maximum depth is a hyperparameter you must choose using validation data, which is the subject of the next lesson.

A useful way to choose between them on a new problem is to start with logistic regression as the baseline, because it takes minutes and tells you how much signal a simple linear rule captures. Then try a shallow tree. If the tree is much better, the data probably contains thresholds or interactions (for example "long-standing members only lapse when visits drop to almost nothing") that a single straight boundary cannot express. If the two perform about the same, prefer the one your audience can understand and maintain.

Common mistakes and the symptoms they cause

• Letting a tree grow without limits. Training accuracy hits 1.0 and validation accuracy lags well behind; the tree has a leaf for nearly every row.
• Reading raw logistic weights as importance without scaling. A weight of 0.8 on kilometres and 0.8 on euros means very different things; compare weights only on standardised inputs.
• Treating 0.5 as sacred. Many problems need a lower or higher threshold, a point developed in the lesson on imbalance and metrics.
• Using `math.exp` on a very negative score. math.exp(-z) with a large negative z raises OverflowError: math range error; real libraries use a numerically stable form of the sigmoid.

How this shows up in real projects and interviews

Logistic regression is the default baseline for classification in industry because it is quick to train, hard to break and gives calibrated-looking probabilities when the data is reasonable. Trees are the building block of random forests and gradient-boosted trees, which dominate tabular data problems. In interviews you will be asked to compare them: say that trees handle non-linear interactions and need no scaling but overfit easily, while logistic regression is stable and interpretable but only draws linear boundaries unless you engineer features. Being able to say what Gini impurity measures, and what happens as depth grows, is a reliable signal that you know more than the library calls.