Linear Regression and Gradient Descent From Scratch in Python

A second-hand marketplace for electric scooters wants to suggest a resale price from one fact: how many months the scooter has been used. You can see the trend in a scatter of past sales: older means cheaper, roughly in a straight line. Regression turns that eyeball judgement into a precise rule, and gradient descent is the general-purpose method for finding it, the same method that trains neural networks with billions of parameters.

A line, a loss and a downhill walk

The model is price = slope * months + intercept. Two numbers, the slope and the intercept, define every possible line. To pick the best one we need a score for how wrong a line is, called the loss. The usual choice is mean squared error (MSE): for each sale take the gap between predicted and actual price, square it so that overshoots and undershoots both count and big misses count extra, and average.

Picture the loss as a landscape over (slope, intercept) pairs, with a single valley bottom where the loss is lowest. The gradient at any point is the direction of steepest climb, so stepping the opposite way goes downhill. Repeating that step, scaled by a small number called the learning rate, walks to the bottom. For MSE the gradient with respect to the slope is the average of 2 * error * x, and with respect to the intercept it is the average of 2 * error.

# Used e-scooters: predict resale price (EUR) from months of use
months = [3, 6, 9, 12, 18, 24, 30, 36, 42, 48]
price  = [640, 610, 560, 540, 470, 430, 370, 340, 300, 250]

# 1. Standardise x so one learning rate suits both parameters
mean_x = sum(months) / len(months)
std_x = (sum((m - mean_x) ** 2 for m in months) / len(months)) ** 0.5
xs = [(m - mean_x) / std_x for m in months]

def mse(w, b):
    return sum((w * x + b - y) ** 2 for x, y in zip(xs, price)) / len(xs)

# 2. Gradient descent on w (slope) and b (intercept)
w, b, lr = 0.0, 0.0, 0.1
for step in range(1, 201):
    grad_w = sum(2 * (w * x + b - y) * x for x, y in zip(xs, price)) / len(xs)
    grad_b = sum(2 * (w * x + b - y) for x, y in zip(xs, price)) / len(xs)
    w -= lr * grad_w
    b -= lr * grad_b
    if step in (1, 5, 20, 50, 200):
        print(f"step {step:>3}: w={w:8.3f} b={b:8.3f} mse={mse(w, b):10.2f}")

# 3. Convert back to the original units: price = slope * months + intercept
slope = w / std_x
intercept = b - slope * mean_x
print(f"gradient descent: price = {slope:.3f} * months + {intercept:.2f}")

# 4. Closed-form least squares for comparison
my = sum(price) / len(price)
slope_cf = sum((m - mean_x) * (p - my) for m, p in zip(months, price)) / sum((m - mean_x) ** 2 for m in months)
print(f"closed form     : price = {slope_cf:.3f} * months + {my - slope_cf * mean_x:.2f}")

# 5. How much variation does the line explain? (R squared)
ss_res = sum((slope * m + intercept - p) ** 2 for m, p in zip(months, price))
ss_tot = sum((p - my) ** 2 for p in price)
print("R squared:", round(1 - ss_res / ss_tot, 4))

step   1: w= -25.425 b=  90.200 mse= 140727.54
step   5: w= -85.470 b= 303.216 mse=  23782.94
step  20: w=-125.662 b= 445.800 mse=    236.79
step  50: w=-127.126 b= 450.994 mse=    207.61
step 200: w=-127.127 b= 451.000 mse=    207.61
gradient descent: price = -8.502 * months + 644.86
closed form     : price = -8.502 * months + 644.86
R squared: 0.9873

Read the trace like a story. The first steps cut the loss from the hundreds of thousands to tens of thousands; by step 20 the line is nearly right; by step 50 the loss has stopped changing at 207.61, and steps 50 to 200 are wasted effort. That plateau is what convergence looks like. The two parameter sets agree with the closed-form least-squares answer to three decimals: each extra month of use costs about 8.50 euros, and a brand-new scooter is worth roughly 645. The R squared of 0.987 says the line explains almost all the variation in this tiny sample.

If least squares has a formula, why bother with descent? Because the formula only exists for a few simple models. Once you have a thousand features, non-linear models or a different loss, there is no closed form, but the gradient can still be computed. Gradient descent is the path from this lesson to every larger model.

When the walk goes wrong: learning rate and scaling

The learning rate is the one dial you must set. Too small and progress is glacial; too large and each step overshoots the valley and lands higher on the opposite slope, so the loss grows instead of shrinking. We can see all three behaviours by training for only 30 steps at different rates, and then by removing the standardisation step from the same data.

months = [3, 6, 9, 12, 18, 24, 30, 36, 42, 48]
price  = [640, 610, 560, 540, 470, 430, 370, 340, 300, 250]
mean_x = sum(months) / len(months)
std_x = (sum((m - mean_x) ** 2 for m in months) / len(months)) ** 0.5
xs = [(m - mean_x) / std_x for m in months]

def train(inputs, lr, steps=30):
    w = b = 0.0
    n = len(inputs)
    for _ in range(steps):
        gw = sum(2 * (w * x + b - y) * x for x, y in zip(inputs, price)) / n
        gb = sum(2 * (w * x + b - y) for x, y in zip(inputs, price)) / n
        w -= lr * gw
        b -= lr * gb
    return sum((w * x + b - y) ** 2 for x, y in zip(inputs, price)) / n

for lr in (0.001, 0.1, 1.05):
    print(f"scaled input, lr={lr:<6} mse after 30 steps: {train(xs, lr):.4g}")
print("raw months,   lr=0.1    mse after 30 steps:", f"{train(months, 0.1):.4g}")

scaled input, lr=0.001  mse after 30 steps: 1.949e+05
scaled input, lr=0.1    mse after 30 steps: 207.9
scaled input, lr=1.05   mse after 30 steps: 6.685e+07
raw months,   lr=0.1    mse after 30 steps: 1.446e+135

At 0.001 the loss is still near 195,000 after 30 steps: it is moving the right way, just slowly. At 0.1 it has essentially arrived. At 1.05 it has exploded into the tens of millions. The last line is the sneakiest: with raw months between 3 and 48 the same rate of 0.1 is far too big, because the gradient for the slope is multiplied by x and so is dozens of times larger than for the intercept. The loss reaches 10 to the power 135 and, with more steps, Python would eventually give inf and then nan. Standardising the input is what made a single learning rate workable.

Common mistakes and the symptoms they cause

• Skipping feature scaling. Training diverges or crawls, and the learning rate that works for one feature breaks for another. See the 1.446e+135 above.
• Stopping on a fixed step count without watching the loss. You either waste time (steps 50 to 200 changed nothing) or stop before convergence and report a half-trained line.
• Extrapolating. A line fitted on 3 to 48 months predicts a price below zero at around 76 months (645 divided by 8.5). The model has no evidence outside its range and a real price floor does not exist in the formula.
• Trusting a high R squared on ten points. With so few rows a good fit proves little; evaluate on held-out data as explained in the fundamentals course.

How this shows up in real projects and interviews

Linear regression is still the first model most teams try, because it is fast, rarely overfits with few features and its coefficients can be read as "each extra month costs about 8.5 euros". It is also the baseline a fancier model has to beat. In interviews, expect to be asked to explain gradient descent in plain words, to say what happens if the learning rate is too high, and why features are scaled. Mentioning that the loss function is chosen to match the cost of errors (squared error punishes big misses; absolute error is more robust to outliers) shows you think beyond the recipe.