Cybersecurity Fundamentals: Incident Response and Day One of a Breach

An online bakery ships custom cakes. On a Saturday morning its owner notices that a customer received a password-reset email nobody asked for, and that three orders now list a delivery address in another country. She has no security team, no runbook and a launch event at noon. What she does in the next hour matters more than anything she did before it. Good incident response is mostly staying orderly while under pressure, and orderly is something you can rehearse in advance.

The shape of an incident

Most frameworks, including the widely referenced NIST guidance, describe the same loop with slightly different names. It is useful to hold it in your head as five steps:

1. Prepare: decide who is in charge, keep contact details for your hosting provider, payment processor and lawyer, and make sure logs and backups exist before you need them.
2. Detect and triage: notice that something is wrong, decide how serious it is, and open a written timeline.
3. Contain: stop the damage from spreading without destroying evidence: revoke sessions, disable compromised accounts, isolate a machine from the network.
4. Eradicate and recover: remove the cause (the stolen credential, the vulnerable plugin), restore from known-good data, and bring services back carefully.
5. Learn: a blameless review that produces a short list of changes, with owners and dates.

Detection: logs you can actually read

You can only respond to what you can see. At minimum, record logins (success and failure with time, user and source address), changes to accounts and permissions, and administrative actions. Retention matters as much as collection: if logs vanish after a day, a slow intrusion is invisible. The script below reads a few sample log lines and applies two ideas many real detection rules use: many failures from one address, and a success right after those failures. A real system would use a log platform and alerting, but the thinking is identical.

from collections import Counter

log_lines = """\
2025-03-02T09:14:01Z LOGIN_FAIL user=priya ip=203.0.113.9
2025-03-02T09:14:03Z LOGIN_FAIL user=priya ip=203.0.113.9
2025-03-02T09:14:04Z LOGIN_FAIL user=sam ip=203.0.113.9
2025-03-02T09:14:06Z LOGIN_FAIL user=lee ip=203.0.113.9
2025-03-02T09:14:08Z LOGIN_FAIL user=priya ip=203.0.113.9
2025-03-02T09:15:30Z LOGIN_OK user=maria ip=198.51.100.4
2025-03-02T09:16:11Z LOGIN_FAIL user=maria ip=198.51.100.4
2025-03-02T09:16:40Z LOGIN_OK user=priya ip=203.0.113.9
""".splitlines()

THRESHOLD = 3
fails_by_ip = Counter()
ok_after_fail = []

for line in log_lines:
    _, event, *fields = line.split()
    ip = dict(f.split("=") for f in fields)["ip"]
    if event == "LOGIN_FAIL":
        fails_by_ip[ip] += 1
    elif event == "LOGIN_OK" and fails_by_ip[ip] >= THRESHOLD:
        ok_after_fail.append(line)

for ip, count in fails_by_ip.items():
    if count >= THRESHOLD:
        print(f"ALERT: {count} failed logins from {ip}")
for line in ok_after_fail:
    print("INVESTIGATE FIRST:", line)

ALERT: 5 failed logins from 203.0.113.9
INVESTIGATE FIRST: 2025-03-02T09:16:40Z LOGIN_OK user=priya ip=203.0.113.9

The interesting line is the last one. Failures alone are noise, but a success from the same address after a burst of failures is the account to look at first. The addresses here come from ranges reserved for documentation, so they are safe to use in examples.

Day one: a checklist you can run while stressed

FIRST HOUR
  [ ] Name one incident lead. Everyone else takes instructions from them.
  [ ] Start a shared timeline (who noticed what, at what time, UTC).
  [ ] Do not wipe or reinstall anything yet; you may need the evidence.
  [ ] Contain: disable the suspect account, revoke active sessions and
      API keys, disconnect the affected machine from the network.
  [ ] Reset credentials for any account that may have been exposed,
      starting with admin and email accounts.

FIRST DAY
  [ ] Preserve logs and snapshots (copy them somewhere the attacker
      cannot reach).
  [ ] Establish scope: which accounts, which data, which time window.
  [ ] Check for persistence: new admin users, forwarding rules,
      scheduled jobs, unknown API keys.
  [ ] Confirm which backups pre-date the problem and test one restore.
  [ ] Tell the people who must be told: hosting provider, payment
      processor, insurer, and legal counsel about notification duties.

BEFORE YOU RESTORE
  [ ] Remove the cause, not only the symptom.
  [ ] Rotate every secret that touched the affected systems.

Keep that list somewhere you can reach when the main system is down, such as a printed page or a shared document outside your own infrastructure. Back at the bakery, the owner's first hour would look like this: she disables the compromised admin account, signs everyone out, resets the passwords of staff accounts from a clean device, and writes down the time she noticed the odd reset email. Only then does she look at which orders were altered, and she asks her payment processor whether any of the changed orders were already charged. None of those steps needs special skill. They need an order of operations that was agreed before the panic.

Two points deserve emphasis. First, contain before you clean: wiping a machine too early destroys evidence about how the intruder got in, so you may fix the symptom and leave the door open. Second, notification duties vary by country and by the kind of data involved, and the timelines can be short. A lawyer or data-protection contact should be on your contact list before an incident, not found during one.

Backups: the part that decides how bad it gets

A widely repeated rule of thumb is 3-2-1: three copies of your data, on two different kinds of storage, with one copy kept offsite or offline. Add two refinements. Make at least one copy immutable, meaning it cannot be altered or deleted for a retention period even with admin credentials. And practise restoring. A backup you have never restored is a hope, not a control. Record how long the restore took; that number is your real recovery time.

Common mistakes

• Reinstalling immediately. Symptom: the same compromise returns next week because nobody learned how the intruder got in. Fix: preserve evidence, find the entry point, then rebuild.
• Resetting only the one obvious password. Symptom: logins keep succeeding from the same unknown address after the reset. Fix: revoke sessions and tokens too, and check for added forwarding rules and recovery emails.
• Coordinating over the compromised channel. Symptom: an intruder with mailbox access reads your response plan. Fix: agree a fallback channel in advance.
• No written timeline. Symptom: by day three nobody agrees what happened when, and notifications are late or inconsistent. Fix: one shared log, timestamps in UTC.
• Blame in the review. Symptom: next time people hide mistakes for hours. Fix: treat the review as a search for system fixes.

How this shows up in real projects and interviews

Interviewers use incident scenarios to test judgement, not memorised acronyms: "production credentials were pushed to a public repo, what happens in the first hour?" The strong answer rotates the key first, then checks what the key could reach, then cleans the history, and writes up the timeline. On a real team, the unglamorous work (an on-call rota, log retention, an up-to-date contact sheet) is what makes responses calm. For related thinking about explaining what happened under pressure, see problem-solving frameworks and the root cause analysis lesson.