Prompt Engineering: Step-by-Step Reasoning and Iterative Refinement

An online shop wants an assistant to answer return requests: a customer bought a desk lamp on 3 March, asks on 19 March, and the policy says 14 days. The first prompt, "Reply to this customer", works on easy cases and then, on some run, cheerfully accepts a return that is two days late. The model was asked to read the policy, do the date maths, make the decision and write kindly, all at once. Each of those is a separate job.

Decompose: one job per step

Decomposition means turning one big prompt into a chain of small ones, where each step has one job and its output feeds the next. It helps for three reasons. Each step is easier to get right. Each step can be tested alone, so a failure points at one place. And some steps stop being prompts at all, because plain code does them better.

1. EXTRACT   customer message  -> {item, purchase_date, request_date}
             (model, or a form field if you control the UI)
2. DECIDE    dates + policy    -> {days, within_window}
             (plain code: no model needed)
3. WRITE     decision + facts  -> 2-3 sentence reply
             (model: tone and wording only)
4. CHECK     reply             -> mentions the decision? no refund
             promise when declined? (code or a second prompt)

Step 2 is the lesson inside the lesson. Counting days between two dates is exact work, and ordinary code does it exactly every time. Asking a model to do it invites rare, hard-to-reproduce mistakes. Use the model for language and judgement, and use code for arithmetic, lookups and rules.

from datetime import date

RETURN_WINDOW_DAYS = 14

def check_window(purchase_iso, request_iso, window=RETURN_WINDOW_DAYS):
    """Deterministic step: do the date arithmetic in code, not in the prompt."""
    days = (date.fromisoformat(request_iso) - date.fromisoformat(purchase_iso)).days
    return {"days_since_purchase": days, "within_window": 0 <= days <= window}

def build_reply_prompt(facts, verdict):
    # Step 3 of the chain: the model only has to write, because the decision is already made.
    return (
        "You write short, polite replies for an online shop's support team.\n"
        "Decision (already made, do not change it): "
        f"{'return ACCEPTED' if verdict['within_window'] else 'return DECLINED'}\n"
        f"Days since purchase: {verdict['days_since_purchase']} (window is {RETURN_WINDOW_DAYS} days)\n"
        f"Item: {facts['item']}\n"
        "Write 2-3 sentences. Explain the decision using the numbers above."
    )

facts = {"item": "desk lamp", "purchase": "2025-03-03", "request": "2025-03-19"}   # step 1: extracted by a model or form
verdict = check_window(facts["purchase"], facts["request"])                        # step 2: plain code
print(verdict)
print()
print(build_reply_prompt(facts, verdict))

{'days_since_purchase': 16, 'within_window': False}

You write short, polite replies for an online shop's support team.
Decision (already made, do not change it): return DECLINED
Days since purchase: 16 (window is 14 days)
Item: desk lamp
Write 2-3 sentences. Explain the decision using the numbers above.

By the time the writing prompt runs, the decision is fixed and the numbers are supplied, so the model has nothing to get wrong except wording. Note the line "already made, do not change it": it tells the model its role in this step.

Asking for step-by-step reasoning

When a task does need the model to work something out, such as comparing several conditions or following a multi-step rule, asking it to show intermediate steps often improves accuracy on multi-step problems, because later tokens can build on earlier written ones. It also gives you something to read when the answer is wrong. It is not magic: it costs extra tokens and time, it does little for simple lookups, and the written steps are a useful trace but not a guaranteed picture of how the model actually arrived at its answer. Some providers also ship models with built-in reasoning modes, so what you need to ask for varies by model.

Decide whether the return is allowed, then write the reply.

Work in this order:
1. List the purchase date, the request date and the return window.
2. Count the days between purchase and request. Show the count.
3. Compare the count with the window. State ACCEPTED or DECLINED.
4. Only then write a 2-3 sentence reply that matches the decision.

Put steps 1-3 under the heading REASONING and the reply under REPLY.

The fixed headings matter: your program can show only the REPLY part to the customer and log the REASONING for support staff. Ordering matters too. If you ask for the verdict first and the explanation afterwards, the explanation tends to justify a verdict already committed to.

The fourth step in the chain is easy to forget. A cheap check on the finished reply, written as code or as a second short prompt, catches the embarrassing failures before a customer sees them: does the reply state the same decision, does it avoid promising a refund when declined, is it under the length limit? Checks are far easier to write than a prompt that never fails, and they give you a number you can track over time.

Refine like an engineer, not a gambler

Beginners tweak a prompt, try one input, see a nicer answer and move on. The better habit is to keep a short list of failing cases, change one thing at a time, and rerun the whole list so a fix for one case does not quietly break another. Keep a log; it takes a minute and pays for itself the first time someone asks why the prompt says what it says.

v1  "Reply to this customer."
    Fails: case 4 (16 days) accepted. Case 6 (exactly 14 days) declined.
v2  Added the 14-day rule and "count the days".
    Fixes case 4. Case 6 still wrong: boundary unclear.
v3  Added: "Day 14 is still inside the window."
    Cases 1-6 pass. New failure: case 7 mentions a refund
    when declined.
v4  Added constraint: "If DECLINED, do not promise a refund."
    Cases 1-7 pass. Keep.

Common mistakes and the symptom each one causes

• One mega-prompt for everything. Symptom: it works on easy cases and fails unpredictably on others, and you cannot tell which instruction was ignored. Fix: split into steps and test each.
• Making the model do exact arithmetic or lookups. Symptom: rare off-by-one errors that cannot be reproduced on demand. Fix: compute in code and pass the result in.
• Reasoning requested after the verdict. Symptom: confident explanations that defend a wrong answer. Fix: reasoning first, verdict second, or compute the verdict in code.
• Changing several things at once. Symptom: results improve or worsen and you do not know why. Fix: one change per version, with a log.
• Tuning on one example. Symptom: perfect on the case you stared at, worse on the rest. Fix: always rerun the full case list.
• Showing raw reasoning to customers. Symptom: internal rule text, guesses and hedging leak into the message. Fix: separate REASONING and REPLY and display only REPLY.

How this shows up in real projects and interviews

Production systems rarely use a single prompt. They look like pipelines: extract, decide, write, check, with ordinary code between the model calls, which is also the seed of an AI agent when the model chooses the next step itself. The cost of a chain is extra calls, so latency and spend go up; measure whether the accuracy gain justifies it. In interviews, a strong answer to "how would you improve a prompt that fails sometimes" is: gather the failing cases, decide whether the step needs a model at all, split the task, ask for reasoning only where it helps, change one thing at a time and regression-test. The next lesson shows how to build that test set.