Using LLMs via APIs: Latency, Cost Drivers and Choosing a Model

A team builds a contract-clause summariser, tests it on ten clauses, loves it, and launches. The first monthly bill is several times what anyone expected. Nothing malfunctioned. Every call re-sent a long set of instructions and three worked examples, and nobody had multiplied the per-call tokens by the number of calls. This lesson is the arithmetic you do before launch.

What you pay for, and the lever for each

Hosted models are typically billed by tokens, with input and output often priced differently, and output usually costing more per token. Prices and limits change and differ across providers, so the code below uses placeholders that you replace with the real figures from your provider's price sheet.

# Back-of-envelope cost and latency model for a contract-clause summariser.
# Every number below is a PLACEHOLDER. Replace with your provider's real
# price sheet and with latencies you measure yourself.
PRICE_PER_MILLION_INPUT = 1.00     # currency units per 1M input tokens (placeholder)
PRICE_PER_MILLION_OUTPUT = 4.00    # output is usually priced higher than input (placeholder)

def request_cost(input_tokens, output_tokens):
    return (input_tokens * PRICE_PER_MILLION_INPUT
            + output_tokens * PRICE_PER_MILLION_OUTPUT) / 1_000_000

def estimate_latency(output_tokens, time_to_first_token=0.8, tokens_per_second=60):
    """Latency is roughly: wait for the first token, then stream the rest."""
    return time_to_first_token + output_tokens / tokens_per_second

clauses_per_day = 5_000
instructions = 350        # fixed system prompt, resent on every call
clause_text = 600         # the clause itself
summary = 120             # generated summary

per_call = request_cost(instructions + clause_text, summary)
print(f"cost per call:  {per_call:.6f}")
print(f"cost per day:   {per_call * clauses_per_day:.2f}")
print(f"latency (est.): {estimate_latency(summary):.1f}s")

# What if we also paste 3 worked examples (900 tokens) into every prompt?
with_examples = request_cost(instructions + 900 + clause_text, summary)
print(f"with examples:  {with_examples * clauses_per_day:.2f} per day "
      f"({with_examples / per_call:.2f}x)")

# Output tokens dominate latency; capping them is the cheapest speed-up.
print(f"capped at 60 tokens: {estimate_latency(60):.1f}s")

cost per call:  0.001430
cost per day:   7.15
latency (est.): 2.8s
with examples:  11.65 per day (1.63x)
capped at 60 tokens: 1.8s

The numbers are placeholders, so the useful part is the shape. Pasting three examples into every prompt raised the daily cost by more than half, which is invisible in a ten-clause demo and obvious at five thousand calls a day. Cutting output from 120 tokens to 60 shortened the estimated latency by a full second. These are the two cheapest optimisations in most projects: shorter prompts and shorter answers.

Latency has two parts

Response time is roughly the time to first token (network, queueing and processing your input) plus the generation time, which is the number of output tokens divided by the generation speed. Long outputs dominate. Practical consequences: stream tokens to the user so the interface feels responsive long before the answer is complete, put the most important information first, cap output length, run independent calls in parallel, and consider a smaller model for steps that do not need a large one. Measure real latency from your own region and at your own load rather than trusting a demo.

Failing gracefully

Network timeouts and rate-limit responses are routine on any hosted API. The standard answer is retries with exponential backoff and jitter, bounded in number, plus a timeout on every call and a fallback when retries run out. Retry only transient errors (timeouts, rate limits, temporary server failures), never malformed requests, which will fail the same way every time. Related ideas appear in rate limiting.

import random
import time

class TransientError(Exception):
    """Stands in for a timeout or 'rate limited' response from any provider."""

def with_retries(fn, attempts=4, base_delay=0.5, sleep=time.sleep):
    for attempt in range(1, attempts + 1):
        try:
            return fn()
        except TransientError:
            if attempt == attempts:
                raise                                  # give up, let the caller decide
            delay = base_delay * 2 ** (attempt - 1)    # 0.5, 1.0, 2.0 ...
            delay += random.uniform(0, delay / 4)      # jitter so clients do not retry in lockstep
            print(f"attempt {attempt} failed, waiting {delay:.2f}s")
            sleep(delay)

calls = {"n": 0}
def flaky_model_call():
    calls["n"] += 1
    if calls["n"] < 3:
        raise TransientError("timeout")
    return "summary text"

random.seed(1)
print(with_retries(flaky_model_call, sleep=lambda s: None))   # no real waiting in the demo

attempt 1 failed, waiting 0.52s
attempt 2 failed, waiting 1.21s
summary text

The demo passes a no-op sleep so it runs instantly; in production you would use the real one. If a request has side effects, such as sending an email, make it safe to repeat before you retry it.

Choosing between models on evidence

Public leaderboards and marketing claims measure other people's tasks. For your decision, build a test set from your real inputs (the method from the prompt testing lesson), run two or three candidate models with the same prompt and settings, and record for each: task accuracy, how often output fails validation, latency at the percentile you care about, and cost per successful task. The last metric matters because a cheaper model that needs two retries may cost more.

Common mistakes and the symptom each one causes

• Estimating cost from a demo. Symptom: a surprise bill. Fix: multiply tokens per call by real volume before launch.
• Ignoring output length. Symptom: slow responses and high cost from rambling answers. Fix: ask for concise output and cap max tokens.
• Retrying everything. Symptom: duplicated side effects, or a retry storm that worsens a rate-limit problem. Fix: retry only transient errors, with backoff, jitter and a limit.
• No timeout. Symptom: requests hang and tie up workers. Fix: set a timeout on every call.
• Choosing a model from a leaderboard. Symptom: it underperforms on your data. Fix: evaluate on your own cases.
• Hard-coding a model into many places. Symptom: switching or upgrading means a risky multi-file change. Fix: wrap calls behind one function and keep the model name in config.

How this shows up in real projects and interviews

Cost and latency estimates are part of any design discussion for an LLM feature, and a quick back-of-envelope calculation like the one above is a strong move in a system-design round. Expect questions such as "how would you cut cost by half without hurting quality?" (shorter prompts, caching, smaller model for easy cases, fewer calls) and "how do you choose a model?" (your own evaluation set, measured on quality, latency and cost per successful task). Caching of repeated requests is covered in the caching lesson. Be honest about what you cannot know in advance: real prices, limits and speeds change, so you build the measurement habit rather than memorise numbers.