REST API Pagination, Filtering, Sorting and Versioning Explained

Here is a bug that passes every test: a marketplace app lists four second-hand textbooks per page. A user reads page one, a seller posts a new listing, the user taps "next", and the book at the bottom of page one appears again at the top of page two. Nothing crashed, and no exception was logged; the pagination simply does not do what the user assumes. Offset pagination has that flaw built in, and the fix is a different kind of pagination, not a bug fix. We will reproduce it first, then fix it.

Offset and cursor pagination

Offset pagination takes limit and offset (or page and page_size): skip the first N rows and return the next limit. It is trivial to build and lets users jump to page 40. But position is counted from the top of a list that keeps changing, and on a large table the database still has to walk past all the skipped rows.

Cursor (keyset) pagination returns an opaque next_cursor that encodes where the last item was, for example "the id of the last listing you saw". The next request says "give me items older than that". Position is anchored to a row rather than to a count, so inserts at the top cannot shift your page, and the database can use an index to seek straight to the position. The cost: no jumping to an arbitrary page, and the sort order has to be stable and unique (often the id or a timestamp plus the id).

import requests

BASE = "http://localhost:5103"


def ids(resp):
    return [item["id"] for item in resp.json()["items"]]


def new_listing(title):
    body = {"title": title, "subject": "maths", "condition": "good", "price": 150}
    requests.post(f"{BASE}/v1/listings", json=body).raise_for_status()


print("--- offset pagination")
page1 = requests.get(f"{BASE}/v1/listings", params={"limit": 4, "offset": 0})
print("page 1:", ids(page1))
new_listing("posted between the two requests")
page2 = requests.get(f"{BASE}/v1/listings", params={"limit": 4, "offset": 4})
print("page 2:", ids(page2), " <- 9 appears twice")

print("--- cursor pagination")
first = requests.get(f"{BASE}/v1/listings", params={"limit": 4, "paginate": "cursor"}).json()
print("page 1:", [i["id"] for i in first["items"]])
new_listing("posted again")
second = requests.get(f"{BASE}/v1/listings", params={"limit": 4, "cursor": first["next_cursor"]}).json()
print("page 2:", [i["id"] for i in second["items"]], " <- no repeats")

--- offset pagination
page 1: [12, 11, 10, 9]
page 2: [9, 8, 7, 6]  <- 9 appears twice
--- cursor pagination
page 1: [13, 12, 11, 10]
page 2: [9, 8, 7, 6]  <- no repeats

The offset client saw listing 9 twice. The cursor client asked for "everything before id 10" and received 9, 8, 7, 6 even though listing 14 had been posted in between. The client script above talks to the server below, which supports both styles.

The listings server

import base64
import json
from flask import Flask, jsonify, request

app = Flask(__name__)
app.json.sort_keys = False

SUBJECTS = ["physics", "maths", "chemistry", "literature"]
CONDITIONS = ["new", "good", "worn"]
# 12 second-hand textbooks; id 12 is the newest listing.
LISTINGS = [
    {"id": i, "title": f"Textbook {i:02d}", "subject": SUBJECTS[i % 4],
     "condition": CONDITIONS[i % 3], "price": 100 + (i * 37) % 400}
    for i in range(1, 13)
]
SORTABLE = {"price", "id", "title"}


def shape(item, version):
    out = dict(item)
    if version == "v2":  # breaking change: price becomes an object
        out["price"] = {"amount": item["price"] * 100, "currency": "INR"}
    return out


def bad(message):
    return jsonify(title="Bad Request", detail=message, status=400), 400


@app.get("/<version>/listings")
def listings(version):
    if version not in ("v1", "v2"):
        return jsonify(title="Not Found", status=404), 404
    args = request.args
    items = list(LISTINGS)

    # filtering
    for field in ("subject", "condition"):
        if field in args:
            items = [i for i in items if i[field] == args[field]]
    if "max_price" in args:
        items = [i for i in items if i["price"] <= int(args["max_price"])]

    # sorting: ?sort=-price,title  (a leading minus means descending)
    for key in reversed(args.get("sort", "-id").split(",")):
        name = key.lstrip("-")
        if name not in SORTABLE:
            return bad(f"cannot sort by '{name}'; allowed: {', '.join(sorted(SORTABLE))}")
        items.sort(key=lambda i: i[name], reverse=key.startswith("-"))

    limit = min(int(args.get("limit", 4)), 50)
    body = {}
    if "cursor" in args or args.get("paginate") == "cursor":
        # keyset pagination, newest first; the cursor is opaque to clients
        items.sort(key=lambda i: i["id"], reverse=True)
        if args.get("cursor"):
            last_id = json.loads(base64.urlsafe_b64decode(args["cursor"]))["id"]
            items = [i for i in items if i["id"] < last_id]
        page = items[:limit]
        body["items"] = [shape(i, version) for i in page]
        more = len(items) > limit
        body["next_cursor"] = (
            base64.urlsafe_b64encode(json.dumps({"id": page[-1]["id"]}).encode()).decode()
            if more else None
        )
        resp = jsonify(body)
    else:
        offset = int(args.get("offset", 0))
        page = items[offset:offset + limit]
        body["items"] = [shape(i, version) for i in page]
        body["page"] = {"limit": limit, "offset": offset, "total": len(items)}
        resp = jsonify(body)
        resp.headers["X-Total-Count"] = str(len(items))
        if offset + limit < len(items):
            resp.headers["Link"] = (
                f'</{version}/listings?limit={limit}&offset={offset + limit}>; rel="next"'
            )
    if version == "v1":
        resp.headers["Deprecation"] = "true"
        resp.headers["Sunset"] = "Thu, 31 Dec 2026 23:59:59 GMT"
    return resp


@app.post("/v1/listings")
def add_listing():
    body = request.get_json()
    new_id = max(i["id"] for i in LISTINGS) + 1
    LISTINGS.append({"id": new_id, **body})
    return jsonify(id=new_id), 201


if __name__ == "__main__":
    app.run(port=5103)

Three details in that file are worth copying. First, limit is capped at 50, because an API that lets clients ask for a million rows will eventually be asked for them. Second, sort is checked against a whitelist (SORTABLE) before it is used; passing user input straight into an ORDER BY or a dynamic attribute lookup is how injection bugs and accidental data exposure happen. Third, the cursor is opaque base64 JSON, so clients cannot depend on its internals and you can change the encoding without a version bump.

Filtering and sorting in practice

Filters are query parameters named after fields (subject=physics, max_price=200); sorting uses a single sort parameter with comma-separated fields and a leading minus for descending (sort=-price,title). Keep the vocabulary small and predictable, validate every value, and reject unknown ones with a 400 that names the valid options rather than silently ignoring them.

curl -s -i "http://localhost:5103/v1/listings?subject=physics&sort=price&limit=2"

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 216
X-Total-Count: 3
Link: </v1/listings?limit=2&offset=2>; rel="next"
Deprecation: true
Sunset: Thu, 31 Dec 2026 23:59:59 GMT

{"items":[{"id":12,"title":"Textbook 12","subject":"physics","condition":"new","price":144},{"id":4,"title":"Textbook 04","subject":"physics","condition":"good","price":248}],"page":{"limit":2,"offset":0,"total":3}}

The body carries a page block, and the headers repeat the useful parts: X-Total-Count is a common (not standardised) convention, while Link with rel="next" follows the web-linking format, so clients can follow pages without constructing URLs. The two extra headers at the bottom are covered in the versioning section. Now a bad sort field:

curl -s -w "\n-> %{http_code}\n" "http://localhost:5103/v1/listings?sort=colour"

{"title":"Bad Request","detail":"cannot sort by 'colour'; allowed: id, price, title","status":400}

-> 400

Versioning without breaking your clients

Versioning exists to let you make a breaking change without breaking the apps already in users' pockets, which you cannot force to update. First decide what counts as breaking.

Path versioning is the most common choice and the one used above. The server serves v1 and v2 from the same code; only the shape function differs. Compare the same listing from both versions:

curl -s "http://localhost:5103/v1/listings?limit=1&sort=price"
curl -s "http://localhost:5103/v2/listings?limit=1&sort=price"

{"items":[{"id":11,"title":"Textbook 11","subject":"literature","condition":"worn","price":107}],"page":{"limit":1,"offset":0,"total":12}}
{"items":[{"id":11,"title":"Textbook 11","subject":"literature","condition":"worn","price":{"amount":10700,"currency":"INR"}}],"page":{"limit":1,"offset":0,"total":12}}

In v1 the price is a plain number; in v2 it is a money object, which is exactly the kind of breaking change that justifies a new version. Old versions should announce their retirement: the v1 response in the earlier example carried Deprecation: true and a Sunset header with a date, which tooling and attentive client developers can read. Choose a sunset date you will actually honour, tell consumers through the docs and the headers, and watch the traffic to v1 fall before removing it.

Common mistakes and the symptoms they produce

• Offset pagination on a fast-changing feed. Symptom: duplicates and missing items between pages, as in the drift demo. Use a cursor for feeds, activity logs and anything sorted by newest.
• No upper bound on `limit`. Symptom: one client request returns the whole table, memory spikes, and the response times out.
• Cursor pagination with a non-unique sort key. Symptom: items sharing a timestamp are skipped or repeated at page boundaries. Break ties with the id.
• Passing `sort` straight into a query. Symptom: sorting by an internal column reveals data that was never meant to be exposed, or a crafted value produces a database error. Whitelist the allowed fields.
• Silently ignoring unknown filters. Symptom: a typo like subjct=physics returns every listing and the client believes the filter worked. Return 400.
• Creating `v2` for every small change. Symptom: you maintain five versions and bugs get fixed in two of them. Add optional fields without a new version, and save versions for real breaks.

How this shows up in real projects and interviews

Almost every list endpoint in a real product ends up with limit, a filter or two, and a sort, and the pagination choice is a classic review discussion. In system design interviews, "how would you paginate the feed?" is a favourite, and the answer that stands out is cursor-based pagination with a unique sort key and a reasoned remark about offset cost on large tables. Versioning comes up as "how do you evolve an API with mobile clients you cannot force to update?". For the database side of why a seek is cheap, see Database Indexing. Last in this course: making the API dependable when networks and clients misbehave, in REST API Errors, Idempotency, Headers and OpenAPI.