SapixDBSapixDB/Docs
Home
Community · SaQL

Cursor Pagination

Page through large result sets without gaps or duplicates using after_hlc — a cursor derived from the timestamp_hlc of the last record on each page.

ℹ Why cursor pagination beats offset paginationOffset-based pagination (SKIP N) re-scans records on every page and drifts when new records are written between requests — you can skip records or see them twice. Cursor pagination anchors to a specific point in the strand using an HLC timestamp, so new writes never disturb in-progress pages and every record appears exactly once.

How It Works

SapixDB assigns every record a timestamp_hlc — a 64-bit Hybrid Logical Clock value that is strictly ordered within a strand. Passing this value as after_hlc in a subsequent request tells the engine to return only records after that position. The comparison is exclusive (>), so the cursor record itself is never repeated.

  1. Send a scan with a limit. Receive up to limit records.
  2. Read timestamp_hlc from the last record in the response.
  3. Send the next request with after_hlc set to that value.
  4. Repeat until the response contains fewer records than limit — that signals the end of results.

Page 1 — First Request

Fetch the first page without after_hlc and extract the cursor from the last record.

bash — fetch page 1 and capture cursor
LAST_HLC=$(curl -s -X POST http://localhost:7475/v1/agents/events/query \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer spx_root_YOUR_ROOT_KEY" \
  -d '{"type": "scan", "limit": 10}' \
  | python3 -c "
import sys, json
r = json.load(sys.stdin)['records']
print(r[-1]['timestamp_hlc'])
")

echo "Cursor: $LAST_HLC"

Page 2 — Using the Cursor

Pass the captured timestamp_hlc as after_hlc in the next request body.

bash — fetch page 2
curl -s -X POST http://localhost:7475/v1/agents/events/query \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer spx_root_YOUR_ROOT_KEY" \
  -d "{\"type\": \"scan\", \"limit\": 10, \"after_hlc\": $LAST_HLC}" \
  | python3 -m json.tool
⚠ Cursor field nameThe cursor you pass in requests is after_hlc. The value you read from each record is timestamp_hlc. These are different names — do not use ts_hlc, which does not exist.

Detecting End of Results

SapixDB does not return a separate "has more" flag. The signal is simple: when the number of records returned is less than your limit, you have reached the end. An empty records array also indicates the end.

SaQL body — end detection logic
{
  "type":      "scan",
  "limit":     10,
  "after_hlc": 1751900065536000
}

// If len(response.records) < 10  →  last page reached

Pagination with Filters

after_hlc and filter compose naturally. The filter is applied first (inside the strand), and after_hlc acts as an additional lower bound on the HLC timestamp. Both constraints are respected simultaneously.

SaQL — paginated filtered scan
{
  "type":      "scan",
  "limit":     10,
  "after_hlc": 1751900065536000,
  "filter": {
    "field": "type",
    "op":    "eq",
    "value": "page_view"
  }
}

Python: Paginate All Records

The following pattern pages through an entire agent strand, collecting every record regardless of total size.

Python — full pagination loop
import requests

BASE    = "http://localhost:7475"
AGENT   = "events"
HEADERS = {
    "Content-Type": "application/json",
    "Authorization": "Bearer spx_root_YOUR_ROOT_KEY",
}
PAGE_SIZE = 100

all_records = []
cursor = None

while True:
    body: dict = {"type": "scan", "limit": PAGE_SIZE}
    if cursor is not None:
        body["after_hlc"] = cursor

    resp = requests.post(
        f"{BASE}/v1/agents/{AGENT}/query",
        json=body,
        headers=HEADERS,
    )
    resp.raise_for_status()
    page = resp.json()["records"]

    all_records.extend(page)

    if len(page) < PAGE_SIZE:
        break  # last page

    cursor = page[-1]["timestamp_hlc"]

print(f"Total records fetched: {len(all_records)}")

Field Reference

FieldInTypeDescription
after_hlcrequest bodyintegerExclusive lower bound. Returns only records whose timestamp_hlc is strictly greater than this value.
timestamp_hlcresponse recordintegerThe HLC timestamp of each record. Use the value from the last record as the next after_hlc.
limitrequest bodyintegerPage size. When the response contains fewer records than this value, the final page has been reached.
✓ Combine with sort orderYou can paginate in descending order by adding "order": "desc". The cursor still comes from timestamp_hlc of the last record in each page response. See Sort Order.