Scheduler API

HTTP contract for the cluster coordinator: ingest, proxy, view registration, SSP lifecycle and job dispatch.

The Scheduler is the central orchestrator that manages SSP sidecars and coordinates data distribution.

Base URL

Default: http://localhost:9667

The bind address comes from the scheduler’s YAML config:

ingest_host: 0.0.0.0   # optional; falls back to 0.0.0.0
ingest_port: 9667

A subset of fields can also be overridden with SPKY_* env vars (SPKY_DB_URL (or legacy SPKY_DB_WS), SPKY_DB_NS, SPKY_DB_NAME, SPKY_DB_USER, SPKY_DB_PASS, SPKY_SCHEDULER_ID, SPKY_SNAPSHOT_UPDATE_INTERVAL_SECS, SPKY_BACKENDS (or legacy SPKY_SCHEDULER_BACKENDS)). The bind host/port are YAML-only.

Ports

The scheduler listens on three ports, and the distinction matters for how you expose it:

Port	Env	Serves	Exposure
`9667`	`ingest_port` (YAML)	`/ingest`, `/proxy/`, `/view/`, `/ssp/`, `/job/`, `/metrics`, `/health`, `/info`, `/backup/`, `/handover/*`, `/db/reconnect`	Private network only
`9668`	`SPKY_ADMIN_PORT`	`/admin` (dashboard) and `/admin/api/*`	Safe to publish
`9669`	`SPKY_POOL_PORT`	`/pool/v1/*`, the machine pool agents’ connection	Publish when pool machines live outside the scheduler’s network

Authentication

Port 9667 is unauthenticated. That includes POST /proxy/query, which runs arbitrary SurrealQL against the scheduler’s replica. Treat reachability of this port as equivalent to full read access to your data, and keep it inside a private network, VPC or firewall.

Port 9668 is authenticated. Every endpoint under /admin/api requires a bearer token except GET /admin/api/config. See Admin dashboard.

This split is why the dashboard has its own listener: it is the one surface meant for a browser, so it can be published without publishing the ingest port alongside it.

Data Ingestion

POST /ingest

Ingest a record change from the database. This endpoint receives database events and broadcasts them to all ready SSP sidecars.

Each event also names one SSP to run any job it creates, sent to the SSPs as job_assignee: a ready SSP by the load-balancing strategy, or a lagging one when none is ready. A lagging SSP (one that missed a delivery and is catching up from its queue) gets the event later, in order and with the same job_assignee, so a job created during a lag runs once that SSP reaches it rather than waiting for a ready SSP. Every copy of an event names the same SSP, and only that one picks the job up.

Request Body:

{
  "table": "users",
  "op": "CREATE",
  "id": "user:123",
  "record": {
    "name": "Alice",
    "email": "alice@example.com"
  }
}

Fields:

table (string, required) - Table name
op (string, required) - Operation: CREATE, UPDATE, or DELETE
id (string, required) - Record ID
record (object, required) - Record data

Response:

200 OK - Durably accepted into the WAL and queued for SSP delivery
400 Bad Request - Invalid operation or malformed request
500 Internal Server Error - Failed to apply to replica or broadcast
503 Service Unavailable - Scheduler is in Cloning state

Example:

curl -X POST http://localhost:9667/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "table": "users",
    "op": "CREATE",
    "id": "user:alice",
    "record": {"name": "Alice", "email": "alice@example.com"}
  }'

Proxy Endpoints

SSPs use these endpoints during bootstrap to query the scheduler’s snapshot replica directly, instead of receiving pushed chunks.

POST /proxy/query

Execute a SurrealQL query against the scheduler’s frozen snapshot replica. SSPs use this to self-bootstrap by pulling the data they need.

Request Body:

{
  "query": "SELECT * FROM users"
}

Response:

200 OK - Query results from the snapshot replica
400 Bad Request - Invalid query
500 Internal Server Error - Query execution failed

Example:

curl -X POST http://localhost:9667/proxy/query \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT * FROM users"}'

POST /proxy/ranges

A table’s id-range hashes: the table’s x3: content hash split into ranges of about 1000 rows in record-id order (at most 1024 ranges, so a bigger table gets bigger ranges), each with its own hash and row count. A warm SSP whose table hash differs from the registration’s compares its own rows range by range and reads only the ranges that differ; a cold bootstrap pages by them. The ranges always add up to the table hash the scheduler hands out at registration.

The scheduler keeps them current with every applied event, persists them with its snapshot state, cuts them whenever it hashes a table from content, and builds them in the background (one range at a time, so draining never waits on it) for tables that have none, such as every table of a replica from before ranges existed.

Request Body:

{ "table": "game" }

Response:

200 OK - { "table", "hash", "starts", "hashes", "counts" }. Range i holds the record keys from starts[i] up to starts[i + 1]; starts[0] is "".
404 Not Found - No ranges for this table: not built yet, its hash is about to be recomputed, or its keys are not plain strings (numeric, uuid, array or object ids). The SSP then lists the whole table.
503 Service Unavailable - The scheduler is restoring from a backup

POST /proxy/signin

No-op endpoint for SurrealDB client compatibility during SSP bootstrap. Returns success without performing authentication.

Response:

200 OK - Always succeeds

POST /proxy/use

No-op endpoint for SurrealDB client compatibility during SSP bootstrap. Returns success without changing namespace/database context.

Response:

200 OK - Always succeeds

View Management

POST /view/register

Register a new view (live query) with the scheduler. The scheduler will assign it to an SSP using the configured load balancing strategy.

Request Body:

{
  "id": "query:abc123",
  "surql": "SELECT * FROM users WHERE active = true",
  "clientId": "client-456",
  "ttl": "30s",
  "params": null,
  "lastActiveAt": "2024-01-01T00:00:00Z",
  "format": null,
  "ownerHash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}

Fields:

id (string, required) - Unique view identifier (e.g. query:abc123)
surql (string, required) - SurrealQL query to materialize
clientId (string, required) - Client identifier
ttl (string, required) - Time-to-live for the view (e.g. "30s")
params (object, optional) - Query parameters
lastActiveAt (string, optional) - ISO 8601 timestamp of last activity
format (string, optional) - Response format
ownerHash (string, optional) - Proof of ownership for an anonymous view, relayed to the SSP and its standbys unchanged. See the SSP’s POST /view/register.

Response:

200 OK - View registered and assigned to an SSP

{
  "query_id": "query:abc123",
  "ssp_id": "ssp-primary-01",
  "assigned_at": 1707654321
}

400 Bad Request - The SSP rejected the payload (rejected); relayed verbatim
403 Forbidden - The SSP refused the shape under the query allowlist (not_allowlisted); relayed verbatim
409 Conflict - The SSP found the view registered under another identity (auth_mismatch), or the view is anonymous and the request does not carry its owner’s hash (owner_mismatch); relayed verbatim
503 Service Unavailable - No SSPs available

A 4xx from the SSP that already holds the view leaves its assignment as it was: someone else’s refused registration of a live view does not move or orphan it.

The SSP’s own verdict is relayed with its status: a 4xx is the client’s problem and reaches it as such, not folded into a 500 that reads as an outage. The body is the SSP’s {"error": "...", "message": "..."} unchanged. See the SSP’s POST /view/register.

Example:

curl -X POST http://localhost:9667/view/register \
  -H "Content-Type: application/json" \
  -d '{
    "id": "query:abc123",
    "surql": "SELECT * FROM users WHERE active = true",
    "clientId": "client-456",
    "ttl": "30s"
  }'

POST /view/unregister

Unregister a view from its assigned SSP.

Request Body:

{
  "id": "query:abc123"
}

Response:

200 OK - Teardown accepted. The tracker is cleared before the response; the SSP is told in the background, so a caller (the _00_dbsp_cleanup DB event under the http transport, which runs inside the transaction that deleted the _00_query row) never waits on the SSP.
An unknown view is also 200 (idempotent).

A teardown is dropped when the same view id was registered again after the delete it stems from (a tab closed and reopened): the registration’s timestamp wins over an older delete. Under the changefeed transport this route is not called by the database at all; the scheduler reads the _00_query DELETE from the feed and applies the same rule.

SSP Lifecycle Management

POST /ssp/register

Register a new SSP sidecar with the scheduler. The scheduler will immediately mark the SSP as bootstrapping and begin sending replica data asynchronously.

Request Body:

{
  "ssp_id": "ssp-primary-01",
  "url": "http://localhost:8667"
}

Fields:

ssp_id (string, required) - Unique identifier for the SSP
url (string, required) - HTTP URL of the SSP (must start with http:// or https://)
replaces (string, optional) - Blue/green: the id of a serving SSP this one is meant to replace (SPKY_SSP_REPLACES). See Blue/green handover.

Response:

202 Accepted - Registration accepted, bootstrap starting
```
{ "snapshot_seq": 123, "table_hashes": { "game": "b3:..." }, "standby": false }
```
standby: true means the scheduler took the SSP in as the standby for replaces: it must not write to the database until it is promoted.
400 Bad Request - Invalid SSP ID (empty) or invalid URL format
409 Conflict - This SSP id was replaced by a successor that is still in the pool (blue/green). The SSP exits; it must not serve beside its successor.

Flow:

SSP sends registration request
Scheduler validates and adds SSP to pool
Scheduler marks SSP as “bootstrapping”
SSP bootstraps itself from the scheduler’s /proxy/query endpoint
Scheduler polls SSP /health until it reports ready
Once ready, scheduler replays buffered events to SSP

Example:

curl -X POST http://localhost:9667/ssp/register \
  -H "Content-Type: application/json" \
  -d '{
    "ssp_id": "ssp-primary-01",
    "url": "http://localhost:8667"
  }'

The optional registration field ingest_batch_limit advertises the number of ordinary records the SSP accepts at /ingest/batch (maximum 128). Missing or zero means single-record delivery. The scheduler preserves this capability through handover, coalesces only already queued events, and checks each final JSON batch against the 1 MiB wire limit. Job, user and _00_* records stay on /ingest. A failed batch and every later undelivered row are buffered in order; replay uses single records, preserving each event’s WAL sequence and assigned job runner.

POST /ssp/heartbeat

Send a heartbeat from an SSP to maintain health status.

Request Body:

{
  "ssp_id": "ssp-primary-01",
  "timestamp": 1707654321,
  "views": 5,
  "cpu_usage": 45.2,
  "memory_usage": 512.5
}

Fields:

ssp_id (string, required) - SSP identifier
timestamp (number, required) - Unix timestamp in seconds
views (number, required) - Number of active views
cpu_usage (number, optional) - CPU usage percentage
memory_usage (number, optional) - Memory usage in MB

Response:

200 OK - Heartbeat accepted
404 Not Found - SSP not registered (SSP should re-register)
503 Service Unavailable - The scheduler is cloning or restoring. A restarted scheduler starts with an empty pool, so it answers 503 rather than 404 until it can accept registrations; the SSP keeps heartbeating with its circuit intact
409 Conflict - The SSP must re-bootstrap. The body is a JSON ResyncDirective:

{ "reason": "Clean restart requested. SSP must drop its snapshot and re-bootstrap.", "clean": true }

clean: true asks the SSP to delete its row checkpoints and arena before it exits, so the relaunch is a cold rebuild. Without it the SSP re-bootstraps in place, keeping its rows and verifying them against the replica table by table. A buffer overflow, an integrity-check resync and an operator restart from the admin dashboard all arrive this way. Older SSPs read the body as free text and restart anyway, and an older scheduler sends free text, which parses as clean: false; both directions degrade to a plain restart, never to a refused one.

Example:

curl -X POST http://localhost:9667/ssp/heartbeat \
  -H "Content-Type: application/json" \
  -d '{
    "ssp_id": "ssp-primary-01",
    "timestamp": 1707654321,
    "views": 5,
    "cpu_usage": 45.2,
    "memory_usage": 512.5
  }'

POST /ssp/bootstrap-progress

Report how far an SSP has got loading its circuit. Advisory only: the value is surfaced on /info and the admin dashboard and is read by nothing that makes a bootstrap decision, so it always answers 200 and is never retried.

Sent by the SSP’s bootstrap loop rather than its heartbeat, because the heartbeat stays silent until the SSP reaches ready — which is exactly the window this describes.

Request Body:

{
  "ssp_id": "ssp-primary-01",
  "tables_done": 12,
  "tables_total": 30,
  "rows_loaded": 148230,
  "current_table": "message"
}

Fields:

ssp_id (string, required) - SSP identifier
tables_done (number) - Tables fully loaded so far
tables_total (number) - Tables to load in total; 0 when not yet known
rows_loaded (number) - Rows loaded across all tables
current_table (string, optional) - Table currently loading

Response:

200 OK - Always, including for an unknown or already-ready SSP

Job Scheduling

POST /job/dispatch

Dispatch a job to be executed on an SSP.

Request Body:

{
  "job_id": "job:123",
  "table": "job",
  "payload": {
    "path": "/api/process",
    "body": {"data": "value"}
  }
}

Fields:

job_id (string, required) - Job record ID
table (string, required) - Job table name
payload (object, required) - Job payload data

Response:

200 OK - Job dispatched, returns the assigned SSP ID string
503 Service Unavailable - No SSPs available

POST /job/result

Report job execution results from an SSP back to the scheduler.

Request Body:

{
  "job_id": "job:123",
  "status": "completed",
  "result": {"output": "processed"},
  "error": null
}

Fields:

job_id (string, required) - Job record ID
status (string, required) - Job status: pending, running, completed, or failed
result (object, optional) - Job result data
error (string, optional) - Error message if failed

Response:

200 OK - Result recorded

Job operator actions

Outbox jobs are routed by the ingest path rather than /job/dispatch, so the scheduler does not know which SSP holds a given job. These three routes work around that; they are what fn::job::kill, fn::job::retry, the recovery sweep and the admin dashboard call.

Endpoint	Body	Behaviour
`POST /job/kill`	`{"id": "job:abc"}`	Broadcast to every ready or lagging SSP, since both run jobs. A kill is idempotent and harmless on SSPs that do not host the job: only the one with the request in flight cancels it, the rest set a kill flag. `200 {id, dispatched, ssps}`; `503 no_ssp` with none ready or lagging
`POST /job/retry`	`{"id": "job:abc"}`	Picks exactly one SSP (a ready one, else a lagging one) and forwards, because a retry re-enqueues into that SSP’s runner and broadcasting would run the job N times. The SSP’s own verdict is relayed with its status: `409 not_terminal` for a job that is still pending or processing, `404` for an unknown one. `200 {id, status: "pending", assigned_to}`
`POST /job/recover`	(SSP route)	Called on one SSP (a ready one, else a lagging one) by the scheduler’s 30 s recovery sweep for `pending` rows nothing picked up and `processing` rows that went stale; the SSP stamps itself as assignee and enqueues them. A `pending` job whose create event is still queued for its assigned SSP is not recovered: it has an owner, which claims it when its queue gets there

Impersonation

Two routes behind the fn::_00_impersonate::* SurrealDB functions (see Admin impersonation). Both return 404 unless SPKY_IMPERSONATION=on and SPKY_AUTH_SECRET is non-empty. The scheduler serves them in cluster mode, where the functions call it instead of an SSP. The rest of the ingest port is unauthenticated; these two routes check the bearer themselves, in constant time.

Authentication: Required (the shared bearer)

Endpoint	Body	Behaviour
`POST /impersonate/mint`	`{"session", "target", "admin", "access", "ns", "db", "ttl_secs", "session_remaining_secs"}`	Signs an HS256 token for the `_00_impersonate` access method, bound to the `_00_impersonation` row in `session`. Expiry is `min(ttl_secs, 1h, session_remaining_secs)`. Stateless: every revocation check runs in the access method’s `AUTHENTICATE` block. `200 {token, exp}`; `400 invalid` for a session that is not an `_00_impersonation` record, a target equal to the admin, or an expired session
`POST /impersonate/users`	`{"table", "fields", "search", "limit"}`	Root-backed search for the DevTools user picker: rows of `table` whose id or any of `fields` contains `search` (case-insensitive), at most `limit` (max 100), each with `is_admin`. `table` and `fields` must be plain identifiers (`400 invalid` otherwise)

Monitoring

GET /metrics

Get scheduler metrics and SSP pool status.

Response:

{
  "scheduler": {
    "total_ssps": 2,
    "ready_ssps": 2,
    "total_queries": 10,
    "running_jobs": 3,
    "uptime_seconds": 3600
  },
  "ssps": [
    {
      "id": "ssp-primary-01",
      "query_count": 5,
      "views": 3,
      "cpu_usage": 45.2,
      "memory_usage": 512.5,
      "last_heartbeat_seconds_ago": 2
    }
  ]
}

Example:

curl http://localhost:9667/metrics

GET /health

Health check endpoint.

Response:

200 OK - At least one SSP is ready
```
{"status": "healthy"}
```
503 Service Unavailable - No SSPs are ready
```
{"status": "unavailable"}
```

The body also carries scheduler (replica lag), heartbeat (the end-to-end probe) and changefeed:

{
  "changefeed": {
    "enabled": true,
    "cursor_ms": 1789416134292,
    "lag_ms": 7,
    "records_1m": 80,
    "doorbell": "connected",
    "doorbell_reconnects": 0,
    "gap": false,
    "stalled": false,
    "last_error": null
  }
}

enabled is false under SPKY_INGEST_TRANSPORT=http. lag_ms is the age of the tail’s cursor, i.e. how far behind the newest commit it can be at most. gap means the cursor fell out of the feed’s retention and the replica is being re-cloned; stalled means no poll has completed for a while. Either one reports the stack degraded.

push is the Web Push engine this scheduler hosts. It is informational and never changes status:

{
  "push": {
    "enabled": true,
    "kid": "0995a9a91bc5dcc9",
    "public_key": "BMeKUJgMb9oF...",
    "subject": "mailto:ops@example.com",
    "rules": 2,
    "rule_names": ["new-message", "alert-nudge"],
    "config_loaded": true,
    "vapid_published": true,
    "totals": { "matched": 12, "messages": 4, "sent": 16, "failed": 0, "dropped": 0, "throttled": 6, "retried": 3, "deduped": 0, "skipped_repair": 0 },
    "last_minute": { "matched": 3, "messages": 1, "sent": 4, "failed": 0, "dropped": 0, "throttled": 1, "retried": 0, "deduped": 0, "skipped_repair": 0 },
    "queues": { "retry": 0, "trailing": 1, "throttle_slots": 2, "messages_in_flight": 0, "dedupe_entries": 12, "cached_users": 2 },
    "last_error": null,
    "providers": {
      "web": true,
      "apns": { "configured": true, "ready": true },
      "fcm": { "configured": true, "ready": false, "error": "fcm service account has no `private_key`" }
    }
  }
}

enabled is false with a reason when nothing can be delivered (no VAPID key because SPKY_AUTH_SECRET and SPKY_VAPID_PRIVATE_KEY are unset, and no usable push.apns / push.fcm credential), SPKY_PUSH=off, or push.enabled: false. providers says what each delivery path can do: web a VAPID key is loaded, apns / fcm a credential is stored (configured) and parses (ready), with the parse error otherwise; the counters also include no_provider (native devices skipped because their provider is not ready). matched counts rule matches, messages direct pushes, sent/failed HTTP sends to push services, dropped pushes over a limit, throttled pushes folded into a trailing one, retried sends put on the retry queue, skipped_repair rows drift repair re-emitted (those never push). The same block is in /info and the admin overview.

Example:

curl http://localhost:9667/health

GET /health/snapshot

Per-table replica record counts and content hashes, plus the scheduler’s own replica-vs-upstream verdict. spky verify reads this.

The replica is cloned from upstream once and then fed only by the per-row ingest events, so rows written while nothing was listening (a bulk migration with the stack down) never reach it, and every SSP that bootstraps from it inherits the gap. The scheduler compares row counts against upstream at startup and after every snapshot drain that leaves nothing buffered. A table that is empty in the replica but not upstream is acted on at once, any other mismatch once it persists across consecutive checks. The table is repaired in place (the rows that differ go through the ingest pipeline); only a repair that cannot fix it re-clones the replica and re-bootstraps every SSP. A table new to the replica (added upstream while the scheduler ran, or with @nosync removed) is backfilled instead: uncapped, in the background, never re-cloned for.

The replica also follows upstream’s table set on the same tick: a table dropped upstream, or marked @nosync, is dropped from the replica (rows and hash) once two consecutive probes agree, and a table’s opaque fields are re-read whenever the schema changes. Both are recorded as schema_change incidents.

Response:

200 OK

{
  "tables": { "thread": 12, "user": 3 },
  "hashes": { "thread": "ab12…", "user": "cd34…" },
  "total_records": 15,
  "snapshot_seq": 230046,
  "latest_seq": 230046,
  "lag": 0,
  "drift": {
    "enabled": true,
    "auto_reclone_enabled": true,
    "checked_at": 1788310193130,
    "tables": { "thread": { "upstream": 12, "replica": 12 } },
    "mismatched": [],
    "stuck": [],
    "last_auto_reclone": null,
    "auto_reclones": 0,
    "last_auto_repair": null,
    "auto_repairs": 0,
    "repair_max_rows": 2000,
    "backfilling": [],
    "last_error": null
  }
}

drift.mismatched lists tables whose counts differed on the last check. drift.stuck lists tables an automatic repair or re-clone did not fix; those need an operator (a row the replica schema rejects, typically). drift.auto_repairs counts in-place table repairs (backfills included), drift.auto_reclones the full re-clones a repair escalated to. drift.backfilling lists the new tables being backfilled right now. drift.upstream is null for a table whose upstream count could not be read.

Example:

curl http://localhost:9667/health/snapshot

GET /info

Get entity information for the scheduler and all registered SSPs.

Response:

200 OK - Entity list

[
  { "entity": "scheduler", "id": "scheduler-abc", "status": "ready", "views": 8 },
  { "entity": "ssp", "id": "ssp-01", "status": "ready", "views": 3 }
]

Example:

curl http://localhost:9667/info

Bootstrap Flow

When an SSP registers, the following poll-based bootstrap process occurs:

sequenceDiagram
    participant SSP
    participant Scheduler

    SSP->>Scheduler: POST /ssp/register
    Scheduler->>Scheduler: Validate & Add to Pool
    Scheduler->>Scheduler: Mark as Bootstrapping
    Scheduler-->>SSP: 202 Accepted {snapshot_seq}

    Note over SSP: SSP bootstraps itself
    SSP->>Scheduler: GET /proxy/query (fetch data)
    Scheduler-->>SSP: Query results

    loop Poll until ready (every ssp_poll_interval_ms)
        Scheduler->>SSP: GET /health
        SSP-->>Scheduler: {"status": "bootstrapping"}
    end

    Scheduler->>SSP: GET /health
    SSP-->>Scheduler: {"status": "ready"}
    Scheduler->>Scheduler: Mark as Ready

    loop Replay buffered events
        Scheduler->>SSP: POST /ingest
        SSP-->>Scheduler: 200 OK
    end

    Note over Scheduler,SSP: SSP is now ready for live updates

    loop Every 5 seconds
        SSP->>Scheduler: POST /ssp/heartbeat
        Scheduler-->>SSP: 200 OK
    end

After the replay and before it routes live traffic to the SSP, the scheduler verifies the catch-up: for every table the replayed events touched it folds those events into the table’s maintained hash (one batched read of the rows they touch, their before-images at the cut, XORed out; each row’s final state XORed in) and compares the result with the hash the SSP reports on GET /info. Only a dirty table, or one without a maintained hash, is read whole for that. A persistent mismatch is checked against the whole-table projection before it counts, then diffed row by row. The Catch-up verification passed line carries verify_ms, rows_read, fast_tables and whole_tables; the final replay logs since_verify_ms and lock_wait_ms.

Message Buffering

While an SSP is bootstrapping, the scheduler buffers incoming messages:

Maximum buffer size: 50,000 messages per SSP (max_buffer_per_ssp in sp00ky.yml, or SPKY_MAX_BUFFER_PER_SSP). A registration always starts with an empty buffer, so an overflow is reported once and the re-bootstrap it asks for begins clean
Buffer exists both globally (event_buffer) and per-SSP
If buffer overflows: SSP marked for re-bootstrap, buffer cleared
Heartbeat returns 409 Conflict when buffer overflow occurs

`POST /db/reconnect`

Bearer SPKY_AUTH_SECRET. SurrealDB moved to another container, as it does in a database upgrade: every upstream handle reconnects now, instead of after minutes of failed probes. Answers 200. The SSP answers the same route with its own bearer secret.

Blue/green handover

Upgrades and restarts replace the scheduler and its SSPs without a moment in which nobody answers. Sp00ky Cloud drives both handovers; a self-hosted setup can drive them the same way.

Scheduler

Start the new scheduler next to the running one, on the same data volume and reachable under the same address, with SPKY_HANDOVER_FROM set to the running one’s base URL and the same SPKY_AUTH_SECRET:

The new process binds its ports at once and holds every request it gets.
It calls POST /handover/prepare on the old one. The old scheduler holds its own traffic, lets in-flight requests finish, stops taking events at a clean boundary, stops its background work, closes its replica and answers with its SSP pool and view assignments. From then on it relays every request to the new one.
The new scheduler opens the replica (no _00_query wipe, no startup drift pass), takes the state over and serves what it held. SSPs keep their registration. Requests are delayed by the handover, typically well under a second, and never refused.
Stop the old scheduler. On SIGTERM a relaying scheduler stops accepting, finishes what it is relaying and exits.

A predecessor on canary.300 or later is handed over in two steps instead, so the hold shrinks to the in-memory export: the new scheduler relays every request to the old one while it calls POST /handover/release (the old one releases its replica and keeps serving), opens the replica and connects upstream, and only then holds and calls POST /handover/commit (the old one holds, exports its state and starts relaying). A predecessor that released and never sees a commit exits after two minutes.

A predecessor that predates handovers answers the prepare with a 404. The new scheduler then relays to it, waits for the replica lock, and boots the ordinary way as soon as you stop the old one.

GET /handover/status

Answered by the process itself whatever its state, even while it holds traffic. Poll it on the new scheduler to know when the old one can go.

{ "supported": true, "role": "active", "phase": "serving", "peer": "http://old:9667", "error": null, "version": "0.0.1-canary.295" }

role: starting (phase relaying, preparing, holding, opening, committing, or waiting_for_lock when the predecessor cannot hand over), active (phase serving, or releasing/released/committing on a predecessor mid-handover), retired (handed over, relaying), failed.

POST /handover/prepare

POST /handover/release and POST /handover/commit take the same body and auth and split this into its two steps; release answers {"released":true}, commit the handover state.

Called by the new scheduler on the old one; bearer SPKY_AUTH_SECRET. Body { "successor": "<host>", "version": "...", "ports": { "main": 9667 } } (ports only when the successor listens elsewhere). Answers the handover state, or 409 {"code":"busy"} while an SSP bootstraps, a snapshot drain, an SSP promotion or a backup runs (the caller retries), or 409 {"code":"gone"} once it has handed over.

SSPs

Start the new SSP with its own SPKY_SSP_ID and advertise address, the same row-checkpoint directory (SPKY_SSP_SNAPSHOT_DIR), its own SPKY_SSP_ARENA_DIR, and SPKY_SSP_REPLACES set to the id of the SSP it replaces. It registers as that SSP’s standby: it loads its rows (warm, from the shared checkpoints), follows every event and receives a copy of every view registration, but gets no views or jobs and writes nothing. Once it has caught up the scheduler promotes it:

View registrations are held for the swap.
The old SSP is taken off the live path and asked to retire (POST /handover/retire): it drains its publication queue and answers with a membership digest per view.
The standby is promoted (POST /handover/promote): it republishes only the views whose membership differs from the old SSP’s, then publishes like any SSP. Views move to it and registrations resume.
The old SSP stays retired (it keeps heartbeating so jobs it is still running are not re-run) until you stop it.

If the swap fails, the old SSP is resumed and gets back every event it missed.

A standby whose predecessor is still coming up (re-registering after a scheduler restart) waits for it. One whose predecessor is not registered at all takes over as an ordinary SSP, and the predecessor is refused with a 409 if it tries to register again, so the two never serve side by side.

GET /handover/ssps

The pool as a handover watches it, plus the last promotions:

{
  "ssps": [
    { "id": "ssp-0", "url": "http://ssp-0:8667", "state": "retired", "standby": false, "replaces": null, "version": "..." },
    { "id": "ssp-0-g1a2b3c", "url": "http://...:8667", "state": "ready", "standby": false, "replaces": null, "version": "..." }
  ],
  "promotions": [
    { "standby": "ssp-0-g1a2b3c", "predecessor": "ssp-0", "phase": "done", "error": null, "started_ms": 0, "finished_ms": 0, "republished": 0 }
  ]
}

state is one of bootstrapping, replaying, ready, lagging, retiring, retired. The swap is complete when the new SSP is ready with standby: false and the old one is retired or gone.

Configuration

Configure the scheduler via sp00ky.yml or environment variables:

# Database connection
db:
  url: "localhost:8000/rpc"
  namespace: "sp00ky"
  database: "sp00ky"
  username: "root"
  password: "root"

# Load balancing strategy
load_balance: "least_queries"  # Options: round_robin, least_queries, least_load

# SSP heartbeat monitoring
heartbeat_interval_ms: 5000
heartbeat_timeout_ms: 15000

# Bootstrap configuration
bootstrap_chunk_size: 1000
bootstrap_timeout_secs: 120

# Replica storage
replica_db_path: "./data/replica"

# WAL (Write-Ahead Log): append-only segments in `wal/` next to this path
# (`./data/wal/event_wal.<seq>.seg`). A JSON Lines log found at the path
# itself (the format before canary.297) is migrated once and renamed `.migrated`.
wal_path: "./data/event_wal.log"

# Server configuration
ingest_host: "0.0.0.0"
ingest_port: 9667

# SSP polling and buffering (the poll is how the scheduler learns a
# bootstrapping SSP is ready, so its interval is paid on every restart)
ssp_poll_interval_ms: 500
max_buffer_per_ssp: 50000

# Snapshot update interval (seconds)
snapshot_update_interval_secs: 300

# Job tables (tables that trigger job execution)
job_tables:
  - "job"

Environment Variables (overrides for YAML fields):

SPKY_SCHEDULER_ID - Unique scheduler identifier (defaults to scheduler-<uuid>)
SPKY_DB_URL - SurrealDB URL (overrides db.url; HTTP engine, ws:// values are accepted and normalized). SPKY_DB_WS is read as a legacy fallback.
SPKY_DB_NS - SurrealDB namespace (overrides db.namespace)
SPKY_DB_NAME - SurrealDB database (overrides db.database)
SPKY_DB_USER - SurrealDB username (overrides db.username)
SPKY_DB_PASS - SurrealDB password (overrides db.password)
SPKY_IMPERSONATION - on serves /impersonate/mint and /impersonate/users (default: off; set from impersonation.enabled)
SPKY_SNAPSHOT_UPDATE_INTERVAL_SECS - Override for snapshot_update_interval_secs (default: 300)
SPKY_WAL_SEGMENT_MB - Size at which the event WAL rotates to a new segment file. A drain unlinks the segments it covered instead of rewriting the log, so a bigger segment only means a few more already-applied events are kept until the next drain (default: 16)
SPKY_DRIFT_CHECK - Compare replica row counts against upstream at startup and after each snapshot drain (default: true). Readiness waits only for the tables the replica holds no row of, the one mismatch acted on at first sight; the full count runs right after the scheduler turns ready, so SSPs can register while it does
SPKY_DRIFT_AUTO_RECLONE - Act on a confirmed mismatch (default: true; false reports only, via /health/snapshot, /metrics and spky verify). The table is repaired in place: its ids and _00_rvs are compared with upstream, and the rows that are missing, stale or deleted go through the ingest pipeline as ordinary events, so the replica and every SSP converge without a bootstrap. Only a repair that finds no difference, would exceed SPKY_DRIFT_REPAIR_MAX_ROWS, or fails falls back to re-cloning the replica and re-bootstrapping every SSP. A table still off after either is reported as stuck until its counts change
SPKY_DRIFT_CONFIRM_TICKS - Consecutive checks a non-zero count mismatch must persist before it is acted on; an empty replica table upstream has rows for is acted on at first sight (default: 2)
SPKY_DRIFT_RECLONE_COOLDOWN_SECS - Minimum spacing between automatic re-clones (default: 3600)
SPKY_DRIFT_REPAIR_MAX_ROWS - Most rows one in-place repair of a table the replica already held may send through the ingest pipeline (default: 2000). A table new to the replica is backfilled with no cap
SPKY_DRIFT_REPAIR_TIMEOUT_SECS - Deadline for one table repair, and the longest any repair (a backfill included) may go without finishing a page; on the deadline it is abandoned, the upstream connection is replaced and the repair is retried on the next check (default: 300)
SPKY_DRIFT_CHECK_TIMEOUT_SECS - Deadline for one drift check. The check is the last step of the snapshot updater’s tick, so a check that never answers would stop the replica draining; on the deadline it is abandoned, the upstream connection is replaced and the tick finishes (default: 120)
SPKY_DRIFT_COUNT_CONCURRENCY - Upstream count() queries one drift check keeps in flight. Each is a full scan of its table, so counting one at a time makes the check as slow as the sum of every table; 1 restores that, for an upstream with no CPU to spare (default: 4)
SPKY_BACKENDS - JSON-encoded backend health-check list (SPKY_SCHEDULER_BACKENDS accepted as legacy fallback)
SPKY_BOOTSTRAP_PAGE_SIZE - Rows per page when the replica clone or a drift repair reads a table from upstream; the table is cut into id windows of three quarters of a page (default: about 32 MiB of rows by a one-row probe, at most 2000)
SPKY_CLEAR_VIEWS_ON_START - Wipe _00_query at startup (default: true). false keeps registered views across a scheduler restart: the replica clones them, SSPs re-register them at bootstrap, clients never notice; the SSP TTL sweep retires expired ones. Single-SSP tenants only until scheduler-owned view assignment lands.
SPKY_AUTH_SECRET - Optional shared secret; when set, SSP clients are required to send Authorization: Bearer <secret> on /proxy/*

Every other field (ingest_host, ingest_port, load_balance, heartbeat_*, bootstrap_chunk_size, bootstrap_timeout_secs, replica_db_path, wal_path, ssp_poll_interval_ms, max_buffer_per_ssp, job_tables) is set via sp00ky.yml. Only bootstrap_timeout_secs (SPKY_BOOTSTRAP_TIMEOUT_SECS) and max_buffer_per_ssp (SPKY_MAX_BUFFER_PER_SSP) also answer to an environment variable.

Error Handling

Common Status Codes

200 OK - Request successful
202 Accepted - Request accepted for async processing
400 Bad Request - Invalid request format or parameters
404 Not Found - Resource not found (e.g., unregistered SSP)
409 Conflict - State conflict (e.g., buffer overflow)
403 Forbidden - Relayed from the SSP on /view/register: the query shape is not allowlisted (not_allowlisted)
500 Internal Server Error - Server error
503 Service Unavailable - No SSPs available

SSP Health Monitoring

The scheduler monitors SSP health via heartbeats:

SSPs should send heartbeats every 5 seconds (configurable)
Scheduler marks SSPs as stale after 15 seconds without heartbeat (configurable)
Stale SSPs are removed from the pool
Queries assigned to stale SSPs are reassigned to healthy SSPs

Persistent incident API

These authenticated, read-only endpoints are on the admin listener:

GET /admin/api/incidents: newest-first history. Optional filters: component, state (open, recovered, interrupted, failed, recorded), severity (warning, info), since and before (start-time epoch milliseconds). offset defaults to 0; limit defaults to 50 and is clamped to 1-200. Returns incidents, total, offset, limit, storage_error, and server_time_ms.
GET /admin/api/incidents/:id: returns incident, or 404 after retention removed it.
GET /admin/api/overview includes incidents with open, total, retention_days, and storage_error.

kind names what happened. Besides the SSP and scheduler lifecycle kinds, two are worth knowing because nothing else reports them: backend_down opens when a backend has failed its health check for longer than a minute and recovers when it answers again (its component is the backend name), and schedule_quarantined opens when a schedule’s per-key failure budget stops firing one forEach key (its component is schedule:<name>/<key>) and recovers when that key fires again, after a passing probe or a release. See Failing keys. hot_views (component views) opens when a registered view updates at least SPKY_ADMIN_HOT_VIEW_UPDATES_PER_SEC times a second and recovers once every view has stayed under that rate for a minute. See Hot views.

An incident carries id, component, kind, severity, state, started_at, ended_at, max_buffered_events, event_count, and events. Each event has at, component, kind, state, summary, operation_id, and version. Times are epoch milliseconds. The backlog is sampled, so it is a lower bound on the true peak. Event summaries are sanitized descriptions, not raw SQL. MCP tools incidents_list and incident_get share these endpoints and permissions. See incident retention and restart semantics.

Incident records may also include publication, the last nonempty SSP heartbeat snapshot sampled while the episode was open, plus max_publication_operations, max_publication_bytes, and max_publication_age_ms. These are sampled peaks, not a complete transaction trace. publication.worst_views contains at most eight query IDs and backlog sizes, with no source rows or query bindings. Older history files and SSPs omit these optional details. Each SSP in the overview exposes the latest publication snapshot, or null when unavailable.

Slow sync operation history

GET /admin/api/views/history is an authenticated, read-only admin endpoint. Optional filters: kind (registration, ingest, ingest_batch), since (inclusive epoch milliseconds) and limit (default 50, clamped to 1–100). It returns samples newest first, total after filtering, limit, capacity (100), slow_ms (250), server_time_ms and a measurement description. An unavailable upstream database returns 503; invalid kinds return 400.

The private _00_sync_evidence:slow_operations record retains the newest 100 recorded successful registration and ingest requests taking at least 250 ms across SSP/scheduler restarts and view TTL cleanup. Recording is best effort: at most four evidence writes run concurrently per process; excess requests increment ssp_slow_evidence_dropped_total and do not wait or spawn another writer. Samples contain masked query shapes, software version and registration/ingest stages; registration samples also include selected indexes and result cardinality. They exclude raw SQL, bindings, authentication and session identifiers. Older deployments return an empty history until their first slow request. Live view inspection includes optional registration_stages and ingest_stages; publication queue age and DB transaction timing are separate, available in metrics and incident history. MCP view_history uses the same endpoint and permissions.