Jobs

A job is one backend call, stored as a row. Watch it, retry it, kill it, from your app or the CLI.

Overview

A job is a row in an outbox table that says “call this backend route with this payload”. Your app writes the row and moves on; the SSP delivers it, retries it on failure, and records the outcome back on the row.

Because the row is ordinary synced data, the whole thing is reactive: your UI can watch a job’s status the same way it watches anything else, offline writes queue like any other mutation, and nothing depends on the client staying connected while the work runs.

// Call a backend route. Returns as soon as the row is written, the SSP
// delivers it to your backend, retrying on failure.
await db.run('api', '/spookify', { id: thread.id }, { assignedTo: thread.id });

spky add api scaffolds the outbox table and wires the backend into sp00ky.yml. See Backend Setup for that walkthrough.

Calling a backend route

db.run(backend, path, payload, options?) takes the backend name from sp00ky.yml, a route path from that backend’s OpenAPI spec, and the payload for it. The payload is validated against the spec, so a missing required argument throws immediately rather than failing later in the runner.

// Idempotent work: retry generously.
await db.run('api', '/generate-content', data, {
  max_retries: 5,
  retry_strategy: 'exponential',
});

// Not idempotent: never retry.
await db.run('api', '/charge-payment', data, { max_retries: 0 });

// Slow work (needs timeoutOverridable: true on the backend).
await db.run('agent', '/chat', data, { timeout: 120 });

// Run no sooner than 5 minutes from now.
await db.run('api', '/send-reminder', { id: reminder.id }, { delay: 5 * 60 * 1000 });

Reading job status

// Jobs are ordinary records, so the query builder reads them, and because the
// query is reactive, the UI follows the job's status without any polling.
const threadQuery = db.query('thread')
  .where({ id: 'thread:abc123' })
  .related('jobs', (q) => q
    .where({ path: '/spookify' })
    .orderBy('created_at', 'desc')
    .limit(1))
  .one()
  .build();

const thread = useQuery(db, () => threadQuery);

const job = () => thread()?.jobs?.[0];
const isRunning = () => ['pending', 'processing'].includes(job()?.status);
const failed = () => job()?.status === 'failed';

status moves pending → processing → success | failed. On success the backend’s response body is stored on the job’s result field, so spky jobs get shows what a job actually returned. On failure each attempt appends { code, reason } to errors, and retries counts what has been tried.

Delayed jobs

A delay keeps a job pending for at least that long. It is a minimum, not a precise schedule: the job becomes eligible when the delay elapses, then runs as soon as the runner picks it up.

A delayed job is just a pending job, so it stays visible and cancellable while it waits. That makes it the natural way to build an undo window:

// Queue the send 30 seconds out, linked to the email record.
await db.run('api', '/send-email',
  { to: email.to, subject: email.subject, body: email.body },
  { assignedTo: email.id, delay: 30_000 }
);

// "Undo": the user owns the email record, so a normal update works, and the
// /send-email handler bails out when the job finally runs.
await db.update('email', 'email:abc123', { retracted: true });

// Backend handler:
if (email.retracted) return { skipped: true };
await mailer.send(email);

Scheduled and multi-step work

Everything above is about jobs your app creates. Work that should happen on a clock is declared in sp00ky.yml instead, and runs whether or not anyone is connected:

# sp00ky.yml, runs server-side, with no client involved.
schedules:
  nightly-cleanup:
    cron: "0 3 * * *"
    backend: api
    route: /cleanupExpired

Each cycle creates a new job row in this same table, so everything on this page still applies to it.

• Schedules: cron and interval cadences, fan-out (one job per row of a query), concurrency policies, spky schedules
• Workflows: several jobs as a DAG, passing each step’s output to its dependents, and the live spky workflows watch view

Inspecting jobs from the CLI

spky jobs reads the outbox tables straight from SurrealDB, so you can watch and manage the queue without building any UI:

spky jobs                      # live dashboard (status counts, recent jobs)
spky jobs list                 # static, scriptable table (alias: ls)
spky jobs list --status failed
spky jobs list --limit 1000 --json
spky jobs get job:abc123       # full payload, result, and error history
spky jobs kill job:abc123      # cancel in-flight, or drop if still queued
spky jobs retry job:abc123     # re-run a terminal job
spky jobs clear                # delete terminal (success/failed) jobs, in batches

# Every command targets the local dev stack by default; --cloud targets the
# deployed database, resolved from "spky login" plus the slug in sp00ky.yml.
spky jobs --cloud

How execution works

The job runner is built into the SSP. There is nothing to install or implement:

1. Pickup is create-only. Writing a pending row fires an event that hands the job to the runner, which is why db.run() creates a row rather than updating one.
2. The runner is the only writer of status. Everything else (kill, retry, recovery) goes through guarded statements, so two writers can never disagree about a job’s state.
3. Retries happen in-process, with linear or exponential backoff. A job that exhausts them is marked failed with its error history intact.
4. A recovery sweep catches what events miss: rows created while the SSP was down, jobs whose delay elapsed during a restart, and processing rows orphaned by a crash. In cluster mode the scheduler owns that sweep and uses assignee to tell whether the owning SSP is still alive.

Concurrency

By default a job table runs one job at a time. Raise it per table:

# sp00ky.yml
apps:
  api:
    type: backend
    method:
      type: outbox
      schema: schema/src/outbox/api.surql
      table: job
      concurrency: 8     # at most 8 of this table's jobs run at once

Above the limit, nothing is queued in memory: the extra rows simply stay pending in the outbox and are admitted in created_at order as slots free. The outbox is the queue. That is what makes a spike safe — a fan-out can create 10,000 rows in one fire, and the table will still only ever have concurrency of them in flight. Because the backlog is rows in the database, it survives a restart, needs no new status, and is visible with the same spky jobs list --status pending you already use.

This is the knob to reach for when a fan-out is more than the backend behind it can take. Everything funnels through the outbox — ad-hoc db.run() calls, scheduled fires, and workflow steps alike — so one limit per table governs all three.

The limit is exact within one SSP process. Across a multi-replica deployment it is a best-effort ceiling: each node also counts the processing rows the others hold and takes the tighter of the two views, so two nodes admitting in the same instant can briefly overshoot before converging. A single-node deployment never runs that count and is exact.

Like retention, the policy lives in the database (_00_job_policy:<table>), written by spky deploy. A throttle is the setting you least want to need a redeploy for:

-- retune live, no redeploy
UPDATE _00_job_policy:job SET concurrency = 20;

Retention

Finished jobs are deleted automatically. Retention is asymmetric by outcome, because the two are read for different reasons: a successful job is inspected once if at all, while a failure is the thing you went looking for.

# sp00ky.yml
retention:
  success: 6h        # job rows that succeeded
  failed: 14d        # job rows that failed
  runSuccess: 24h    # schedule/workflow run history for success|skipped|replaced
  runFailed: 30d     # ... and for failed|killed

pending and processing rows are never pruned, at any age. skipped and replaced follow the success window on purpose: with the default concurrency: skip, a slow wide fan-out writes one skipped row per item per fire, and those are the highest-volume, lowest-information rows in the system.

What survives the delete

Before each batch is deleted it is folded into permanent hourly counters, so “how much ran, and how much of it failed” outlives the rows themselves. That is what makes a 6-hour window acceptable rather than lossy. spky schedules get <name> shows the totals:

$ spky schedules get sync-every-tenant
  ...
  pruned      : 41230 ok  12 failed  388 skipped

The counters are bounded by construction — 24 rows a day per schedule or table — and are written as a by-product of a delete that was happening anyway, so they cost no extra pass over the data.

The row cap

An age window bounds volume at rate x window, which is normally enough. For a fan-out so wide that even a short window holds more than you want to carry, maxRows is a hard ceiling:

retention:
  success: 6h
  maxRows: 50000     # hard ceiling on successful rows per table (0/unset = off)

It is off by default, and deliberately narrow: it only ever trims successful rows (never failed, killed, pending, or processing), it trims the oldest first, and it still folds what it removes into the counters. A table can legitimately sit above the cap if failures alone exceed it. Because it has to count rows, it runs on a slow multiple of the prune cadence rather than every pass.

A single schedule can override the run-history windows, which is what a noisy fan-out needs:

schedules:
  sync-every-tenant:
    every: 1m
    backend: api
    route: /sync
    forEach:
      query: SELECT id FROM tenant
      key: id
    history:
      success: 15m     # this schedule alone keeps far less
      failed: 30d

Job table schema

spky add api generates this. The permission rule is the important part: scope jobs to their owner so one user can neither read nor queue work on another’s records.

DEFINE TABLE job SCHEMAFULL
  PERMISSIONS
    FOR select, create, update, delete
    WHERE $access = "account" AND assigned_to.author.id = $auth.id;

-- What to call, and for whom
DEFINE FIELD assigned_to ON TABLE job TYPE record<thread>
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;
DEFINE FIELD path ON TABLE job TYPE string
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;
DEFINE FIELD payload ON TABLE job TYPE any
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;

-- Retry policy
DEFINE FIELD retries ON TABLE job TYPE int DEFAULT ALWAYS 0
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;
DEFINE FIELD max_retries ON TABLE job TYPE int DEFAULT ALWAYS 3;
DEFINE FIELD retry_strategy ON TABLE job TYPE string DEFAULT ALWAYS "linear"
  ASSERT $value IN ["linear", "exponential"]
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;

-- Optional per-job overrides
DEFINE FIELD delay ON TABLE job TYPE int DEFAULT ALWAYS 0
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;
DEFINE FIELD timeout ON TABLE job TYPE option<int>
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;

-- Lifecycle, written by the platform
DEFINE FIELD status ON TABLE job TYPE string DEFAULT ALWAYS "pending"
  ASSERT $value IN ["pending", "processing", "success", "failed"]
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;
DEFINE FIELD errors ON TABLE job TYPE array<object> DEFAULT ALWAYS []
  PERMISSIONS FOR create WHERE true FOR select, update WHERE false;
-- The element must be FLEXIBLE, or a SCHEMAFULL table rejects the runner's
-- { code, reason } entries as unknown fields.
DEFINE FIELD errors[*] ON TABLE job TYPE object FLEXIBLE;
DEFINE FIELD result ON TABLE job TYPE any
  PERMISSIONS FOR select WHERE true FOR create, update WHERE false;
DEFINE FIELD assignee ON TABLE job TYPE option<string>
  PERMISSIONS FOR select WHERE true FOR create, update WHERE false;

DEFINE FIELD updated_at ON TABLE job TYPE datetime DEFAULT ALWAYS time::now()
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;
DEFINE FIELD created_at ON TABLE job TYPE datetime VALUE time::now()
  PERMISSIONS FOR create, select WHERE true FOR update WHERE false;

See also