Admin dashboard
The operator dashboard the scheduler serves at /admin: sign-in, cluster overview, restart and backup actions, backend health, live logs and workflow runs.
Every scheduler serves an operator dashboard. It shows the health of the scheduler and its SSPs, the end-to-end sync latency, the status of your backends, live logs, and workflow runs as they happen.
Incident history
The Incidents navigation item keeps automatic lag, heartbeat failures, bootstrap integrity failures, removal/re-registration, recovery, and operator actions in one timeline. Every navigation entry has a labeled icon on desktop and mobile. The overview carries an Incidents tile: the open count, the last 24 hours as a strip with one marker per episode (as wide as it lasted, pulsing while open), and a link to the latest one.
The page opens on a rail (open now, count in the range, longest episode, latest) above a day-grouped feed. Filters are chips that apply the moment they are clicked: range (24 h, 7 days, 30 days), state and component, each with its count, plus a search over kind and summary. The chips filter the newest 400 episodes of the range on the client, so switching between them costs no request. Open an incident to see its duration, largest sampled event backlog, server version, operation id when present, and event timeline. Recovered means the scheduler observed recovery. Interrupted means the scheduler restarted before it could observe recovery. Lag followed by replay is described separately from an SSP re-registering; re-registration alone does not prove a container crash.
History is stored in incidents.json beside the scheduler WAL, independently
of SurrealDB. Keep that directory on persistent storage. Retention is bounded
by 30 days, 2,000 incidents, 64 events per incident (first and newest retained),
and an 8 MiB file cap. Counts include compacted events. Ordinary events flush
once per second; a scheduler restart or cloud restart request flushes its action
before exit/dispatch. Abrupt termination can lose the last unflushed second.
A clean restart that wipes the scheduler volume also wipes its incident history.
A missing or unreadable history file is reported in the dashboard; malformed
history is not overwritten. The incident logging target stays enabled even
with RUST_LOG=warn. A dropped log-feed segment becomes a visible history-gap
record. Stored summaries contain no query bindings, SQL, or credentials.
Detailed logs remain available through the Logs page and are independently
subject to their retention limits. History starts when this version is deployed;
it does not reconstruct older incidents from arbitrary log text.
The read-only MCP tools incidents_list and incident_get expose the same
history as the dashboard. See the scheduler API.
Opening it
The dashboard is bundled into the official mono424/spooky-scheduler image and
served on its own port:
http://<scheduler-host>:9668/adminWhy a second port
Everything on the ingest port (9667) is unauthenticated by design — including
POST /proxy/query, which runs arbitrary SurrealQL against the scheduler’s
replica. That port is meant to be reachable only from inside your deployment’s
private network.
The dashboard is the first surface intended for a browser, so it gets a listener
of its own. Publish 9668; keep 9667 private.
docker run -p 9668:9668 mono424/spooky-scheduler # dashboard reachable
# 9667 stays on the internal networkSigning in
Two ways in on the scheduler’s own page, and a third from the Sp00ky Cloud console.
Admin accounts (the normal path)
Any account on the _00_admin roster can sign in with its ordinary application
credentials. Grant access with the CLI:
spky admin add alice
spky admin list
spky admin remove aliceThe scheduler signs in to your database through your app’s own record-access
method, then checks the resulting user against _00_admin. Nothing extra to
configure, and no second password to manage.
The access method is discovered from INFO FOR DB. If your schema defines
several record accesses and you want to pin one, set SPKY_ADMIN_ACCESS. Both
$username and $email are tried as the identifier parameter, so either
spelling of a SIGNIN clause works.
Break-glass password
Set SPKY_ADMIN_PASSWORD to enable a password-only login that needs no
database:
SPKY_ADMIN_PASSWORD='…'This exists because the roster path needs a working database, and a database that is down is exactly when you most want the dashboard. A session established this way is flagged with a persistent banner in the UI.
Leave the variable unset to disable this path entirely; there is no default password.
From the Sp00ky Cloud console
On a cloud-linked scheduler, a member of the project’s workspace can open its
dashboard from console.sp00ky.cloud without an account
on the roster. The control plane signs a one-minute grant for that one
project (an EdDSA JWT), and the console posts it to POST /admin/api/session
as {grant}. The scheduler accepts it only if it is signed with a key from
the control plane’s /.well-known/jwks.json, names this project as its
audience, was issued moments ago, and has not been used before; it answers
with a cloud session of 15 minutes, which the console renews from a fresh
grant. Removing someone from the workspace therefore ends their access within
15 minutes. Self-hosted schedulers have no cloud link and refuse grants.
Session behaviour
How long a session lives depends on whether the scheduler has a cluster secret:
- With
SPKY_AUTH_SECRETset (always the case on Sp00ky Cloud), tokens are self-describing and HMAC-signed with a key derived from that secret. They verify without any server-side state, so they survive a scheduler restart, including one you trigger from the dashboard. Signing out adds the token to an in-memory revocation set; a restart forgets that set, so a revoked token comes back for the remainder of its own expiry. That trade is deliberate and is the whole cost of not logging you out when you press Restart. - Without a secret there is nothing to sign with that outlives the process, so sessions are random tokens held in memory and a restart means signing in again.
GET /admin/api/config reports which mode is active as sessions_persistent.
GET /admin/api/me reports the session’s mode (roster, breakglass, or
mcp for a token minted on the Access page) and its scope.
Failed logins are rate limited to 10 attempts per address per 5 minutes, and every failure returns the same response regardless of cause, so the endpoint cannot be used to discover which usernames exist.
Standalone use
The same bundle runs outside a scheduler. Opened from anywhere else, the sign-in form grows an endpoint field; point it at a scheduler’s admin port and it behaves identically.
pnpm --filter @spooky-sync/dashboard dev
# then enter http://your-scheduler:9668 in the endpoint fieldThe dashboard authenticates with a bearer token in localStorage, keyed per
endpoint, rather than a cookie — a cross-origin cookie would need
SameSite=None; Secure, which browsers refuse over plain HTTP, and plain HTTP
at an IP address is exactly how a scheduler is usually reached.
What it shows
Overview — a grid of tiles, one per question: the end-to-end sync latency
with a 30-minute chart as the large tile, then SSPs ready, backends healthy,
ingest lag and the scheduler’s own details beside it. The page paints from the
last state it saw the moment it opens (marked refreshing) and swaps in the
live poll a moment later, so a reload never shows an empty frame. A tile whose subject is
degraded is tinted, so the problem is the first thing on the page. The
latency is the scheduler’s own heartbeat probe: it writes _00_heartbeat:probe
upstream and times the full round trip through /ingest, the WAL, the
broadcast and every SSP’s circuit. It measures the real sync path, not a
synthetic ping.
SSPs — per-processor status, live view count, uptime and environment. An SSP that is bootstrapping or replaying shows a progress bar: tables loaded and rows read when the SSP reports it, and otherwise the phase with its elapsed time against the scheduler’s bootstrap budget.
Backends — status and response time for everything in SPKY_BACKENDS, with
a detail page carrying a ~30-minute response-time history and the resolved
healthcheck URL. Environment values are masked by the scheduler before they
leave it. On Sp00ky Cloud each deploy pushes the list; the scheduler stores the
last pushed list (env values masked) in _00_scheduler_state and reads it back
when it starts, so a restart or reclone between deploys keeps its backends. A
list in SPKY_BACKENDS always takes precedence. A backend that runs on a machine pool (run_on: { pool: ... }) is
not probed: between jobs there may be no machine to answer, and that is not an
outage. Its status is what the pool sweep last saw: idle at zero machines with
nothing queued, starting while a machine boots or a job waits for one,
healthy with a ready machine, unhealthy only when the pool’s boot-failure
breaker is open or its last pass failed. No backend_down incident is ever
opened for it; the pool’s own pool_breaker_open incident carries the reason
when it cannot get a machine.
Runs — schedules, workflow runs and jobs on one page, because they are one machine seen from three sides: a schedule fires either a job (a row in an outbox table that a backend runs at a path) or a workflow run, and every step of a workflow run is itself a job. On separate pages one failure showed up three times; here it shows up once.
Top to bottom:
- The numbers: running, waiting (with the oldest wait), jobs done per minute,
and failures in the last hour. Stalled is called out separately from
running: a
processingrow whose lease has expired, meaning nobody is working on it and a recovery sweep will reclaim it on its own clock. - Schedules: each one’s cadence in words, its last twelve fires as squares (red where a fire failed; click one to open what it ran), the last and next fire, and Run now and Pause on hover.
- Runs: everything that executed, running and waiting first. A workflow run
is one row and its step jobs live inside it; a job of its own (from a
kind: jobschedule, a durable instance, or your own code, which belongs to no schedule and no workflow) is one row. Each row carries its failure reason, so four failed runs do not cost four clicks to tell one problem from four. Filter by status, kind, what started it, schedule or queue, or search by name, id or error. A filter the scheduler can answer is pushed into SurrealQL rather than applied to the page already fetched, so failures on a busy table are found even when they are older than the newest page. - Queues: one row per outbox table, running against its
_00_job_policyconcurrency, its pace, what waits and what failed.
A run, job or schedule opens in a sheet beside the list. The sheet is part of
the address (/admin/runs?open=run:<id>, job:<id>, schedule:<name>), so it
survives a reload, and the old /admin/jobs/<id>, /admin/workflows/<id> and
/admin/schedules/<name> links land on it.
- A workflow run draws every step on the run’s own time axis, so where the time went is visible at a glance. A step still waiting on its dependencies is a dashed line rather than a bar, because it has not run. Click a step to open its job, which is where a failure says what actually happened: the step row keeps only the attempt that ended the job, while the job keeps every attempt.
- A schedule shows its definition, the last 24 hours of outcomes, recent
fires with why each ended as it did, and any quarantined keys with a way to
release them. The tally comes from
_00_run_rolluprather than counting run rows, so it stays accurate after retention has pruned them.
The scheduler runs one shared poller behind the live run list and one job
sampler behind the queue figures, so ten open dashboards cost your database
exactly what one does, and a closed one costs nothing. Counts cover work in
flight plus the last hour of terminal work, never all of history: once
retention keeps failures longer than successes, an all-time ratio reports a
failure rate that climbs as successes age out. A bounded window is the only
honest denominator, and it is the one spky jobs uses too.
Every job aggregate is a scan of one of your own tables, so the scheduler takes them once
, in a single sampler that every reader serves from — the same bargain the presence sampler
strikes. It reads every 15 seconds while nobody has the page open and every 2 seconds while
someone does, so ten open dashboards cost what one does and an idle cluster costs almost nothing.
Tune both with SPKY_ADMIN_JOB_INTERVAL_SECS and
SPKY_ADMIN_JOB_LIVE_INTERVAL_SECS, or set the first to 0
to switch the sampler off entirely: the list and its filters still work, the totals and the
Overview then say they are not being measured rather than showing zeros.
Logs — an explorer over the scheduler’s or any SSP’s log output. Opening
it loads the source’s whole log ring (up to 10,000 lines), then follows the
live tail. Above the lines, a histogram shows their volume over time with
warnings and errors stacked in their colours. Drag across it to pick a time
window (drag inside the window to move it, drag its edges to resize it), and
the pill over the window zooms into it, back out, or clears it. The window
becomes a time chip in the search beside the level and target filters, and
the table follows as you drag. Lines are listed newest first; click one to read
it whole beside the table, with its fields one per row and how often a line of
the same shape appears in the buffer. Keys: / focuses the search, ↑/↓
move through the lines, Escape closes the line, then clears the window, then
zooms out. If a viewer falls behind the stream, the count of skipped lines is
shown rather than hidden.
Backends have no log stream: the scheduler reaches them with HTTP health checks and has no access to their output.
Resources (Sp00ky Cloud only) — what the deployment runs on, as the control plane measures it: every container with its CPU, memory (against its limit), disk and network, with the last 15 minutes of CPU and memory as sparklines; the dedicated machines and machine pool VMs; and the bucket volume with how much of it is used and free. A container’s disk is what its volumes take (the database’s data, the scheduler’s replica and event log, an SSP’s snapshot and arena) against that plus what is still free on the disk of the VM they run on, so a VM filling up shows on every container on it. On a Pro project with its own VMs the database is on the database VM and the rest on the project host. The scheduler cannot see its neighbours’ containers or the database’s volume, so all of it comes from Sp00ky Cloud through the cloud link: container samples every 15 seconds, disk and the bucket volume every 5 minutes. The Overview’s database and scheduler tiles and each SSP’s row show the same disk figures. The page is not in the navigation of a scheduler that is not linked.
Actions
Everything the dashboard can do, as opposed to show, goes through the
scheduler. Every action logs the signed-in subject, and the long-running ones
answer 202 with an operation the dashboard watches (see below).
Restarting an SSP
The scheduler never touches a container. Restarting an SSP means flagging it
in the pool; the SSP’s next heartbeat is answered with 409 and a JSON
ResyncDirective, and the SSP goes through the same register, bootstrap and
verify path every SSP takes on boot.
| Mode | What happens |
|---|---|
restart | Flag, 409, re-bootstrap in place. The process keeps running and keeps its rows: it drops its views, registers again, keeps the tables that still match the replica and repairs the rest, then rebuilds its views. |
clean | The directive carries clean: true, so the SSP deletes its row checkpoints and its arena directory (and a snapshot.json an older build left), then exits. The relaunch is a cold rebuild from the database. Use it when the rows themselves are suspect. |
reload | No exit. The scheduler calls the SSP’s POST /admin/reload, which rebuilds the circuit in-process from the database. Schema changes do not need it (they are applied live), but it is there for a suspect circuit. |
Nothing happens until the next heartbeat (5 s by default). An operation is
marked done when the SSP has re-registered and reached ready, and
failed if that has not happened within one heartbeat interval plus the
scheduler’s bootstrap timeout plus 30 s. An SSP older than this release reads
the directive as free text and restarts anyway, so a clean on it degrades to
a plain restart; the operation records the SSP’s version so you can tell.
Restart all defaults to rolling: one SSP is flagged, the scheduler waits for it to come back ready, then flags the next, in id order. Queries are pinned to an SSP, so taking one down is already an outage for its clients; taking all of them down at once is an outage for everyone. All-at-once is still offered for the case where you want exactly that. A roll stops at the first SSP that does not come back, because continuing would take down the next one while this one is still missing. Only one roll runs at a time.
Restarting the scheduler
| Mode | What happens |
|---|---|
restart | The scheduler answers 202, waits for the response to flush, and calls exit(0). Sp00ky Cloud runs it under Docker’s unless-stopped, which relaunches on any exit code, and the startup integrity check runs again on boot. |
reclone | Resets the local replica and re-fetches everything from upstream SurrealDB, then flags every SSP to re-verify. The same thing POST /admin/resync on the ingest port does. Runs as an operation, because it can take minutes. |
rehash | Keeps the replica’s rows and recomputes the persisted snapshot hashes from them, then flags every SSP. Repairs hash-metadata drift without a refetch. |
A scheduler run from a checkout has no supervisor and simply stops.
GET /admin/api/config reports supervised (true when SPKY_ENV=cloud) so
the dashboard can say so before you press the button rather than after.
Reclone and rehash are refused with 503 while the scheduler is cloning or
restoring, and with 409 while another reclone is running.
Cloud-only actions
Some things are not the scheduler’s to do: pulling a newer image, wiping its
own volume (RocksDB holds the files open), bouncing SurrealDB, and the backup
catalog with its schedule and retention. Those belong to Sp00ky Cloud, which
owns the containers and the backup rows. When the scheduler is linked (see
Configuration) the dashboard offers them and the scheduler
forwards the request; when it is not, every such action answers 409 with
Not linked to Sp00ky Cloud, and the dashboard shows the option disabled with
that sentence beside it.
If Sp00ky Cloud refuses the scheduler’s credentials, the dashboard shows a
502 with code: cloud_auth and the sentence from the control plane. It is
never relayed as a 401: that status means “not signed in” to the dashboard,
and a cloud refusal has nothing to do with your session.
POST /admin/api/cloud/restart takes {roles, upgrade, clean, surreal} with
exactly the meaning of spky restart: upgrade force-pulls the latest
scheduler and SSP images, clean wipes the scheduler’s volume (and always
includes the scheduler in the role set), surreal also bounces SurrealDB.
Empty roles means scheduler plus SSPs. A request that recreates this
scheduler will, by definition, not be reported done by it; the dashboard’s
reconnect loop observes the outcome instead.
Operations
Every asynchronous action records an operation: kind, target, who asked,
start and finish times, status (running, done, failed), the server’s
own closing message, and free-form progress ({done, total, current} for a
rolling restart, the registry state for a backup).
GET /admin/api/operationslists the most recent ones, newest first.GET /admin/api/operations/streamis an SSE stream (eventoperations) that sends the whole list on every change.GET /admin/api/overviewembeds the running ones, so the poll the dashboard already makes shows activity without a second request.
Operations live in memory, bounded at 50. Finished ones are evicted oldest first; a running one is never evicted, however old, because a stuck operation is exactly what you still need to see. They are a UI convenience, not an audit log: the audit trail is the scheduler’s own log output.
Workflows, schedules and jobs
| Action | What happens |
|---|---|
| Cancel a run | Sets kill_requested and runs the kill immediately: every dispatched step’s job gets /job/kill broadcast to the ready SSPs. Sub-second, with the flag as the durable fallback the next sweep would act on. 409 unless the run is running. |
| Rerun | A brand-new ad-hoc run with the same frozen DAG, input and target table, trigger: manual, and rerun_of pointing at the source. It has no owning schedule run, never counts towards concurrency: skip, and rolls up under (ad-hoc). Any source status is allowed. |
| Retry from failed | Only for a failed or killed run. Resets failed and skipped steps to blocked, keeps successful steps and their outputs, reopens the run (and its owning schedule run, so the outcome is mirrored there) and dispatches the reset steps again under new job ids of the form <table>:wf_<run key>_<step>_r<n>. Old job rows are left as history; a job row is never deleted or re-created under the same id. Refused with 409 step <name>'s job is still processing; wait for the kill to settle while a killed step’s job has not gone terminal on its SSP, because resetting it before the kill lands would run the step twice. |
| Pause / resume a schedule | Writes paused on the _00_schedule row, the same field spky schedules pause writes. |
| Run a schedule now | Writes trigger_requested_at. 409 while the schedule is paused (pause wins over a queued trigger, so the fire would silently never happen) or disabled in its config. |
| Kill / retry a step’s job | The cluster /job/kill (broadcast to every ready SSP) and /job/retry (one SSP is picked). The SSP’s own verdict is relayed: a 409 for a job that is not terminal, a 404 for one it does not know. |
Retrying the latest fire of a schedule leaves _00_schedule.last_run_status at failed even when
the retry succeeds: the engine only lets a failure overwrite an equal last_run_at. The run and
schedule-run rows are correct; only that summary field lags.
A scheduler upgraded before spky deploy has migrated the schedule tables
answers 409 asking you to run it: rerun and retry write fields
(trigger, rerun_of, retry_count as option<int>, last_retry_at on _00_workflow_run)
that older schemas do not have.
Access
The Access page mints long-lived tokens for MCP clients, so an AI agent can
do everything this dashboard does through the
scheduler admin MCP at
/admin/api/mcp. A token has a label, a scope and a lifetime of up to 365
days, and is shown once; the page prints the claude mcp add command and the
Cursor and VS Code snippets with everything filled in. Paste a token back to
revoke it.
| Scope | Meaning |
|---|---|
read | May only make GET requests to the API. Any other method answers 403 This token is read-only |
full | Everything the signed-in operator can do |
The bearer middleware enforces that on every request, so a read token binds an
agent exactly as it binds a browser. POST /admin/api/mcp is the single
exception, and it has to be: MCP carries reads over POST as well, so the
endpoint is exempted from the method check and enforces scope per tool
instead. A read token’s tools/list lists only read tools, and calling a
write tool anyway comes back as a tool error rather than a result. The reach
is the same either way, only the place it is checked differs.
Roster and break-glass sessions are always full. An MCP token cannot mint
further tokens; only a person’s own session can. Tokens follow the
session rules: signed and restart-proof when
SPKY_AUTH_SECRET is set, in memory otherwise.
Backups
The Backups tab merges two sources rather than choosing between them:
- This scheduler executes backups and restores. Its registries know the live state of anything in flight, including restore stages nothing else can see. They are in memory and forget on restart, which the tab says under Restore history when it is empty.
- Sp00ky Cloud, when linked, owns the catalog, the schedule, retention and deletion.
Linked, the catalog is the control plane’s list with each row joined to the
scheduler’s registry entry by id (the cloud worker uses its own row id as the
scheduler’s backup id, so the join is exact). Unlinked with S3 configured, the
catalog is a listing of {slug}/*.surql.gz in the bucket, where the slug is
SPKY_CLOUD_PROJECT or, failing that, SPKY_PROJECT_SLUG (default
default). That is enough to create and restore; there is no schedule,
retention or delete, and the tab says so instead of hiding the panels.
A restore reports the stages the scheduler actually goes through:
queued → running → main_db → replica → done
↘ failedmain_db means the main SurrealDB has been wiped and re-imported; replica
means the scheduler’s own replica followed. A restore that ends failed with
the main database restored but the replica not is the deliberately stuck
Restoring state described under Cloud → Backups; the
tab calls it out with the recovery step.
A restore wipes the database, evicts every SSP (they re-bootstrap against the restored state) and
requires your migrations to be run again. The dialog asks you to type restore before it
proceeds.
Who is connected, and what they registered
Two surfaces, split by the question each answers.
Overview carries the rollup, because “is anyone using this?” is a landing-page question: a band of four tiles — users, sessions and registered views, each with its count and its history, and a view-health tile with the hot / large / slow / errored counts. Each chart keeps its own scale; views outnumber users by one to two orders of magnitude, and a shared axis would flatten the users line onto the baseline.
Views is the tab you open to ask which queries are registered: a filterable, sortable table of every live query with its owner, SSP, row count, subscribers, materialization percentiles and time to expiry. Clicking one opens its SurrealQL and params, every subscribing session with its age, that view’s heap inside its SSP, and the other live sessions running the identical query.
Everything is derived from _00_query, the registry of live queries. One row
there is one client session x query — the query id is hashed with a
per-browser-session salt, so two tabs of the same person register two rows.
A reload keeps its tab’s salt and re-registers onto the rows it already had.
That is what makes the rollup possible: distinct auth_id counts people,
distinct clientId counts open tabs, and the row count is registered views.
A client proves it is still watching by refreshing lastActiveAt on a0.9 × ttl
timer — about nine minutes at the default
10m. A row counts as live while lastActiveAt + ttl > time::now(), the
same rule the TTL sweep uses. A browser tab that closes normally hands its rows back on
pagehide: fn::query::leave pulls their expiry in to one minute, and the sweep
removes them after that. The minute is a grace for a reload of the same tab, which re-registers
onto the same rows and finds its views still warm. A tab that never sends that release (a crash, a
mobile tab killed in the background, a lost request) decays out over the full ttl instead. So
does every signed-out tab: an anonymous row answers only to the WebSocket sessions that
registered it, and the release goes out as a separate HTTP request.
Every row on the Views tab carries its own expires so you can
see which ones are on the way out. There is no finer signal to be had: the client’s WebSocket goes
to SurrealDB, not to the scheduler, so there is no connection to count.
One background sampler feeds both surfaces, so ten open dashboards cost the database exactly what one does, and the totals ride the overview poll the app already makes rather than a request of their own.
Hot views
A view whose updateCount climbs at least SPKY_ADMIN_HOT_VIEW_UPDATES_PER_SEC
times a second (default 5) is hot. The count on _00_query is a running
total, so the sampler reads it on every tick and divides the difference by the
gap since the last one. The Views tab shows each row’s current rate next to its
update count, marks hot rows, and opens with a banner naming the hottest ones;
Hot only (?hot=true) narrows the table to them. A row that was hot earlier
but has calmed down keeps its peak (peaked 125/s) for as long as it stays
registered. The view’s own page shows the rate and the peak with its time.
The shape this catches is a bulk write into a table somebody has open: an import, a backfill, a script. Every update is a delta the SSP publishes and every subscriber downloads, so one view moving a hundred times a second can fill the SSP’s publication queue and stall registrations for everyone. Pace the writer, or window the view so most of those rows never land in it.
While any view is hot, one hot_views incident stays open under the component
views, so a burst nobody was watching still leaves a record. It closes once
every view has stayed under the rate for a minute (a pause between import
slices does not split one episode in two) and its summary names the peak, the
view key and the user. A rate needs two samples, so a view registered less than
one presence interval ago has none yet, and the rate is measured over that
interval (15s by default).
Live updates and caching
The dashboard keeps one WebSocket open to GET /admin/api/v2/live instead of
polling. It subscribes to topics (the overview, presence, the metrics
series, jobs, workflow runs, incidents, operations, the scheduler’s log tail);
the scheduler computes each topic once however many dashboards watch it and
sends a snapshot followed by small patches, so an idle cluster sends nothing
and ten open tabs cost what one does. The session token is the socket’s first
message, never part of its URL. After a network blip the dashboard resumes
from the last patch it holds; the scheduler keeps a minute of them.
On https pages the dashboard caches the topics in the browser (SQLite in the Origin Private File System), so a reload paints at once and then asks only for what changed. Signing out clears that cache. A scheduler reached over plain http keeps them in memory only. If a proxy blocks WebSockets the dashboard falls back to polling on its own.
Configuration
| Variable | Default | Meaning |
|---|---|---|
SPKY_ADMIN_ENABLED | true | Set to 0/false/off to disable the admin listener entirely |
SPKY_ADMIN_PORT | 9668 | Admin listener port |
SPKY_ADMIN_HOST | 0.0.0.0 | Admin listener bind address |
SPKY_ADMIN_DIR | /usr/share/spooky/dashboard | Where the built dashboard lives |
SPKY_ADMIN_PASSWORD | unset | Break-glass password. Unset disables password-only login |
SPKY_ADMIN_ACCESS | auto | Pins the record-access method used for sign-in |
SPKY_ADMIN_SESSION_TTL_SECS | 28800 | Session lifetime (8 hours) |
SPKY_ADMIN_PRESENCE_INTERVAL_SECS | 15 | How often the presence sampler reads _00_query. One sampler serves every open dashboard |
SPKY_ADMIN_SLOW_VIEW_MS | 250 | Materialization p99 at which a view counts as slow. ?slow_ms= overrides it per request |
SPKY_ADMIN_LARGE_VIEW_ROWS | 1000 | Row count from which a view is flagged large on the Views tab and the Overview. Every row of an unwindowed live view is republished as an edge on each cold registration, so a few thousand at once can stall the database; ?large=true filters the list to them |
SPKY_ADMIN_HOT_VIEW_UPDATES_PER_SEC | 5 | Updates per second, between two presence samples, from which a view is flagged hot on the Views tab and the Overview and opens a hot_views incident. ?hot=true filters the list to them. See Hot views |
SPKY_ADMIN_PRESENCE_MAX_ROWS | 20000 | Ceiling on rows one presence or views query may pull back |
SPKY_ADMIN_JOB_INTERVAL_SECS | 15 | How often the job sampler reads the outbox tables while nobody has the Jobs page open. 0 switches the sampler off: the Jobs list still works, the totals and the Overview tile report that they are not measured |
SPKY_ADMIN_JOB_LIVE_INTERVAL_SECS | 2 | The cadence while at least one dashboard holds /admin/api/jobs/stream open. Clamped to no slower than the idle interval |
SPKY_ADMIN_ALLOWED_ORIGINS | unset (any) | Comma-separated browser origins allowed to call the API and open the live socket. Unset or * allows any (the bearer token, never a cookie, is what authenticates); the scheduler’s own page is always allowed. Sp00ky Cloud sets it to the console and the project’s admin host |
SPKY_ADMIN_CLOUD_LOGIN | true | Whether console grants are accepted on a cloud-linked scheduler. 0/false/off refuses them |
SPKY_ADMIN_CLOUD_JWKS | unset | The control plane’s grant keys as a JWKS document. Without it (or for an unknown key) they are fetched from {SPKY_CLOUD_API_URL}/.well-known/jwks.json |
SPKY_ADMIN_CLOUD_ISSUER | SPKY_CLOUD_API_URL | The issuer a grant must name |
SPKY_ADMIN_CLOUD_SESSION_TTL_SECS | 900 | Lifetime of a session opened with a grant (capped by SPKY_ADMIN_SESSION_TTL_SECS) |
SPKY_ADMIN_DATA | true | Set to 0/false/off/no to refuse /admin/api/data and hide the console’s Database, Users and Storage pages for this project. On Sp00ky Cloud, set it under deployment.env for the scheduler |
SPKY_BUCKET_DIR | unset | A directory holding file: buckets, mounted read-only at the same path SurrealDB uses. Buckets under it list their subfolders in the Storage page. Writes still go through SurrealDB |
SPKY_CLOUD_API_URL | unset | Sp00ky Cloud API base URL. Injected into the scheduler container by the control plane |
SPKY_CLOUD_PROJECT | unset | The project slug, injected alongside it |
SPKY_PROJECT_SLUG | default | Storage prefix for backups on an unlinked scheduler |
Backup storage is configured separately, and those five variables carry no
SPKY_ prefix: S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, S3_BUCKET
and S3_REGION. The backup plane switches on as soon as any of the first four
is present, and each one falls back to a local-MinIO default when it is not, so
set them deliberately rather than partially. Sp00ky Cloud injects them for you;
self-hosters set them by hand, and
Self-hosting lists them with their defaults.
Without them the Backups tab still loads and says storage is unconfigured, and
POST /admin/api/backups answers 503.
The scheduler counts as linked when SPKY_CLOUD_API_URL,
SPKY_CLOUD_PROJECT and SPKY_AUTH_SECRET are all set. The first two are not
secrets; the credential is the SPKY_AUTH_SECRET the scheduler already holds
as its cluster identity, which the control plane accepts on a small,
allow-listed route family for that one project. No new credential is minted
or stored. GET /admin/api/config reports the result as cloud_linked.
Running from a checkout
The dashboard is shipped as files rather than embedded in the binary, so
cargo build -p scheduler needs no Node toolchain. A scheduler with no bundle
starts normally, logs one warning, and serves a short placeholder page at
/admin; the API on that port is unaffected.
To build and serve it locally:
pnpm --filter @spooky-sync/dashboard build
SPKY_ADMIN_DIR=apps/dashboard/dist cargo run -p schedulerAPI
Every endpoint is under /admin/api on the admin port. All require
Authorization: Bearer <token> except GET /admin/api/config.
Two routes sit outside that prefix: the built dashboard itself, served from
/admin, and GET /, which answers a 301 to /admin/ so typing the host
and port with no path lands on the dashboard.
| Endpoint | Purpose |
|---|---|
GET /admin/api/config | Scheduler id, version, breakglass_available, cloud_linked, cloud_login, supervised, sessions_persistent, project_slug, and live (the live protocol version) with its topics. Unauthenticated |
POST /admin/api/session | Sign in. {username, password}, {password} for break-glass, or {grant} from the Sp00ky Cloud console |
GET /admin/api/v2/live | WebSocket. First message {"op":"auth","token":…}; then {"op":"sub","ref","topic","args","since"} per topic. The host answers snap (full state) and diff (from, seq, patch), and expiring before the session ends. Closes 4401 for a lost session |
GET /admin/api/v2/snapshot | The same topic states for a reader that polls. ?subs=[{"topic":"overview"},{"topic":"metrics","args":{"name":"sync"}}] |
POST /admin/api/logout | Revoke the current token |
GET /admin/api/me | Current session’s subject, label, mode and scope |
GET /admin/api/overview | Scheduler, SSP and backend entities plus totals, and the presence rollup |
GET /admin/api/presence | Live users, sessions and views, the sample ring behind the charts, the heaviest users and the per-SSP split. Served from the sampler’s memory |
GET /admin/api/views | Registered live queries. ?limit=, ?user=, ?ssp=, ?sort=, ?slow_ms=, ?q=, ?shared=, ?large=, ?hot=, ?include_expired=. Every row carries update_rate (per second, null until sampled twice), update_rate_peak, update_rate_peak_at_ms and hot |
GET /admin/api/views/:key | One view: SurrealQL, params, subscribers, percentiles, its SSP’s memory attribution, and the other sessions running the identical query |
GET /admin/api/backends | Backend list |
GET /admin/api/backends/:name | One backend with probe history and masked env |
GET /admin/api/logs | SSE. ?source=scheduler or ssp:<id>, &tail=, &backfill= |
GET /admin/api/workflows/runs | Run list. ?name=, ?schedule=, ?status=, ?rerun_of=, ?limit= |
GET /admin/api/workflows/runs/:id | One run with its steps and DAG |
GET /admin/api/workflows/stream | SSE. Run list, pushed on change |
GET /admin/api/jobs | Outbox jobs across every table, newest activity first, with the queue totals and a per-table breakdown. ?status=, ?table=, ?origin=schedule|workflow|app, ?q=, ?limit=. Unfiltered it is served from the sampler’s memory and costs the database nothing |
GET /admin/api/jobs/stream | SSE. Event jobs, the list and totals pushed on change. Holding it open is what puts the sampler on its live cadence. 501 when the sampler is disabled |
GET /admin/api/jobs/:id | One outbox job: status, path, payload, result, every attempt in errors, retry budget, assignee, lease, and the origin block naming the schedule fire or workflow step it came from. 404 unless the id’s table is in _00_retention.job_tables |
GET /admin/api/schedules | Schedule definitions with next/last fire |
GET /admin/api/schedules/:name | One schedule, its recent fires, and its _00_run_rollup tally |
GET /admin/api/operations | Recent operations, newest first |
GET /admin/api/operations/stream | SSE. Event operations, the whole list on every change |
POST /admin/api/ssps/:id/restart | {mode: "restart" | "clean" | "reload"}. 202 with the operation |
POST /admin/api/ssps/restart-all | {mode: "restart" | "clean", rolling: true}. 409 while a roll is running |
POST /admin/api/scheduler/restart | {mode: "restart" | "reclone" | "rehash"} |
POST /admin/api/cloud/restart | {roles, upgrade, clean, surreal}, forwarded to Sp00ky Cloud. 409 when unlinked |
GET /admin/api/cloud/deployment | The deployment as Sp00ky Cloud sees it. 409 when unlinked |
GET /admin/api/cloud/stats | The Resources page: services (status, allocation, newest sample, cpu_series, mem_series, disk), machines, pool_machines, bucket_volume (size_gb, usage.fs_bytes/used_bytes/free_bytes). 409 when unlinked |
GET /admin/api/backups | Catalog, local registry, restores, schedule config, storage status |
POST /admin/api/backups | {name}. 202 with the operation and backup_id. 503 without S3 storage |
PUT /admin/api/backups/config | {enabled, schedule, retention}. 409 when unlinked |
DELETE /admin/api/backups/:id | Soft-delete in the catalog. 409 when unlinked |
POST /admin/api/backups/:id/restore | 202 with the operation and restore_id. 503 unless the scheduler is Ready |
GET /admin/api/backups/:id/restore | {cloud, local, stage} for the latest restore of that backup |
POST /admin/api/workflows/runs/:id/cancel | 202 {run, status: "killed" | "kill_requested"} |
POST /admin/api/workflows/runs/:id/rerun | 201 {run, rerun_of} |
POST /admin/api/workflows/runs/:id/retry | 200 {run, retry_count, reset, kept} |
POST /admin/api/schedules/:name/pause | {name, paused: true} |
POST /admin/api/schedules/:name/resume | {name, paused: false} |
POST /admin/api/schedules/:name/trigger | {name, triggered_at}. 409 when paused or config-disabled |
GET /admin/api/pools | Every machine pool: sizing, pause and breaker state, machines by state, busy slots, jobs waiting for a machine |
GET /admin/api/pools/:name/machines | One pool’s machines, newest first, terminal ones included: state, slots in use, last contact, why it was taken away |
POST /admin/api/pools/:name/pause | No new jobs assigned and no machines created; running jobs finish. Survives redeploys |
POST /admin/api/pools/:name/resume | Resume a paused pool |
POST /admin/api/machines/:id/drain | The machine takes no new jobs, finishes what it runs, and is destroyed |
POST /admin/api/jobs/:id/kill | The cluster kill, relaying the SSPs’ answer |
POST /admin/api/jobs/:id/retry | The cluster retry, relaying the SSP’s answer |
POST /admin/api/jobs/clear | {table, all}. Batch-deletes terminal rows, never processing ones. 200 {cleared, total, more} |
POST /admin/api/tokens | {label, scope: "read" | "full", ttl_days}. 201 with the token, shown once. 403 for an MCP or read-only session |
DELETE /admin/api/tokens | {token}. Revokes it. 204 |
POST /admin/api/mcp | The scheduler admin MCP server: JSON-RPC over Streamable HTTP, stateless. GET and DELETE answer 405 |
A 503 on the login screen —
“Cannot reach the tenant database, so no sign-in can be checked” — is not a credential
problem and no password will fix it. The roster path signs in through your database, so
it needs that database reachable; when it is not, the scheduler says so instead of answering
401. Check the scheduler’s log for the reason and the database container’s health. A
tenant database can reach a state where it keeps answering existing connections and its REST
endpoints while refusing new SDK sessions, so “the app still works” does not rule this out;
restarting the database container clears it.
retry_count on _00_workflow_run became
option<int>: as a plain int every update to a run created before
the field existed failed validation, which blocked cancel and the engine’s own finalize on those
rows. spky migrate applies the new definition and a one-time backfill; run it once
after upgrading.
The workflow endpoints are served by the scheduler rather than read from the
database directly because _00_workflow_run, _00_step_run and
_00_schedule_run are PERMISSIONS NONE — no record-auth token can read them.
The scheduler holds root and is the cluster’s only schedule ticker, which makes
it the right place to serve them.
Why a row ended the way it did
Every skip, kill and failure the engine records carries a structured cause on the
row’s error field — a stable code, a sentence, and whatever ids make the
sentence actionable — and the dashboard renders that rather than a JSON payload.
The ones worth knowing by name:
code | On | Means |
|---|---|---|
concurrency_skip | _00_schedule_run | The fire was suppressed by concurrency: skip. blocked_by is the run holding the key and since is when it started. A run wedged in running suppresses that key’s fires for as long as it stays there, so the blocking run is what needs acting on — not the skipped rows |
upstream_failed | _00_step_run | The step is below a failure under on_failure: continue-independent. blocked_by names the dependency one hop up, so the chain can be walked a step at a time |
run_halted | _00_step_run | The step had not started when the run halted under on_failure: halt. failed_steps names what actually broke, which is elsewhere in the DAG |
not_started | _00_step_run | The run was killed or reaped before this step started. run_error carries the run’s own reason |
never_dispatched | _00_step_run | The step was promoted but its job row was never created, and the run stopped before recovery re-dispatched it |
lost_dispatch | _00_step_run | The step exhausted its dispatch attempts without ever getting a job |
job_failed | _00_step_run, _00_schedule_run | The step’s job failed. Normally the job’s last errors entry verbatim; synthesized, and saying so, when the job recorded none |
job_missing | _00_step_run, _00_schedule_run | The job row is gone — deleted, or pruned by retention while the step still pointed at it |
step_failed | _00_workflow_run | A step failed and the run failed with it. The step’s own row carries the real reason |
deadline_exceeded | _00_workflow_run, _00_schedule_run | The reaper stopped a run that passed its deadline: (or the project’s run_deadline_secs) without finishing, and killed its jobs |
killed | _00_workflow_run, _00_schedule_run | An operator cancelled the run |
spawn_failed | _00_workflow_run, _00_schedule_run | The fire could not create what it needed — the job row, or the run’s steps |
bad_dag | _00_workflow_run | The frozen DAG on the run does not validate. A redeploy cannot fix a run already in flight; its DAG is frozen at spawn |
no_target_table | _00_step_run | The step names no outbox table and its schedule has no target_table |
_00_schedule.last_error is the exception: it is a bare string, because it is why
a schedule could not be planned or fanned out at all (a bad cron, a failing
forEach) — which happens before any run row exists to carry a cause.
Each step in GET /admin/api/workflows/runs/:id carries both created_at and
started_at, and the distinction matters: the engine creates every step row when
the run is spawned, so created_at on a step deep in a DAG is the run’s start,
not the step’s. started_at is stamped when the step’s job is created and is the
only honest answer to “when did this begin”. The Timeline draws from it, and
falls back to the moment the step’s last dependency finished — marked as an
estimate — for runs recorded before the engine stamped it.
Incident details show publication backlog when an SSP heartbeat sample was available: peak queued operations and bytes, longest observed wait, parked batches, last successful publication, and related view links. Bulk database session generation and reconnect timing distinguish database recovery from circuit evaluation. The SSP page also shows its current publication backlog. These are sampled measurements; a spike between samples can be missed.
Project data
/admin/api/data is what the console’s Database, Users, Feature flags and Storage
pages read. The scheduler and the SSP node serve the same routes from one
implementation (schedule_core::admin_data). GET /admin/api/config says whether a
host has them, as data, and the flag routes, as flags. These pages are in the console only for now; the
scheduler-hosted /admin does not show them.
A read-only session gets the GET routes. An MCP token gets none of them, whatever
its scope, and no MCP tool reaches them. Every change is logged with the session’s
subject, and the SQL editor’s log line names the first word of the statement, never
its text.
| Endpoint | Purpose |
|---|---|
GET /data/tables | Names, kinds and shapes of the tables. Nothing is counted |
GET /data/tables/:table | Its fields (with credential, required, readonly), indexes and events |
GET /data/tables/:table/rows | ?limit= (500 at most), ?start=, ?sort=, ?dir=, ?filter=[{"field","op","value"}] (16 at most). op is eq, ne, gt, gte, lt, lte, contains, starts_with, is_none or not_none. A value is cast by the field’s type. Top-level fields only, and not a credential: filtering on one is a way to read it |
GET /data/tables/:table/count | ?filter=. A scan, stopped at 10 s: {"count": null, "timed_out": true} |
GET /data/tables/:table/row?id= | One row, with long values kept |
POST /data/tables/:table/rows | {id?, set}. A relation takes set.in and set.out |
PUT /data/tables/:table/row?id= | {set, unset}. null and unset both remove a value |
DELETE /data/tables/:table/row?id= | One row |
POST /data/tables/:table/rows/delete | {ids}, 500 at most |
POST /data/query | {sql}, as root. A lone SELECT is given a TIMEOUT 30s and a LIMIT 1001 when it has neither (one past what is returned, so truncated can say there was more), and 1000 rows of any one statement come back. Refused to read-only sessions |
GET /data/users | ?q=, ?filter=admins|online, ?limit=, ?start=, ?sort=, ?dir=, ?source=. Where the users are comes from the SIGNIN clause of a record access (?source= picks one), and each carries is_admin, views and sessions. Newest first only when an index on the created field makes that free (created_indexed); otherwise ask with ?sort= |
GET /data/users/stats | total, admins, online. null for one it could not read |
GET /data/users/user?id= | The row, roster entry, live sessions and feature flags: features lists every flag with the variant the user gets and the one they are pinned to, features_assignable whether flags apply to their table |
POST /data/users | {identifier, password, fields?} |
PUT /data/users/user/password?id= | {password}, 8 characters or more, hashed with the algorithm SIGNIN compares with. 501 when SIGNIN does not show one |
PUT /data/users/user/admin?id= | {admin, note?}: what spky admin add and remove do. Nobody can remove themselves |
DELETE /data/users/user?id= | The row, its roster entry and its feature flags. Nobody can delete themselves |
GET /data/flags | Every flag with its rollouts and how many users are pinned, and whether the schema has the flag tables (installed), fn::feature::assign (evaluator) and a user table (users_table) |
POST /data/flags | {key, description?, variants?, default_variant?}, as spky flag create: on, no rules. Answers evaluate: true |
GET /data/flags/flag?key= | The flag, users per variant (counts, null when the count timed out) and the pinned users by name, 500 at most (pins_total counts them all) |
PUT /data/flags/flag?key= | Any of {enabled, description, default_variant, variants, rollouts: [{variant, percent}]}. rollouts replaces every rollout rule. A variant the default or a rule still uses cannot be removed. evaluate says whether assignments can have moved |
DELETE /data/flags/flag?key= | The flag and every assignment of it |
POST /data/flags/flag/evaluate?key= | Re-evaluate one page of users in id order: ?limit= (500 at most), ?after= the next of the page before. total on the first page, next: null after the last |
PUT /data/flags/flag/user?key=&id= | {variant} pins a user record to a variant (an allowlist rule), {variant: null} unpins them. Only that user is re-evaluated |
GET /data/buckets | Each bucket’s backend and limit, whether it can be browsed, and lists_folders: false for a file: bucket whose subfolders this host cannot see |
GET /data/buckets/:bucket/files | ?prefix=, ?start=, ?limit=. One folder’s files and folders, and next when there is more. counts gives each folder’s entries, up to 1000, when the folder was read from disk |
GET /data/buckets/:bucket/file?key= | {key, size, updated, data}, data in base64, up to 32 MB. &meta=1 leaves data out |
PUT /data/buckets/:bucket/file?key= | {data} in base64. 409 when the key is in use, unless &overwrite=1 |
DELETE /data/buckets/:bucket/file?key= | One file |
POST /data/buckets/:bucket/rename | {key, target}. 409 when the target is in use |
POST /data/buckets/:bucket/delete | {keys}, 500 at most |
What the browse routes keep back:
- Credentials. A field named like one (
password,secret,token,private_key,api_key, in any case and spelling), or the field aSIGNINclause compares a password against, reads as{"$redacted": true}whatever its shape, and cannot be written, filtered or sorted on. Only a number undertokenstays:tokens_usedis a count. This goes by name. A value carrying a mask cannot be written back at all, so a form cannot replace a secret with its mask.POST /data/queryreturns what the statement selects. - Definitions. A credential’s
default,valueandassert, every event’sthenand every bucket’spermissionsstay behind: any of them can hold a literal. _00_push_credentialand_00_push_subscription, which hold push keys and device tokens, are not listed and answer403.- Long values. Over 4 KB in a list (512 KB on one row) a value reads as
{"$trimmed": true, "bytes": n, "preview": "..."}. _00_*tables answer403to every write. The roster moves through its own route.
Regenerating these screenshots
They are captured from the real dashboard driven by a fixed fixture, not by hand:
See apps/dashboard/screenshots/README.md for what to update when a route is
added or a response shape changes.