Admin dashboard

The operator dashboard the scheduler serves at /admin: sign-in, cluster overview, restart and backup actions, backend health, live logs and workflow runs.

Every scheduler serves an operator dashboard. It shows the health of the scheduler and its SSPs, the end-to-end sync latency, the status of your backends, live logs, and workflow runs as they happen.

Incident history

The Incidents navigation item keeps automatic lag, heartbeat failures, bootstrap integrity failures, removal/re-registration, recovery, and operator actions in one timeline. Every navigation entry has a labeled icon on desktop and mobile. The overview carries an Incidents tile: the open count, the last 24 hours as a strip with one marker per episode (as wide as it lasted, pulsing while open), and a link to the latest one.

The page opens on a rail (open now, count in the range, longest episode, latest) above a day-grouped feed. Filters are chips that apply the moment they are clicked: range (24 h, 7 days, 30 days), state and component, each with its count, plus a search over kind and summary. The chips filter the newest 400 episodes of the range on the client, so switching between them costs no request. Open an incident to see its duration, largest sampled event backlog, server version, operation id when present, and event timeline. Recovered means the scheduler observed recovery. Interrupted means the scheduler restarted before it could observe recovery. Lag followed by replay is described separately from an SSP re-registering; re-registration alone does not prove a container crash.

History is stored in incidents.json beside the scheduler WAL, independently of SurrealDB. Keep that directory on persistent storage. Retention is bounded by 30 days, 2,000 incidents, 64 events per incident (first and newest retained), and an 8 MiB file cap. Counts include compacted events. Ordinary events flush once per second; a scheduler restart or cloud restart request flushes its action before exit/dispatch. Abrupt termination can lose the last unflushed second. A clean restart that wipes the scheduler volume also wipes its incident history.

A missing or unreadable history file is reported in the dashboard; malformed history is not overwritten. The incident logging target stays enabled even with RUST_LOG=warn. A dropped log-feed segment becomes a visible history-gap record. Stored summaries contain no query bindings, SQL, or credentials. Detailed logs remain available through the Logs page and are independently subject to their retention limits. History starts when this version is deployed; it does not reconstruct older incidents from arbitrary log text.

The read-only MCP tools incidents_list and incident_get expose the same history as the dashboard. See the scheduler API.

Opening it

The dashboard is bundled into the official mono424/spooky-scheduler image and served on its own port:

http://<scheduler-host>:9668/admin

Why a second port

Everything on the ingest port (9667) is unauthenticated by design — including POST /proxy/query, which runs arbitrary SurrealQL against the scheduler’s replica. That port is meant to be reachable only from inside your deployment’s private network.

The dashboard is the first surface intended for a browser, so it gets a listener of its own. Publish 9668; keep 9667 private.

docker run -p 9668:9668 mono424/spooky-scheduler   # dashboard reachable
# 9667 stays on the internal network

Signing in

Two ways in.

Admin accounts (the normal path)

Any account on the _00_admin roster can sign in with its ordinary application credentials. Grant access with the CLI:

spky admin add alice
spky admin list
spky admin remove alice

The scheduler signs in to your database through your app’s own record-access method, then checks the resulting user against _00_admin. Nothing extra to configure, and no second password to manage.

The access method is discovered from INFO FOR DB. If your schema defines several record accesses and you want to pin one, set SPKY_ADMIN_ACCESS. Both $username and $email are tried as the identifier parameter, so either spelling of a SIGNIN clause works.

Break-glass password

Set SPKY_ADMIN_PASSWORD to enable a password-only login that needs no database:

SPKY_ADMIN_PASSWORD='…'

This exists because the roster path needs a working database, and a database that is down is exactly when you most want the dashboard. A session established this way is flagged with a persistent banner in the UI.

Leave the variable unset to disable this path entirely; there is no default password.

Session behaviour

How long a session lives depends on whether the scheduler has a cluster secret:

• With SPKY_AUTH_SECRET set (always the case on Sp00ky Cloud), tokens are self-describing and HMAC-signed with a key derived from that secret. They verify without any server-side state, so they survive a scheduler restart, including one you trigger from the dashboard. Signing out adds the token to an in-memory revocation set; a restart forgets that set, so a revoked token comes back for the remainder of its own expiry. That trade is deliberate and is the whole cost of not logging you out when you press Restart.
• Without a secret there is nothing to sign with that outlives the process, so sessions are random tokens held in memory and a restart means signing in again.

GET /admin/api/config reports which mode is active as sessions_persistent. GET /admin/api/me reports the session’s mode (roster, breakglass, or mcp for a token minted on the Access page) and its scope.

Failed logins are rate limited to 10 attempts per address per 5 minutes, and every failure returns the same response regardless of cause, so the endpoint cannot be used to discover which usernames exist.

Standalone use

The same bundle runs outside a scheduler. Opened from anywhere else, the sign-in form grows an endpoint field; point it at a scheduler’s admin port and it behaves identically.

pnpm --filter @spooky-sync/dashboard dev
# then enter http://your-scheduler:9668 in the endpoint field

The dashboard authenticates with a bearer token in localStorage, keyed per endpoint, rather than a cookie — a cross-origin cookie would need SameSite=None; Secure, which browsers refuse over plain HTTP, and plain HTTP at an IP address is exactly how a scheduler is usually reached.

What it shows

Overview — a grid of tiles, one per question: the end-to-end sync latency with a 30-minute chart as the large tile, then SSPs ready, backends healthy, ingest lag and the scheduler’s own details beside it. The page paints from the last state it saw the moment it opens (marked refreshing) and swaps in the live poll a moment later, so a reload never shows an empty frame. A tile whose subject is degraded is tinted, so the problem is the first thing on the page. The latency is the scheduler’s own heartbeat probe: it writes _00_heartbeat:probe upstream and times the full round trip through /ingest, the WAL, the broadcast and every SSP’s circuit. It measures the real sync path, not a synthetic ping.

SSPs — per-processor status, live view count, uptime and environment. An SSP that is bootstrapping or replaying shows a progress bar: tables loaded and rows read when the SSP reports it, and otherwise the phase with its elapsed time against the scheduler’s bootstrap budget.

Backends — status and response time for everything in SPKY_BACKENDS, with a detail page carrying a ~30-minute response-time history and the resolved healthcheck URL. Environment values are masked by the scheduler before they leave it. On Sp00ky Cloud each deploy pushes the list; the scheduler stores the last pushed list (env values masked) in _00_scheduler_state and reads it back when it starts, so a restart or reclone between deploys keeps its backends. A list in SPKY_BACKENDS always takes precedence. A backend that runs on a machine pool (run_on: { pool: ... }) is not probed: between jobs there may be no machine to answer, and that is not an outage. Its status is what the pool sweep last saw: idle at zero machines with nothing queued, starting while a machine boots or a job waits for one, healthy with a ready machine, unhealthy only when the pool’s boot-failure breaker is open or its last pass failed. No backend_down incident is ever opened for it; the pool’s own pool_breaker_open incident carries the reason when it cannot get a machine.

Workflows — runs updating in real time, and a run page built around a timeline: every step plotted on one shared axis, so where the time went is visible at a glance rather than reconstructed from timestamps. Each row in the list carries the run’s own failure reason, so four failed runs do not cost four clicks to tell one problem from four. Click a step to expand its job, output and error inline. A step still waiting on its dependencies is drawn hatched at the origin rather than as a bar, because it has not run. The scheduler runs a single shared poller behind the live list, so ten open dashboards cost your database exactly what one does, and a closed one costs nothing.

An expanded step also loads its job, which is where a failure says what actually happened: the step row keeps only the attempt that ended the job, while the job’s errors array keeps every attempt. A step that failed after two timeouts and a 502 is a different problem from one that 502’d immediately, and only the job can tell them apart. The job is fetched per opened step rather than with the run, so a fan-out run with hundreds of steps costs nothing extra.

Schedules — the definition, the next fire, recent fires, and a success/failure tally. The tally comes from _00_run_rollup rather than counting run rows, so it stays accurate after retention has pruned them. Each fire shows why it ended as it did, and a schedule whose fires are being suppressed says so at the top of the page, naming the run that is holding the key — a wall of skipped rows looks like “nothing to do” and means “something is still running”.

Jobs — every outbox job in one list, whatever created it. This is the only place two whole classes of work are visible at all: a kind: job schedule’s fires, and jobs your own code created, which belong to no schedule and no workflow. Each row says where it came from, and links back to the schedule or the workflow step when the owning run still exists — a terminal run is pruned on its own retention window well before the jobs that outlived it, so “scheduled work whose fire has aged out” and “a job nobody can account for” are deliberately not drawn the same.

The top of the page is the queue itself: a Queue depth tile with the pending count and its history from the sampler, throughput, failures in the last hour, in-flight and succeeded beside it, then Queues, one lane per outbox table drawn on a shared scale (in flight, stalled, pending and failed as segments, a tick where the table’s concurrency sits). Clicking a lane’s name filters the list below to that table.

Filter with the chips by origin (schedule, workflow, app), by status, by table, or by a substring of the path. The filter is pushed into SurrealQL rather than applied to the page that was already fetched, so asking for application jobs on a busy table returns them instead of reporting none.

The tiles above the list are the queue itself: pending depth with the age of the oldest one, in-flight against each table’s _00_job_policy concurrency ceiling, failures over the last hour, and jobs succeeded per minute. Stalled is called out separately from in-flight and is worth knowing: it is a processing row whose lease has expired, meaning nobody is working on it and a recovery sweep will reclaim it on its own clock. The schema has no status for that state, which is exactly why it earns a name here.

Counts cover work in flight plus the last hour of terminal work, never all of history. Once retention keeps failures longer than successes, an all-time ratio reports a failure rate that climbs as successes age out; a bounded window is the only honest denominator, and it is the one spky jobs uses too.

Logs — a live tail of the scheduler’s or any SSP’s log output, with level and text filters. If a viewer falls behind, the gap is shown explicitly rather than hidden.

Backends have no log stream: the scheduler reaches them with HTTP health checks and has no access to their output.

Actions

Everything the dashboard can do, as opposed to show, goes through the scheduler. Every action logs the signed-in subject, and the long-running ones answer 202 with an operation the dashboard watches (see below).

Restarting an SSP

The scheduler never touches a container. Restarting an SSP means flagging it in the pool; the SSP’s next heartbeat is answered with 409 and a JSON ResyncDirective, the SSP exits, and its supervisor relaunches it through the same register, bootstrap and verify path every SSP takes on boot. A restart from the dashboard is indistinguishable from one by Docker.

Nothing happens until the next heartbeat (5 s by default). An operation is marked done when the SSP has re-registered and reached ready, and failed if that has not happened within one heartbeat interval plus the scheduler’s bootstrap timeout plus 30 s. An SSP older than this release reads the directive as free text and restarts anyway, so a clean on it degrades to a plain restart; the operation records the SSP’s version so you can tell.

Restart all defaults to rolling: one SSP is flagged, the scheduler waits for it to come back ready, then flags the next, in id order. Queries are pinned to an SSP, so taking one down is already an outage for its clients; taking all of them down at once is an outage for everyone. All-at-once is still offered for the case where you want exactly that. A roll stops at the first SSP that does not come back, because continuing would take down the next one while this one is still missing. Only one roll runs at a time.

Restarting the scheduler

A scheduler run from a checkout has no supervisor and simply stops. GET /admin/api/config reports supervised (true when SPKY_ENV=cloud) so the dashboard can say so before you press the button rather than after. Reclone and rehash are refused with 503 while the scheduler is cloning or restoring, and with 409 while another reclone is running.

Cloud-only actions

Some things are not the scheduler’s to do: pulling a newer image, wiping its own volume (RocksDB holds the files open), bouncing SurrealDB, and the backup catalog with its schedule and retention. Those belong to Sp00ky Cloud, which owns the containers and the backup rows. When the scheduler is linked (see Configuration) the dashboard offers them and the scheduler forwards the request; when it is not, every such action answers 409 with Not linked to Sp00ky Cloud, and the dashboard shows the option disabled with that sentence beside it.

If Sp00ky Cloud refuses the scheduler’s credentials, the dashboard shows a 502 with code: cloud_auth and the sentence from the control plane. It is never relayed as a 401: that status means “not signed in” to the dashboard, and a cloud refusal has nothing to do with your session.

POST /admin/api/cloud/restart takes {roles, upgrade, clean, surreal} with exactly the meaning of spky restart: upgrade force-pulls the latest scheduler and SSP images, clean wipes the scheduler’s volume (and always includes the scheduler in the role set), surreal also bounces SurrealDB. Empty roles means scheduler plus SSPs. A request that recreates this scheduler will, by definition, not be reported done by it; the dashboard’s reconnect loop observes the outcome instead.

Operations

Every asynchronous action records an operation: kind, target, who asked, start and finish times, status (running, done, failed), the server’s own closing message, and free-form progress ({done, total, current} for a rolling restart, the registry state for a backup).

• GET /admin/api/operations lists the most recent ones, newest first.
• GET /admin/api/operations/stream is an SSE stream (event operations) that sends the whole list on every change.
• GET /admin/api/overview embeds the running ones, so the poll the dashboard already makes shows activity without a second request.

Operations live in memory, bounded at 50. Finished ones are evicted oldest first; a running one is never evicted, however old, because a stuck operation is exactly what you still need to see. They are a UI convenience, not an audit log: the audit trail is the scheduler’s own log output.

Workflows, schedules and jobs

A scheduler upgraded before spky deploy has migrated the schedule tables answers 409 asking you to run it: rerun and retry write fields (trigger, rerun_of, retry_count as option<int>, last_retry_at on _00_workflow_run) that older schemas do not have.

Access

The Access page mints long-lived tokens for MCP clients, so an AI agent can do everything this dashboard does through the scheduler admin MCP at /admin/api/mcp. A token has a label, a scope and a lifetime of up to 365 days, and is shown once; the page prints the claude mcp add command and the Cursor and VS Code snippets with everything filled in. Paste a token back to revoke it.

The bearer middleware enforces that on every request, so a read token binds an agent exactly as it binds a browser. POST /admin/api/mcp is the single exception, and it has to be: MCP carries reads over POST as well, so the endpoint is exempted from the method check and enforces scope per tool instead. A read token’s tools/list lists only read tools, and calling a write tool anyway comes back as a tool error rather than a result. The reach is the same either way, only the place it is checked differs.

Roster and break-glass sessions are always full. An MCP token cannot mint further tokens; only a person’s own session can. Tokens follow the session rules: signed and restart-proof when SPKY_AUTH_SECRET is set, in memory otherwise.

Backups

The Backups tab merges two sources rather than choosing between them:

• This scheduler executes backups and restores. Its registries know the live state of anything in flight, including restore stages nothing else can see. They are in memory and forget on restart, which the tab says under Restore history when it is empty.
• Sp00ky Cloud, when linked, owns the catalog, the schedule, retention and deletion.

Linked, the catalog is the control plane’s list with each row joined to the scheduler’s registry entry by id (the cloud worker uses its own row id as the scheduler’s backup id, so the join is exact). Unlinked with S3 configured, the catalog is a listing of {slug}/*.surql.gz in the bucket, where the slug is SPKY_CLOUD_PROJECT or, failing that, SPKY_PROJECT_SLUG (default default). That is enough to create and restore; there is no schedule, retention or delete, and the tab says so instead of hiding the panels.

A restore reports the stages the scheduler actually goes through:

queued → running → main_db → replica → done
                                   ↘ failed

main_db means the main SurrealDB has been wiped and re-imported; replica means the scheduler’s own replica followed. A restore that ends failed with the main database restored but the replica not is the deliberately stuck Restoring state described under Cloud → Backups; the tab calls it out with the recovery step.

Who is connected, and what they registered

Two surfaces, split by the question each answers.

Overview carries the rollup, because “is anyone using this?” is a landing-page question: a band of four tiles — users, sessions and registered views, each with its count and its history, and a view-health tile with the shared / slow / errored counts. Each chart keeps its own scale; views outnumber users by one to two orders of magnitude, and a shared axis would flatten the users line onto the baseline.

Views is the tab you open to ask which queries are registered: a filterable, sortable table of every live query with its owner, SSP, row count, subscribers, materialization percentiles and time to expiry. Clicking one opens its SurrealQL and params, every subscribing session with its age, that view’s heap inside its SSP, and the other live sessions running the identical query.

Everything is derived from _00_query, the registry of live queries. One row there is one client session x query — the query id is hashed with a per-browser-session salt, so two tabs of the same person register two rows. That is what makes the rollup possible: distinct auth_id counts people, distinct clientId counts open tabs, and the row count is registered views.

One background sampler feeds both surfaces, so ten open dashboards cost the database exactly what one does, and the totals ride the overview poll the app already makes rather than a request of their own.

Configuration

Backup storage is configured separately, and those five variables carry no SPKY_ prefix: S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, S3_BUCKET and S3_REGION. The backup plane switches on as soon as any of the first four is present, and each one falls back to a local-MinIO default when it is not, so set them deliberately rather than partially. Sp00ky Cloud injects them for you; self-hosters set them by hand, and Self-hosting lists them with their defaults. Without them the Backups tab still loads and says storage is unconfigured, and POST /admin/api/backups answers 503.

The scheduler counts as linked when SPKY_CLOUD_API_URL, SPKY_CLOUD_PROJECT and SPKY_AUTH_SECRET are all set. The first two are not secrets; the credential is the SPKY_AUTH_SECRET the scheduler already holds as its cluster identity, which the control plane accepts on a small, allow-listed route family for that one project. No new credential is minted or stored. GET /admin/api/config reports the result as cloud_linked.

Running from a checkout

The dashboard is shipped as files rather than embedded in the binary, so cargo build -p scheduler needs no Node toolchain. A scheduler with no bundle starts normally, logs one warning, and serves a short placeholder page at /admin; the API on that port is unaffected.

To build and serve it locally:

pnpm --filter @spooky-sync/dashboard build
SPKY_ADMIN_DIR=apps/dashboard/dist cargo run -p scheduler

API

Every endpoint is under /admin/api on the admin port. All require Authorization: Bearer <token> except GET /admin/api/config.

Two routes sit outside that prefix: the built dashboard itself, served from /admin, and GET /, which answers a 301 to /admin/ so typing the host and port with no path lands on the dashboard.

The workflow endpoints are served by the scheduler rather than read from the database directly because _00_workflow_run, _00_step_run and _00_schedule_run are PERMISSIONS NONE — no record-auth token can read them. The scheduler holds root and is the cluster’s only schedule ticker, which makes it the right place to serve them.

Why a row ended the way it did

Every skip, kill and failure the engine records carries a structured cause on the row’s error field — a stable code, a sentence, and whatever ids make the sentence actionable — and the dashboard renders that rather than a JSON payload. The ones worth knowing by name:

_00_schedule.last_error is the exception: it is a bare string, because it is why a schedule could not be planned or fanned out at all (a bad cron, a failing forEach) — which happens before any run row exists to carry a cause.

Each step in GET /admin/api/workflows/runs/:id carries both created_at and started_at, and the distinction matters: the engine creates every step row when the run is spawned, so created_at on a step deep in a DAG is the run’s start, not the step’s. started_at is stamped when the step’s job is created and is the only honest answer to “when did this begin”. The Timeline draws from it, and falls back to the moment the step’s last dependency finished — marked as an estimate — for runs recorded before the engine stamped it.

Incident details show publication backlog when an SSP heartbeat sample was available: peak queued operations and bytes, longest observed wait, parked batches, last successful publication, and related view links. Bulk database session generation and reconnect timing distinguish database recovery from circuit evaluation. The SSP page also shows its current publication backlog. These are sampled measurements; a spike between samples can be missed.

Regenerating these screenshots

They are captured from the real dashboard driven by a fixed fixture, not by hand:

pnpm --filter @spooky-sync/dashboard screenshots

See apps/dashboard/screenshots/README.md for what to update when a route is added or a response shape changes.