Machine pools

Run a backend's jobs on machines that exist only while there is work. Autoscaling, warm buffers, one machine per job if you want it, and nothing left running when the queue is empty.

Overview

Every backend normally runs as one always-on container next to your database, and the SSPs hand it one job after another. That is right for most work. It is wrong for work that needs a machine of its own: a render that pins every core for twenty minutes, a transcode, a headless browser, a model run, anything whose CPU, memory or bandwidth would starve the rest of the project.

A machine pool gives a backend’s jobs their own machines, created when there is work and destroyed when there is not. You declare the pool in sp00ky.yml and point a backend at it with runOn. Nothing about the backend changes: it still answers a POST per job, and your app still creates jobs with db.run() or a schedule.

# sp00ky.yml
mode: cluster                     # pools are run by the scheduler

pools:
  render:
    provider: hetzner             # a VM per machine on Sp00ky Cloud (spky dev uses containers)
    machine: { type: cx33, locations: [fsn1, nbg1] }
    min: 0                        # nothing runs while there is no work
    autoscale: true
    max: 2                        # the ceiling, and so the bill
    idleTimeout: 5m               # an idle machine goes after five minutes

apps:
  renderer:
    type: backend
    runOn: { pool: render }       # this backend's jobs run on the pool
    spec: ./renderer/openapi.yml
    baseUrl: http://127.0.0.1:8080
    method:
      type: outbox
      table: render_job
      schema: ./src/outbox/render.surql
    deploy:
      dockerfile: ./renderer/Dockerfile
      port: 8080
      healthcheck: /health

// renderer: an ordinary backend. On each machine an agent POSTs every job
// to it over loopback and waits for the answer, however long it takes.
app.get('/health', (_req, res) => res.send('ok'));

app.post('/render', async (req, res) => {
  const { gameId, format } = req.body;
  const file = await renderGame(gameId, format);   // minutes of CPU
  res.json({ url: await upload(file) });            // stored on the job row as `result`
});

// From your app, exactly like any other job:
await db.run('renderer', '/render', { gameId, format: 'mp4' });

With that configuration nothing runs while the queue is empty. The first job makes the scheduler ask for a machine; about a minute later the machine is up, runs the job, and after five idle minutes it is gone again. Two jobs at once get two machines. A third waits for one of them.

How a job runs

1. Your app (or a schedule) writes a pending row into the backend’s outbox table, as for any job.
2. The scheduler sizes the pool against what is queued and asks the provider for machines it lacks.
3. On each machine, a small agent starts your backend’s container, waits for its healthcheck to answer, and reports the machine ready.
4. The scheduler binds the oldest queued job to a free slot. The agent picks it up on its next poll and sends it to your backend as POST http://127.0.0.1:<port><path> with the job’s payload as the JSON body, and holds the request open until the backend answers.
5. The answer goes back to the scheduler, which records it on the job row exactly as it would for any other job.

The agent dials out: it polls the scheduler every few seconds and nothing ever connects to the machine. Every poll also renews the lease of the jobs it is running, so a job may run for hours without anyone tuning a timeout for it.

Sizing

A pool sizes itself one of three ways, decided by min, autoscale, max and buffer:

pools:
  # Fixed: always two machines. Jobs beyond the free slots wait for one.
  steady:
    min: 2

  # Baseline plus on demand: one machine always, more while jobs queue up,
  # never more than eight. A job that finds no free slot waits one boot.
  elastic:
    min: 1
    autoscale: true
    max: 8

  # Warm buffer: two ready machines always wait ON TOP of the busy ones,
  # so a job starts at once and the buffer refills behind it.
  instant:
    min: 0
    autoscale: true
    max: 10
    buffer: 2

  # Scale to zero: no machine at all until a job arrives.
  on-demand:
    min: 0
    autoscale: true
    max: 3

The rule behind all three, evaluated every two seconds:

capacity = busy slots + queued jobs + buffer x slots
needed   = ceil(capacity / slots)
desired  = autoscale ? clamp(max(min, needed), min, max) : min

A pool with min: 0 and buffer: 0 scales to zero: no machine runs until a job arrives, and each burst of work waits one boot. Surplus machines are only ever taken away once they have been idle for idleTimeout, the longest-idle first, and never a machine that is running a job.

Machines

Providers

spky dev runs every pool on docker, whatever the file says, so the same sp00ky.yml works locally and in the cloud. A deployed pool that should create VMs sets provider: hetzner.

Machine shape

pools:
  render:
    provider: hetzner
    machine:
      type: cpx42                 # any Hetzner Cloud x86 server type
      locations: [fsn1, nbg1, hel1]   # tried in order; empty = Hetzner decides
    slots: 2                      # two jobs at once on each machine

machine.type is a Hetzner Cloud server type name. It defaults to cx33. Use an x86 type (the CX, CPX and CCX lines): your backend image is built for linux/amd64, so the Arm types (CAX) cannot run it.

machine.locations lists Hetzner locations (fsn1, nbg1, hel1, ash, hil, sin) in order of preference. A location that is out of capacity, or does not offer the server type, is skipped for the next one. Capacity in single locations does run out now and then, so list more than one. Leave it empty and Hetzner picks.

slots is how many jobs one machine runs at the same time (default 1). Your backend must be able to serve that many concurrent requests; the machine’s cores are what they share.

Boot time

About a minute on Sp00ky Cloud from the scheduler asking to the machine being ready (the VM boots, installs the agent, downloads and starts your backend’s image), and a couple of seconds under spky dev. Keep images small, and use buffer when jobs must not wait for a boot.

Writing a pool backend

A pool backend is an ordinary backend with an outbox method. spky lint checks what a pool needs from it:

• an outbox method with a table: pool machines run outbox jobs;
• deploy.port: the port the agent sends jobs to;
• deploy.dockerfile: the image each machine runs (also what spky dev builds);
• deploy.healthcheck (recommended): a path that answers 2xx once the backend can take work. Without it, the backend counts as ready as soon as its port accepts connections.

The contract

A job’s deadline is maxJobDuration (default 8 hours). A job’s own timeout, when it has one (Jobs), shortens it but never extends it. The backend’s deploy.timeout is the core-host job runner’s setting and plays no part here.

Environment

Each machine runs your backend with:

• the backend’s own env:, resolved at deploy exactly as for any backend: inline values, env files and vault references;
• the image’s own ENV;
• on Sp00ky Cloud, SPKY_ENV, SPKY_DB_NS and SPKY_DB_NAME.

The environment reaches the machine through the agent’s authenticated connection to your scheduler, never through the cloud provider’s metadata, which anything on a VM can read back.

Networking

On Sp00ky Cloud, machines accept no inbound connections: every pool machine sits behind a firewall with no inbound rules, and your backend’s port is never published, only the agent inside the machine talks to it. Outbound traffic is open, so a job can fetch its input from and upload its output to anywhere. Each machine has its own network interface, which is the point for bandwidth-heavy work.

Machine lifecycle

A machine moves through these states, left to right; a machine that never becomes ready or stops answering is failed instead, and is destroyed all the same.

A machine starts draining when:

• it has been idle for idleTimeout and the pool has more machines than it needs;
• it has lived maxLifetime (default 24 hours), so a long-lived pool is renewed piece by piece;
• a deploy changed what it runs: a new image, command, environment or setting. Old machines finish their jobs and are replaced; a deploy never interrupts a running job;
• an operator drained it by hand.

recycle decides what happens between jobs on one machine. job (the default) restarts your backend’s process once the machine has no job running, so every job starts from a fresh process and nothing leaks from one job into the next. never keeps the process up, for backends that keep a useful cache in memory. Either way the machine itself stays until one of the reasons above.

When things go wrong

Worth knowing, because it is what makes a pool safe to leave alone:

• A machine dies or loses its network. Each job attempt holds a lease (default 90 seconds) that the machine’s polls renew. When the polls stop, the attempt goes back to the queue once the lease runs out and runs again on another machine; that counts as an attempt against max_retries. The agent keeps its own clock too and stops its jobs a safety margin before the lease can run out, so the same attempt never runs on two machines at once. A machine out of contact for long enough shuts itself down.
• The scheduler restarts. Nothing is lost. Which job runs where is recorded on the job rows, and the machines keep working and reconnect.
• A job is killed. spky jobs kill sends the cancel to the machine: the agent drops the request and, when it was the machine’s only job, restarts your backend so the work really stops. The job is failed with code cancelled.
• Machines keep failing to boot. After three failed boots in a row the pool stops creating machines for a minute, doubling up to fifteen, and opens a pool_breaker_open incident on the admin dashboard. A machine that boots fine closes it again. Typical causes: an image that does not start, a healthcheck that never answers, a server type the provider does not have.
• A limit is reached. Sp00ky Cloud caps how many machines a project may have (see below). A machine asked for beyond it is refused, and a refusal counts as a failed boot, so a pool whose max is above your plan’s limit opens its breaker whenever it tries to grow past the limit. Keep max within your plan.

Sp00ky Cloud also watches from its side, whatever your scheduler does: a pool never gets more than twice its size limit plus one machines (max, or min for a fixed pool; the headroom covers a deploy’s old machines draining while new ones start), machines of a scheduler that has been silent for half an hour are destroyed, no machine lives longer than 72 hours, and destroying a project destroys its machines first.

Local development

spky dev runs every pool as containers next to your dev stack, with the same scheduler logic as production. It builds each pool backend’s image from deploy.dockerfile (as sp00ky-dev-pool-<backend>), gives the scheduler the Docker socket so it can start and stop machine containers, and removes machine containers left over from a previous session.

A machine is ready in a couple of seconds, so scale-up, idle timeouts and a killed job are all easy to watch with docker ps. The agent and your backend log to the container, docker logs <container>.

On Sp00ky Cloud

Deploy with spky deploy. It sends the pool definitions along, keeps pool backends off the core host (their image only runs on pool machines), and publishes the scheduler’s pool listener at <slug>-pool.<domain> for machines to dial.

The limit counts machines in every pool of the project together, including ones still booting or draining.

Machine time is recorded per machine from creation to destruction and shows up in your project’s usage as pool_machine_hours. A pool with min: 0 costs nothing while it has no work.

Operating pools

# Jobs on a pool are ordinary jobs:
spky jobs                          # dashboard, pool tables included
spky jobs get render_job:abc       # payload, result, every attempt
spky jobs kill render_job:abc      # stops it on the machine
spky jobs retry render_job:abc

For the pools themselves, the scheduler’s admin API and admin MCP server have five operations:

Pausing is the big red button: a deploy never un-pauses a pool, just as it never un-pauses a schedule.

Self-hosting

A self-hosted scheduler serves pools with the docker provider. It needs the Docker socket and a pool listener its machine containers can reach:

# The scheduler's pool listener, which pool machines dial:
SPKY_POOL_PORT=9669
SPKY_POOL_PUBLIC_URL=https://pool.example.com   # as a MACHINE reaches it
SPKY_POOL_SECRET=<long random string>           # signs machine tokens; keep it stable

# The docker provider: machines are containers on the scheduler's docker host.
SPKY_POOL_DOCKER=1
SPKY_POOL_DOCKER_NETWORK=my-network

The full list is in Self-hosting → Machine pools. The hetzner provider needs Sp00ky Cloud, which holds the cloud credentials and enforces the limits on its side.

Limitations

• One backend per pool. A machine runs exactly one backend’s image. Give each backend its own pool.
• Deploy with spky deploy. Git-linked deploys do not carry pools yet.
• x86 only. See Machine shape.
• No database connection variables on the machine. See Environment.
• Hetzner is the only cloud provider.
• Cluster mode only. Pools are run by the scheduler, so the project needs mode: cluster.
• Jobs, not services. A pool runs a backend for its jobs and takes the machines away when there are none. For one always-on backend on a VM of its own, use a dedicated machine.

Validation

spky lint checks pools with the rest of the config:

• every runOn.pool names a declared pool, and every pool is used by exactly one backend;
• a pool backend has an outbox method with a table, and a deploy.port;
• autoscale: true has a max of at least 1, min is not above max, and buffer is not above max;
• a fixed pool has min of at least 1 and no max or buffer;
• slots is at least 1, lease at least 30 seconds, and maxLifetime not shorter than maxJobDuration;
• every duration parses.

Reference

Pool fields

Durations take the same form as elsewhere in sp00ky.yml: 90s, 10m, 1h30m, 8h.

On a backend

Job errors written by a pool

See also