Dedicated machines

Give always-on backends a Hetzner VM of their own, one or several per machine. Sp00ky Cloud creates it, keeps it healthy, updates each app on it in place, and proxies traffic to them, so exposure and custom domains work as for any backend.

A backend normally runs as a container on your deployment’s host: the shared host, or for a Pro project its own project host, next to the scheduler and the SSPs. On a Pro project the database has a VM of its own. A dedicated machine gives always-on backends a Hetzner VM of their own instead: its own CPU and memory, nothing of the platform on the box, and a fixed address. One backend can have a machine to itself, or several can share one.

Sp00ky Cloud owns the machine from start to finish: it creates it, keeps every app on it running, updates each app in place when you deploy it, replaces the VM when it dies, and proxies traffic to the apps, so deploy.expose, spky domain add --app, outbox jobs and the operator dashboard all work exactly as for any other backend.

This is not a machine pool. A pool runs a backend for its jobs and takes the machines away when the queue is empty. A dedicated machine runs backends as services, always.

Declaring one

# sp00ky.yml
machines:
  media:
    provider: hetzner             # the only provider
    type: cpx32                   # Hetzner server type
    fallbackTypes: [cx33]         # when cpx32 cannot be had
    locations: [fsn1, nbg1]       # tried in order

apps:
  livekit:
    type: backend
    runOn: { machine: media }     # lives on the media VM
    deploy:
      dockerfile: ./livekit/Dockerfile
      port: 7880
      healthcheck: /              # required: the app takes traffic only once this answers
      ports: ["7881", "7882/udp"] # raw ports on the VM's own address (WebRTC media)
      expose: true                # https://<slug>-livekit.<domain>
  recorder:
    type: backend
    runOn: { machine: media }     # same VM, its own container
    deploy:
      dockerfile: ./recorder/Dockerfile
      port: 3676
      healthcheck: /health
      expose: true

machines.<name> is the VM; runOn: { machine: <name> } on a backend puts it there. Any number of backends may live on one machine, and runOn takes either pool or machine, never both. spky lint checks all of it, and spky deploy refuses a control plane that does not offer dedicated machines before building anything.

What a backend on it needs

• deploy.port: the port the service listens on. Sp00ky Cloud forwards it (and deploy.grpc_port, if set) to the machine through the core host.
• deploy.ports (optional): raw ports the machine opens to the internet on its own public address, for traffic the HTTP forwarder cannot carry: WebRTC media, RTMP, anything UDP. Each entry is published by docker on the VM and allowed in the machine’s firewall from anywhere ("7882/udp", "7881", "443:4433/udp", same syntax as in spky dev). They are not proxied by the core host, so clients need the machine’s public address: SPKY_MACHINE_IP (below) is it.
• deploy.healthcheck: an HTTP path answering 2xx when the service is ready. A new version of the app takes the place of the old one only once this answers.
• hosting: cloud (the default): Sp00ky Cloud runs it there. A hosting: external backend has its own host already.
• An image with a command: the VM imports the exported image, so deploy.cmd is needed when the Dockerfile has no CMD.

Apps on one machine share its one address, so no two of them may use the same port: each app’s port, grpc_port and deploy.ports entries must be distinct from every other app’s on that machine. spky lint says which two collide.

The backend’s environment reaches the machine the same way as on the core host, with the usual SPKY_* variables, except that SPKY_DB_URL and SPKY_DB_WS point at the published database host (https://<slug>-db.<domain>) rather than the internal one, and SPKY_SCHEDULER_URL and SPKY_SSP_ADDR are not set: those addresses have no public twin. Two more are added:

• SPKY_MACHINE_IP: the machine’s public IPv4 address, for a service that has to advertise it (a WebRTC SFU, a QUIC server, an RTMP URL).
• SPKY_MACHINE: the machine’s name.

Apps on the same machine reach each other directly by name: every app is on the machine’s own network as http://<app>:<port> (the recorder above talks to http://livekit:7880), without a trip through the platform. deploy.resources is ignored; the machine type is the size, and the apps on it share it.

What a deploy does

A machine has two kinds of change, and they are handled differently:

• An app changes (image, env, command, ports, healthcheck): only that app’s container is replaced, on the running VM. The other apps keep running untouched, and the machine keeps its address. The old container is stopped, the new one started, and the deploy waits until it answers its healthcheck. If it never does, the old version is put back, the deploy fails, and the machine keeps serving what it served before.
• The machine changes (type, locations): a fresh VM boots beside the running one with every app on it, and traffic switches over once all of them answer; the old VM is retired after that. A machine that never gets there is destroyed, the deploy fails, and the previous machine keeps serving. fallbackTypes are not part of this: they only matter when a VM is made.

A new VM is started as soon as the deploy is accepted, so it boots while your migrations run. In the apps phase Sp00ky Cloud waits for the machine to run everything, then routes to it.

  SERVICE        IP                 STATUS
  ● surrealdb    172.18.0.4         running
  ● scheduler    172.18.0.5         running
  ● ssp          172.18.0.6         running
  ◐ machine/media  65.21.4.17       starting (booting)
  ...
  ● machine/media  65.21.4.17       running (serving)
  ● livekit      172.18.0.9         running
  ● recorder     172.18.0.10        running

Each backend row is that backend’s presence on the core host: a small forwarder that carries the backend’s name and points at the machine. That is what keeps job routing, health probes and custom domains unchanged.

What spky status shows

  MACHINE        IP                 TYPE       STATUS         BACKEND
  ──────────────────────────────────────────────────────────────────────────────
  ● media g3     65.21.4.17         cpx32 fsn1 serving        livekit, recorder
      ● livekit              running
      ✗ recorder             failed, previous version kept
        exited with code 1

g3 is the generation: the third VM this machine name has had. The IP is the machine’s public address. serving means traffic goes there; booting and healthy mark a newcomer that has not taken over yet; unhealthy means the machine has stopped reporting and a replacement is on its way. Under each machine is every app on it, as the machine last reported it.

Failure and replacement

An agent on each machine reports every few seconds what runs and whether it answers. When it stops reporting for three checks in a row (30 seconds apart), or the VM is gone at the provider, a replacement starts: a fresh machine with the same shape and every app, promoted the moment all of them answer. The one that failed is retired afterwards. This happens at most three times per hour per machine; past that the machine is reported down and you get the uptime alert email. The serving machine is never touched before its replacement is ready, so a false alarm costs a machine-hour, not an outage.

A crashing app is not a reason to replace the machine: docker’s restart policy on the machine brings it back, and its healthcheck is watched through the forwarder like any backend’s.

When Hetzner has no capacity for the machine type (or its fallbackTypes) in any of the listed locations, a replacement cannot be made and the machine is reported down; Sp00ky Cloud keeps trying every 30 seconds until capacity is back, and those attempts do not count against the three per hour. Listing more than one location, and a fallback type or two, is the cheapest insurance against this.

spky restart <app> on a backend on a dedicated machine restarts that app’s container, on the same machine; the other apps on it are not touched.

Every switch to a new machine recreates the small forwarder containers on the core host, which costs about three seconds of 404 at the platform hostnames (measured on staging), the same as any other backend’s container restart.

Local development

spky dev ignores runOn.machine. Each backend is started like any other app, from its dev: command, and the local SSP dispatches its jobs to its baseUrl. There is no machine network in dev: an app that talks to another app on its machine by name needs a dev: env pointing at that app’s local address.

Limitations

• Hetzner only. provider must be hetzner; the machine type must be an x86 type the account can order.
• Git-linked deploys place the backends and carry deploy.ports like spky deploy does; spky restart <app> remains a CLI action.
• No logs from the machine yet. spky logs <app> shows the forwarder’s log, not the backend’s.
• deploy.ports are not forwarded, only port and grpc_port reach the machine through the core host; the others are open on the VM’s own public address. deploy.resources is ignored.
• A backend that calls the scheduler (SPKY_SCHEDULER_URL) does not work from the machine, as on pool machines.
• Machines count against your plan’s machine allowance, together with pool machines, and bill per machine-hour. A machine shared by several apps counts once.

Reference

Machine fields

On a backend