Sp00ky Logo SP00KY_SYNC
Github|...

Backend Functions

Backend functions allow you to execute server-side logic from your Sp00ky application. Currently, Sp00ky supports backend integration through the outbox pattern, where client operations create job records that are processed asynchronously by a backend service.

How It Works

The outbox pattern decouples your frontend from backend services:

  1. Your client calls db.run() to create a job record in SurrealDB
  2. A separate job runner service polls for pending jobs
  3. The job runner executes the HTTP request to your backend API
  4. The job status is updated in the database
  5. Your client can query the job status reactively
Note

Direct backend calls may be supported in future versions. For now, the outbox pattern is the recommended approach for all backend integrations.

Configuration

Backend functions are configured in your sp00ky.yml file in the schema directory.

yaml 
mode: sidecar
backends:
api:
  type: http
  baseUrl: http://host.docker.internal:3660
  spec: ../api/openapi.yml
  dev:
    type: npm
    script: dev
    workdir: ../api
  auth:
    type: token
    token: THIS_IS_TOP_SECRET
  method:
    type: outbox
    table: job
    schema: ./src/outbox/api.surql

Configuration Fields

  • type: The backend type (currently only http is supported)
  • baseUrl: The base URL of your HTTP backend service
  • spec: Path to the OpenAPI specification file that defines your API routes
  • auth (optional): Authentication configuration for the backend
    • type: Authentication type (currently only token is supported)
    • token: The bearer token to include in HTTP requests
  • method.type: The method for calling the backend (use outbox for the outbox pattern)
  • method.table: The name of the job table in SurrealDB (e.g., job)
  • method.schema: Path to the SurrealQL schema file that defines the job table structure
  • dev (optional): A dev server command that spooky dev will spawn alongside infrastructure (see Dev Server below)

Authentication

When you configure an auth block in your backend configuration, the job runner will automatically include the token in all HTTP requests to that backend using the Authorization: Bearer <token> header.

This is useful for securing your backend APIs and ensuring that only authorized job runners can execute requests.

yaml 
# Backend with authentication
backends:
api:
  type: http
  baseUrl: https://api.example.com
  spec: ../api/openapi.yml
  auth:
    type: token
    token: ${API_TOKEN}  # Can use environment variables
  method:
    type: outbox
    table: job
    schema: ./src/outbox/api.surql

On your backend server, validate the bearer token in your middleware:

TypeScript 
import { bearerAuth } from 'hono/bearer-auth';

app.use('/*', bearerAuth({
token: process.env.API_AUTH_TOKEN
}));

OpenAPI Specification

Your backend routes are defined using an OpenAPI specification. Each path becomes a callable route from your Sp00ky client.

yaml 
openapi: 3.1.0
info:
version: 1.0.0
title: example
paths:
/spookify:
  post:
    requestBody:
      content:
        application/json:
          schema:
            type: object
            properties:
              id:
                type: string
                example: thread:kv9b3b...
            required:
              - id
    responses:
      '200':
        description: ok
      '404':
        description: Thread not found
      '500':
        description: Internal server error

The request body schema defines the parameters that will be validated when you call db.run(). In this example, the /spookify endpoint requires an id parameter.

Job Table Schema

The job table stores outbox records and tracks their execution status. Here’s the required schema:

sql 
DEFINE TABLE job SCHEMAFULL
PERMISSIONS
  FOR select, create, update, delete 
  WHERE $access = "account" AND assigned_to.author.id = $auth.id;

-- Link to parent record (e.g., thread, user)
DEFINE FIELD assigned_to ON TABLE job TYPE record<thread>
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

-- API route path
DEFINE FIELD path ON TABLE job TYPE string
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

-- JSON payload for the request
DEFINE FIELD payload ON TABLE job TYPE any
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

-- Retry configuration
DEFINE FIELD retries ON TABLE job TYPE int DEFAULT ALWAYS 0
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

DEFINE FIELD max_retries ON TABLE job TYPE int DEFAULT ALWAYS 3;

DEFINE FIELD retry_strategy ON TABLE job TYPE string DEFAULT ALWAYS "linear"
ASSERT $value IN ["linear", "exponential"]
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

-- Job status lifecycle
DEFINE FIELD status ON TABLE job TYPE string DEFAULT ALWAYS "pending"
ASSERT $value IN ["pending", "processing", "success", "failed"]
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

-- Error tracking
DEFINE FIELD errors ON TABLE job TYPE array<object> DEFAULT ALWAYS []
PERMISSIONS
  FOR create WHERE true
  FOR select, update WHERE false;

-- Timestamps
DEFINE FIELD updated_at ON TABLE job TYPE datetime
DEFAULT ALWAYS time::now()
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

DEFINE FIELD created_at ON TABLE job TYPE datetime
VALUE time::now()
PERMISSIONS
  FOR create, select WHERE true
  FOR update WHERE false;

Key Fields

  • assigned_to: Links the job to a parent record (optional, but useful for querying jobs by entity)
  • path: The API route path from your OpenAPI spec
  • payload: JSON payload containing the request parameters
  • status: Current job state: pending, processing, success, or failed
  • retries: Number of retry attempts so far
  • max_retries: Maximum number of retry attempts before marking as failed
  • retry_strategy: Retry timing strategy (linear or exponential)
  • errors: Array of error objects from failed execution attempts

Calling Backend Functions

Use the db.run() method to create a job and call your backend function:

TypeScript 
// Call a backend function
await db.run('api', '/spookify', { id: threadData.id }, { 
assignedTo: threadData.id 
});

Method Signature

TypeScript 
db.run(
backend: string,      // Backend name from sp00ky.yml
path: string,          // Route path from OpenAPI spec
payload: object,       // Request parameters
options?: {
  assignedTo?: string,           // Link to parent record
  max_retries?: number,          // Override default max retries
  retry_strategy?: 'linear' | 'exponential'  // Override retry strategy
}
)

Parameters

  • backend: The name of the backend from your sp00ky.yml (e.g., 'api')
  • path: The API route path (e.g., '/spookify')
  • payload: An object containing the request parameters defined in your OpenAPI schema
  • options (optional):
    • assignedTo: Record ID to link this job to (useful for querying related jobs)
    • max_retries: Override the default maximum retry attempts (default: 3)
    • retry_strategy: Override the retry strategy (default: 'linear')
Warning

The db.run() method validates the payload against your OpenAPI schema. Missing required parameters will throw an error.

Querying Job Status

Jobs are stored as regular SurrealDB records, so you can query them using the query builder. The most common pattern is to use .related() to fetch jobs for a specific entity:

TypeScript 
const threadQuery = db.query('thread')
.where({ id: 'thread:abc123' })
.related('jobs', (q) => {
  return q
    .where({ path: '/spookify' })
    .orderBy('created_at', 'desc')
    .limit(1);
})
.one()
.build();

const thread = useQuery(db, () => threadQuery);

// Access the job status
const job = thread()?.jobs?.[0];
const isProcessing = ['pending', 'processing'].includes(job?.status);

This pattern is reactive - when the job runner updates the job status in the database, your UI will automatically reflect the changes.

Complete Example

Here’s a complete example of implementing a “spookify” feature that generates AI content:

TypeScript 
import { useQuery } from '@spooky-sync/client-solid';
import { db } from './db';
import { createSignal } from 'solid-js';

function ThreadDetail() {
const [isSubmitting, setIsSubmitting] = createSignal(false);

// Query thread with related jobs
const threadQuery = db.query('thread')
  .where({ id: 'thread:abc123' })
  .related('jobs', (q) => {
    return q
      .where({ path: '/spookify' })
      .orderBy('created_at', 'desc')
      .limit(1);
  })
  .one()
  .build();

const thread = useQuery(db, () => threadQuery);

// Check if job is in progress
const isJobRunning = () => {
  const job = thread()?.jobs?.[0];
  return ['pending', 'processing'].includes(job?.status);
};

// Trigger backend function
const handleSpookify = async () => {
  setIsSubmitting(true);
  try {
    await db.run('api', '/spookify', 
      { id: thread()?.id },
      { assignedTo: thread()?.id }
    );
  } catch (err) {
    console.error('Failed to spookify:', err);
  } finally {
    setIsSubmitting(false);
  }
};

return (
  <div>
    <h1>{thread()?.title}</h1>
    <p>{thread()?.content}</p>

    <button 
      onClick={handleSpookify}
      disabled={isSubmitting() || isJobRunning()}
    >
      {isJobRunning() ? 'Spookifying...' : 'Spookify Thread'}
    </button>

    {thread()?.jobs?.[0]?.status === 'failed' && (
      <div class="error">
        Job failed after {thread()?.jobs?.[0]?.retries} attempts
      </div>
    )}
  </div>
);
}

Job Runner Setup

The job runner is a separate service that processes outbox jobs. You can use the example job runner from the Sp00ky repository or implement your own.

The job runner:

  1. Polls the job table for records with status = 'pending'
  2. Updates status to 'processing'
  3. Makes the HTTP request to your backend
  4. Updates status to 'success' or 'failed'
  5. Implements retry logic based on retry_strategy
Note

See the packages/job-runner directory in the Sp00ky repository for a reference implementation in Rust.

Dev Server

Each backend in sp00ky.yml can declare an optional dev field. When you run spooky dev, these commands are spawned alongside the rest of the infrastructure (SurrealDB, SSP, scheduler), so you don’t need to start backend services in separate terminals.

Each backend gets a unique color in the terminal and its output is prefixed with [backend.<name>.dev], making it easy to distinguish logs from different services.

String Form

The simplest option is a raw shell command:

yaml 
backends:
api:
  dev: "node server.js"
  # ...other fields...

This runs the command from the project root using sh -c.

npm Type

Run an npm/pnpm script:

yaml 
backends:
api:
  dev:
    type: npm
    script: dev           # runs `pnpm run dev`
    workdir: ../api       # optional: directory to run from (relative to sp00ky.yml)
  # ...other fields...

Docker Type

Build and run a Dockerfile:

yaml 
backends:
worker:
  dev:
    type: docker
    file: Dockerfile.dev   # Dockerfile to build
    workdir: ../worker     # optional: build context directory
    port: "3000:3000"      # optional: port mapping
    env:                   # optional: environment variables
      - NODE_ENV=development
  # ...other fields...

uv Type (Python)

Run a Python script via uv:

yaml 
backends:
ml-service:
  dev:
    type: uv
    script: server.py      # runs `uv run server.py`
    workdir: ../ml-service  # optional: directory to run from (relative to sp00ky.yml)
  # ...other fields...

This uses uv run to execute the script inside the project’s virtual environment, automatically installing dependencies from pyproject.toml if needed.

env-file

All typed forms support an optional env-file field that loads environment variables from a dotenv-style file (one KEY=VALUE per line, # comments and blank lines are ignored). The path is resolved relative to the project root.

yaml 
backends:
api:
  dev:
    type: npm
    script: dev
    env-file: .env.development   # loaded into the process environment
    workdir: ../api
  # ...other fields...

For the docker type, the file is passed directly via Docker’s --env-file flag. For all other types (npm, uv), the variables are parsed and injected into the spawned process environment.

workdir

All typed forms support an optional workdir field that is resolved relative to the project root (the directory containing sp00ky.yml). If omitted, commands run from the project root.

Note

Backend dev processes are automatically stopped when you press Ctrl+C to shut down spooky dev.

Best Practices

Always link jobs to their parent entity using assignedTo. This makes it easy to query related jobs:

TypeScript 
await db.run('api', '/process-order', 
{ orderId: order.id },
{ assignedTo: order.id }  // Link job to order
);

Handle Failed Jobs in Your UI

Check the job status and provide feedback to users:

TypeScript 
const job = thread()?.jobs?.[0];

if (job?.status === 'failed') {
return <ErrorMessage>
  Operation failed after {job.retries} attempts. 
  Please try again later.
</ErrorMessage>;
}

Set Appropriate Retry Policies

For idempotent operations, use higher retry counts. For non-idempotent operations, use lower retry counts or disable retries:

TypeScript 
// Safe to retry multiple times
await db.run('api', '/generate-content', data, {
max_retries: 5,
retry_strategy: 'exponential'
});

// Should not retry
await db.run('api', '/charge-payment', data, {
max_retries: 0
});

Use Permissions to Secure Jobs

The job table should have permissions that prevent users from seeing or modifying other users’ jobs:

sql 
DEFINE TABLE job SCHEMAFULL
PERMISSIONS
  FOR select, create, update, delete 
  WHERE $access = "account" AND assigned_to.author.id = $auth.id;

This ensures users can only access jobs assigned to records they own.