Reactive Queries

A reactive query in Sp00ky is a query you declare once and forget about. As soon as the underlying data changes — whether the change happens in the same component, another tab, another device, or a backend job — the query result is updated and your UI re-renders automatically. You never write a refetch, a subscription teardown, or an invalidation rule.

How it works in theory

Reactivity in Sp00ky is driven by an end-to-end pipeline made of four coordinated pieces: the Scheduler, one or more Streaming Processors (SSPs), SurrealDB as the system of record, and a client-side Stream Processor running in the browser. Every reactive query follows the same five-stage lifecycle:

1. Build. You compose a query with the fluent API (db.query('table').where(...).build()). This produces a plan — not a request.
2. Register. useQuery hashes the plan into a stable queryHash, registers it with the local client, and subscribes a callback to that hash. The plan is also propagated to the Scheduler, which assigns it to an SSP.
3. Materialize. The assigned SSP compiles the plan into an incrementally-maintained view inside its in-memory circuit, executes it once to produce initial results, and marks the view live.
4. Propagate. When a write lands in SurrealDB, the Scheduler observes the change event and fans it out to every SSP over HTTP /ingest. Each SSP feeds the change into its circuit, which emits deltas for any affected views.
5. Notify. SurrealDB pushes a LIVE notification on the user’s _00_list_ref_user_<id> system table to every connected client over WebSocket (in refMode: dedicated, the default; the legacy refMode: single uses a shared _00_list_ref table — see Architecture). Each client’s Stream Processor re-derives which registered queryHash values are affected, reads fresh data from the local store, and invokes the useQuery callback. Your component re-renders.

The key property: every stage is incremental. The Scheduler does not re-broadcast full snapshots, the SSP circuit does not re-run full queries, and the client Stream Processor does not re-evaluate unrelated subscriptions.

The Fluent Query API

The db.query() method provides a fluent interface to construct queries. It is fully type-safe based on your schema.

// Build a query - returns a query object, not the results
const query = db.query('user')
  .where({ age: { $gt: 18 } })
  .orderBy('created_at', 'desc')
  .limit(10)
  .build();

// Execute with useQuery hook (reactive)
const usersQuery = useQuery(db, () => query);
const users = usersQuery.data();

Relationships

Sp00ky makes fetching related data simple with the .related() method. It automatically handles the underlying graph traversals or joins.

Unified Query Syntax

Whether you are fetching a simple 1:1 link, a 1:N collection, or traversing a complex N:M graph, the syntax remains the same. Sp00ky uses your schema definition to determine how to fetch the data.

// Fetch threads with their authors (1:1)
const threadsQuery = db.query('thread')
  .related('author')
  .orderBy('created_at', 'desc')
  .limit(10)
  .build();

// Use in a component with useQuery
const threads = useQuery(db, () => threadsQuery);

// Access related data
threads.data()?.forEach(thread => {
  console.log(thread.title);
  console.log(thread.author?.username); // Automatically typed!
});

Type Safety

When you use .related(), the return type of your query is automatically adjusted.

• Without .related(): the field is a string (Record ID).
• With .related(): the field becomes the full related object with selected fields.

const post = posts[0];

// ✅ TypeScript knows this is available
console.log(post.author.username);

// ❌ Error: Property 'email' does not exist on type 'User' (we only selected username/avatar)
console.log(post.author.email);

This ensures you can never accidentally access a property on a relation that hasn’t been fetched. TypeScript catches the error at compile time.

Using useQuery

For reactive applications, Sp00ky provides hooks that automatically update your component when the data changes.

import { useQuery } from '@spooky-sync/client-solid';
import { For } from 'solid-js';
import { db } from './db';

function UserList() {
  // Build query
  const query = db.query('user')
    .orderBy('created_at', 'desc')
    .limit(20)
    .build();
  
  // Hook automatically subscribes to updates
  const usersResult = useQuery(db, () => query);
  
  return (
    <ul>
      <For each={usersResult.data() || []}>
        {(user) => <li>{user.username}</li>}
      </For>
    </ul>
  );
}

Under the hood, useQuery calls subscribe(queryHash, callback, { immediate: true }). The immediate: true flag fires the callback synchronously with whatever is already in the local store so the first render has data; subsequent invocations come from Stream Processor deltas. UPDATE events are debounced by streamDebounceTime (default 50ms, configurable — see Configuration); CREATE and DELETE fire immediately. Subscriptions are torn down automatically on component unmount.

Pagination & infinite scroll

Pagination is just .limit() and .offset() on the query. The important part: every page is a live query, so once a page is on screen it keeps itself current — an insert, edit, or delete on the server reconciles the affected page automatically, with no refetch and no manual invalidation.

Load more (growing window)

The simplest approach: drive .limit() from a signal and grow it. Because the query factory reads the signal, useQuery re-registers the larger window when you call “load more”. Best for small-to-medium lists where keeping every loaded row live is fine.

import { useQuery } from '@spooky-sync/client-solid';
import { createSignal, For } from 'solid-js';
import { db } from './db';

function Feed() {
  const [limit, setLimit] = createSignal(20);

  // Reading limit() inside the query factory grows the live window:
  // useQuery re-registers whenever the resulting query changes.
  const posts = useQuery(db, () =>
    db.query('post')
      .orderBy('created_at', 'desc')
      .orderBy('id', 'asc')        // unique tiebreaker -> stable total order
      .limit(limit())
      .build()
  );

  return (
    <>
      <For each={posts.data() || []}>{(p) => <PostRow post={p} />}</For>
      <button onClick={() => setLimit((n) => n + 20)}>Load more</button>
    </>
  );
}

Windowed infinite scroll

For large lists, render one live query per page window (LIMIT PAGE START page*PAGE) and mount more windows as the user scrolls. Pass { deregisterOnCleanup: true } so a window that scrolls off-screen tears down its synced view — this keeps the number of live queries (and memory) bounded no matter how far the user scrolls.

import { useQuery } from '@spooky-sync/client-solid';
import { createSignal, createMemo, createEffect, For, onCleanup } from 'solid-js';
import { createStore } from 'solid-js/store';
import { db } from './db';

const PAGE = 50;

// One live window: LIMIT PAGE START page*PAGE. It reports its rows to the parent
// and drops them on unmount. deregisterOnCleanup cancels the synced view when
// this window scrolls off-screen (scrolling back re-registers it).
function Window(props) {
  const result = useQuery(
    db,
    () =>
      db.query('post')
        .orderBy('created_at', 'desc')
        .orderBy('id', 'asc')        // unique tiebreaker -> total order
        .limit(PAGE)
        .offset(props.page * PAGE)
        .build(),
    { deregisterOnCleanup: true }
  );
  // Pass a FRESH array so a same-length change (a delete shifts the next row in)
  // still updates the parent store.
  createEffect(() => props.onRows(props.page, (result.data() || []).map((r) => r)));
  onCleanup(() => props.onRows(props.page, undefined));
  return null;
}

function InfiniteList() {
  const [pages, setPages] = createSignal(1);          // grow as the user scrolls
  const [windows, setWindows] = createStore({});      // page -> rows | undefined

  // Contiguous rows from page 0; stop at the first gap or short (final) page.
  const rows = createMemo(() => {
    const out = [];
    for (let p = 0; p < pages(); p++) {
      const r = windows[p];
      if (r === undefined) break;     // window still loading
      out.push(...r);
      if (r.length < PAGE) break;     // short page = end of the list
    }
    return out;
  });

  const loadMore = () => setPages((p) => p + 1);

  return (
    <>
      <For each={Array.from({ length: pages() }, (_, p) => p)}>
        {(p) => <Window page={p} onRows={(page, r) => setWindows(page, r)} />}
      </For>
      <For each={rows()}>{(post) => <PostRow post={post} />}</For>
      {/* Call loadMore() from an IntersectionObserver sentinel or scroll handler */}
      <button onClick={loadMore}>Load more</button>
    </>
  );
}

Each window stays individually live. Offset windows are materialized server-side from the SSP’s window id-set rather than by re-applying START n against your local store, so a delete in an earlier page correctly shifts later rows up across window boundaries — you don’t get stale gaps or duplicated rows (see Architecture). For long lists, pair this with a virtualizer (e.g. @tanstack/solid-virtual) keyed by record id so only the visible rows render.

Behind the scenes: the Spooky Scheduler

The Scheduler is a Rust service (apps/scheduler/) that acts as the central coordinator for the reactive pipeline. It owns three things:

• A Snapshot Replica — an embedded SurrealDB instance backed by RocksDB that mirrors the authoritative database and is used to bootstrap new SSPs without hammering the primary.
• A Write-Ahead Log — every change event is appended to the WAL before being fanned out, giving the system crash-recovery and replay semantics.
• An SSP Pool — a live registry of Streaming Processor workers with per-worker health (heartbeats every 5s; stale after 15s). Queries are assigned to SSPs via a pluggable load-balancing strategy: Round Robin, Least Queries, or Least Load.

When an SSP dies mid-flight, the Scheduler’s monitors detect the missed heartbeats and reassign affected queries to healthy workers, so reactive subscriptions keep flowing without manual intervention. See Architecture and the Scheduler API for full details.

Behind the scenes: Streaming Processors (SSP)

A Streaming Processor (apps/ssp/) is a Rust worker that holds live materialized views in memory. Each view is compiled from a registered query plan into an incremental circuit: a dataflow graph that consumes record-level CREATE/UPDATE/DELETE events and emits only the delta for each view, rather than re-executing the query from scratch.

An SSP has three lifecycle states:

• Bootstrapping — on startup, the SSP registers with the Scheduler and self-bootstraps by paging SurrealQL queries against the Scheduler’s snapshot replica via POST /proxy/query. Events the Scheduler observed during bootstrap are buffered and replayed in this state too.
• Ready — the SSP now accepts POST /ingest calls in real time, routes each change through its circuit, and updates every affected view.
• Failed — bootstrap or replay surfaced an unrecoverable error; the Scheduler removes the SSP from the pool and reassigns its queries.

Registered view plans are persisted in the _00_query system table, so they survive restarts and can be re-registered across the pool. See the SSP API Reference for the full protocol.

Live sync across tabs, windows, and clients

Sp00ky does not use BroadcastChannel, SharedWorker, or ServiceWorker to coordinate tabs. Sync is server-mediated, and it takes the same path whether the two participants are two tabs on the same laptop or two devices on opposite sides of the planet.

Every SyncedDb instance — one per tab — opens its own WebSocket to SurrealDB and issues a single LIVE SELECT * FROM _00_list_ref_user_<auth_id> subscription (or LIVE SELECT * FROM _00_list_ref in the legacy refMode: single layout). _00_list_ref[_user_*] is a system table that records, for each registered query, which records are currently in its result set. Any change to that membership — a new row enters the result, an existing row leaves — produces a LIVE notification that every subscribed client receives.

sequenceDiagram
    participant TabA as Tab A (Device 1)
    participant TabB as Tab B (Device 1)
    participant ClientC as Client C (Device 2)
    participant DB as SurrealDB
    participant Sched as Scheduler
    participant SSP

    TabA->>DB: Mutation (WebSocket)
    DB->>Sched: Change event
    Sched->>SSP: POST /ingest
    SSP->>SSP: Circuit updates materialized views
    DB-->>TabA: LIVE notification on _00_list_ref
    DB-->>TabB: LIVE notification on _00_list_ref
    DB-->>ClientC: LIVE notification on _00_list_ref
    Note over TabA,ClientC: Each client's Stream Processor<br/>derives affected queryHashes<br/>and re-renders subscribers

A few consequences worth internalizing:

• There is no “same-origin fast path.” A mutation in Tab A reaches Tab B the same way it reaches Client C — via SurrealDB’s LIVE notification. This keeps the semantics predictable but means every tab pays for its own WebSocket.
• LIVE notifications signal membership, not content. When a client receives a LIVE event, its sync layer reconciles version arrays (localArray vs remoteArray) to pull any content changes. This separation means big record updates don’t have to ride on the LIVE channel itself.
• Local writes are optimistic. A mutation is applied to the local store immediately and enqueued to the server. When the server confirms and the LIVE notification comes back, the local state is reconciled.

Performance

Sp00ky’s reactive pipeline is built from pieces chosen for throughput:

• Rust Scheduler and Rust SSP. Both services are native Rust binaries. Change-event fan-out, view maintenance, and load balancing run without a GC.
• Rust → WASM Stream Processor on the client. The component that decides which local queryHash values changed after an ingest is a WebAssembly module compiled from Rust (@spooky-sync/ssp-wasm), not JavaScript interpreting deltas.
• WebSockets via SurrealDB LIVE queries. There is no HTTP polling layer and no secondary pub/sub broker on the hot path — clients talk directly to SurrealDB using its native WebSocket protocol, and invalidation piggy-backs on LIVE SELECT.
• Incremental everywhere. Neither the server-side circuit nor the client-side processor re-evaluate a query from scratch on each change; only the affected portions of each view are recomputed.

How this compares to other realtime frameworks

Sp00ky’s distinguishing combination is (a) a Rust Scheduler + SSP pool that maintains materialized views incrementally on the server, (b) a Rust→WASM Stream Processor that mirrors that incremental model on the client, and (c) SurrealDB LIVE queries over WebSocket as the invalidation bus. Most frameworks in the space implement one or two of these; none combine all three.

The framing above is about architectural differences, not about which tool is better. Pick Sp00ky when you want server-side incremental views plus a matching client-side processor over a single WebSocket to SurrealDB; pick an alternative when their different trade-offs fit your stack better.