Sync and working offline

A script that adds a note, reads the week's meals or uploads a photo needs none of this. It sends a command or asks a query over HTTP, and the answer comes from the household at that moment.

Sync is for a client that keeps its own copy of a household, so it can read without a network round trip and keep working offline. Rigel's web app does this: it holds every row of its household in a local database, shows a change the instant a person makes it, and sends the change when it can. Android isn't built yet, so the web app is the only client that syncs today.

The pieces are separate, and you can use one without the others.

• Pull reads every row that changed since a version. Use it to mirror a household.
• Push sends commands a device queued while it was offline, in order, safely to repeat.
• The socket tells you when to pull. See the household socket.

The routes are in the sync area of the reference.

A household has a version, an integer that goes up by one for every command it applies. Every command response carries the version after the command, as version. Every stored row records the version of the command that last wrote it. Version 0 is before anything happened.

GET /api/v1/households/{household_id}/pull takes the version you have, in from, and returns every row written after it and every row deleted after it. Send from=0 for everything.

curl "https://rigel.house/api/v1/households/hh_01JC8Q4W2N7XK5R3VYTZ9D6HBE/pull?from=41" \
  -H "Cookie: rigel_session=$RIGEL_SESSION"

{
  "from": 41,
  "version": 44,
  "rows": {
    "person": [
      {
        "id": "per_01JC8Q5X0RT4M8ZVB2HNQ6EK1Y",
        "created_at": 1794232800000,
        "updated_at": 1794240000000,
        "version": 43,
        "name": "Nia",
        "birth_date": "2019-04-01",
        "birth_date_precision": "day",
        "user_id": null,
        "sort_order": 2
      }
    ]
  },
  "tombstones": [{ "table_name": "family_note", "row_id": "fnt_01JC9A4B6D3KXQ8TW2VZ5NMRHE" }]
}

That response is cut down to one table. A real one has an entry in rows for every synced table, with an empty list when nothing in it changed.

• rows maps a table name to the rows that changed, oldest version first. Upsert each by its id.
• tombstones lists rows that were deleted. Remove each from your copy. Deletes are real, so there is no flag to check.
• version is where you are now. Keep it, and send it as from next time.

A pulled row is a stored row, not the shape a query returns. Columns are snake_case, times are integer UTC milliseconds (other formats use ISO 8601), and true and false are 1 and 0. This reference doesn't list table columns yet, so read rows by column name and ignore any you don't know.

A mirror is a loop around one request:

const base = "https://rigel.house/api/v1";
const householdId = "hh_01JC8Q4W2N7XK5R3VYTZ9D6HBE";
const headers = { Cookie: `rigel_session=${process.env.RIGEL_SESSION}` };

type Row = { id: string } & Record<string, unknown>;
type Pulled = {
  version: number;
  rows: Record<string, Row[]>;
  tombstones: { table_name: string; row_id: string }[];
};

const copy = new Map<string, Map<string, Row>>(); // your copy, by table then id
let cursor = 0; // keep this next to your copy

export async function pull(): Promise<void> {
  const response = await fetch(`${base}/households/${householdId}/pull?from=${cursor}`, { headers });
  if (!response.ok) throw new Error(`pull failed with ${response.status}`);
  const pulled = (await response.json()) as Pulled;
  for (const { table_name, row_id } of pulled.tombstones) copy.get(table_name)?.delete(row_id);
  for (const [name, rows] of Object.entries(pulled.rows)) {
    const table = copy.get(name) ?? new Map<string, Row>();
    for (const row of rows) table.set(row.id, row);
    copy.set(name, table);
  }
  cursor = pulled.version;
}

A device that can be offline keeps a queue. It makes a client_id once, such as cli_01JC9A2ZPD6KQ3W8YT5MXE1RNB, and keeps it for good. It numbers every command it queues with client_seq, starting at 1 and never skipping. It gives each command a command_id (cmd_…) that stays the same on every retry, which is the Idempotency-Key of the HTTP route.

POST /api/v1/households/{household_id}/sync/push takes the queue in client_seq order, up to 500 commands. Each has the command's name, its input exactly as the command's route takes it, and issued_at, when the person did it, in UTC milliseconds.

curl -X POST https://rigel.house/api/v1/households/hh_01JC8Q4W2N7XK5R3VYTZ9D6HBE/sync/push \
  -H "Cookie: rigel_session=$RIGEL_SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "cli_01JC9A2ZPD6KQ3W8YT5MXE1RNB",
    "commands": [
      {
        "client_seq": 7,
        "command_id": "cmd_01JC9A3M5K8WTR2XD4VQ7NB6HF",
        "name": "family.note.record",
        "input": { "person_id": "per_01JC8Q5X0RT4M8ZVB2HNQ6EK1Y", "text": "Not into plain chicken lately" },
        "issued_at": 1794239998000
      },
      {
        "client_seq": 8,
        "command_id": "cmd_01JC9A3M5MQ2X7TD4NE8WR5KBH",
        "name": "family.note.forget",
        "input": { "note_id": "fnt_01JC9A4B6D3KXQ8TW2VZ5NMRHE" },
        "issued_at": 1794240005000
      }
    ]
  }'

{ "last_client_seq": 8, "resend_from": null }

The household takes the commands in order and runs each one as you.

• A client_seq at or below the last one it processed is skipped. If a response is lost, send the whole queue again. Nothing runs twice.
• A client_seq that skips ahead is a gap. It stops there and answers with resend_from, the number to start again from.
• Applied or rejected, a command moves last_client_seq forward. A command whose input doesn't parse, whose name isn't a command, or whose command_id was already used for something else is rejected and recorded, with the code validation_failed, unknown_command or idempotency_key_reused. It never blocks the commands behind it. A name that isn't one of the commands in this reference is unknown_command.
• A client_id belongs to the first person who uses it. Another person gets 403 forbidden_client and nothing runs. Make a new client_id for a new device.

A push says how far the household got, not how each command went. The next pull says that. Add your client_id and after_seq, the highest client_seq you already know the outcome of, and the pull also returns what happened to your commands after it.

curl "https://rigel.house/api/v1/households/hh_01JC8Q4W2N7XK5R3VYTZ9D6HBE/pull?from=44&client_id=cli_01JC9A2ZPD6KQ3W8YT5MXE1RNB&after_seq=6" \
  -H "Cookie: rigel_session=$RIGEL_SESSION"

{
  "from": 44,
  "version": 45,
  "rows": {
    "family_note": [
      {
        "id": "fnt_01JC9A3M5KZZ0ZC3WE1E8ZESPG",
        "created_at": 1794239998000,
        "updated_at": 1794239998000,
        "version": 45,
        "person_id": "per_01JC8Q5X0RT4M8ZVB2HNQ6EK1Y",
        "text": "Not into plain chicken lately",
        "app": null,
        "noted_at": 1794239998000
      }
    ]
  },
  "tombstones": [],
  "last_client_seq": 8,
  "outcomes": [
    { "client_seq": 7, "command_id": "cmd_01JC9A3M5K8WTR2XD4VQ7NB6HF", "status": "applied", "error": null },
    {
      "client_seq": 8,
      "command_id": "cmd_01JC9A3M5MQ2X7TD4NE8WR5KBH",
      "status": "rejected",
      "error": { "code": "note_not_found", "message": "That note isn't there any more." }
    }
  ],
  "schema_version": 34
}

Here rows is cut down to the one table that changed, and the note was made by command 7. last_client_seq is the highest of your commands the household has processed, 0 if none. outcomes has one entry per command after after_seq. A rejected command carries the error's stable code and a message written for the person who did it, so a client can show it as it is. Drop everything up to last_client_seq from your queue, and send it as after_seq next time. Without client_id, after_seq does nothing and the response has only from, version, rows and tombstones.

A person should see their change at once, whatever the connection. So Rigel's web app runs a command against its local copy first, as a guess, and queues it. When a pull arrives it works in this order:

1. It takes back every change its still-queued commands made to the local copy.
2. It applies the household's deleted rows and changed rows.
3. It drops the queued commands the household has processed, and shows the rejected ones with their reason.
4. It runs the commands still waiting on top of the new state.

What a person sees is their own change first, then the household's answer, in one step. The screen is never left showing a state the household will contradict. A queued command that fails when it runs again stays in the queue with no local effect until the household rejects it and says why.

A command that creates rows makes the same IDs on the device and on the household, because the IDs are derived from the command's ID. A row created offline keeps its ID when the household confirms it, so there is nothing to swap.

There is no merge step. The household runs each command's intent against the state it has, in the order it receives them. Two people using up the same item both count. Two people renaming the same recipe both succeed and the later one wins, because an update changes only the fields it names. Text that people type into at the same moment, such as a document, is different: it merges as they type instead of one change winning. That text travels over the socket, not as commands, and the household settles it into the document before any command reads it. See editing a document together.

Running a command's logic locally takes the code behind it, and Rigel's apps share that code. It isn't published, so an outside client can't replay commands on its own copy. It can keep its queue, show those commands next to the pulled rows as waiting, and treat each pull as the answer. A few commands don't run on a device first even in Rigel's own apps, such as email.draft.send and documents.google.import. Their effects appear after the household answers.

Pulling on a timer works. To learn about a change within moments, connect to the household socket. After every commit, and once after a push that applied anything, the household sends every connected socket a poke:

{ "type": "poke", "version": 45 }

A poke carries no data. If its version is higher than your cursor, pull. The socket's first message, welcome, carries the current version in the same way. A client that was disconnected may have missed pokes, so it pulls after it reconnects.

Commands travel over HTTP today, not over the socket. Pushing over the socket isn't built.

A pull that includes client_id also returns schema_version, the number of household database changes the household has applied. A client that keeps rows in its own tables was built for one number. If the household reports a higher one, there may be new tables or columns you haven't seen.

Changes to the stored shape are made in two steps: the new shape is added and the old one keeps working, and only a release later is the old one removed. A client one step behind keeps working, as long as it ignores columns and tables it doesn't know. Rigel's web app compares the number to the one it was built with. When the household is ahead, it says so, keeps its queue and updates itself.

If your client is newer than the household, because a deploy is in progress or was rolled back, wait and try again.