How does sync and local-first work with IndexedDB?

how does the sync / local first stuff work? does it use indexedb in the client side?

Now I have a thorough understanding of the entire sync architecture. Here's the full picture:

##How InstantDB's Sync / Local-First Architecture Works

Yes, it uses IndexedDB on the client side. Here's a detailed breakdown of the entire system:

###1. Client-Side Storage: IndexedDB

The primary persistence layer is IndexedDB, implemented in client/packages/core/src/IndexedDBStorage.ts.

• The DB is named instant_${appId}_${version} (currently version 6).
• It has 3 object stores:
  • kv — General key-value store (holds pendingMutations, currentUser, local tokens, etc.)
  • querySubs — Cached query subscription results (the actual data for each query)
  • syncSubs — Data for the SyncTable feature

Rather than using IndexedDB's built-in versioning (which can't roll back and causes issues with multiple tabs), they create a new IDB database per version and manually migrate data (e.g., upgrade5To6).

There's also an InMemoryStorage (client/packages/core/src/InMemoryStorage.ts) used for server-side rendering or testing, providing the same StoreInterface but backed by a Map.

###2. The PersistedObject Abstraction

client/packages/core/src/utils/PersistedObject.ts is the bridge between in-memory state and IndexedDB. It handles:

• Lazy loading: Data is loaded from IDB asynchronously. If set() is called before loading finishes, it updates the in-memory value first, then merges with storage on load via a custom onMerge function.
• Throttled writes: Saves to IDB are throttled (e.g., 100ms for the kv store) and use requestIdleCallback to avoid blocking the main thread.
• Garbage collection: Query caches are GC'd with configurable maxAgeMs (1 year), maxEntries (1000), and maxSize (1 million triples).
• Serialization/parsing: Custom serialize/parse per key type (e.g., pendingMutations is stored as [...map.entries()] and parsed back into a Map).

The Reactor creates two PersistedObject instances (in _initStorage, line ~455):

this.querySubs = new PersistedObject({ persister: new Storage(appId, 'querySubs'), ... });
this.kv = new PersistedObject({ persister: new Storage(appId, 'kv'), ... });

###3. The Triple Store (In-Memory)

The actual query-able data lives in an in-memory triple store (client/packages/core/src/store.ts):

• Data is stored as triples: [entity, attribute, value, timestamp]
• Indexed in 3 ways for fast lookups:
  • eav (Entity → Attribute → Value)
  • aev (Attribute → Entity → Value)
  • vae (Value → Attribute → Entity — for reverse lookups / relationships)
• The store is immutable — mutations produce new store instances via the mutative library (like Immer but faster).

The transact() function (line 891) applies tx-steps (like add-triple, retract-triple, deep-merge-triple) to a store immutably.

###4. The Reactor: Orchestrating Sync

The Reactor (client/packages/core/src/Reactor.js) is the central coordinator. Here's the sync flow:

####Queries (Server → Client)

1. subscribeQuery(q, cb) — Registers a query subscription.
2. If a cached result exists in querySubs (loaded from IndexedDB), it's returned immediately — this is the "local-first" instant load.
3. The query is sent to the server via add-query over the WebSocket.
4. Server responds with add-query-ok containing triples and pageInfo.
5. The triples are loaded into an in-memory store, optimistic updates are layered on top, and instaql runs the query against the store.
6. The result is persisted back to IndexedDB for next time.

####Mutations (Client → Server) — Optimistic Updates

1. pushTx(chunks) — The user calls transact(...).
2. The transaction chunks are transformed into tx-steps via instaml.transform().
3. The mutation is immediately stored in pendingMutations (persisted to IndexedDB!) with a unique eventId.
4. notifyAll() is called — every active query re-computes its result by:
   • Taking the last server-confirmed store
   • Applying all pendingMutations on top via _applyOptimisticUpdates() (line 1384)
   • This gives the user instant local feedback
5. The mutation is sent to the server via WebSocket (op: 'transact').
6. Server responds with transact-ok + a tx-id.
7. The mutation is marked as confirmed but stays in pendingMutations until all queries have processed it (tracked via processedTxId).
8. When a refresh-ok arrives (server pushes updated query results), mutations with tx-id <= processedTxId are cleaned up.

####Offline Resilience

• If offline, mutations are enqueued (not sent), staying in pendingMutations in IndexedDB.
• A NetworkListener monitors connectivity.
• When back online, _startSocket() reconnects, _flushPendingMessages() re-sends all unconfirmed mutations.
• The merge logic in _onMergeKv (line 553) handles merging storage mutations with in-memory ones on load, and re-sends any that don't yet have a tx-id.

###5. Transport Layer: WebSocket + SSE Fallback

client/packages/core/src/Connection.ts provides two transport options:

• WSConnection — Primary, using native WebSocket.
• SSEConnection — Fallback using Server-Sent Events (for environments where WebSocket fails).

The Reactor first tries WebSocket. If it fails (_wsOk === false), it falls back to SSE on the next reconnect.

The connection flow:

1. Open WebSocket to wss://api.instantdb.com/runtime/session?app_id=${appId}
2. On open, send init message with app ID and refresh token
3. Server responds with init-ok + schema attrs
4. Flush pending queries and mutations

###6. Cross-Tab Sync

A BroadcastChannel('@instantdb') is used to sync auth state across browser tabs (line ~327). When one tab signs in/out, others are notified.

###Summary Diagram

User Code
    │
    ▼
┌──────────────┐
│   Reactor    │ ◄── Central orchestrator
├──────────────┤
│ pushTx()     │──► pendingMutations ──► IndexedDB (kv store)
│              │──► optimistic update ──► notify subscribers immediately
│              │──► WebSocket send ──────► Server
│              │
│ subscribeQ() │──► check IndexedDB cache (querySubs) ──► return cached if exists
│              │──► WebSocket send ──────► Server
│              │◄── WebSocket receive ◄── Server (triples)
│              │──► build in-memory triple store
│              │──► apply optimistic mutations on top
│              │──► run instaql query ──► notify callback
│              │──► persist to IndexedDB (querySubs)
└──────────────┘
        │
   ┌────┴────┐
   │ Storage │  IndexedDB (web) / InMemoryStorage (SSR)
   │ Stores: │  • kv (pendingMutations, currentUser, ...)
   │         │  • querySubs (cached query results as triples)
   │         │  • syncSubs (SyncTable data)
   └─────────┘

The architecture is elegantly designed so that the same StoreInterface abstraction can be backed by IndexedDB (browser), in-memory (SSR/tests), or potentially other backends (React Native also uses IndexedDB via the same code path, as seen in client/packages/react-native/src/Storage.js).

how do mutations communicate queries that something's updated? does a query track "all the items that match, or may match"?

Now I have the full picture. Here's the answer:

##How Mutations Communicate Updates to Queries

There are two very different mechanisms at play — one client-side (optimistic) and one server-side (reactive invalidation). They work fundamentally differently.

###Client-Side: Brute-Force Replay (No Tracking)

On the client, there is no targeted invalidation. When a mutation happens, every active query is re-evaluated against every pending mutation.

Here's the flow in Reactor.js:

1. pushTx() (line 1441) saves the mutation into pendingMutations and then calls notifyAll() (line 1486).

2. notifyAll() (line 1428) iterates over every query hash in queryCbs and calls notifyOne(hash) for each.

3. notifyOne(hash) calls dataForQuery(hash) (line 1335), which:

   • Takes the query's last server-confirmed store (the triple store from the most recent refresh-ok or add-query-ok)
   • Replays ALL pending mutations on top via _applyOptimisticUpdates() (line 1384):

     _applyOptimisticUpdates(store, attrsStore, mutations, processedTxId) {
       for (const [_, mut] of mutations) {
         if (!mut['tx-id'] || (processedTxId && mut['tx-id'] > processedTxId)) {
           const result = s.transact(store, attrsStore, mut['tx-steps']);
           store = result.store;
           attrsStore = result.attrsStore;
         }
       }
       return { store, attrsStore };
     }
     

   • Runs instaql() (the query engine) against the resulting store
   • Does a deep equality check against the previous result — if nothing changed, the callback is skipped

So the client doesn't track "which entities match which query." It simply replays all mutations against each query's store and lets the query re-evaluate. The deep equality check at the end prevents spurious UI updates.

Each query has its own independent triple store — it's not a single global store. Each querySub has a result.store containing only the triples relevant to that query (as returned by the server).

###Server-Side: Topic-Based Invalidation (The Smart Part)

The server is where the actual "what queries are affected" logic lives. It uses a topic matching system.

####What is a "Topic"?

A topic is a 4-tuple: [index-type, entity, attribute, value] where each component is either:

• A set of specific values (e.g., #{uuid-1 uuid-2}) — "matches any of these"
• The symbol _ (wildcard) — "matches anything"

From server/src/instant/reactive/topics.clj:

;; A triple insert [entity=abc, attr=name, value="Alice"] 
;; produces a topic like:
[#{:ea :eav :av :ave :vae}  #{abc}  #{name-attr-id}  #{"Alice"}]

####Queries Register Topics

When a query is executed, the server extracts coarse topics from the query's datalog patterns (server/src/instant/db/datalog.clj, line 333):

(defn pats->coarse-topics [pats]
  (->> pats
       flatten-pats-for-coarse-topics
       (mapv pat->coarse-topic)))

Each pattern variable (?x) becomes _ (wildcard), and concrete values become singleton sets. So a query like "get all users where name = 'Alice'" would produce topics like:

[#{:ea}  _  #{name-attr-id}  #{"Alice"}]

These topics are stored alongside the datalog query in the reactive store.

####Mutations Produce Topics

When a write hits the database, the WAL (Write-Ahead Log) invalidator (server/src/instant/reactive/invalidator.clj, line 65) picks it up:

(defn- invalidate! [_process-id store {:keys [app-id tx-id isn] :as wal-record}]
  (let [topics      (topics/topics-for-changes wal-record)
        session-ids (rs/mark-stale-topics! store app-id tx-id isn topics wal-record)
        ...])

topics-for-changes converts WAL changes (inserts/updates/deletes on the triples table) into topics. For example, inserting a triple [entity=abc, attr=name, value="Bob"] produces:

[#{:ea :eav :av :ave :vae}  #{abc}  #{name-attr-id}  #{"Bob"}]

####Topic Matching = Intersection

The key function is match-topic? (server/src/instant/reactive/store.clj, line 1039):

(defn match-topic?
  [[iv-idx iv-e iv-a iv-v]    ;; invalidation topic (from mutation)
   [dq-idx dq-e dq-a dq-v]]  ;; datalog query topic (from query)
  (and
   (match-topic-part? iv-idx dq-idx)
   (match-topic-part? iv-e   dq-e)
   (match-topic-part? iv-a   dq-a)
   (match-topic-part? iv-v   dq-v)))

Where match-topic-part? (line 994) handles wildcards:

• _ (symbol) matches anything
• Sets match if they intersect
• Supports comparison operators ($gt, $lt, $like, etc.)

So mark-stale-topics! (line 1372) scans all registered datalog queries for the app and checks if any of their topics match the mutation's topics. Matching queries get marked stale?.

####Then: Recompute & Push

When a session is told to refresh, handle-refresh! (line 459):

1. Gets all stale queries for that session: (rs/get-stale-instaql-queries store app-id sess-id)
2. Re-executes each stale query against the database
3. Compares with the previous result — if the result actually changed (result-changed?), it's included in computations; otherwise it's logged as "spam" and dropped
4. Sends refresh-ok with only the changed computations

###Summary: Two-Level System

The topics are intentionally coarse (over-approximate). A query for users where name = "Alice" will be invalidated if any user's name changes, not just Alice's. This is safe because the server re-runs the query and only sends refresh-ok if the result actually changed. The filter-queries function (line 1352) in store.clj also does some smarter filtering using "instaql topics" that can check whether the changed entities actually match the query's WHERE clause, pruning some false positives before re-execution.