Rebuilding Linear’s delta sync read path

Linear is a local-first application. Each client maintains a local database so that creating an issue, changing its status, or navigating a workspace doesn’t require a network round trip. That makes the app feel immediate, but the tradeoff is that a client returning online needs a way to catch up, fast.

Rather than download the entire workspace again, the client sends a checkpoint containing the ID of the last change it applied. Delta sync uses that checkpoint to retrieve only what has changed in the meantime.

Some of our largest workspaces produce close to one million sync actions per day, and a client that’s been offline for only a few hours can return hundreds of thousands of sync actions behind. Those results must also be filtered by what the user can access and has subscribed to. Across more than 20 TB of sync actions, delta sync becomes a large, permission-aware set intersection that is increasingly difficult to serve. We built a new read path with Turbopuffer to keep that query fast and predictable, even as our biggest workspaces continue to grow.

What a delta sync query actually does⁠

In practice, every change that Linear clients are concerned with creates a sync action. Each sync action has an ordered ID, the affected model, the type of change, routing metadata, and the data a client must apply.

These sync actions form an application-level log that clients replay into their local databases. It operates at a level of abstraction higher than the Postgres write-ahead log and describes changes in terms the client understands, such as updating an issue, deleting a comment, or archiving a project. Each workspace has its own immutable ordered sync action log where new sync actions are appended.

Linear filters the log by what the user can access and has subscribed to before returning the relevant sync actions.

A simplified delta-sync query looks like this:

For context, sync groups encode access to parts of a workspace, and sync subscriptions narrow that further to the models and views the client currently needs.

Why the Postgres read path stopped scaling⁠

Our previous delta sync serving path used a second Postgres table designed for these reads. As workspaces began producing thousands of sync actions between client checkpoints, each request combined widening ID range with array-overlap predicates for access and subscriptions, and Postgres spent a growing amount of CPU testing and discarding irrelevant rows.

This caused four problems:

• Tail latency became increasingly volatile, even when median latency remained healthy.
• Adding read replicas did not reduce the amount of intersection work required per request.
• Database maintenance and replica lag could delay how quickly a returning client caught up.
• Extending the query with additional filtering dimensions made an already CPU-intensive path even more expensive.

We could continue tuning indexes and adding replicas, but neither would change the fundamental shape of the query. We needed a serving index designed to combine very large sets with low latency and cost that remained predictable for our largest workspaces. While we investigated using a Postgres GIN index, the cost at write for our scale didn’t make sense.

Changing the shape of the query⁠

Turbopuffer is designed around inverted indexes that can efficiently evaluate large filter intersections. For each filterable attribute, it maintains a mapping from an attribute value to the sorted IDs of documents that contain it. That sorted set of IDs is called a posting list.

For delta sync, each document represents the metadata for a sync action, and its document ID is the sync action ID itself. A sync group, therefore, has a posting list containing every sync action routed to that group. A sync subscription has another, containing every sync action relevant to that subscription.

When a client requests a delta, ContainsAny unions the posting lists for the user’s sync groups. It does the same for the client’s sync subscriptions. Turbopuffer then intersects those sets with the requested sync action ID range and any remaining filters.

In the Postgres path, the database examined candidate rows from a potentially large range and repeatedly tested them against the request’s permission and subscription arrays. With inverted indexes, those filters are already represented as sorted sets of matching sync action IDs. The query can combine those sets directly and narrow in on the small intersection the client needs.

The surrounding workload also aligns well with Turbopuffer’s architecture:

• Sync actions are immutable, so the index is dominated by appends rather than updates.
• Each workspace maps to its own Turbopuffer namespace, keeping tenant data and query work isolated.
• Only fields used for filtering are indexed, while large sync action payloads remain in Postgres.
• Turbopuffer’s object-storage architecture allows the index to grow without requiring the full dataset to remain in memory.

Putting Turbopuffer in the read path⁠

We built a custom change-data-capture pipeline that reads committed sync actions from a Postgres publication and writes their metadata to Turbopuffer. From the moment a sync action commits in Postgres to the moment it becomes available in Turbopuffer, replication latency is roughly one second at p50 and a few seconds at p95.

On the read side, we re-designed sync as a two-stage pipeline that includes a metadata scan followed by late enrichment.

The metadata scan returns only the ordered IDs and routing fields needed to decide whether a sync action belongs in the response. While the actions are still represented by this lightweight metadata, the server applies access checks, subscription filters, packet transformations, and deduplication. Only after that filtering is complete do we fetch the full payloads. The server batches the surviving sync action IDs, retrieves their data from Postgres, and streams the enriched sync actions to the client.

Late enrichment is an important part of the design. In the old path, Postgres could read and decode large JSON payloads for candidate sync actions that were later rejected by an access or subscription check. The new path carries compact metadata through most of the server process and enriches only the sync actions that are about to leave for the client.

This keeps the expensive part of the pipeline focused on the sync actions the client will actually receive. As a result, we reduce Postgres I/O, JSON processing, and memory use while preserving an ordered streaming response.

Handling replication lag safely⁠

Moving reads to a secondary index raises the question of what happens to sync actions that have been committed in Postgres but are not yet visible in Turbopuffer.

We do not assume replication is perfectly current. For every delta request, Postgres serves a small, authoritative slice at the head of the sync action log, while Turbopuffer serves the larger historical range behind it. The two ranges intentionally overlap, and the server deduplicates the combined result by sync action ID. If Turbopuffer is unavailable or cannot cover the requested range, the request falls back to Postgres.

This makes the serving path tolerant of replication delay, out-of-order index visibility, deploys, and restarts. The replicator uses durable progress tracking and idempotent writes, while the read path treats Postgres as the final authority for the most recent actions.

Before sending production traffic through the new path, we ran it in shadow mode. Each request was executed against both Turbopuffer and the existing Postgres path, and we compared the resulting sync action IDs. Delta sync is a correctness-critical path; a faster query is only useful if it returns exactly the sync actions the client is supposed to receive.

Predictable latency at scale⁠

The clearest difference between the two read paths showed up in tail latency as workspaces grew. On the Postgres path, larger candidate ranges and access filters steadily increased p95 and p99 latency. With Turbopuffer’s posting-list indexes, however, both remained largely flat as workspace size increased.

In production, that flatter tail made catch-up much more predictable.

This matters most for our largest customers. A workspace generating close to one million sync actions per day should not make reconnecting progressively slower as it grows, nor should large permission and subscription sets force clients into a full bootstrap.

The broader lesson for us was that storing a change log and serving a change log are different problems. Postgres is the right place to commit and retain our client-facing WAL. The reconnect query, however, is an ID range intersected with very large permission and subscription sets.

Representing those sets as posting lists changed the economics of that query. Turbopuffer gave us a serving path whose latency stays predictable as both the log and the customer grow, while Postgres remains the authoritative source of the underlying data.