Local First Development with Replicache - Building smooth user experiences | Shootmail

What are we building?

Demo Link: sveltekit-replicache-todo.pages.dev

Repo: https://github.com/subhendupsingh/sveltekit-replicache-todo

More on local first development: https://localfirstweb.dev/

You probably know even before I say it. A TODO App.

Though it may not seem so, but this decision is a thoughtful one. To understand a concept that is new and a bit different from the traditional methods of reading and writing data from and to a database, I thought to start with a simple app will be more suitable where the readers and me, both can focus on Replicache and local first development concepts rather than focussing on the application logic.

Yeah, I know I sound smart now ;-)

What is local first development?

Forget replicache for a moment. Let’s think about how would we normally write our TODO app. Here are the tools we need:

  1. Database - Supabase
  2. Web framework - Next, SvelteKit, Remix etc
  3. Optionally, we can separate our backend and frontend by using something like Express or Hono just for backend. I am not going to do that here.
  4. CSS - Tailwind CSS (Don’t tell me tailwind css is not vanilla css)

Todo app the traditional way

Here I am just outlining the steps

  1. We will define our database schema, say a *todo table with following columns - i d, task, created_at, updated_at, is_complete
  2. We will define our CRUD operations.
  3. We will make UI to add todos, mark them complete, delete and update them and will style them nicely and probably add the dark mode.

This app will work just fine. Now, imagine a use case where your database is hosted in Singapore, your application code is deployed on a server in Germany (think Hetzner).

One day, a Indie-hacker named SPS decides to pack his bags and work on his next Facebook from the a quaint village somewhere in the mountains. The internet connection is patchy. SPS decides to make an outline of his project on your TODO application. Somethings SPS might expect from your app is, it should work fast and changes he makes are not lost in case of internet fluctuation.

With your current setup, if there is an internet connection, SPS will be able to load his TODOs, but it will be quite slow because of the slow internet and multiple network round trips to reach the application server in Germany and database in Singapore. Also, in case the internet gets disconnected while he saves his TODOs, his changes will not be saved and he will have to do it again.

Result, SPS might switch to someone else’s TODO app who has already read this article and implemented replicache.

Todo app the replicache way

Replicache way or the local-first way in general means the Create, Update and Delete operations are performed using simple javascript methods called mutators, and the Read is done using replicache subscriptions or query.

Mutated data or mutations are always applied locally first, optimistically, which means that the changes are done locally and are reflected in the UI without informing the server yet. These mutations are then replayed on the server and the data changes in our database.

Now, even if the internet gets disconnected, replicache will try to replay he mutation on the server when it next gets connected, and it will keep on trying until the server replies with a confirmation that the mutations were applied. Hence, SPS can rest assured that his work will not be lost and all the changes are visible to him immediately because everything is read locally avoiding the network trips.

This makes SPS happy. That’s the purpose of local-first web development, making SPS happy.

Why replicache?

  1. Doesn’t offer separate storage, keep using your existing database.
  2. You don’t need to run any separate service to use it.
  3. Handles a lot automatically, exposes only a set of rules for you to implement
  4. Free - well, there is pricing but not until your business earns more than $200k in ARR.
  5. Has pink logo

In short, has so much to offer with very less expectations, be like replicache.

Replicache terminology

For local storage, replicache uses indexed db that is available with all modern browsers. Here are a few useful terms:

  1. Mutations: Simple javascript functions that change data
  2. Space: Logical separation of data. For example, in your TODO app, when SPS signs up, his user_id can be used to separate his data from the other users. So, this user_id is a space.
  3. Space version: Every space (every user) has a version number that starts from 1. Every time a mutation happens, this version is incremented.
  4. Push endpoint: Is a simple API endpoint that is called by replicache automatically whenever it needs to apply a mutation on the server.
  5. Pull endpoint: This is also a simple API endpoint that replicache calls automatically after a configurable interval (default : 60s).
  6. Query: To read data stored locally using replicache, you use replicache queries.
  7. Subscriptions: You can subscribe to a query. Whenever the data related to query changes, the data is automatically received in the subscription callback, from there, you can update your UI.
  8. Client: A browser tab is a client.
  9. Client group: A browser, a mobile device etc is a client group. Multiple clients (tabs) belong to one client group.

TODO app implementation, with replicache

Schema

replicache_space: For each user, there will be a replicache_space. Typically, user_id is space_id

CREATE TABLE replicache_space (
  id TEXT PRIMARY KEY,
  version INTEGER NOT NULL
);

replicache_client: Multiple clients

CREATE TABLE replicache_client (
  id TEXT PRIMARY KEY,
  client_group_id: TEXT NOT NULL,
  last_mutation_id INTEGER NOT NULL,
  last_modified_at: TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

todo: A todo is added by a user. For each user, there is a corresponding space. Space has its version.

CREATE TABLE todos (
  id text PRIMARY KEY,
  space_id TEXT NOT NULL,
  text TEXT NOT NULL,
  completed BOOLEAN NOT NULL DEFAULT FALSE,
  created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
  space_version INTEGER NOT NULL,
  deleted BOOLEAN NOT NULL DEFAULT FALSE
);

Generate replicache license

Run this command, it will return a random string, save it in your .env or similar

npx replicache@latest get-license

Disable ssr

Since replicache needs indexed db to run, which is only available in the browser, make sure to set ssr=false in wherever you want to use replicache. In Sveltekit, I am setting it in my +page.ts

export const ssr = false;

Define mutators

Create a separate file called mutators.ts. Here we will add all our mutators.

export const mutators = {
    createTodo: (tx, args) => {...},
    updateTodo: (tx, args) => {...},
    deleteTodo: (tx, args) => {...},
}

export type M = typeof mutators;

Initialise Replicache

import { mutators, type M } from './mutators';

let replicache = new Replicache<M>({
      licenseKey: license,
      name: spaceId,
      pushURL: `/api/push?spaceID=${spaceId}`,
      pullURL: `/api/pull?spaceID=${spaceId}`,
      mutators
})

Implementing push endpoint

This endpoint pushes the local mutations to server, and if the mutations are applied successfully, increases the version of the space.

  1. We get the current version of the space. If the space doesn’t exist, that means the user is new, we create a new space for the user with userId being the spaceId.
  2. We calculate what the next space version would be when all the mutations are applied successfully.
  3. We retrieve all the client Ids present in the push request.
  4. We get the lastMutationId for each client belonging to the clientGroupId we receive in the request.
  5. If a particular clientId doesn’t exist in the database, we create a new replicache_client.
  6. We start applying mutations to the database.
  7. We increment the lastMutationId for each client that pushed the mutation.
  8. Lastly, we prepare the response with the new version number and push it.

Implementing poke

Poke request in an indication to replicache that the database has updated or new records and it should immediately pull the changes.

Implementing pull endpoint

The pull request sent by replicache has the shape containing properties for updated records and their statuses.

UI Implementation

UI is simple. I will explain the relevant parts here:

Mutators

Mutators are the javascript functions that create, update or delete records on the local and then the same mutations are replayed on the server via push endpoint.

Query subscription

You can subscribe to any query with replicache to receive changes live as they happen and to update the UI in response.

Conclusion

Getting hang of the entire setup takes some time. But I recommend reading and re-reading this and once you implement this yourself, you will start to understand the concepts.

Also, the strategy explained here is known as “Per space version strategy”. There are 2 more, and you can read about them here. Select the one that makes most sense to you and your use case.

If you want to implement replicache in your existing application, I recommend adopting it incrementally.