Silicon JungleSilicon Jungle

Simple Syncing

So far we've covered how to create a map of last-writer-wins registers .

When building offline-first software users need to be able to make changes while they are offline that can easily be synced with changes from the server when they come back online.

In this article we will cover a simple protocol for exchanging changes over a network.

High level overview

On connection the client will send all changes it has made since it was last online and the server will respond by returning the entire state.

Whenever a change is made by a client, it will be sent to the server and distributed to the other connected clients.

Operations

Operations are data types that express changes to data. There are many common operations used by CRDT's including (but not limited to):

  • Insert
  • Delete
  • Create
  • Increment
  • Decrement
  • Set

For the purposes of this article we will only focus on the set operation.

The set operation will set the value and version at a key regardless of whether it currently exists in the map or not. If it doesn't exist, it will create the value, if it does it will replace its value.

export const OPERATION = {
  SET: 'set',
}

export const createOperation = {
  set: (key, version, value) => [OPERATION.SET, key, version, value],
}

Messages

For this example, we only need a connect and a patch message to communicate changes between clients and the server.

The connect and patch messages are just a list of operations, the only difference is that the server responds to them in different ways:

export const MESSAGE = {
  CONNECT: 'connect',
  PATCH: 'patch',
}

export const createMessage = {
  connect: (operations) => [MESSAGE.CONNECT, operation],
  patch: (operations) => [MESSAGE.PATCH, operations],
}

On connection

Connection handshake

The client will send any local changes that were made whilst they were offline to the server using the connect message.

const handleConnection = (client, operations) => {
  client.send(createMessage.create(operations))
}

The server will then respond by getting all of the values from the last-writer-wins map and sending them to the client using the patch message.

// Versions and values should have the same keys.
export const getSnapshotOperations = (versions, values) =>
  Object.keys(versions).map((key) =>
    createOperation.set(key, versions[key], values[key])
  )

export const handleMessage = (client, message, versions, values) => {
  const [type] = message
  switch (type) {
    case MESSAGE.CONNECT: {
      client.send(createMessage.patch(getSnapshotOperations(versions, values)))
    }
  }
}

Local changes

Client changes

Following on with the code in the last article , we can now send the local change over the network.

const client = new Client(SERVER_URL)

const change = applyLocalChange(
  versions,
  values,
  'jungle',
  userId,
  'Hello world'
)

if (change !== null) {
  client.send(createMessage.patch([change]))
}

We could also expand applyLocalChange to work with multiple changes.

Remote changes on the server

The server code needs to be modified to include support for patches. When a patch is received, we attempt to apply the operations to the map. Each of the operations which are successful are then broadcast to the other connected clients.

Filter out any operations which won't be applied

Operations which have an earlier version than the current version won't be applied and need to be filtered out.

const filterRemoteChanges = (operations, versions, values) =>
  operations.filer(([key, version]) =>
    lwwMap.shouldSetChild(versions, key, version)
  )

Apply the filtered operations

Since we are sending multiple operations over the network, we need to be able to apply the changes to the map.

const applyRemoteChanges = (operations, versions, values) => {
  operations.forEach(([key, version, value]) => {
    applyRemoteChange(versions, values, key, version, value)
  })
}

Handle the response to receiving a patch

When we receive a message from a client, we need to attempt to apply those changes to the map. Then we can broadcast the applied changes to all other connections.

export const handleMessage = (client, message, versions, values) => {
  const [type] = message
  switch (type) {
    case MESSAGE.CONNECT: {
      client.send(createMessage.patch(getSnapshotOperations(versions, values)))
    }
    case MESSAGE.PATCH: {
      const [_, operations] = message
      const filteredOperations = filterRemoteChanges(
        operations,
        versions,
        values
      )
      applyRemoteChanges(filteredOperations, versions, values)
      broadcastMessageExcluding(createMessage.patch(filteredOperations), client)
    }
  }
}

Remote changes on the client

The client side is simpler, it only needs to respond to patch messages and it doesn't need to broadcast any remote changes it receives.

export const handleMessage = (client, message, versions, values) => {
  const [type] = message
  switch (type) {
    case MESSAGE.PATCH: {
      const [_, operations] = message
      const filteredOperations = filterRemoteChanges(
        operations,
        versions,
        values
      )
      applyRemoteChanges(operations, versions, values)
    }
  }
}

Further reading

Learn about last-writer-wins user ID spoofing: user-id-spoofing .