Fedor Indutny's Blog

Resource Decryption On-the-Fly in Electron

My team at Signal constantly faces challenges that go beyond what one would expect while working on practically any other messaging application. Consider the following facts about Signal:

It'd take a long time to discuss all of the implications of everything mentioned above, and the innumerable innovations that my colleagues have created. In this post, however, we will concentrate on how attachments and other user data are stored on the Signal Desktop client.

Existing Code #

Based on feedback from the community, we recently introduced an important change to how dynamic resources are stored in Signal Desktop. Beginning in the Signal Desktop v7.18.0, attachments, avatars, stickers, and other media are all individually encrypted on disk with the encryption key securely stored in a local SQLCipher database.

Given that Signal Desktop is an Electron app, this presents a challenge: How does one migrate existing references that point directly to attachments on the file system (using tags like <img/>, <video/>, etc.) and transition to a storage system where the displayed files can no longer be loaded from disk without an explicit decryption step?

Custom Protocol Handler #

The breadth of the UI elements that would be affected by this change makes loading and decrypting stored data within the React components intractable. However, we identified a simpler path forward that leverages custom protocol handlers in Electron. Because each stored attachment is separately encrypted with a random key, the client only needs to generate a simple formatted URL:

attachment://v2/ab/abcde?size=<num>&key=<base64>

When Electron's main process receives a request with such a URL, it verifies the supplied params and uses stream decryption to send back the decrypted attachment via a Web API Response object and Node's Readable.toWeb() helper. This means that the UI layer only has to be modified at the stage where these URLs are generated, and individual components can remain practically unchanged!

Naturally, there are some quirks about how Electron handles requests for resources that support streaming (like video and audio files), and we had to implement a Range request handler that only returns the requested data chunks to the Renderer process instead of returning the whole file.

Originally, we created a Node.js stream for every request and then sliced the desired data out of it with a simple 11-line Transform stream:

let offset = 0;
const transform = new Transform({
  transform(data, _enc, callback) {
    if (offset + data.byteLength >= start && offset <= end) {
      this.push(data.subarray(Math.max(0, start - offset), end - offset));
    }

    offset += data.byteLength;
    callback();
  },
});

However, we quickly discovered that Chromium would issue a whopping 200 requests for sufficiently large files instead of the small handful of requests that we had seen for smaller files. This meant that the simple approach above wouldn't scale but we also noticed that most of these requests were cancelled after only reading a small amount of data, and their offsets were evenly distributed with infrequent overlaps. Knowing this enabled us to write a more nuanced npm module called "range-finder".

Dev Tools Screenshot showing lots of requests

In "range-finder," partially consumed streams are not immediately destroyed and are reused for subsequent offset-based reads. Thus if Chromium requests a stream starting from byte zero of the file, cancels it after 800kb, and then makes a separate request for data starting from the 1-megabyte mark, then we can just skip some data on the original stream and keep giving Chromium data without opening and decrypting everything again from scratch.

This approach reduces the number of created streams from ~200 to ~3, which is an improvement of more than 98%!

Here's the source code for the full protocol handler at the time of writing this post.

In the future, we can move to a system where we don't have to decrypt the whole file to get the desired chunk (which is usually around ~800kb in size), but more on this later!

Migration #

The exciting story described above wouldn't be complete without also mentioning the existing attachments that users who are updating from older versions of Signal Desktop will still have on their computer. Since the on-disk size of the data that needs to be migrated to the new storage format is potentially large, and users can also close the app at any time, the migration and re-encryption process can't block the app from launching or require everything to happen all at once. Fortunately, as a result of past migrations, Signal Desktop already has an existing system in place to migrate messages in smaller batches whenever the app is idle on the user's computer. This system has now been updated to include attachment re-encryption, and every file in the old format (v1) will be migrated to the new format (v2) automatically.

You might have noticed that the attachment URL above had a v2 in it. Attachments that haven't been migrated yet also have their own special URL format:

attachment://v1/ab/abcde

Similarly, files at these URLs need to enable streaming to the UI with the support of Range requests, and be able to handle all of the compatibility updates for v2 URLs too. Although every new attachment that is sent or received on the latest version of Signal Desktop will start using the new v2 format right away, the versioned URL scheme helps keep things organized during the short period of time where the UI has to display a mix of v1 and v2 attachments during the migration process.

Moving Forward #

Feel free to install our app to see all of the described above come into motion right in front of you!

If the technical details we discussed in this post sound exciting to you — come work with us!