Creating a Nix Cache in an S3 cloud storage

If you're building or distributing software with Nix or NixOS, setting up your own remote binary cache can seriously improve your workflow. By default, Nix uses the official cache at cache.nixos.org, which serves most of the prebuilt packages you'll install—for example, when you run a command like nix-shell -p hello. But when you're working with your own builds, you can store their results in a cloud bucket and share them across machines and with your team. This custom remote cache can help speed up your CI/CD pipeline and make deployments easier. In this post I'll walk you through the steps to create a binary cache stored in an S3-compatible object storage provider. I'll show you how to:

• Create and configure a cloud bucket
• Generate and use signing keys
• Upload Nix store paths to the bucket
• Pull them back down into a fresh environment

Depending on the size and time to build your project, this process can take anywhere from about ten minutes to an hour.

Getting started

In my case, I wanted to document how to provision a binary cache that could be used by the thebacknd. thebacknd is a small proof-of-concept program to spin up a virtual machine in a cloud platform such as DigitalOcean by simply typing a single command. It relies on a binary cache to store and distribute the system.

Let's look at what commands we need to use to accomplish this with two different object storages, one in DigitalOcean Spaces and the other in Backblaze B2. (Note that both use AWS S3-compatible commands, which means what I'm going to show you should work with AWS S3 itself, or MinIO, or really any cloud storage that's compatible with S3.)

Before we dive in, here are some points to consider:

1. I'll assume we're working on a local development machine that has Nix installed (either Nix or NixOS).
2. We'll create an S3-compatible bucket where we can store our cache.
3. We'll create encryption keys to sign the files in the local Nix store on our machine.
4. We'll push the signed files from our local Nix store to the S3 bucket.
5. We'll retrieve them to test them out.

Note that in thebacknd scenario mentioned above, the final retrieval is done on a new virtual machine where we retrieve a new NixOS toplevel and activate it.

Bucket creation and endpoints

Cloud object storage organizes your data into something called buckets. Think of a bucket like a big directory--or even its own hard drive--where you can store a huge amount of files. In this context, the files in the bucket are usually known as "objects." Most cloud providers give you generous limits, like 5 Terabytes of space per bucket, and they usually let you store as many objects as you want, no matter how big they are. Within the buckets you can have something similar to directories, just like on a hard drive. (Technically, each object has a "key" and that key can have paths in it, giving it the look of directories.) And if you want, you can grant public read access to the objects through a URL so they can be easily accessed and downloaded.

Typically you'll create a bucket manually through the cloud provider's web console or through a single CLI command, since it's only done once. If you're doing it through a command, you can use the s3cmd command-line tool in most cases. (Note that with Backblaze B2, you can use s3cmd for most work, except you cannot use it to create buckets. Also with B2, access tokens are associated with buckets, and the buckets need to exist before creating a token. This is less developer-friendly, but it also means that such access tokens can provide for more restricted access rights than on DigitalOcean Spaces. Also, at the time of writing, DigitalOcean has an experiment currently in beta that provides for restricted access rights.)

Let's go ahead and create a bucket. I'm going to use the web console at DigitalOcean. (If you prefer to use B2, you can access it here.) Or if you want to try the command prompt, I'll show you how a bit later in this post.

On DigitalOcean Spaces, I created a new bucket named "demo-store". (I'm using the AMS3 region, for DigitalOcean. For B2, there is no region to select--the region is bound to the account when it is created--and I'm keeping it private, with no default encryption and no object lock.)

After creation, we can view the endpoint used for programmatic access to the buckets. In my case these are demo-store.ams3.digitaloceanspaces.com for DigitalOcean and s3.eu-central-003.backblazeb2.com for B2. Note how the bucket name we choose appears as a prefix in the DigitalOcean case, but not in the B2 case. When we reference the endpoints below, mentioning or not the name of the bucket, and where in the s3:// URL scheme, will matter.

Access keys

For programmatic access to the buckets, in addition to the endpoints mentioned above, we also need credentials in the form of a pair access key id (public) and secret access key (private).

To create keys for DigitalOcean, in the API page of the web interface, there is a dedicated tab for Spaces.

For Backblaze, they can be created in the Application Keys page. The nice thing here is we can have a per-bucket access/secret key pair, and we can also create a read-only pair. We need to click the "Add a New Application Key" button, choose demo-store for the name, and we select to allow access to only the demo-store bucket as read-and-write. We can leave the other options empty.

s3cmd configuration

The standard way to interact with an S3 bucket from the command-line is to use the s3cmd tool. The s3cmd uses a configuration file located at ~/.s3cfg and will complain if it doesn't exist:

$ s3cmd mb s3://demo-store
ERROR: /home/thu/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

(mb stands for "Make bucket".)

We need to create the configuration file, but instead of using its default location, we explicitly pass the filename we want with the -c option. This allows us to have different configurations for both DigitalOcean and B2. The following commands demonstrate this, with the first command using a file for DigitalOcean, and the second command using a file for B2:

$ s3cmd --configure -c s3-config-do
$ s3cmd --configure -c s3-config-bb

When prompted, we don't enter the access key and the secret key, and we rely instead on environment variables provided by a .envrc file (see below](environment-variables)). I think the "region" is only important when using AWS S3 only and I keep "US".

For the S3 endpoint, in the case of DigitalOcean, I'm using ams3.digitaloceanspaces.com, and for the DNS-style template, I'm using %(bucket)s.ams3.digitaloceanspaces.com.

For Backblaze, we took note above of the endpoint s3.eu-central-003.backblazeb2.com and used it for both the endpoint and the DNS template.

For everything else, I use blank or the default value.

Environment variables

To populate environment variables used by s3cmd, I create a .envrc file to be used with the direnv tool. To use both DigitalOcean and Backblaze, I actually create two such files, in two different directories. It looks like this:

$ cat .envrc
export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=xxxx

You can confirm it works with by running one of the two following commands, and it shouldn't issue an error message:

$ s3cmd -c s3-config-do ls s3://demo-store
$ s3cmd -c s3-config-bb ls s3://demo-store

Bucket creation with s3cmd

If you want to use the command line to create the bucket in the DigitalOcean case, you can use the mb subcommand like so:

$ s3cmd -c s3-config-do mb s3://demo-store

Here are some more commands you can try out; the first lists the objects in the top directory of the bucket; the second uploads a file called README.md to the path called /some/path. The third one lists all the files recursively. And finally the fourth deletes the uploaded README.md file.

$ s3cmd -c s3-config-do ls s3://demo-store
$ s3cmd -c s3-config-do put README.md s3://demo-store/some/path/
$ s3cmd -c s3-config-do ls s3://demo-store --recursive
$ s3cmd -c s3-config-do del s3://demo-store/some/path/README.md

Signing keys

Our goal is to send a Nix build artifact located in our local Nix store into the bucket. Before doing so, we want to create signing keys to sign our store content (at least the part we want to push to the bucket), so that we can verify later when we pull that content that it was indeed created by us.

We can create a public/private key pair used to verify and sign store path with the following command:

$ nix-store --generate-binary-cache-key \
  demo-store \
  cache-priv-key.pem \
  cache-pub-key.pem

demo-store is the name of the key. There is no obligation to have it the same name as the bucket, but it helps to remember what it is used for.

The public key can be used with the --option substituters and --option trusted-public-keys options of the nix-xxx and nix xxx commands. The private key is used to sign store paths in the next section.

Signing and uploading to the binary cache

To make sure we're actually using our own cache reliably, we need to build something on our own (as opposed to, for example, something that already exists in cache.nixos.org). Otherwise we might get (and then push) the cache.nixos.org-1 signature instead.

The following command builds a file in the local Nix store, with "asdf" as its content.

$ nix-build -E 'with import <nixpkgs> {}; writeText "example.txt" "asdf"'

This creates a file—also called a build artifact—which Nix stores under a path like /nix/store/.... We can now sign this artifact using the private key generated earlier:

$ nix store sign --recursive --key-file cache-priv-key.pem $(readlink result)

Finally, we use the nix copy command to upload the signed artifact to our binary cache. This ensures the file and its signature are both available to others who want to reuse the build without rebuilding it themselves. Then we can use s3cmd ls --recursive to check that we indeed have some files in the bucket.

For the DigitalOcean case, note that we include the bucket name in the endpoint:

$ nix copy \
  --to "s3://cache?endpoint=demo-store.ams3.digitaloceanspaces.com" \
  $(readlink ../result)
$ s3cmd -c s3-config-do ls s3://demo-store --recursive

And for the Backblaze case, we don't:

$ nix copy \
  --to "s3://demo-store/cache?endpoint=s3.eu-central-003.backblazeb2.com" \
  $(readlink ../result)
$ s3cmd -c s3-config-bb ls s3://demo-store --recursive

If you're curious, you can list the files in the bucket and download the .narinfo one. You should see a line starting with Sig: demo-store: followed by the signature.

Note: it may happen that you forget or fail to properly sign a store path, then push it up. When this happens, delete it and re-upload it. Note that if the file is not signed, you'll see a message like this:

      error: cannot add path '/nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt'
      because it lacks a signature by a trusted key

Downloading from the binary cache

When you built the example.txt file above, a local Nix store path was produced. In my case it looked like this:

$ nix-build -E 'with import <nixpkgs> {}; writeText "example.txt" "asdf"'
  /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt

Before you can download it, you need to remove it from your local Nix store. Type the following as root (because of this issue):

$ nix store delete /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt \
  --ignore-liveness

Now let's make sure everything is working as expected. Now that we've deleted the local copy, we're simulating a fresh environment. So let's pull the artifact back down from the remote cache by using nix copy. Use either of these commands depending on which cloud you're using:

$ nix copy \
  --from "s3://cache?endpoint=demo-store.ams3.digitaloceanspaces.com" \
  --option trusted-public-keys $(cat ../cache-pub-key.pem) \
  /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt

In the Backblaze case:

$ nix copy \
        --from "s3://demo-store/cache?endpoint=s3.eu-central-003.backblazeb2.com" \
        --option trusted-public-keys $(cat ../cache-pub-key.pem) \
        /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt

This command tells Nix to fetch the specific file from the binary cache that's stored in the cloud. The --from option points to the cache, and the trusted-public-keys option ensures that the file's signature is verified using the public key.

Next, we can use nix store with a -r option to "realize" the path and make sure the artifact is available on the disk. (Note that this command requires root privileges.) This command essentially validates that the previous command worked:

$ nix-store -r /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt \
        --option substituters "s3://cache?endpoint=demo-store.ams3.digitaloceanspaces.com" \
        --option trusted-public-keys $(cat ../cache-pub-key.pem)

Note: This command only works when run as root. That's because the Nix daemon, not your user shell, handles the actual download. The daemon runs in its own environment, meaning it doesn't have access to the credentials stored in your environment variables. Instead, it makes use of a key pair that you generate earlier. You configure the daemon to use this key by setting the secret-key-files option in /etc/nix/nix.conf, pointing it to the private key file. After updating the config, restart the daemon so it picks up the change. With the key in place, the daemon can sign store paths as it uploads them. These signatures are stored in Nix's internal database (/nix/var/nix/db/db.sqlite) and become part of the .narinfo metadata used by the binary cache.

Conclusion

And there you have it. You've seen how to create and use a Nix binary cache hosted on a cloud bucket. Together we set up the keys, pushed signed store paths, and pulled them back down into a fresh environment, verifying that everything works end-to-end.

This setup pays off quickly. Whether you're deploying systems remotely, or sharing builds across machines, having your own remote cache means faster workflows and more predictable results. And because we used an S3-compatible provider, you can choose from several different cloud platforms.