Prerequisites

  • You know what a shell is and it’s difference with a terminal
  • You know your way around a terminal

Introduction

When you develop multiple software projects, you may encounter version conflicts between runtime or tools. Perhaps your Todo project requires Node 22 while your Calculator project requires Node 18 or your Weather app requires openapi-generator-cli v7 while your Quizz app requires openapi-generator-cli v5, how do you make sure that you have the correct runtime and tool version while in development?

Existing Solutions and Their Pitfalls

The problem I mentioned is commonly encountered and there already tools that solves it.

NVM

Node Version Manager or commonly known as NVM, is a utility script that manages Node versions and allows you to switch versions by just running 1 command. The problem with this tool is obvious: it only supports Node.

Asdf

The improved version of NVM in my opinion is asdf. Basically works the same like NVM, but supports more runtime. On top of that, it has a feature that switches runtime version automatically based project config. The problem with Asdf is that it’s supported tools is limited which may not cover your needs.

Asdf + External Tooling

Commonly, Asdf only installs the runtime and another program like npm installs the needed tools. You can install your tools as dev dependencies then execute it using npx command. Example, you can install vite as dev dependency then run npx vite to run it. There are just some problems with this: reproducibility and dependency conflicts.

Try to install in a fresh machine and run openapi-generator-cli using npx, it will not run becuase of missing Java JDK dependency. This setup is not reproducible as you need to install an external undecalred dependency!

You might be thinking “just document what JDK version is required and install it, it’s not that big of a deal!” that’s true, but soon you will have dependency conflicts. The v5 of openapi-generator-cli requires JDK 8 while v7 requires JDK 11. So, if you switch in between these tool versions, you will have to make sure to also switch the JDK. On top of that, you might have other programs that depends on these JDK versions. A huge headache and hassle!

Docker Containers

If you’re using Laravel, you might have encountered this pretty neat package called Laravel Sail. Sail allows you to manage isolated containers that you can use to develop your projects in. It solves our previous problem of limited runtime, tool availability and version conflicts. The main problem in this setup is the bloat and complexity it comes with. You’re gonna pull gigabytes of docker image and on top of that, it will also eat a chunk of your RAM. You also have to make some utility scripts to interact seamlessly with the container like what Sail does.

The Ideal Solution

This should be pretty obvious by now: The runtime and tools needed for your project should not be installed on your system; these should be installed (in isolation) in your project instead. This is what Asdf is basically going for and only hindered by the limited packages and complete reproducibility. While the Docker approach solves this, the complexity and bloat it brings is not worth it.

Our ideal solution must have the following features:

  • Able to declare in a config file the runtime and tools required to develop the project (like Asdf).
  • Automatically switch to the correct runtime and tool version when changing the directory to the project based the config file declared (like Asdf).
  • Isolated, so that there will be no conflicts in runtime, tool and dependency versions.
  • Installs the required dependencies like JDKs for our programs to run.
  • Must have a plethora of available runtime / tools / packages available (Like Docker Containers).

Luckily, there is something that will help us achieve that: Nix.

Nix Primer

Nix is a broad topic so I will just explain concepts that are needed for our declarative dev setup. You can think of Nix as a tool that allows us to create isolated terminal sessions or shells. The packages installed in nix shells are isolated, meaning they won’t be installed in your system and will only exist in those shell sessions. With Nix, you can create a shell that has Node 18 then instantiate another one that has Node 22 installed. As of writing this, it has over 120,000 avaiable package!

In the following section, I will gradually introduce you to Nix and demonstate to you it’s power. You can follow along with this if you want. This is not a comprehensive tutorial and is only meant to introduce Nix and it’s capabilities. If you already know what Nix is, feel free to skip this.

Installing

Follow the steps in the official site: https://nixos.org/download/

You may want the multi-user installation.

Enabling Flakes

Enable the flakes feature of Nix. Why? I’ll explain later. For now just put this in ~/.config/nix/nix.conf

experimental-features = nix-command flakes

Checking if it works

Run the following in your terminal:

nix run nixpkgs#hello

If it outputs Hello, world!, then it’s working good!

Your First Isolated Shell

Now that Nix is installed, time for us to make our feet wet! Try to run the following in your terminal:

cowsay "Hello World!"

You will probably receive something like cowsay: command not found. Now run the following:

nix-shell -p cowsay

This will create an instance of Nix shell with cowsay installed. Note that this will take time installing the first time you run it so be patient. Now re-run the command in the same shell:

cowsay "Hello World!"

It should produce something this:

 ______________
< Hello World! >
 --------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

As you can see, our current shell has the package cowsay installed. Now let’s try to exit out in this Nix shell by executing the following:

exit

After that try to execute the same command and you will now receive an error saying the command is not found again as we exited the nix-shell session.

cowsay "Hello World!"

cowsay: command not found

This is powerful as we can install other packages like the latest Node and it’s isolated in your nix-shell session!

Declarative Shells

Let’s make it declarative so that you won’t have to type -p cowsay. Create a file called shell.nix with the following content:

let
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz";
  pkgs = import nixpkgs { config = {}; overlays = []; };
in

pkgs.mkShellNoCC {
  packages = [
    pkgs.cowsay
    pkgs.hello
  ];
}

This is a language called Nix Language. For our purposes, you don’t need to learn it. Don’t worry if you can’t understand every single line for now, just pay attention on the packages block. The packages block accepts an array of packages it should install. In this config, it will install cowsay and hello. Now you only need to run the following in the same directory of shell.nix file:

nix-shell

You will have the same shell as before (with the new hello package installed).

Flakes

Nix flakes is very similar to Declarative shells but flakes produces a lock file upon installing packages to ensure reproducibility similar to package-lock.json file in Node projects.

If you use a normal shell.nix file, it might install 3.8.2 version of cowsay today then after a few months when you execute nix-shell again, it might install 3.8.3 as the package got updated.

Some notable difference compared to normal Nix Shells:

  • Uses flake.nix file instead of shell.nix.
  • Uses nix develop command instead of nix-shell.
  • Produces a lock file called flake.lock to ensure that we always get the same version of packages.
  • The flake.nix file must be staged or committed in a repo.
  • Different syntax when declaring a shell.

This is the same shell.nix file from before converted into a flake.

{
  description = "A flake that's used to develop this project";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs}:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
    in
    {
      devShells.default = pkgs.mkShell {
        buildInputs = [
            pkgs.cowsay
            pkgs.hello
        ];
      };
    };
}

To ensure that it will work on different systems like Mac or ARM devices, we can do something like this:

{
  description = "A flake that's used to develop this project";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
        in
        {
          devShells.default = pkgs.mkShell {
            buildInputs = [
                pkgs.cowsay
                pkgs.hello
            ];
          };
        }
      );
}

If you want to update the packages, just run nix flake update. We will explore it more in depth in the incoming sections.

Where to find packages

You might be wondering, how did I knew the cowsay and hello packages indeed exists? You can use their official package search website: NixOS Search.

Further Reading

I highly recommend reading nix.dev if you want to learn more about Nix. The previous sections were basically ripped off from this site.

My Usual Project Setup

I will detail here my workflow to make my project’s dev tool reproducible.

Workflow

  1. Go to your project folder.
  2. Make sure a git repo for the project is already initiated.
  3. Run nix flake init.
  4. Use my template.
  5. Add packages as needed.
  6. Commit the flake.

After that, you can now use the nix develop command to enter your isolated shell.

Template

This is the template that I use in my new projects where I need the latest version of the runtime or tool.

{
  description = "A flake that's used to develop this project";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
        in
        {
          devShells.default = pkgs.mkShell {
            buildInputs = [
                #packages here
            ];
          };
        }
      );
}

Version Pinning

What if the project requires a previous or specific version? In that case, we need to pin the version of our nix channel. For example, this side project of mine requires PHP 7.4.29 which is not available anymore in the nixos-unstable channel.

{
   description = "A flake that's used to develop this project";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/99fcf0ee74957231ff0471228e9a59f976a0266b";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
        in
        with pkgs;
        {
          devShells.default = pkgs.mkShell {
            buildInputs = [php74 nodejs];
          };
        }
      );
}

The interesting part is the inputs block:

  #...other code
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/99fcf0ee74957231ff0471228e9a59f976a0266b";
    flake-utils.url = "github:numtide/flake-utils";
  };
  #...the rest

As you can see, it uses a hash instead of nixos-unstable. This is how you install a specific version of Nix packages.

To know what hash to put:

  1. Head over to nixhub.io
  2. Search the package you want, in this example search php
  3. Click on the package php
  4. You will see a list of versions of that package, just look for the exact version you want.
  5. You will see the hash and name of that package in the Nixpkgs Reference column.

Multiple Package Version Pinning

You will notice that the nodejs is not the latest package there is, because the package channel is pinned into the php’s version. If you want the latest or previous versions, you should also pin the nodejs package like so:

{
   description = "A flake that's used to develop this project";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/99fcf0ee74957231ff0471228e9a59f976a0266b";
    nixpkgs-node.url = "github:nixos/nixpkgs/<your node's hash here>";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, nixpkgs-node, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
          pkgs-node = nixpkgs-node.legacyPackages.${system};
        in
        {
          devShells.default = pkgs.mkShell {
            buildInputs = [pkgs.php74 pkgs-node.nodejs];
          };
        }
      );
}

Automatic Loading of Environment

Up until this point, we still need to execute nix develop command everytime we want to enter our environment. We want something like Asdf where it loads the runtime and tools when we change the directory into the project’s folder and in order to achieve that, we will use direnv. Direnv basically hooks into your shell and if it detects a file called .envrc, it wil execute scripts or load env variables inside it.

Installation

  1. Install it via your package manager
  2. Hook it into your shell
  3. Install nix-direnv by executing the following:
nix profile install nixpkgs#nix-direnv
  1. Then add the following to ~/.config/direnv/direnvrc:
source $HOME/.nix-profile/share/nix-direnv/direnvrc

Now we’re ready to use it in our projects.

Project Integration

  1. Go to your project’s root directory
  2. Create a file called .envrc with the following contents:
use flake
  1. Execute the following in the project’s root directory: direnv allow
  2. Done!

Now try to go out of the directory then re-enter it. Your environment should be now automatically loaded! Do note that I recommend to ignore .envrc file from your git as this file might be different from other devs.

In the demo below, I have elixir 1.18.1 installed on my machine, while the flake in my project has 1.16.3. direnv will automatically switch to that version upon changing directory to ensure you’re using the correct version!

Recap

Just to recap, I use the following setup:

  1. Nix Flake to declare and reproduce my runtime and tools in an isolated shell.
  2. direnv to automatically load the shell whenever I change the directory into the project folder.

Did we achieve our goal of an ideal solution? Let’s check.

  1. Able to declare in a config file the runtime and tools required to develop the project (like Asdf).
  • Yes, we are able to create a config file (flake.nix) that contains all of our required runtime and tools for our project.
  1. Automatically switch to the correct runtime and tool version when changing the directory to the project based on the config file declared (like Asdf).
  • Yes, through direnv.
  1. Isolated, so that there will be no conflicts in runtime, tool and dependency versions.
  • Yes, nix-shells are isolated.
  1. Installs the required dependencies like JDKs for our programs to run.
  • Yes, Nix makes sure to grab all of the dependencies needed for our tools and installs it in the shell.
  1. Must have a plethora of available runtime / tools / packages available (Like Docker Containers).
  • Yes, as mentioned, it contains 120,000 packages as of writing this.

Drawbacks

In Software Engineering, there is no such thing as “perfect solution” only ideal solution with tradeoffs. That also applies to our current setup.

Learning Curve

Nix is very large and complex. Just look at the official tutorial. This might overwhelm you if you’re a beginner.

Overkill

Sometimes Asdf + External tooling is enough. Look at this flake, it only requires php 7.4 and node to run. If you have written this in Asdf, this will just be 2 lines of text in the config. If your project does not require tools that needs complex dependency, Nix approach is definitely an overkill.

Disk Usage

To ensure Nix shells are reproducible, it has a unique way to handle dependencies which I won’t dicuss here as it’s complex. Essentially, it will take more disk space if you install a package in Nix compared to a native install.

Windows Support

As of writing this, Nix only supports Linux, Mac and partially, Windows. For Nix to function in Windows, you have to run it in WSL.

Alternative

Many emerging solution tries to simplify this whole process with Nix. You can check this project called devenv.sh if you find writing your own flake complex. Personally, I want deep understanding of my tools and try to avoid the “complex made easy” solution.

Conclusion

Wether Nix is the “ideal” solution for you really depends on the projects you are working with.

If you just stay on a single tech stack like Node, you most likely won’t find that much value as their respective community already have a standard way in solving this problem. In this case, just use Asdf.

If you’re like me who likes to explore a lot of tech, languages and have a lot of side projects, it’s god-sent. It brings me the following benefits:

  • A unified way to do “dev environment” on every single language I explore. Every language community that I encountered with have a lot ways in dealing with this. The Node community heavily favors NVM and npx, Elixir community likes Asdf and hex, Rust community have their own rustup and cargo. Me, I just use Nix flakes on every project regardless of language to manage the runtime and tooling.
  • Confidence that I always use the correct runtime and tools when working on a project.
  • Isolated shells mean I don’t have to deal with dependency version conflicts.
  • Peace of mind that I won’t struggle with the availability of tools with the huge amount of packages Nix has.
  • I can just give the flake.nix and flake.lock file to another dev (or future me) and they will have the same environment as mine down to the dependencies.
  • Able to try new tools without worrying of messing up my previous installations.