Tyblog

If you’re like me, you: have to look up Dockerfile quirks each time you write one, never know whether a container you exec into will have bash, sh, or whatever other shell, and are never sure which container init is currently the best practice. I definitely still don’t know what the hell Moby is.

Overall, packaging applications into Docker container images isn’t very ergonomic, and the “repeatable” part isn’t all that reliable (have you ever run into outdated distribution repositories? It’s annoying). That said, container primitives are cool and useful - resource constraints like cgroups are nice, and the fundamental principle of shipping your entire runtime is definitely good for deployment consistency. Let’s mess around with other ways to build executable artifacts to toss into a container orchestration system or workload scheduler.

Also: this title is clickbait. I know you can break pretty much any build. This is just a way to make the process a little better.

The Runtime

I use Hashicorp Nomad in my homelab to schedule workloads because it’s uncomplicated and flexible. Nomad actually supports numerous types of task drivers aside from Docker - in addition to other containerized drivers like podman, Nomad can also natively run things like Java jars or qemu virtual machines.

Nomad can also sandbox simple executables with the exec driver. Coupled with the ability to fetch and extract artifacts, I like to pop programs into a minio instance somewhere and let Nomad clients retrieve their workloads, and it works pretty well.

Nomad supports all the typical bells and whistles you’d expect from an “orchestrator” like secrets management, environment variable injection, and more. Moving on.

The Executable

You could probably come up with some way to get a portable executable across to your Nomad node, but one of the biggest boons to packaging an application into a container image is that everything comes with it. Shared libraries, dependent executables, all of it. Of course, you do end up with abominations like this, and you know in your gut that every piece of this line feels like an annoying anti-pattern.

RUN apt-get update && \
    apt-get -y --no-install-recommends install curl \
        ca-certificates && \
    <curl something>
    apt-get purge -y curl \
        ca-certificates && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

The container primitives are awesome, but if I’m being honest, I really do not care about whatever wrapping it all in Docker gets me. I’m sure there are some capabilities a Docker runtime gets you, but a totally standalone executable that leans on basic container primitives to put some safety bars around it is what I actually need.

The Builder

You might stop reading here, and that’s okay, but give it a chance: my answer is nix. Nix has a reputation for being opaque and inscrutable, and that’s not wrong. But people still love to use it, and there’s a reason why people push through the pain.

This is broken into three parts: wrapping, my application, building my application, and packaging my application. We’ll use a dumb python webserver as an example. Assumptions: nix is working, and you have niv as well.

Nix Gonna Nix

We initialize a new project with niv. This pins our forthcoming nix operations to a specific revision of the nixpkgs package repository.

$ mkdir demo
$ cd demo
$ niv init

Next we define a shell.nix. We throw in nix-bundle for later.

{ sources ? import ./nix/sources.nix
, pkgs ? import sources.nixpkgs {}
}:
pkgs.mkShell {
  buildInputs = [
    pkgs.nix-bundle
    pkgs.python39
    pkgs.python39.pkgs.poetry
  ];
}

You can drop into the shell environment this defines with nix-shell, but I much prefer to use direnv for this. It’s what separates man from beast. In .envrc:

source_up
use nix

Beautiful. Now when you cd into the directory, nix sets up your environment for you. I use source_up because I have a general-purpose .envrc in $HOME.

$ direnv allow
<nix configures everything for you>

Once nix concludes, you’ll have a version-pinned copy of python 3.9 and poetry available. You can lean on nix to pull in python dependencies, but I find poetry (as a native python tool) a little easier to manage python dependencies with. Initialize poetry and install dependencies.

$ poetry init
$ poetry add flask

We have an eminently-reproducible python dev environment, and here’s the VC-backed $5M series A blockchain application we want to run:

$ mkdir demo
$ cat demo/__init__.py

from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
    return "I will require a $10M series B to write a blockchain-based pet taxi app."

def main():
    app.run()

if __name__ == '__main__':
    main()

This runs and is an incredible testament to our capabilities as a real full-stack developer:

$ poetry run python demo/__init__.py_
$ http --body :5000

I will require a $10M series B to write a blockchain-based pet taxi app.

At this point we could commit the files here and, as long as a colleague has nix and niv working, get a carbon copy of the relevant requirements to replicate our environment.

Builders Build

Okay: we know that nix and niv are pedantic about being Functional in the sense that we have expressed the inputs required to make our program run. Assuming we define the total set of necessary dependencies for a working application, nix should be able to take that output and capture it, right?

Two things make this possible. The first is poetry2nix. Because poetry itself pins specific revisions of dependencies that we’re relying on for our project, we can translate those specific, pinned python modules into something nix can work with.

Here’s what we put in default.nix:

{ sources ? import ./nix/sources.nix
, pkgs ? import sources.nixpkgs {}
}:

pkgs.poetry2nix.mkPoetryApplication {
  projectDir = ./.;
  python = pkgs.python39;
}

Again niv lets us pin nixpkgs to something predictable, and mkPoetryApplication is a function we can invoke that parses poetry files. Now drop a line into pyproject.toml to tell poetry how to run our project as a script.

[tool.poetry.scripts]
app = 'demo:main'

Building this derivation will kick out our script in a form that has all of its dependencies expressed in a “sandboxed” sort of way.

$ nix build

Behold, a nix-built python application:

$ ./result/bin/app
$ http --body :5000

I will require a $10M series B to write a blockchain-based pet taxi app.

Artifacts Art

Here’s where things get a little more bleeding edge. If you poke around result, you’ll see how it works: nix has setup its maze of symlinks and /nix/store files to define the total set of nuts and bolts to make the application run. It turns out that you can wrap all of that up into something deployable.

nix-bundle is a cool project that wraps up the output of a nix function into a self-contained executable. You can read the details on the GitHub page, but the tl;dr is that we can ask it to take our built application and bundle it into what you can sort of think of as a statically-built executable.

$ nix-bundle '(import ./default.nix {})' /bin/app

The first argument tells nix-bundle what to build - which is just our poetry2nix function - and the second is what it should run from the result of the build (basically, what to invoke inside of result if it were a chroot).

Run what you’ve built! Startup time isn’t instant, because the single-file artifact is a compressed archive, but it should run as a standalone program.

$ ./app
$ http --body :5000

I will require a $10M series B to write a blockchain-based pet taxi app.

We’ve built repeatable python artifact and bundled it into a single-file and self-contained executable. Very cool.

Deploy Boy

At this point we can use this executable however we choose if we want to deploy it somewhere. We’ve already digested the meat of this post, so here’s the extra dessert if you’re interested. I upload my artifact to the minio cluster in my homelab:

$ mc cp app lan/artifacts/app-0.1.0-x86_64

Now let’s write a lil’ Nomad job definition to run it:

job "app" {
  datacenters = ["lan"]
  region = "global"
  type = "service"
  task "app" {
    artifact {
      source = "https://my.local.domain/artifacts/app-0.1.0-x86_64"
      options {
        checksum = "sha256:deadbeefc0ffee"
      }
    }
    driver = "exec"
    env {
      PYTHONUNBUFFERED = "yes"
    }
    config {
      command = "./local/app-0.1.0-x86_64"
    }
    resources {
      cpu    = 500
      memory = 512
    }
  }
}

All done.

$ nomad run app.nomad

It’s a simplified example, but my slightly-more-fleshed out real-world deployment is running great.

Final Thoughts

I’ve covered these points already, but to summarize:

Nix has a tremendous amount of utility in a few different ways. A reproducible and consistent development environment. A way to build and package an application for consistent outputs. Perhaps the most important point is that this is generally language agnostic. My example was python, but there are lots of similar projects! Go? Use go2nix. Haskell? cabal2nix. Each variant turns an ordinary project into something that can leverage the utility of the Nix ecosystem. (nix-bundle is the example, here but are other cool services like cachix)

There are interesting options out there for building deployment artifacts. I doubt Docker images are going anywhere, but I know that I’ve been envious of go’s ability to kick out easily portable executables, and general-purpose wrapping like this is pretty nice.