Remember when we all used to commit secrets to source code repositories? In the bygone software engineering paleolithic era, public cloud key management services didn’t exist and neither did OSS solutions like Vault. Managing sensitive credentials has gotten much better over the years, and there’s very little reason to ever even store passwords or tokens in plaintext files any more.
Somewhere in-between the Big, Burly Public Cloud solutions and “encrypt files in your repo with
gpg” there exists a pocket of utility for some very focused tooling.
Doppler is interesting because it solves the “where to put my secrets” problem and fleshes out the space a little bit more to fully encompass “managing potentially-sensitive key/values” at the product level.
I’m going to go in blind and see what it’s all about, and how Doppler gels with the types of tooling and development patterns that someone like myself is accustomed to.
Disclosure: Doppler contacted me about an opportunity to take their product for a spin, so I’m being compensated for writing up this experience. However, their developer relations team is honorable and “write up your experience” is the only direction they gave, so there won’t be any astroturfing here. Everything you’ll read is solely from my own mouth! Err, fingers.
- First Steps
- The Illustrative Case(s)
I don’t need to repeat the instructions provided by Doppler, but there are some interesting traits to note.
Over at Installing the Doppler CLI there’s plenty of packaging options, and I was frankly surprised that the CLI tool is present in nixpkgs!
I just sort of assume that vendor packaging will overlook esoteric ecosystems like the cadre of NixOS users that I’m a part of, but
doppler is present in nixpkgs and even at the latest stable version.
If you’re a nix user and on flakes, you can invoke the following to run
doppler right now:
$ nix run 'github:NixOS/nixpkgs/nixpkgs-unstable#pkgs.doppler' -- --version v3.38.0
Kudos to whoever is in charge of packaging and distribution here;
nixpkgs is sort of the “last mile” when it comes to package repositories so the effort is appreciated.
Nothing excessively exciting here either. Again, I would consider myself an edge case user (developing on a Linux/NixOS desktop) but the signup process and coupling my account to the CLI was smooth. A few beats later and we’re ready to go (I used GitHub OAuth).
The Illustrative Case(s)
The value proposition for Doppler is to easily manage environment variables at scale. I find that the best way for me to digest new tools is to start using them, so let’s give this a shot with some examples.
The simplest case of “I need a secret” is probably shell commands that expect secrets from the environment.
We’re familiar with typical environment variables like
Assuming that we aren’t going whole-hog yet with a skeleton project and git repository, can we use
doppler to arbitrarily inject env vars into commands when we need them?
Doppler relies on organizing values within Projects, so let’s try making a general-purpose one:
$ doppler projects create shells ┌────────┬────────┬─────────────┬──────────────────────────┐ │ ID │ NAME │ DESCRIPTION │ CREATED AT │ ├────────┼────────┼─────────────┼──────────────────────────┤ │ shells │ shells │ │ 2022-03-22T22:08:16.069Z │ └────────┴────────┴─────────────┴──────────────────────────┘
Look at that ASCII table. Lovely. What’s in there?
$ doppler --project shells --config dev secrets ┌─────────────────────┬────────┐ │ NAME │ VALUE │ ├─────────────────────┼────────┤ │ DOPPLER_CONFIG │ dev │ │ DOPPLER_ENVIRONMENT │ dev │ │ DOPPLER_PROJECT │ shells │ └─────────────────────┴────────┘
Ah, a lot of informational environment variables. Cool.
Note: In addition to organizing environment variables by project, Doppler further segments these values into a “config”, (i.e., development, production, staging). We’ll use the default “development” config for now.
One of the all-time most common uses that I’ve had personally is interacting with AWS over the command line.
The overly-verbose environment variables
AWS_SECRET_ACCESS_KEY are the default environment variables that
aws will search out in order to authenticate against the AWS API.
Let’s give it a shot!
Fortunately the CLI docs are helpful enough to the extent that I don’t need to leave the terminal:
$ doppler secrets set --help Set the value of one or more secrets. There are several methods for setting secrets: 1) stdin (recommended) $ echo -e 'multiline\nvalue' | doppler secrets set CERT 2) interactive stdin (recommended) $ doppler secrets set CERT multiline value <snip>
Neato. Currently these values are set in my shell, so let’s get them into Doppler first.
$ echo $AWS_ACCESS_KEY_ID | doppler --project shells --config dev secrets set AWS_ACCESS_KEY_ID ┌───────────────────┬──────────────────────┐ │ NAME │ VALUE │ ├───────────────────┼──────────────────────┤ │ AWS_ACCESS_KEY_ID │ AKIA.....<snip>..... │ └───────────────────┴──────────────────────┘ $ echo $AWS_SECRET_ACCESS_KEY | doppler --project shells --config dev secrets set AWS_SECRET_ACCESS_KEY ┌───────────────────────┬──────────────────────────────────────────┐ │ NAME │ VALUE │ ├───────────────────────┼──────────────────────────────────────────┤ │ AWS_SECRET_ACCESS_KEY │ .................<snip>................. │ └───────────────────────┴──────────────────────────────────────────┘
All right. Let’s peek at the “shells” project in the Doppler dashboard because I’m curious about what that looks like now:
Everything seems right!
To ensure I’m working in a naked shell, we can either open a new terminal or unset the old variables, either way (with
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY).
Now to try injecting these values at runtime.
First, a failing test - to ensure that
awscli really can’t find credentials that I need.
$ aws s3 ls s3://blog-tjll-net Unable to locate credentials. You can configure credentials by running "aws configure".
Sackcloth and ashes; no AWS access. As expected.
“Wrapping” commands in
doppler run appears to be a first-class citizen, so let’s try this out:
$ doppler --project shells --config dev run -- aws s3 ls s3://blog-tjll-net | tail -n3 2015-02-15 01:13:39 286 humans.txt 2022-02-04 12:02:08 8548 index.html 2015-02-15 01:13:39 107 robots.txt
I store the blog in S3, so this these files aren’t secret, but interacting with them over the S3 API is a privileged operation.
It certainly seems like the
doppler run use case isn’t hard to get running with.
I mentioned at the beginning of this post that Doppler seems to sit between “DIY in your own encrypted repository” and “go all-in with something like a KMS service”.
At least in my modern development workflow, I never commit sensitive information directly to disk, and values like API keys are either encrypted or fetched on the fly when needed.
This seems to fill that niche fairly well, and as a bonus, the Doppler docs even support
direnv as a first-class citizen, which is hugely convenient - I lean on
direnv heavily for this exact sort of thing, so we can even codify the
doppler run semantics (we need to use the
download positional argument) in a top-level
.envrc so the secrets are fetched transparently.
$ cat .envrc export $(doppler --project shells --config dev secrets download --no-file --format env-no-quotes)
And the environment gets populated. Again, a common workflow that I’m glad is supported.
That covers what injecting arbitrary values looks like, but I’m also curious about the more fully-fledged, project-oriented, multiple-environment scenarios that accompany typical software development and deployment. Whether this takes the form of a program that hits an API or consumes some sort of token, we’re likely to need some sort of method to slip in these dynamically-fetched values.
As an illustrative case, let’s build a toy application that relies on an API that requires a token. Full speed ahead!
For this micro-project we’ll use python’s flask library to make an application that tells us about the weather. It’s boring! Which is a good thing, in this case. We’re concerned with getting familiar with how Doppler works and not necessarily the myriad other bits and pieces of the python ecosystem.
Let’s call our project
cloudler because it blends The Cloud, weather, and Doppler in a particularly obnoxious way:
$ mkdir cloudler $ cd !$ $ git init
If you’re in a typical development environment, I’ll assume that you can get your necessary prerequisites installed. If you’re like me and use nix heavily, here’s some files I got started with:
$ cat flake.nix
$ cat .envrc
$ cat setup.py
$ git add flake.nix .envrc setup.py $ direnv allow $ which python /nix/store/nrx2jhajrgxplg8x4j3mwm9hhc8z5m0g-devshell-dir/bin/python
In case you’re not a nix devotee: these files set up a sandboxed shell in
cloudler/ that installs python with the libraries I need,
httpie for ad-hoc API testing, and the
doppler CLI tool as well, so we’re guaranteed to have everything we need.
It also defines the build steps to generate a Docker container, which we’ll use as our deployment artifact later.
If you’re playing along at home and don’t use nix, you can achieve similar results with a simple
Let’s get cookin’!
To begin, let’s first interact directly with the API we’re interested in. I signed up for an account with OpenWeatherMap and snagged an API key. Create a project in Doppler:
$ doppler projects create cloudler
Now that we’re working in a proper project, we can configure
$ doppler setup ? Select a project: cloudler ? Select a config: dev ┌─────────┬──────────┬────────────────────────────┐ │ NAME │ VALUE │ SCOPE │ ├─────────┼──────────┼────────────────────────────┤ │ config │ dev │ /home/tylerjl/src/cloudler │ │ project │ cloudler │ /home/tylerjl/src/cloudler │ └─────────┴──────────┴────────────────────────────┘
Works as expected! Time to store our API key. Crucial to this workflow, I’m not persisting my API key any more spots than it needs to be to reduce risk of exposure. I copied my key out of the welcome email from OpenWeatherMap and into my clipboard and proceeded to:
$ doppler secrets set API_KEY
…and follow the prompts.
Once our secret is saved, it’ll be available when we ask for it.
Let’s use a simple
httpie command coupled with the secret we want to retrieve weather data:
$ http \ http://api.openweathermap.org/data/2.5/weather \ q==Boise \ appid==$(doppler secrets get --plain API_KEY) <big blob of json>
I get a nice snapshot of the current weather. In a more readable form;
$ http \ http://api.openweathermap.org/data/2.5/weather \ q==Boise \ appid==$(doppler secrets get --plain API_KEY) \ | jq '.weather.main' "Clear"
We have an API endpoint, an API key, and now need to integrate the dynamic secret injection into a real application.
For a simple weather reporting application, we’ll create a very basic web app with Flask. Consider:
This returns a very simple plaintext response with the weather in a given city.
Per the python Doppler docs, there isn’t (currently) a native integration/library for python, so this application assumes that it’ll yank the credentials out of the environment.
We’ll get hit with a
HTTP 401 if we send requests without an
appid parameter so we better include it!
Fortunately, that ends up looking pretty simple in practice:
$ doppler run -- flask run * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
How about sending requests to OpenWeatherMap with our API key?
$ http :5000/Boise | cowsay _______________________ < The weather is Clear! > ----------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
That’s great, and it works as expected.
Per the previous paragraphs, we can actually make the
doppler invocation automatic and transparent by leveraging
$ cat .envrc use flake export $(doppler secrets download --no-file --format env-no-quotes) $ direnv allow $ flask run ...
Now we don’t even need to think about importing variables!
This experimentation has been all well and good, but I’m equally interested in the deployment story with Doppler. Retrieving secrets as an actual human isn’t complicated in terms of trust, but if I want to grant a service account somewhere the ability to grab secrets from the environment, that’s a different matter. Morever, we can investigate differing behavior between development and production.
flyctl present, so let’s bootstrap a quick project on Fly:
$ flyctl launch
I’ll answer the questions to create an app called
cloudler in the
fly.toml needs a few edits, so it ends up thusly:
This next part is crucial: our docker image is wrapping the python application within
doppler run, so we need to inject the Doppler service token into the application process.
We can do that cleanly:
flyctl secrets set DOPPLER_TOKEN=$(doppler configs tokens create fly-io --plain)
I’m particularly proud of the fact that we never even see the token first-hand; it just gets generated by Doppler and seamlessly slipped into
flyctl without any intermediate steps.
Ready to launch?
flake.nix, we’ll build and load the container image, then ask
flyctl to roll it out:
$ nix build '.#container' $ docker load < result $ flyctl deploy
Is that all there is to it?
Wait a beat for
flyctl deploy to resolve and then try the generally-available endpoint.
$ http https://cloudler.fly.dev/Adelaide | cowsay ________________________ < The weather is Clouds! > ------------------------ \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
That is all there is to it!
Our API key is being dynamically populated via
doppler run inside the running container on Fly.
All is right with the world.
From everything I’ve seen, the
doppler run wrapper is pretty transparent and there’s no interruption in log flow from
I’ve experienced issues in the past with other methods that buffer either
stderr so that’s an important consideration using this approach.
Judging from the Doppler documentation, there’s a fair number of features that I haven’t kicked the tires on. In particular, “branch configs” seem interesting from the perspective of juggling many variables that might otherwise become tangled in a maze of different environments. That and some public cloud integrations (such as for AWS) are almost certainly a few layers of abstraction above just populating the environment variables for a running process, though that operational method is more than sufficient for lots of applications. Injecting an environment variable should work anywhere regardless of your platform of choice.
My take-away opinion is that there’s certainly some good niches this fills. If we start from the baseline precaution of “commit encrypted files” to a repository there are a number of undesirable tradeoffs (forward secrecy, plaintext values on-disk following decryption, and so on). I’ve typically operated one level up by running my own personal Vault deployment, but if we liken Vault to a operational primitive like a programming language’s implementation of a data structure, then Doppler is the higher level abstraction that gels a little more smoothly with the needs of a software development team - native support for different environments, cloud integrations, and handling the intricacies about how principals are authenticated against the secret store.
So there you have it. It does what is says on the box, is widely available, and works as expected in my most common workflows; namely, shell tools and running containers in production. I appreciate the folks at Doppler for funding this experiment and offer them a virtual high-five to a positive experience and promoting the “never expose secrets” mindset with their tooling.