The DX Evolution: From local setup to the cloud

Since the inception of cloud developer environments, like GitHub Codespaces, I've been enamoured with the thought of never having to manage dependencies ever again.

Imagine the onboarding time saved by having a fully configured environment, ready for development, one URL away. What a dream! Unfortunately, for reasons like cost, not all companies are willing to adopt cloud dev environments. So what can we do to reduce onboarding time, without having to pay for it?

Enter: containers.

Over the years, I've seen a shift away from detailed, often times out of date, local setup steps in a README. Towards a more containerized setup using Docker containers and Docker compose. What used to take days or weeks getting your codebase setup, could now be finished in mere minutes.

But are containers enough?

Let’s explore the evolution of DX from local environment setup to cloud developer environments and all the steps in between.

The problem with local setup

There are a many reasons why local setup is great. Apps run faster on bare-metal. You can customize your dev environment exactly how you want. And there's less abstraction, meaning you have more insight into what dependencies or tooling you're using.

However, at a company with a large or quickly scaling engineering team, there are many pain points with local setup:

• Devs are on their own if they need to upgrade dependencies. Often running into the "it works on my machine" problem.

• You can't easily dispose of an environment. Meaning, switching between codebases with different environments can cause unintended side effects.

• As your team grows, you run into inconsistencies across developer environments. Such as machines with different tooling versions or even OS versions.

• Documentation or scripts that help set up a complicated codebase can become quickly out of date. Making local setup not repeatable.

There are options to solve some of the pain points above. Tools like Nix that can make your dev setup disposable and repeatable. But it has a steep learning curve, and added complexity.

So where do containers fit into our journey?

Are containers enough?

Using containers will give us repeatable and disposable environments. By containerizing our codebase, we can ensure that setup steps are the same across developer machines. And if we want to switch from one codebase to another. All we need to do is spin up new containers for that codebase.

With Docker compose, we explicitly define a set of dependencies that are required for a codebase to run. Meaning tasks that were difficult with local setup, like upgrading a DB version, become easy. Devs no longer need to manually run upgrade steps by themselves.

Coming back to our journey of DX evolution, containers are like discovering stone tools. They're usable, and much better than not having stone tools. But the experience still isn't great.

The problem with plain ol' containers is the abstraction mismatch.

For example, let's say we run our Rails app in a container. When we start the app, the logs tell us to run bundle install because it's missing gems. Unfortunately, that won't work outside the container. You have to remember to prepend the command with docker exec or docker compose run.

This issue might seem trivial, but it can catch out a developer, especially someone new on the team. And if they're not careful, they could waste hours going down the rabbit hole of “accidental” local setup.

One workaround is to start up a terminal inside a container. That way, we don't need to prepend our command with anything Docker specific. But this approach adds extra friction when developing. The container has none of your dev tooling setup. Fortunately, there's a better way.

Development containers to the rescue

A development container aka devcontainer is a container that comes fully configured for development. Meaning they come pre-installed with the tooling you're used to and configured with your dotfiles.

Under the hood, a devcontainer is defined by a specification, devcontainer.json. This specification gets read by tools like editors, or the devcontainer CLI. Which then handle the lifecycle of a devcontainer.

The tight editor integration with devcontainers is what makes them so powerful. VSCode has the tightest integration, but other editors are adding support for it too. Jetbrains has added it, and it’s on the roadmap for Zed, just to name a few.

We can imagine a devcontainer like a plain ol' container with added layers:

In blue, is the container configuration that we would have set up, moving from local setup to containers. Aka your Dockerfile, and Docker compose setup.

In yellow, are the devcontainer specific configuration:

• Devcontainer Features

• Editor Configuration

• Local dev customizations

Each layer adds some development capabilities to our container. Let's break these down.

Devcontainer features

From the Devcontainer Features docs:

Features are installed on top of a base image. The purpose is to quickly and easily add more tooling to your dev containers for development. Let's take a look at the Git feature. It runs an install.sh script on top of your base image, which installs the latest Git (by default). Nothing too fancy happening here.

The power of Features is that it keeps your Dockerfile lean, and reusable across environments outside of development. And you can modify Features without touching the Dockerfile.

Editor configuration

The decontainer.json allows us to set editor customizations. We can specify which extensions to install, and what settings to apply by default. This configuration ensures all devs using the devcontainer will have the same baseline editor experience.

Here's an example of a VSCode configuration, that will install the ruby-lsp, rubocop, and rails db schema VSCode extensions on the devcontainer by default. As well as setting the Ruby LSP formatter to “rubocop” in settings.

{
  "customizations": {
    "vscode": {
      "extensions": [
        "shopify.ruby-lsp",
        "rubocop.vscode-rubocop",
        "aki77.rails-db-schema"
      ],
      "settings": {
        "rubyLsp.formatter": "rubocop"
      }
    }
  }
}

Integrated Terminal customizations

The last layer of a devcontainer is customization of your terminal through dotfiles. This step is ultimately enforced by tooling. For example, both VSCode and the devcontainer CLI support dotfiles, but Jetbrains IDE doesn't have this ability yet (source).

In VSCode, dotfiles are installed by passing a repository URL into the dotfiles settings. Under "Dotfiles: Repository":

Now when we spin up a devcontainer, VSCode will clone the dotfiles repository to the target path, and run the install command. The result is an integrated terminal setup to a developers liking.

With these layers configured, we now have a development environment that combines the reproducibility of containers with the comfort of our local setup.

What's next?

There's still one drawback with devcontainers, and containers in general. If you have a large, resource hungry codebase, you need a powerful machine.

This drawback is what cloud dev environments aim to solve. They have all the benefits of devcontainers, and the added benefit of standardizing machine specs across your team. The tradeoffs being cost, and requiring internet connectivity.

In 2021, GitHub moved their engineering team to Codespaces. From the article, these quotes stuck out to me:

As teams grow and codebases become more complex, the ability to onboard developers quickly, becomes paramount. Cloud developer environments can be the answer to this challenge. They represent the next step in our DX journey—though likely not the final one, as our tools continue to evolve.

This newsletter is kicking off a bunch of content I have planned around devcontainers and cloud dev environments. Stay tuned for more! I’m curious to hear your thoughts, have you used devcontainers or cloud dev environments like Codespaces before?