IaaC guide with NixOS on EC2

I've been refining the way I do IaaC and immutable infrastructure for years. Before I switched to NixOS, I had enormous Ansible playbooks that configured everything imperatively.

My playbooks are much simpler now, with most of the configuration declaratively specified in configuration.nix. Here are the main tools in my current workflow:

• Packer (for building AMI images)
• Ansible (for handling credentials)
• NixOS (for setting up applications within the instance)
• Cloudformation (for deploying the AMI images)

The rest of this article is a walkthrough of how I do IaaC when I need to deploy something to AWS. I stay away from the serverless AWS services because it's harder to migrate an application away from those specialized services. The ability to easily self host is important to me, so I stick to EC2.

Step 1: repository structure

I'll use the example of this very website you're reading (www.fossable.org). First of all, the IaaC repo is always named according to the domain it configures:

fossable.org
├── ansible.cfg
├── group_vars
│   └── all
│       └── vault
├── hosts
│   └── www.fossable.org
│       ├── cloudformation.yml
│       ├── configuration.nix
│       ├── playbook.yml
│       └── www.pkr.hcl
└── inventory.yml

• group_vars/all/vault

This YAML file is encrypted in the repo and contains the secrets necessary to deploy the hosts.

ansible-vault edit group_vars/all/vault

• hosts/www.fossable.org/www.pkr.hcl

This configuration file tells packer how to build the image.

packer {
  required_plugins {
    amazon = {
      version = ">= 0.0.2"
      source  = "github.com/hashicorp/amazon"
    }
    ansible = {
      source  = "github.com/hashicorp/ansible"
      version = "~> 1"
    }
  }
}

data "amazon-ami" "nixos" {
  filters = {
    name = "nixos/24.11.*-aarch64-linux"
    root-device-type    = "ebs"
    virtualization-type = "hvm"
  }
  owners      = ["427812963091"]
  most_recent = true
  region      = "us-east-2"
}

source "amazon-ebs" "www" {
  ami_name      = "www.fossable.org-${formatdate("YYYY-MM-DD-hhmmss", timestamp())}"
  instance_type = "t4g.large"
  region        = "us-east-2"
  source_ami    = "${data.amazon-ami.nixos.id}"
  ssh_username  = "root"
}

build {
  sources = ["source.amazon-ebs.mail"]

  # Add a bootstrap configuration to get python
  provisioner "file" {
    destination = "/etc/nixos/configuration.nix"
    content = <<EOT
      { pkgs, ... }:

      {
        imports = [ <nixpkgs/nixos/modules/virtualisation/amazon-image.nix> ];
        ec2.hvm = true;
        environment.systemPackages = with pkgs;
          [ (python3.withPackages (ps: with ps; [ requests cryptography psutil ])) ];
      }
    EOT
  }

  provisioner "shell" {
    inline = ["nixos-rebuild switch"]
  }

  # Install final configuration
  provisioner "ansible" {
    playbook_file = "hosts/www.fossable.org/playbook.yml"
    inventory_file = "inventory.yml"
    use_proxy = false
    extra_arguments = ["--extra-vars", "ansible_host=${build.Host}", "--extra-vars", "ansible_user=root", "--vault-password-file", ".password", "--diff"]
  }

  # Cleanup
  provisioner "shell" {
    inline = ["nix-collect-garbage -d", "rm -rf /etc/ec2-metadata /etc/ssh/ssh_host_* /root/.ssh"]
  }
}

• hosts/www.fossable.org/configuration.nix

This file configures the instance with whatever application it needs to run.

{ pkgs, ... }:

{
  imports = [ <nixpkgs/nixos/modules/virtualisation/amazon-image.nix> ];
  ec2.hvm = true;

  # Don't replace configuration.nix with user data on boot
  systemd.services.amazon-init.enable = false;

  i18n.defaultLocale = "en_US.UTF-8";
  networking = { hostName = "www"; };
  networking.firewall.allowedTCPPorts = [ 80 443 ];

  nix.optimise.automatic = true;
  nix.gc = {
    automatic = true;
    options = "--delete-older-than 30d";
  };

  users = {
    mutableUsers = false;
    users = {
      root = {
        hashedPassword = "{{ hosts.www.users.root.password | password_hash }}";
      };
    };
  };

  # Never change!
  system.stateVersion = "24.11";
}

Step 2: deploy

Deployment can be done from a CI pipeline or from the command line. For example, to build a new image:

packer build hosts/www.fossable.org/www.pkr.hcl

And to deploy:

aws cloudformation create-stack --stack-name www-fossable-org --template-body file://$(realpath hosts/www.fossable.org/cloudformation.yml)

Or to update a running host:

ansible-playbook -i inventory.yml --ask-vault-pass hosts/www.fossable.org/playbook.yml