NixOps: Towards The Final Frontier

NixOps Source Page
NixOps Source Page

NixOps (Nix Operations) is a program for deploying and provisioning multiple NixOS machines in a network or cloud environment. This write up is by no means an Consult your friendly NixOps documentation. guide but rather a commentary on my The current procedure of moving my entire external and personal infrastructure to NixOS. nixops approach. You can also check out krops, morph, and deploy-rs as community alternatives to nixops. You may also want to use the simpler nixos-rebuild command.

shell
nixos-rebuild switch \
  --target-host "nix@remote.host" \
  --build-host "localhost" \
  --no-build-nix

Bootstrapping Remote Machines

NixOps NixOps version 2.0 supports non-root deployments. ssh access to the root user, therefore you should bootstrap a remote machine using its configuration.nix. Power users will use NixOS generators to Provision a specific NixOS configuration and convert between the many (too many) virtualization and cloud vendor formats. up this process.

nix
{
  services.openssh = {
    enable = true;
    permitRootLogin = "prohibit-password";
    passwordAuthentication = false;
    challengeResponseAuthentication = false;
    extraConfig = "Compression no";
  };

  users.users.root = {
    openssh.authorizedKeys.keyFiles = [ ./ssh-key.pub ];
  };
}

The user’s authorized_keys is users.users.<name?> where <name?> is the actual user. set using the option users.users.<name?>.openssh.authorizedKeys.keyFiles. Do not use users.users.<name?>.openssh.authorizedKeys.keys to configure public keys as it replaces the authorized_keys that nixops sets on the remote machine.

Once nixops interfaces successfully with a remote machine it sets its public key, as well as a private NixOps was first introduced as Charon [pdf]. called id_charon_vpn in the user’s .ssh folder. The remote machines are now ready to accept connections from nixops.

Directory Structure

Let’s take a look at my directory Use a directory structure that fits your use case. before creating a deployment using nixops.

text
|__ configuration.nix
|__ deployments.nixops
|__ configuration
|   |__ heron.nix
|   |__ pigeon.nix
|__ hardware
|   |__ heron.nix
|   |__ pigeon.nix
|__ helpers
|   |__ extra-builtins.nix
|   |__ wrappers
|   	|__ vault
|__ keys
|   |__ thedroneely.com.nix
|__ mailers
|__ packages
|__ programs
|   |__ nix.nix
|__ servers
|__ users
|   |__ pigeon.nix
|   |__ root.nix
|   |__ thedro.nix
|__ virtualizers
|__ websites
    |__ thedroneely.com.nix

The deployment entry point is at configuration.nix. The machine specific configuration.nix and hardware.nix are in the configuration and hardware folders respectively. The deployment The state file is rigid, however options for multiple state backends are in the works. is tracked in deployments.nixops.

This directory contains helpers, program specific configurations, as well as files responsible for secrets management. For example — this website is realized from an entry point that imports and implements every dependency it needs to function.

nix
{ config, lib, pkgs, ... }:

{
  imports = [
    ../../keys/deployments/thedroneely.com.nix
    ../../servers/cgit/service.nix
    ../../servers/goaccess/service.nix
    ../../servers/isso/service.nix
    ../../servers/nginx/service.nix
    ../../servers/phpfpm/service.nix
    ../../servers/postgresql/service.nix
    ../../servers/rainloop/service.nix
  ];
}

NixOps Deployment and Commands

Populate the configuration entry point configuration.nix with a list of machines to provision in the deployment.

nix
{
  network.description = "nixos";
  network.enableRollback = true;

  heron = {
    imports = [ ./hardware/heron.nix ./configuration/heron.nix ];
    deployment.targetHost = "heron.local";
  };

  talon = {
    imports = [ ./hardware/talon.nix ./configuration/talon.nix ];
    deployment.targetHost = "talon.local";
  };

  tiger = {
    imports = [ ./hardware/tiger.nix ./configuration/tiger.nix ];
    deployment.targetHost = "tiger.local";
  };

  hound = {
    imports = [ ./hardware/hound.nix ./configuration/hound.nix ];
    deployment.targetHost = "hound.local";
  };

  pigeon = {
    imports = [ ./hardware/pigeon.nix ./configuration/pigeon.nix ];
    deployment.targetHost = "pigeon.local";
  };
}

This entry point declares five machines to be operated on — each with their own hardware and configuration specific assets under the network nixos. Rollbacks are enabled for the nixops operator to switch generations in the case of a mishap.

Make a deployment using create by referencing the entry point file path and The state file defaults to ~/.nixops/deployments.nixops setting the location of the deployment state file using --state.

shell
nixops create --deployment nixos --state deployments.nixops configuration.nix

You can also adjust the entry point file paths using modify. Below two files configuration.nix and extra.nix are used as entry points.

shell
nixops modify --deployment nixos --state deployments.nixops configuration.nix extra.nix

List all available The output of nixops list depends on the last nixops deploy run. configurations using the list argument.

shell
$ nixops list --state deployments.nixops
+--------------------------------------+-------+-------------+------------+------+
| UUID                                 | Name  | Description | # Machines | Type |
+--------------------------------------+-------+-------------+------------+------+
| f56d276b-af54-11ea-b171-02422b1a33b6 | nixos | nixos       |          5 | none |
+--------------------------------------+-------+-------------+------------+------+

nixops
deployment nixops deployment the deployment nixos on all machines listed in the entry point configuration using the deploy argument.

shell
nixops deploy --deployment nixos --state deployments.nixops

Run the deployment nixos but only on the machine with the cryptonym heron using --include.

shell
nixops deploy --deployment nixos --state deployments.nixops --include heron

Show the state for deployment nixos using the info argument. This command lists the status and current state of each machine.

shell
$ nixops info --deployment nixos --state deployments.nixops

Network name: nixos
Network UUID: f56d276b-af54-11ea-b171-02422b1a33b6
Network description: nixos
Nix expressions: /nixos/configuration.nix
Nix profile: /nix/var/nix/profiles/per-user/thedro/nixops/f56d276b-af54-11ea-b171-02422b1a33b6

+--------+-----------------+------+----------------------------------------------------+------------+
| Name   |      Status     | Type | Resource Id                                        | IP address |
+--------+-----------------+------+----------------------------------------------------+------------+
| heron  |  Up / Outdated  | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-heron  |            |
| hound  |  Up / Outdated  | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-hound  |            |
| pigeon | Up / Up-to-date | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-pigeon |            |
| talon  |  Up / Outdated  | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-talon  |            |
| tiger  | Up / Up-to-date | none | nixops-f56d276b-af54-11ea-b171-02422b1a33b6-tiger  |            |
+--------+-----------------+------+----------------------------------------------------+------------+

Examine the machines more closely with the check argument. This command provides extra information such as load averages and failed units.

shell
$ nixops check --deployment nixos --state deployments.nixops --include tiger pigeon
Machines state:
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
| Name   | Exists | Up  | Reachable | Disks OK | Load avg.      | Units                                  | Notes |
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+
| pigeon | Yes    | Yes | Yes       | N/A      | 0.16 0.07 0.04 | proc-sys-fs-binfmt_misc.mount [failed] |       |
| tiger  | Yes    | Yes | Yes       | N/A      | 0.11 0.09 0.03 | proc-sys-fs-binfmt_misc.mount [failed] |       |
+--------+--------+-----+-----------+----------+----------------+----------------------------------------+-------+

List the generations for deployment nixos. The deployment fleet is currently on generation 203.

shell
$ nixops list-generations --deployment nixos --state deployments.nixops
 200   2020-06-26 00:28:10
 201   2020-06-26 04:14:40
 202   2020-06-26 04:28:12
 203   2020-06-26 21:58:55   (current)

Rollback the deployment to a previous generation with the rollback argument. Use --include to target specific machines.

shell
nixops rollback 201 --deployment nixos --state deployments.nixops --include heron ferret

Delete a deployment with the delete argument but you must first remove the machine resources with destroy.

shell
nixops destroy --deployment nixos --state deployments.nixops --include heron
nixops delete --deployment nixos --state deployments.nixops

NixOps Secrets Management

The nix store at /nix/store is world readable. Secrets entered directly into any nix configuration will be available to all users on the system. Avoid placing secrets into a nix configuration directly — as it will leak to unprivileged users. NixOps provides a powerful secrets management system in the form of password files.

Using a password file provides indirection and guards against writing secrets directly into your nix files. NixOps writes deployment keys to /run/keys. Users must be a part of the key group to access deployment keys on the system.

NixOps can set the permissions, owners, and groups of each key. In my case, the Use any preferred program as a secrets storage. come from a Vault server wrapper defined in builtins.extraBuiltins. This blog post gives an excellent rundown on this technique, and this wiki compares the different secret management approaches. A sample of the nix secrets configuration for this website is shown The secrets snippet in this example is verbose, but can be made very succinct.

nix
{ config, ... }:

let user = "thedroneely"; in

{
  deployment.keys = {

    thedroneely_cockpit_api_token = {
      user = user;
      text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "cockpit_api_token"}";
    };

    thedroneely_pgsql_database_password = {
      user = user; group = "postgres"; permissions = "0640";
      text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "pgsql_database_password"}";
    };

    thedroneely_ssh_known_hosts = {
      inherit user;
      text = "${builtins.extraBuiltins.vault "thedroneely/thedroneely.com" "ssh_known_hosts"}";
    };
  };
}

Enable extraBuiltins by linking directly to the nix-plugins package on the nix operator host. Allow users in @wheel to become trusted users and wield this ultimate power.

nix
{ pkgs, config, ... }:

{
  nix.extraOptions = ''
    trusted-users = root @wheel
    plugin-files = ${pkgs.nix-plugins}/lib/nix/plugins/libnix-extra-builtins.so
  '';
}

Create helpers/extra-builtins.nix and extend the builtins with custom functions. The vault function below accepts a field and a path as arguments to an external wrapper.

nix
{ exec, ... }:

{
  vault = path: field: exec [ ./wrappers/vault field path ];
}

The wrapper returns the secret result from vault as a string. NixOps is now aware of keys available from a vault server. This is done at evaluation time. Environment variables are available to wrappers called from extraBuiltins.

shell
#!/bin/sh -eu
printf '"' && vault kv get -field "$1" "$2" && printf '"'

Load the extraBuiltins with any nixops command by These commands can be shortened by adding the extra options to nix.conf. the --option flag with the file path. Vault keys are called using builtins.extraBuiltins.vault.

shell
nixops reboot --deployment nixos --state deployments.nixops --include pigeon --option extra-builtins-file $PWD/nixos/helpers/extra-builtins.nix

The keys in /run/keys are ephemeral and never touch storage. Do Executing a reboot with nixops on the same nixops host won’t end well if secrets are set. reboot the system through any other means except nixops. Your keys will be lost and any dependent service will Keyless service failures can be avoided with some systemd voodoo. In my case, missing keys from an upstream password manager are replaced with random diced passwords. Use nixops to remotely reboot a machine and its keys will be seeded back into position. Wonderful.

NixOps reboot seeding secrets
NixOps reboot seeding secrets

Upload deployment keys immediately without waiting on a deploy by using the send-keys argument.

shell
nixops send-keys --deployment nixos --state deployments.nixops --include pigeon

NixOps Pinning

NixOps depends on the nix channel set on the host. This is a very interesting Executing nixops on a machine with an uncertain channel is an assured gotcha. target. Shoot it and lock the host’s channel by prefixing or Currently using a nix shell until more of the code is read. exporting a defined NIX_PATH to your nixops commands.

shell
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/2b417708c282d84316366f4125b00b29c49df10f.tar.gz

Check the system’s channel version by running nixos-version or nix-instantiate if you prefer to use nix-channel directly.

shell
nix-instantiate --eval --expr '(import <nixpkgs> {}).lib.version'

NixOps and Nix Channel

Syncing the channel on the host and remote with nix-channel works out To avoid silent drift when working with throw away nix-env or nix-shell environments. for my use case. Note that you can use nix flake (manual) to completely replace the channel mechanism. The channel list is at releases.nixos.org and status at status.nixos.org. In my case, a “lock” file called versions.nix is created containing all sources of truth.

nix
{
  # Channel Status:  https://status.nixos.org
  # Channel List:    https://releases.nixos.org/?prefix=nixos

  pkgs = import <nixpkgs> { };

  "20.03" = { channel = "https://releases.nixos.org/nixos/20.03/nixos-20.03.3061.360e2af4f87";   };
  "20.09" = { channel = "https://releases.nixos.org/nixos/20.09/nixos-20.09.2468.c6b23ba64ae";   };
  "21.05" = { channel = "https://releases.nixos.org/nixos/21.05/nixos-21.05.1486.2a96414d7e3";   };
  "21.11" = { channel = "https://releases.nixos.org/nixos/21.11/nixos-21.11.336020.2128d0aa28e"; };
  "22.05" = { channel = "https://releases.nixos.org/nixos/22.05/nixos-22.05.998.d17a56d90ec";    };

  unstable = import (builtins.fetchTarball {
    url = "https://releases.nixos.org/nixos/unstable/nixos-22.11pre386147.e0a42267f73/nixexprs.tar.xz";
    sha256 = "0y6q1j17lmhxh1pqi2jj6xr21pnmachra48336nnbcpnxizswjgg"; }) { };

  linux_5_6_10 = import (builtins.fetchTarball {
    url= "https://github.com/NixOS/nixpkgs/archive/b0e3df2f8437767667bd041bb336e9d62a97ee81.tar.gz";
    sha256 = "0d34k96l0gzsdpv14vnxdfslgk66gb0nsjz7qcqz1ykb0i7n3n07"; }) { };

  linux_5_7_7 = import (builtins.fetchTarball {
    url= "https://github.com/NixOS/nixpkgs/archive/f761c14fd2f198f64cc5483ebf9f83222f9214aa.tar.gz";
    sha256 = "09w0n8qvldanab1m6ik507nl48aszam2a5ii3z2fvk72s66zmry7"; }) { };

  linux_5_10_13 = import (builtins.fetchTarball {
    url= "https://github.com/NixOS/nixpkgs/archive/75d4d5fe851a.tar.gz";
    sha256 = "0ks99va26jsq1mdr1mk9p9r75zvj6ghlmqkdcf4yak0dwr7j48a6"; }) { };
}

A systemd service is created using restartTriggers to lazily watch versions.nix for attribute changes. The system will sync up as This allows easier operation of nixos-rebuild if nixops is not available or fails.

nix
{ config, pkgs, ... }:

let channel = (import ../versions.nix)."${config.system.stateVersion}".channel; in

{
  systemd.services.nix-channel-update = {
    description = "Update Nix Channels";
    wantedBy = [ "multi-user.target" ];
    after = [ "network.target" ];
    restartTriggers = [ channel ];
    path = [ pkgs.shellcheck pkgs.nix ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1

      nix-channel --add "${channel}" nixos
      nix-channel --update
    '';
    serviceConfig = { RemainAfterExit = "yes"; };
  };
}

The option config.system.stateVersion is the system’s canonical state version — query the current and default value with the command nixos-option system.stateVersion.

29 June 2020 — Written
10 November 2021 — Updated
Thedro Neely — Creator
nixops-towards-the-final-frontier.md — Article

More Content

Openring

Web Ring

Comments

References

  1. https://thedroneely.com/git/
  2. https://thedroneely.com/
  3. https://thedroneely.com/posts/
  4. https://thedroneely.com/projects/
  5. https://thedroneely.com/about/
  6. https://thedroneely.com/contact/
  7. https://thedroneely.com/abstracts/
  8. https://ko-fi.com/thedroneely
  9. https://thedroneely.com/tags/nix/
  10. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#isso-thread
  11. https://thedroneely.com/posts/rss.xml
  12. https://thedroneely.com/images/nixops-towards-the-final-frontier.png
  13. https://github.com/NixOS/nixops
  14. https://nixos.org/
  15. https://github.com/NixOS/nixops/blob/3128b4ca31fe1e7930ce67a115eb131aa1b0b57d/doc/manual/overview.rst#overview
  16. https://cgit.krebsco.de/krops/about/
  17. https://github.com/DBCDK/morph
  18. https://github.com/serokell/deploy-rs/
  19. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#bootstrapping-remote-machines
  20. https://github.com/nix-community/nixos-generators#nixos-generators---one-config-multiple-formats
  21. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-4c4bad4
  22. https://edolstra.github.io/pubs/charon-releng2013-final.pdf
  23. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#directory-structure
  24. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-8d6164e
  25. https://github.com/NixOS/nixops/pull/1264
  26. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-29bd299
  27. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#nixops-deployment-and-commands
  28. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-ddcf6d1
  29. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-eaa371b
  30. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-d3fdbb8
  31. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-cccc1ff
  32. https://thedroneely.com/images/nixops-deploy.gif
  33. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-09995ec
  34. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-077e073
  35. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-b9611f7
  36. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-55ae169
  37. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-8fcc737
  38. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-1a93cd5
  39. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-b92804a
  40. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#nixops-secrets-management
  41. https://github.com/NixOS/nixops/blob/ddeadb5db5d717e84972b8b5efd30681bc836e20/nix/keys.nix#L191
  42. https://github.com/hashicorp/vault
  43. https://elvishjerricco.github.io/2018/06/24/secure-declarative-key-management.html
  44. https://nixos.wiki/wiki/Comparison_of_secret_managing_schemes
  45. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-00bf7a8
  46. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-d386916
  47. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-6a519f9
  48. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-b0052cd
  49. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-afaf3bc
  50. https://github.com/NixOS/nixops/blob/bc67d704f12a55f9708f66613698263a37964a36/doc/overview.rst#filekey-dependencynix-track-key-dependence-with-systemd
  51. https://en.wikipedia.org/wiki/Diceware
  52. https://thedroneely.com/images/nixops-reboot.gif
  53. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-5cd7039
  54. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#nixops-pinning
  55. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-10de4ed
  56. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-541fe2f
  57. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#nixops-and-nix-channel
  58. https://serokell.io/blog/practical-nix-flakes
  59. https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-flake.html#flake-format
  60. https://releases.nixos.org/?prefix=nixos
  61. https://status.nixos.org/
  62. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-1222c1a
  63. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#code-block-a40662a
  64. https://www.thedroneely.com/posts/nixops-towards-the-final-frontier.md
  65. https://thedroneely.com/posts/tweaking-the-bash-shell/
  66. https://thedroneely.com/posts/keeping-up-with-open-source/
  67. https://thedroneely.com/archives/tags/
  68. https://git.sr.ht/~sircmpwn/openring
  69. https://drewdevault.com/2022/11/12/In-praise-of-Plan-9.html
  70. https://drewdevault.com/
  71. https://mxb.dev/blog/the-indieweb-for-everyone/
  72. https://mxb.dev/
  73. https://www.taniarascia.com/simplifying-drag-and-drop/
  74. https://www.taniarascia.com/
  75. https://thedroneely.com/posts/nixops-towards-the-final-frontier#isso-thread
  76. https://thedroneely.com/posts/nixops-towards-the-final-frontier#bootstrapping-remote-machines
  77. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-4c4bad4
  78. https://thedroneely.com/posts/nixops-towards-the-final-frontier#directory-structure
  79. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-8d6164e
  80. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-29bd299
  81. https://thedroneely.com/posts/nixops-towards-the-final-frontier#nixops-deployment-and-commands
  82. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-ddcf6d1
  83. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-eaa371b
  84. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-d3fdbb8
  85. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-cccc1ff
  86. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-09995ec
  87. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-077e073
  88. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-b9611f7
  89. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-55ae169
  90. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-8fcc737
  91. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-1a93cd5
  92. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-b92804a
  93. https://thedroneely.com/posts/nixops-towards-the-final-frontier#nixops-secrets-management
  94. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-00bf7a8
  95. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-d386916
  96. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-6a519f9
  97. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-b0052cd
  98. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-afaf3bc
  99. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-5cd7039
  100. https://thedroneely.com/posts/nixops-towards-the-final-frontier#nixops-pinning
  101. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-10de4ed
  102. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-541fe2f
  103. https://thedroneely.com/posts/nixops-towards-the-final-frontier#nixops-and-nix-channel
  104. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-1222c1a
  105. https://thedroneely.com/posts/nixops-towards-the-final-frontier#code-block-a40662a
  106. https://thedroneely.com/archives/abstracts/
  107. https://thedroneely.com/abstracts/the-art-of-being-right/
  108. https://thedroneely.com/projects/news-aggregator/
  109. https://thedroneely.com/posts/nixos-pins-and-needles/
  110. https://thedroneely.com/posts/my-ts100-settings-and-configuration/
  111. https://thedroneely.com/posts/finding-that-one-percent/
  112. https://drewdevault.com/2022/09/16/Open-source-matters.html
  113. https://mxb.dev/blog/make-free-stuff/
  114. https://thedroneely.com/sitemap.xml
  115. https://thedroneely.com/index.json
  116. https://thedroneely.com/resume/
  117. https://gitlab.com/tdro
  118. https://github.com/tdro
  119. https://codeberg.org/tdro
  120. https://thedroneely.com/analytics
  121. https://thedroneely.com/posts/nixops-towards-the-final-frontier#
  122. https://creativecommons.org/licenses/by-sa/2.0/
  123. https://thedroneely.com/git/thedroneely/thedroneely.com
  124. https://opensource.org/licenses/GPL-3.0
  125. https://www.thedroneely.com/
  126. https://thedroneely.com/posts/nixops-towards-the-final-frontier/#