NixOS in The Wild

NixOS on a T420
NixOS on a T420

Nix and NixOS are two technologies that my eyes have been on for a few years. Nix is the independent package manager that allows for Close to 99% reproducible. and immutable package managment and NixOS is a Linux distribution built from the ground up using Nix.

Keeping my Alpine, Arch, Debian, and CentOS installations around is a must. of my personal infrastructure now resides on NixOS including the server that hosts this website. CentOS, the previous Linux distribution running this server has done Shadowing this NixOS server just in case things blow up.

shell
14:39:28 up 502 days, 15:30,  1 user,  load average: 0.00, 0.01, 0.05

Let’s discuss the Disclaimer: These pain points are fixed in later versions. This post (which is more like a note to myself) tracks nixpkgs 20.03 commits. points, use cases, and general application of NixOS in the real world for both desktops and servers but before that — here’s my experience with the package repository and documentation search discovery.

The Package Repository

Nix Packages Collection
Nix Packages Collection

The nix Package Repository or nixpkgs contains every package derivation. My one pain point is that GitHub chokes on this repository hard. Searching the right term over 4,000+ issues can intermittently For this reason issues and pull requests are not viewed from within GitHub. The interface is just too slow. the page. This is not common but happens often enough to notice. In my experience, GitHub issue discovery is better achieved using a search engine that supports site queries.

text
site:https://github.com/NixOS/nixpkgs/issues <keyword>

Online Documentation and Search Discovery

In my experience, the online documentation has poor search discoverability. Nix’s This post probably makes it worse, but perhaps the title is tricky enough to fool search engines. search discoverability is the result of generating the majority of the documentation as a single HTML page. These pages are impeccable under the hood with anchor id links on each section and subsection, but search Searching
for phases on Bing Searching for phases on Bing are not smart enough to scope out this context and link directly to the anchor id fragments on the search engine results page (SERP). Most search queries send you to the top of the nix package manual.

Searching for phases on Google
Searching for phases on Google

Local Documentation and Troubleshooting

NixOS provides a lot of documentation out of the box that is specific to every installation. Users familiar with man will run man configuration.nix to see Simple and fast. config Use nixos-option if you know what you are looking for. available for a system. This means that the local configuration.nix documentation is more accurate than searching the online NixOS options index.

NixOS configuration specification
NixOS configuration specification

In fact, the entire package repository (nixpkgs) resides on every installation as a channel. In the above manpage nixpkgs/nixos/modules/config/appstream.nix declares the logic behind appstream.enable. View this file by listing the channel name as root and navigating to the per user channel source tree.

shell
$ sudo nix-channel --list
nixos https://releases.nixos.org/nixos/20.03/nixos-20.03.2351.f8248ab6d9e

$ tree -L 1 /nix/var/nix/profiles/per-user/root/channels/nixos
|__ COPYING
|__ default.nix
|__ doc
|__ flake.nix
|__ lib
|__ maintainers
|__ nixos
|__ nixpkgs -> .
|__ pkgs
|__ programs.sqlite
|__ README.md
|__ svn-revision

$ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
"20.03.2351.f8248ab6d9e"

This is good to know even if you don’t care about the source purely as a reminder that not every system has the same state or exposes the same configuration options.

User Autologin

Most of my desktop machines do not use login managers. The auto login settings of NixOS are login manager dependent. Apparently it’s trivial to override the autovt@tty1 service and force auto login for any user without having to install a login manager. Set restartIfChanged to false to avoid restarting your desktop.

nix
{ pkgs, ... }:

{
  systemd.services."autovt@tty1" = {
    after = [ "systemd-logind.service" ];
    restartIfChanged = false;
    serviceConfig = {
      Type = "simple";
      ExecStart = "${pkgs.utillinux}/sbin/agetty --autologin ${username} --noclear %I $TERM";
      Restart = "always";
    };
  };
}

The above will not work in future versions, as it is simplified to the option services.getty.autologinUser. Be careful though, enabling this option auto logs into every getty and the first activation restarts the currently running session.

nix
{
  services.getty.autologinUser = "username";
}

Updating NixOS

Updating NixOS is tricky business. The short of it is that generally the system wide channel is the repository used to source updates. Most systems have one channel called nixos or nixpkgs under the user root. Every user inherits the root user’s channel unless they have set their own per user channel.

  1. Using root list the channel with nix-channel --list
  2. Override or add a channel with nix-channel --add
  3. Update the channel with nix-channel --update
  4. Upgrade every per user environment with nix-env --upgrade
  5. That’s rather cumbersome manually (but for good reason). My preference is to sync the channel declaratively with nixops and use a per user declarative configuration with nix-env. the system with nixos-rebuild switch --upgrade
shell
# nix-channel --list
nixos https://nixos.org/channels/nixos-20.03
# nix-channel --add 'https://nixos.org/channels/nixos-20.03' nixos
# nix-channel --update
# nix-env --upgrade
# nixos-rebuild switch --upgrade

If you trust the stability, you can use system.autoUpgrade.enable and system.autoUpgrade.channel, but those are scary options for my use case.

Shellcheck and systemd

Every language, system, or framework leads you down a certain path. NixOS coaxes you down the path of writing lots of shell scripts with systemd. Save yourself the pain — set the shell to a stricter mode and run shellcheck upon service execution.

nix
{ pkgs, ... }:

{
  systemd.services.my-service = {
    description = "My service";
    wantedBy = [ "multi-user.target" ];
    path = [ pkgs.shellcheck ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1
      # Code goes here...
    '';
  };
}

LibreOffice and Spell Checking

Spell checking is a commonly used feature of LibreOffice. On my NixOS 20.03 machines, LibreOffice spell checking doesn’t seem to work. Add the missing dictionary package links and expose them using DICPATH. Log out and back in and LibreOffice should find the spell checking modules.

nix
{ pkgs, ... }:

{
  environment.systemPackages = with pkgs; [
    hunspell
    hunspellDicts.en_US-large
    hyphen
  ];

  environment.pathsToLink = [ "/share/hunspell" "/share/myspell" "/share/hyphen" ];

  environment.variables.DICPATH = "/run/current-system/sw/share/hunspell:/run/current-system/sw/share/hyphen";
}

Hashless Git Fetching

Use the function builtins.fetchGit to fetch the HEAD of a repository branch directly without stipulating a hash. The builtin function fetchGit will fetch any git repository at evaluation time — allowing for automation, ssh-agent integration, and other niceties.

nix
{ stdenv }:

let url = "https://github.com/koalaman/shellcheck"; in

stdenv.mkDerivation rec {

  pname = "shellcheck";
  version = "master";

  src = builtins.fetchGit { inherit url; ref = "refs/heads/master"; };

  dontBuild = true;

  installPhase = ''
    runHook preInstall
    mkdir $out
    # Code goes here...
    runHook postInstall
  '';
}

Package Tracking and systemd

In a continuous integration and deployment (CI/CD) environment you can set a systemd service to restart on package changes. Use restartTriggers to track upstream package definitions.

nix
{ pkgs, ... }:

let package = pkgs.callPackage ./default.nix {}; in

{
  systemd.services.my-service = {
    description = "My service";
    wantedBy = [ "multi-user.target" ];
    restartTriggers = [ package ];
    path = [ pkgs.shellcheck ];
    script = ''
      set -euxo pipefail
      shellcheck "$0" || exit 1
      # Code goes here...
    '';
    serviceConfig = { RemainAfterExit = "yes"; };
  };
}

Internet Connectivity in the Nix Sandbox

Nix build environments prohibit internet connections outside the scope of defined functions like fetchgit or fetchurl. You can remove this restriction by setting the correct expected hash recursively before evaluation. This allows us to do “illegal” things like the following.

nix
{ stdenv, pkgs }:

let url = "https://thedroneely.com/git/thedroneely/thedroneely.com.git"; in

stdenv.mkDerivation rec {

  pname = "composer";
  version = "master";

  src = builtins.fetchGit { inherit url; ref = "refs/heads/master"; };

  buildInputs = [ pkgs.cacert pkgs.php74Packages.composer ];

  dontBuild = true;

  installPhase = ''
    runHook preInstall
    composer --no-cache install
    mkdir $out
    cp -r vendor $out/vendor
    runHook postInstall
  '';

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = "0zkqkbwz5vg4k95s83pl0kxvphav1wzmivs5b1kmwf101wnj1m4q";
}

Check out this neat blog for more interesting approaches and tips of this nature.

Kernel Patching for Kids

Applying kernel patches using fetchurl in NixOS is a trivial endeavour. You should manually patch, configure, and compile a kernel at least once to understand what’s happening.

nix
{
  boot.kernelPatches = [

    { name = "ck-5.6"; patch = (builtins.fetchurl {
      url = "http://ck.kolivas.org/patches/5.0/5.6/5.6-ck2/patch-5.6-ck2.xz";
      sha256 = "18rk9023b14x62n0ckbnms6ahq5yjramz7qfjagkaga95i8ha6b2"; });
    }

    { name = "uksm-5.6"; patch = (builtins.fetchurl {
      url= "https://raw.githubusercontent.com/dolohow/uksm/master/v5.x/uksm-5.6.patch";
      sha256 = "021sylwacamh8q26agcp0nmmw3ral2wl7bgibmi379irnvy0c37y"; });
    }

    { name = "userns-overlayfs"; patch = (builtins.fetchurl {
      url= "https://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/patch/?id=0c29f9eb00d76a0a99804d97b9e6aba5d0bf19b3";
      sha256 = "1j4ind31hgkjazbgfd64lpaiqps8hcsqkar4v6nvxrpysmkg9nfd"; });
    }

  ];
 }

Nginx and its Temporary Folders

The permissions and ownership on nginx’s temporary folders can change in peculiar circumstances. There is a chance that the user nginx becomes dissociated from its folders when disabling and re-enabling nginx using the option services.nginx.enable. The temp folders for nginx are in /var/spool/nginx on system version 20.03. Use systemd’s tmpfiles.d to ensure that these permissions always stay consistent. This is solved in version 20.09.

nix
{
  systemd.tmpfiles.rules = [
    "z /var/spool/nginx                  0700 nginx nginx -"
    "z /var/spool/nginx/client_body_temp 0700 nginx nginx -"
    "z /var/spool/nginx/fastcgi_temp     0700 nginx nginx -"
    "z /var/spool/nginx/logs             0700 nginx nginx -"
    "z /var/spool/nginx/proxy_temp       0700 nginx nginx -"
    "z /var/spool/nginx/scgi_temp        0700 nginx nginx -"
    "z /var/spool/nginx/uwsgi_temp       0700 nginx nginx -"
  ];
}

The Hash as Truth

A common mode of failure when working with Nix and NixOS is that That’s a good thing. happens when you try to apply a change. Nix looks at the changes of a package’s derivation hash to decide if something should be rebuilt and applied. This also means that nix knows the hashes of already built resources.

If you want to set a new sha256 and force a rebuild: don’t place a random dummy string or change one of the characters in the original sha256. Always use lib.fakeSha256 or the command nix-prefetch-url to fetch the new sha256 hash.

The reason is simple — if you happen to use an already known hash, then nix may download the correlated binaries from a cached location and leave you with an interesting debugging session. The sha256 attributes always take precedence — changing only the pname or version attributes in most cases will have no effect.

nix
{ lib, stdenv, fetchurl }:

stdenv.mkDerivation rec {

  pname = "puppeteer-docs";
  version = "latest";

  src = fetchurl {
    url = "https://raw.githubusercontent.com/puppeteer/puppeteer/main/docs/api.md";
    sha256 = lib.fakeSha256;
  };

  phases = [ "installPhase" ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out
    cp ${src} $out/api.md
    runHook postInstall
  '';

  meta = with lib; {
    description = "Puppeteer Documentation";
    homepage = "https://github.com/puppeteer/puppeteer/blob/main/docs/api.md";
  };
}

Nix Commands and Local Caching

Nix is slow and caches to improve speed. Local caching can sometimes interfere with automation tasks when using commands like nix-build or nixops. Use the --option argument and set the tarball-ttl to 0 to ensure you are always pulling fresh sources and repositories.

shell
$ nixops deploy -d deployment --option tarball-ttl 0
$ nix-build --option tarball-ttl 0

If you are running a lean system, make the time to live (TTL) settings permanent with nix.extraOptions in the system configuration.

nix
{
  nix.extraOptions = ''
    tarball-ttl = 0
    narinfo-cache-negative-ttl = 0
    narinfo-cache-positive-ttl = 0
  '';
}

Logs and Logrotate

On NixOS, it’s systemd The seething rage. the way down. The important logs for most services route to systemd’s journald. The defaults are okay, but logrotate is a crucial program in my stack.

The abstraction for logrotate is fine, and it’s trivial to mimic my preferred setup as it would be on any traditional Linux distribution. Enable logrotate with a default preset.

nix
{
  services.logrotate = {
    enable = true;
    config = ''
      compress
      create
      daily
      dateext
      delaycompress
      missingok
      notifempty
      rotate 31
    '';
  };
}

Create a per service setup using the config This config attribute is of type types.lines which basically means it will mix down or append all declarations into a single logrotate.conf. anywhere in your configuration, but preferably in the same file as the service in question (nginx).

nix
{
  services.logrotate.config = ''
    /var/spool/nginx/logs/*.log {
      create 644 nginx nginx
      postrotate
        systemctl reload nginx
      endscript
    }
  '';
}

Using the Force

Breaking the assumptions of abstractions and frameworks is a hobby of mine. Sometimes there are special situations where the abstractions get in the way of the desired configuration. Use lib.mkForce as a forcing action to It looks like mkForce is an alias of mkOverride 50 which sets the override priority of a configuration option. any configuration option and prevent other modules from interfering. The name service switch configuration is written to /etc/nsswitch.conf using environment.etc and lib.mkForce to prevent casualties.

nix
{ lib, ... }:

{
  environment.etc."nsswitch.conf".text = lib.mkForce ''
    rpc: files
    shadow: files
    ethers: files
    networks: files
    services: files
    netgroup: files
    publickey: files
    protocols: files
    group: files systemd
    passwd: files systemd
    hosts: files mymachines myhostname libvirt libvirt_guest mdns4_minimal resolve [!UNAVAIL=return] dns
  '';
}
14 September 2020 — Written
15 May 2021 — Updated
Thedro Neely — Creator
nixos-in-the-wild.md — Article

More Content

Openring

Web Ring

Comments

References

  1. https://thedroneely.com/git/
  2. https://thedroneely.com/
  3. https://thedroneely.com/posts/
  4. https://thedroneely.com/projects/
  5. https://thedroneely.com/about/
  6. https://thedroneely.com/contact/
  7. https://thedroneely.com/abstracts/
  8. https://ko-fi.com/thedroneely
  9. https://thedroneely.com/tags/nix/
  10. https://thedroneely.com/posts/nixos-in-the-wild/#isso-thread
  11. https://thedroneely.com/posts/rss.xml
  12. https://thedroneely.com/images/nixos-in-the-wild.png
  13. https://nixos.org/features.html
  14. https://r13y.com/
  15. https://www.centos.org/
  16. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-420f28d
  17. https://nixos.org/manual/nixos/unstable/release-notes.html
  18. https://status.nixos.org/
  19. https://github.com/NixOS/nixpkgs/commits/nixos-20.03
  20. https://thedroneely.com/posts/nixos-in-the-wild/#the-package-repository
  21. https://thedroneely.com/images/nixos-package-repository.png
  22. https://github.com/NixOS/nixpkgs
  23. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-04d8fd6
  24. https://thedroneely.com/posts/nixos-in-the-wild/#online-documentation-and-search-discovery
  25. https://thedroneely.com/images/nixos-bing-search.png
  26. https://nixos.org/nixpkgs/manual/
  27. https://thedroneely.com/images/nixos-google-search.png
  28. https://thedroneely.com/posts/nixos-in-the-wild/#local-documentation-and-troubleshooting
  29. https://search.nixos.org/options
  30. https://thedroneely.com/images/nixos-configuration-manual.png
  31. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-7ff3ea4
  32. https://thedroneely.com/posts/nixos-in-the-wild/#user-autologin
  33. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-1d16e08
  34. https://github.com/NixOS/nixpkgs/commit/8694e7de2593139ea46c3e6d1431d6803a2d5a8a
  35. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-ccd4d9b
  36. https://thedroneely.com/posts/nixos-in-the-wild/#updating-nixos
  37. https://www.thedroneely.com/posts/nixops-towards-the-final-frontier#nixops-and-nix-channel
  38. https://www.thedroneely.com/posts/declarative-user-package-management-in-nixos#declarative-package-management
  39. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-e62c3cc
  40. https://thedroneely.com/posts/nixos-in-the-wild/#shellcheck-and-systemd
  41. https://github.com/koalaman/shellcheck
  42. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-6aebd9b
  43. https://thedroneely.com/posts/nixos-in-the-wild/#libreoffice-and-spell-checking
  44. https://www.libreoffice.org/
  45. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-cdc259f
  46. https://thedroneely.com/posts/nixos-in-the-wild/#hashless-git-fetching
  47. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-733b128
  48. https://thedroneely.com/posts/nixos-in-the-wild/#package-tracking-and-systemd
  49. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-87c2d24
  50. https://thedroneely.com/posts/nixos-in-the-wild/#internet-connectivity-in-the-nix-sandbox
  51. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-648c4f3
  52. http://chriswarbo.net/projects/nixos/index.html
  53. https://thedroneely.com/posts/nixos-in-the-wild/#kernel-patching-for-kids
  54. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-a5514af
  55. https://thedroneely.com/posts/nixos-in-the-wild/#nginx-and-its-temporary-folders
  56. https://github.com/nginx/nginx
  57. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-9c34f3b
  58. https://thedroneely.com/posts/nixos-in-the-wild/#the-hash-as-truth
  59. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-8b48da2
  60. https://thedroneely.com/posts/nixos-in-the-wild/#nix-commands-and-local-caching
  61. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-835d556
  62. https://en.wikipedia.org/wiki/Time_to_live
  63. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-1dfbb0f
  64. https://thedroneely.com/posts/nixos-in-the-wild/#logs-and-logrotate
  65. https://github.com/systemd/systemd
  66. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-46ed117
  67. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-25cea25
  68. https://thedroneely.com/posts/nixos-in-the-wild/#using-the-force
  69. https://github.com/NixOS/nixpkgs/blob/1d4656225d4f1e93ea9801c72eb0b1b0bffa245d/lib/modules.nix#L653
  70. https://en.wikipedia.org/wiki/Name_Service_Switch
  71. https://thedroneely.com/posts/nixos-in-the-wild/#code-block-a14d84e
  72. https://www.thedroneely.com/posts/nixos-in-the-wild.md
  73. https://thedroneely.com/posts/a-few-abstracts/
  74. https://thedroneely.com/posts/a-few-linux-distributions/
  75. https://thedroneely.com/posts/improving-paperless-interface/
  76. https://git.sr.ht/~sircmpwn/openring
  77. https://drewdevault.com/2022/11/12/In-praise-of-Plan-9.html
  78. https://drewdevault.com/
  79. https://mxb.dev/blog/the-indieweb-for-everyone/
  80. https://mxb.dev/
  81. https://www.taniarascia.com/simplifying-drag-and-drop/
  82. https://www.taniarascia.com/
  83. https://thedroneely.com/posts/nixos-in-the-wild#isso-thread
  84. https://thedroneely.com/posts/nixos-in-the-wild#code-block-420f28d
  85. https://thedroneely.com/posts/nixos-in-the-wild#the-package-repository
  86. https://thedroneely.com/posts/nixos-in-the-wild#code-block-04d8fd6
  87. https://thedroneely.com/posts/nixos-in-the-wild#online-documentation-and-search-discovery
  88. https://thedroneely.com/posts/nixos-in-the-wild#local-documentation-and-troubleshooting
  89. https://thedroneely.com/posts/nixos-in-the-wild#code-block-7ff3ea4
  90. https://thedroneely.com/posts/nixos-in-the-wild#user-autologin
  91. https://thedroneely.com/posts/nixos-in-the-wild#code-block-1d16e08
  92. https://thedroneely.com/posts/nixos-in-the-wild#code-block-ccd4d9b
  93. https://thedroneely.com/posts/nixos-in-the-wild#updating-nixos
  94. https://thedroneely.com/posts/nixos-in-the-wild#code-block-e62c3cc
  95. https://thedroneely.com/posts/nixos-in-the-wild#shellcheck-and-systemd
  96. https://thedroneely.com/posts/nixos-in-the-wild#code-block-6aebd9b
  97. https://thedroneely.com/posts/nixos-in-the-wild#libreoffice-and-spell-checking
  98. https://thedroneely.com/posts/nixos-in-the-wild#code-block-cdc259f
  99. https://thedroneely.com/posts/nixos-in-the-wild#hashless-git-fetching
  100. https://thedroneely.com/posts/nixos-in-the-wild#code-block-733b128
  101. https://thedroneely.com/posts/nixos-in-the-wild#package-tracking-and-systemd
  102. https://thedroneely.com/posts/nixos-in-the-wild#code-block-87c2d24
  103. https://thedroneely.com/posts/nixos-in-the-wild#internet-connectivity-in-the-nix-sandbox
  104. https://thedroneely.com/posts/nixos-in-the-wild#code-block-648c4f3
  105. https://thedroneely.com/posts/nixos-in-the-wild#kernel-patching-for-kids
  106. https://thedroneely.com/posts/nixos-in-the-wild#code-block-a5514af
  107. https://thedroneely.com/posts/nixos-in-the-wild#nginx-and-its-temporary-folders
  108. https://thedroneely.com/posts/nixos-in-the-wild#code-block-9c34f3b
  109. https://thedroneely.com/posts/nixos-in-the-wild#the-hash-as-truth
  110. https://thedroneely.com/posts/nixos-in-the-wild#code-block-8b48da2
  111. https://thedroneely.com/posts/nixos-in-the-wild#nix-commands-and-local-caching
  112. https://thedroneely.com/posts/nixos-in-the-wild#code-block-835d556
  113. https://thedroneely.com/posts/nixos-in-the-wild#code-block-1dfbb0f
  114. https://thedroneely.com/posts/nixos-in-the-wild#logs-and-logrotate
  115. https://thedroneely.com/posts/nixos-in-the-wild#code-block-46ed117
  116. https://thedroneely.com/posts/nixos-in-the-wild#code-block-25cea25
  117. https://thedroneely.com/posts/nixos-in-the-wild#using-the-force
  118. https://thedroneely.com/posts/nixos-in-the-wild#code-block-a14d84e
  119. https://thedroneely.com/posts/making-web-pages/
  120. https://thedroneely.com/posts/extreme-ssh-hardening/
  121. https://thedroneely.com/posts/literate-programming/
  122. https://thedroneely.com/posts/declarative-user-package-management-in-nixos/
  123. https://thedroneely.com/posts/writing-strategy/
  124. https://thedroneely.com/posts/programming-sans-internet/
  125. https://drewdevault.com/2022/09/16/Open-source-matters.html
  126. https://mxb.dev/blog/make-free-stuff/
  127. https://thedroneely.com/sitemap.xml
  128. https://thedroneely.com/index.json
  129. https://thedroneely.com/resume/
  130. https://gitlab.com/tdro
  131. https://github.com/tdro
  132. https://codeberg.org/tdro
  133. https://thedroneely.com/analytics
  134. https://thedroneely.com/posts/nixos-in-the-wild#
  135. https://creativecommons.org/licenses/by-sa/2.0/
  136. https://thedroneely.com/git/thedroneely/thedroneely.com
  137. https://opensource.org/licenses/GPL-3.0
  138. https://www.thedroneely.com/
  139. https://thedroneely.com/posts/nixos-in-the-wild/#