diff --git a/.github/README.md b/.github/README.md index 99247e8..99fa38d 100644 --- a/.github/README.md +++ b/.github/README.md @@ -109,7 +109,8 @@
Click here for a summary of my infrastructure -topology +topology + ### Programs diff --git a/.github/workflows/build-and-deploy.yml b/.github/workflows/build-and-deploy.yaml similarity index 100% rename from .github/workflows/build-and-deploy.yml rename to .github/workflows/build-and-deploy.yaml diff --git a/SwarselSystems.org b/SwarselSystems.org index 9b8b8f8..5902178 100644 --- a/SwarselSystems.org +++ b/SwarselSystems.org @@ -37,15 +37,20 @@ That is the reason why I keep this configuration as a literate one: so that I am For a beginner, I recommend to read this file like a book, from start to finish. I will try to explain concepts whenever they first come up, and will regularly link to [[#h:8ea35dcc-ef94-4c10-9112-8be8efd6f424][Appendix C: Explanations to nix functions and operators]] when more context is needed. For the first few times that I am using a new function, I will place such a link again. However, to keep the writing of this file manageable, I will generally only do this no more than three times. +This page offers some utilities to you: you can pin specific headings to the right "pinned" bar by hovering over the heading and clicking =[pin]=. If a section seems uninteresting to you, you can press the =↓= button to skip to the next one. And if you want to send a section to somebody else, you can click the =#= in order to copy its link to the clipboard. Your pinned headings will be saved locally, so you can continue reading in case you take a break. + ** Structure of this file :PROPERTIES: :CUSTOM_ID: h:bcc3ebbe-df8a-46bd-b42d-73aad6fc66e5 :END: -This file is structured as follows: +Now, I will outline how this document is structured. I have segmented this file into the following sections: - [[#h:a86fe971-f169-4052-aacf-15e0f267c6cd][Introduction (no code)]] - This is the block you are currently in. It holds no code that actually builds the system, it just outlines the general approach and explains the rough design mentality. For simply understanding the code in here, reading this should not be necessary (feel free to skip to [[#h:c7588c0d-2528-485d-b2df-04d6336428d7][flake.nix]]) + This is the block you are currently in. It holds no code that actually builds the system, it just outlines the general approach and explains the rough design mentality. For understanding the nix (or Emacs) code in here, reading this should not be necessary if you already know some nix (feel free to skip to [[#h:c7588c0d-2528-485d-b2df-04d6336428d7][flake.nix]]). Otherwise, I will also give a brief introduction to some nix terms here. + +- [[#h:10a6f80d-7e00-4db1-953a-811993c22cb0][An introduction to nix]] + Here I will give a coarse overview over some important concepts in the nix landscape. This is aimed at beginners in the field; others might want to skip this section as they are likely to not find much that will be worth their time. - [[#h:c7588c0d-2528-485d-b2df-04d6336428d7][flake.nix]] This block holds everything related to the heart of the nix side of the configuration - the =flake.nix= file. I am using [[https://github.com/hercules-ci/flake-parts][flake-parts]] to manage this flake, so different aspects of the configuration are handled by flake-part modules in different files. @@ -56,7 +61,7 @@ This file is structured as follows: - [[#h:ed4cd05c-0879-41c6-bc39-3f1246a96f04][Emacs]] This section defines my Emacs configuration. For a while, I considered to use rycee's =emacs-init= module ([[https://github.com/nix-community/nur-combined/blob/master/repos/rycee/hm-modules/emacs-init.nix]]) to manage my Emacs configuration; I have since come to the conclusion that this would be a bad idea: at the moment, even though it might seem as I am very bound to the configuration file that you are currently reading, if I ever decide to change how I run my system, I can simply take the generated =.nix= and =.el= files and put them wherever I need them. This file only simplifies that generation without putting further restrictions on my. If I were however to switch to =emacs-init= then I would be indeed to some level confined to the nix ecosystem with my Emacs configuration, as I would no longer have a valid =.org= file to manage it with, instead generating an =init.el= directly from nix code. I like to keep that level of freedom for potential future use. Also, you will notice there is no package system setup in this configuration. This is because packages are automatically handled on the NixOS side by parsing the generated =init.el= file for package installs. - My emacs is built using the emacs-overlay nix flake, which builds a bleeding edge emacs on wayland (pgtk) with utilities like treesitter support. By executing the below source block, the current build setting can be updated at any time, and you can see my most up-to-date build options (last updated: {{{revision-date}}}) +[[#h:fa377c74-36e5-4aea-bc39-4d3def4b67d1][nix build]] My emacs is built using the emacs-overlay nix flake, which builds a bleeding edge emacs on wayland (pgtk) with utilities like treesitter support. By executing the below source block, the current build setting can be updated at any time, and you can see my most up-to-date build options (last updated: {{{revision-date}}}) #+begin_src emacs-lisp :tangle no :exports both @@ -67,7 +72,7 @@ This file is structured as follows: #+RESULTS: : --prefix=/nix/store/lymgpfqr5dp1wc0khbcbhhjnxq8ccsy9-emacs-pgtk-20240521.0 --disable-build-details --with-modules --with-pgtk --with-compress-install --with-toolkit-scroll-bars --with-native-compilation --without-imagemagick --with-mailutils --without-small-ja-dic --with-tree-sitter --without-xinput2 --with-xwidgets --with-dbus --with-selinux -This file is not loaded by Emacs directly as the configuration (even though this would be possible) - instead, it generates two more files: +This file is not loaded by Emacs directly as the configuration (even though this would be possible - actually I did that in the past!) - instead, it generates two more files: - =early-init.el= This file handle startup optimization and sets up the basic frame that I will be working in. @@ -77,7 +82,7 @@ This file is not loaded by Emacs directly as the configuration (even though this By using the configuration offered by this file, the file you are reading right now (=SwarselSystems.org=) will not be freshly tangled on every file save, as this slows down emacs over time. However, when you clone this configuration yourself and have not yet activated it, you need to tangle the file yourself. This can be done using the general keybind =C-c C-v t= or my personal chord =C-SPC o t=. Alternatively, execute the following block: -#+begin_src emacs-lisp :tangle no :export both :results silent +#+begin_src emacs-lisp :tangle no :export code :results silent (org-babel-tangle) @@ -96,23 +101,29 @@ As such, this served to reduce code duplication in this file. The tangled files An example of using a noweb-ref block: +First we define the block: + #+begin_src nix-ts :tangle no :noweb-ref blockName enable = true; #+end_src which can then be used in a block like: -#+begin_src markdown :tangle no :noweb yes +#+begin_src markdown :tangle no :noweb no :results silent <> #+end_src -#+RESULTS: -: enable = true; +and is finally parsed as: -Note that noweb-reffed blocks will not be indented correctly. You will want to account for that when checking your nix flake with the formatter of your choice. Personally, I have solved this issue using the functions defined in [[#h:59d4306e-9b73-4b2c-b039-6a6518c357fc][org-mode: Upon-save actions (Auto-tangle, export to html, formatting)]]. Originally, I also automatically exported to html there, but it incurred a too high memory penalty which made Emacs become sluggish over time - instead I now build the website version whenever I push to GitHub. +#+begin_src markdown :tangle no :noweb yes :results silent + <> +#+end_src + + +Note that noweb-reffed blocks will not always be indented correctly. You will want to account for that when checking your nix flake with the formatter of your choice. Personally, I have solved this issue using the functions defined in [[#h:59d4306e-9b73-4b2c-b039-6a6518c357fc][org-mode: Upon-save actions (Auto-tangle, export to html, formatting)]]. Originally, I also automatically exported to html there, but it incurred a too high memory penalty which made Emacs become sluggish over time - instead I now build the website version whenever I push to GitHub. - [[#h:8fc9f66a-7412-4091-8dee-a06f897baf67][Appendix B: Supplementary Files]] - This section holds files that are not written in nix but are still referenced in the configuration in some way. This is mostly used for configuration of programs that have no native nix support, like tridactyl. Note that shell scripts are still defined under their respective entry in [[#h:64a5cc16-6b16-4802-b421-c67ccef853e1][Packages]]. Over time, the goal is to reduce this section to a minimum, but things like the aforementioned tridactyl might stay for a long time, until we have a stable interface to configure browser plugins. + This section holds files that are not written in nix but are still referenced in the configuration in some way. This is mostly used for configuration of programs that have no native nix support, like tridactyl, as well as "meta" files like the [[#h:bf3e6fc0-a95a-46d0-9305-0d1068b2f1ec][GitHub Readme]] or the [[#h:abed312c-5bf7-43f5-b0d0-43cce513ccb9][GitHub Workflow: Build-and-deploy]]. Note that shell scripts are still defined under their respective entry in [[#h:64a5cc16-6b16-4802-b421-c67ccef853e1][Packages]]. Over time, the goal is to reduce this section to a minimum, but things like the aforementioned tridactyl might stay for a long time, until we have a stable nix interface to configure browser plugins. - [[#h:8ea35dcc-ef94-4c10-9112-8be8efd6f424][Appendix C: Explanations to nix functions and operators]] When I started to learn about nix, I found that journey quite arduous; while I disagree with the general public in that the documentation is too sparse, I will say that, while it is very good, reading (and understanding!) it requires a certain level of existing nix knowledge that one will problably not have when starging out. Hence, the goal of this document is to explain common nix functions as they come up in this document (I thing I wrote this before :sweat:), in hopes that you will be able to understand most of the code. When a new function appears for the first time, I will try to link to an entry in the appendix. @@ -122,9 +133,11 @@ Note that noweb-reffed blocks will not be indented correctly. You will want to a :CUSTOM_ID: h:12880c64-229c-4063-9eea-387a97490676 :END: -The =.html= version of this page is normally built by a GitHub workflow; it can be generated manually by calling the chord =C-SPC o e=, or by executing the below block +I will also quickly talk about the webpage that (I hope) you are currently viewing works: -#+begin_src emacs-lisp :tangle no :export both :results silent +The =.html= version of this page is built by a GitHub workflow; it can also be generated manually by calling the chord =C-SPC o e=, or by executing the below block + +#+begin_src emacs-lisp :tangle no :export code :results silent (org-html-export-to-html) @@ -135,832 +148,25 @@ The web version is useful because it allows navigation using org-mode links, whi - Functionality to save links to certain headers in a "Pinned" menu that can be toggled - These links are locally saved -I add a javascript bit to the file in order to have a darkmode toggle when exporting to html. NOTE: What I am doing is defining an elisp multiline string which gives me heredoc capabilities for defining blocks that should be both exported to the final =.html= but should also be shown in the document itself (only putting a =#+[begin/end]_export= block would not show the block in the content of the page) which is then output as html and exported to both. +I add a javascript bit to the file in order to have a darkmode toggle when exporting to html (defined in [[#h:cf6a12cf-929b-4080-9945-e6f02afc7990][HTML Export: Darkmode toggle]]). NOTE: What I am doing is defining an elisp multiline string which gives me heredoc capabilities for defining blocks that should be both exported to the final =.html= but should also be shown in the document itself (only putting a =#+[begin/end]_export= block would not show the block in the content of the page) which is then output as html and exported to both. + +#+begin_src elisp :noweb yes :exports both :results html + " + <> + " +#+end_src + + +I also add this javascript to add header pinning functionality to the site, using the same trick as above (this is defined in [[#h:e5f900a0-9d68-4269-a663-53c52434c342][HTML Export: Docs QoL]]): #+begin_src elisp :noweb yes :exports both :results html " - " #+end_src -#+RESULTS: -#+begin_export html - - - -#+end_export - -I also add this javascript to add header pinning functionality to the site, using the same trick as above: - -#+begin_src elisp :noweb yes :exports both :results html - " - - " -#+end_src - -#+RESULTS: -#+begin_export html - - -#+end_export - ** TODO Structure of this flake :PROPERTIES: :CUSTOM_ID: h:2c5529ed-e6d9-44b6-b0d3-5bf96a6bed64 @@ -995,6 +201,18 @@ The structure of this flake as seen many revisions, however lately I have settle } #+end_src +It is also possible to pass it as a function: + + + #+begin_src nix-ts :tangle no + { config }: { + my_value = 2; + my_attrSet = { + enable = config.myconfig.enable; + }; + } + #+end_src + Using the mechanisms in [[#h:82b8ede2-02d8-4c43-8952-7200ebd4dc23][PII management]] (which in turn uses [[#h:87c7893e-e946-4fc0-8973-1ca27d15cf0e][extra-builtins]] and [[#h:315e6ef6-27d5-4cd8-85ff-053eabe60ddb][sops-decrypt-and-cache]]), these files are decrypted during evaluation time and stored under a persistent directory. As the name suggests, I am using these files to store personally identifiable information - these "secrets" are stored world-readable in the nix store. As such, this should not be used to store important secrets, but rather information that you would not like everyone on the internet to easily find in your git repo. Other than that, the =secrets= folder will also be used to store conventional (decryted at activation-time) sops-encrypted secrets in the standard =.yaml= / =.toml= / =.ini= formats. @@ -1139,110 +357,1165 @@ This is a comprehensive list of the services/components ran by my server machine |✉️ **Mail** | [simple-nixos-mailserver](https://github.com/Swarsel/.dotfiles/tree/main/modules/nixos/server/mailserver.nix) | #+end_src -** Manual steps when setting up a new machine +* An introduction to nix :PROPERTIES: -:CUSTOM_ID: h:ed34ee4d-31f9-4d27-bc6e-ba37ee502d5a +:CUSTOM_ID: h:10a6f80d-7e00-4db1-953a-811993c22cb0 :END: -In the [[#h:a86fe971-f169-4052-aacf-15e0f267c6cd][Introduction (no code)]], I mentioned that this is a nearly fully declarative config. In fact, most client configs are in one way or another not fully declarative. I use oneshotting systemd services + sentinel files for most such tasks (which makes them declarative!), but some of them I would rather perform manually once. This mainly concerns work related things. +This is where it gets interesting. -Whenever I encounter a configuration bit that needs manual steps, I use a [[#h:dae0c5bb-edb7-4fe4-ae31-9f8f064cc53c][Appendix A: Noweb-Ref blocks]] to tangle that bit of information into a central place (here). I discern between the following scenarios: - - =setup=: Used in a standard NixOs + home-manager deployment - - =worksetup=: Stuff to be done only on work machines - - =homemanageronlysetup=: Steps that are needed only on machines that are not running NixOs. +In this section, I want to give an overview over some important concepts needed when working with nix. -#+begin_src markdown :noweb yes :exports results :results html - These steps are required when setting up a normal NixOS host: +** Nix, NixOS, Nixpkgs, and Nix +:PROPERTIES: +:CUSTOM_ID: h:6d0a91d4-e478-45ae-82de-0997fa289e7a +:END: - <> +First off, when talking about nix, we need to differentiate between several things: - If the new machine is a work machine, these steps are additionally needed: +- The [[#h:b2222c7c-0bd5-44f8-aa69-17869d6913c0][nix language]] +- The [[#h:f4fa6a46-e016-48f8-9ead-608b739c706a][nix package manager]] +- The linux distribution [[#h:d62af55c-a7f3-47ba-b2a5-78cc60b03aef][NixOS]] +- the =nix-community= input [[#h:f4e89635-e006-4c41-a545-d4b56c7ac293][nixpkgs]] - <> +While these terms are all connected with each other, it is important to keep in mind that funamentally those are separate entities. Let us briefly talk about each one: - If the new machine is home-manager only, perform these steps: +*** nix language +:PROPERTIES: +:CUSTOM_ID: h:b2222c7c-0bd5-44f8-aa69-17869d6913c0 +:END: - <> +Fundamental to all of the other concepts named above is the =nix language=, a functional programming language (that feels like a design language for reasons we will get to). What follows is what you can read at [[https://hydra.nixos.org/build/318367945/download/1/manual/][the nix reference manual]] in detail, condensed to the most important stuff. + + +Here I will give a brief overview over the nix language. While a very deep topic, this aims to only build the essential understanding needed for dealing with the configuration I am using (this should be sufficient to then understand most other configurations online). For deepening your knowledge, you might want to check out the [[https://nix.dev/reference/nix-manual.html][nix reference manual]]. + +**** Derivations and the nix store +:PROPERTIES: +:CUSTOM_ID: h:45135360-96d4-4348-b6fb-e8bc4685611e +:END: + +The main purpose of the nix language is to build packages. These packages are built by functions that can be inaccurately described by something like this: + +~f: { source, compiler, dependencies, etc. } -> package~ + +This function along with all its inputs we call a =derivation=. What is important to realize is that such a derivation can itself depend on other derivations, and the resulting package does not necessarily need to be a callable program (a more correct name for the function result is =outputs=). When we as nix users define a package, we will usually use =pkgs.stdenv.mkDerivation= (ore one of its derived wrappers like =buildRustPackage=), which will run things like install phases and setup hooks for us. It is a wrapper around nix function ~derivation(name, system, builder, args=[],outputs=["out"])~, where =name= is the derivation name, =system= is the system architecture to build for, =builder= is the executable responsible for building the result, =args= are arbitrary arguments passed to =builder=, and =outputs=, which can be thought of a list of locations where we want to put different build artifacts that can be referenced separately when using the package. There are [[https://nix.dev/manual/nix/2.28/language/advanced-attributes][more attributes]], but these five are the most important and commonly used ones. + +Building such a package (in nix terms: "realising") from a derivation causes the package to be created in the =nix store=. This is a read-only filesystem that is usually located at =/nix/store=. When building a package, nix will place its content of each output at =/nix/store/---/=, where == is a unique identifier that changes whenever one of the inputs to the above function changes. That means that it is no problem to store different versions of the same file on one system, and each program can use the correct dependencies that it needs. + +A key observation that follows from this is that packages are declarative; when not changing the inputs, a derivation will always yield the same package with the same store path. + +We call the set of derivations (and, recursively, the derivations that those depend on) that a derivation depends on its =closure=. + +Usually when installing a package, we do not need all the outputs it provides: For example, the [[https://search.nixos.org/packages?channel=25.11&query=pcsclite][pcscliteWithPolkit package on nixpkgs]] that is used for configuring smart cards provides 5 outputs: =out=, =dev=, =doc=, =man=, and =lib=. By default, a NixOS system will install =out=, =man=, =info= and =doc= outputs (the latter three have respective =documentation..enable= NixOS options) as well as any outputs listed in =environment.extraOutputsToInstall= (by default an empty list). However, if we need to debug something, we might need the =dev= output of the package, which can then be installed by referencing =pcscliteWithPolkit.dev=. + +When reading package source code in =nixpkgs= for example, you will often come across =$out= and =$src=. The =$out= represents the root of that build output (=$src= are the build sources). You will often see that in build phases (it is there set as an environment variable). In other places, you will instead see =placeholder =, which will be replaced by the outputs future location in the nix store. + +**** Types in the nix language +:PROPERTIES: +:CUSTOM_ID: h:7bbb4e50-f5e9-4496-a0cd-d8eb34aa2514 +:END: + +The nix language supports the following types and how they look in the wild: +- null: =null= +- bool: =true= +- strings: ="text"= + - strings can also be defined as =multiline strings= + - Those will look for the lowest level of indent shared over all lines and strip it: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate " + '' + indent0 + indent2 + indent1 + '' + " +#+end_src +/(you can ignore the [[#h:95ebfd13-1f6b-427f-950d-e30c1ed6f9fa][swarsel-instantiate]]; it is just a wrapper around =nix-instantiate= that preloads nixpkgs. I will use this to show you the evaluation results of nix calls)/ + - note that tab characters will /not/ be stripped: +#+begin_src bash :tangle no :exports both + swarsel-instantiate " + '' + indentTab + indent1 + indent2 + '' + " #+end_src #+RESULTS: -#+begin_export html -These steps are required when setting up a normal NixOS host: +: \tindentTab\nindent1\n indent2\n -- setup yubikey (automatic yubikey enrollment is not yet supported by `disko`): - - `systemd-cryptenroll --fido2-device=auto /dev/` + - an URI string can assume the type of string without need for quoting (e.g. you can write =http://about.org= instead of ="http://about.org"=) + - Strings can be interpolated by using =${expression]= (read on for an example) + - =expression= can be any valid nix expression that is compatible with the enclosing type (here: string) + - As in many languages, you have =\n=, =\r=, and =\t= available + - in normal strings, you can escape stuff using =\= + - the following characters need to be escaped in normal strings: + - Backslash: =\= (escape using =\\=) + - Double quote: ="= (using =\"=) + - Opening of nix expression: =${= (using =\${=) + - in multiline strings, you escape stuff using one or two apostrophes ='= + - the following characters need to be escaped in multline strings: + - Two apostrophes are escaped with a single apostrophe: =''= (escape with ='''=) + - All other things are escaped using =''=: + - Dollar sigh: =$= (escape with =''$=) + - =\n=, =\r=, and =\t= (=''\n= and so on) + - =''\= is a catchall for all other escaping + - somewhat interestingly, double dollars =$$= (as used in Makefiles) never need to be escaped + - ints: 1 + - floats: 1.1 + - paths: =/home= + - a path can also be given as a relative path (e.g. =./.config=) + - attribute sets: ={ }= + - these hold name value pairs, e.g. ={ a = 3; }= + - a "chain" of attributes, separated by dots, is called an =attribute path=, e.g. =config.environment.systemPackages= + - in such a chain, all attributes but the last will be =attribute sets= + - =config.environment.systemPackages= is equivalent to ~config = { environment = { systemPackages = ; }; };~ + - lists: [ ] + - these hold values, e.g. =[ { a == 3; } ]=. In this example, the list holds a single value, that is, the attribute set ={ a = 3; }=. + - functions: =arg: body= + - =arg= can be any of these data types, including functions + - when =arg= is an attribute set, some special things apply: + - default values can be specified using =arg ? defaultValue=, e.g. ={ name ? "Default", age }: "${name} is ${age} years old"= will yield ="X is 20 years old"= when called using the attribute set ={ name = "X"; age = "20"; }=. Otherwise it will yield "Default is 20 years old" when called using ={ age = "20"; }=. When not passing =age= in this example, it will throw an error (also not that we had to pass =age= as a string as the value is not cast to a string automatically. Alternatively, we could have passed =age = 20;= and updated the function body to ="${name} is ${builtins.toString age} years old"=. But that is just a sidenote) + - the evaluator will error out if it is called with an argument that is not explicitely mentioned in the function definition + - this behaviour can be suppressed by adding an ellipsis =...= to the function arguments (={ name ? "Default", age, ... }:=) -If the new machine is a work machine, these steps are additionally needed: +Let's see this in action: -- setup the work VPN: - - using the laptop certificate `.pem` as User cert and private key (CA cert: none) - - vpn gateway is found in `nixosConfig.repo.secrets.local.work.vpnGateway` -- setup gpgsm for signing of mails using S/MIME: - - `gpgsm --import ~/Certificates/.p12` - - `gpgsm --import ~/Certificates/harica-root.pem` - - `gpgsm --import ~/Certificates/harica-intermediate.pem` - - `gpgsm --list-keys --with-validation "HARICA Client RSA Root CA 2021"` - - trust the certificate and set passphrase -- setup pizauth for microsoft mail sync (account names are possibly `uni` and `work`): - - `pizauth auth ` - - `pizauth dump > ~/.pizauth.state` +Calling with explicit values: -If the new machine is home-manager only, perform these steps: - -- (Optional) Install openssh-server -- Set hostname to the name specified in the home-manager configuration -- Install nix, either: - - (if upgrading existing nix) Install nix version matching with version that `nix-plugins` is compiled against: `nix-env --install --file '' cacert -I nixpkgs=channel:nixpkgs-unstable --attr nixVersions.nix_x_yy` - - (or installing nix freshly): - - Grab the link to the install script of the needed nix version from https://releases.nixos.org/?prefix=nix, e.g. https://releases.nixos.org/nix/nix-2.30.1/install - - `bash <(curl -L https://releases.nixos.org/nix/nix-x-yy-y/install) --daemon` -- add the following to /etc/nix/nix.conf to become a trusted user: `trusted-users = @wheel root swarsel` -- For the first build: - 1) Clone dotfile repo & change into it - 2) `nix --extra-experimental-features 'nix-command flakes' develop` - 3) `home-manager --extra-experimental-features 'nix-command flakes' switch --flake .#$(hostname) --show-trace` -#+end_export - -** TODO Current issues -:PROPERTIES: -:CUSTOM_ID: h:b562adaf-536c-4267-88a5-026d8a0cda61 -:END: - -Besides the manual steps outlined above, sometimes things break when I update this flake. The fix, for me, is most of the times one of these two: - - instead of the broken package, use the package from the latest stable nixpkgs release where the package is still functoning (this is why I pull all of these in as inputs) - - if the broken component is critical, I perform manual patches/overrides. - -In order to keep track of these changes, I gather them here in a similar style to what you saw in [[#h:ed34ee4d-31f9-4d27-bc6e-ba37ee502d5a][Manual steps when setting up a new machine]]. I simply prefix them with the date and check them after a while to see if things got better. TODO: this list is not comprehensive probably - -#+begin_src markdown :noweb yes :exports results :results html - Currently, these adaptions are made to the configuration to account for bugs in upstream repos: - - <> +#+begin_src bash :tangle no :exports both + swarsel-instantiate 'let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { name = "X"; age = "20"; }' #+end_src #+RESULTS: -#+begin_export html -Currently, these adaptions are made to the configuration to account for bugs in upstream repos: +: X is 20 years old + +Calling by using a default value: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate 'let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { age = "20"; }' +#+end_src + +#+RESULTS: +: Default is 20 years old + +Not passing =age= errors out: + +#+begin_src bash :tangle no :exports both :results output + swarsel-instantiate 'let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { }' +#+end_src + +#+RESULTS: +#+begin_example +error: + … from call site + at «string»:1:104: + 1| let lib = import ; in let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { } + | ^ + + error: function 'f' called without required argument 'age' + at «string»:1:44: + 1| let lib = import ; in let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { } + | ^ +[ Babel evaluation exited with code 1 ] +#+end_example + +Passing a superfluous =another= errors out: + +#+begin_src bash :tangle no :exports both :results output + swarsel-instantiate 'let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { age = "2"; another = "0"; }' +#+end_src + +#+RESULTS: +#+begin_example +error: + … from call site + at «string»:1:104: + 1| let lib = import ; in let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { age = "2"; another = "0"; } + | ^ + + error: function 'f' called with unexpected argument 'another' + at «string»:1:44: + 1| let lib = import ; in let f = {name ? "Default", age }: "${name} is ${age} years old"; in f { age = "2"; another = "0"; } + | ^ +[ Babel evaluation exited with code 1 ] +#+end_example + + +When adding the ellipsis =...= to the function definition, we can now pass =another=: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate 'let f = {name ? "Default", age, ... }: "${name} is ${age} years old"; in f { age = "2"; another = "0"; }' +#+end_src + +#+RESULTS: +: Default is 2 years old + +The latter is useful mainly in the NixOS module system, where a lot of things will be passed by default. More on that later. + +Finally, we can make the values in =...= available by using =@= as in =extra @ { name, ... }: "${name} likes ${extra.object}"=: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate 'let f = extra @ { name, ... }: "${name} likes ${extra.object}"; in f { name = "Nyx"; object = "nix"; }' +#+end_src + +#+RESULTS: +: Nyx likes nix + +Note that it is equivalent to write ={} @ extra=: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate 'let f = { name, ... } @ extra: "${name} likes ${extra.object}"; in f { name = "Nyx"; object = "nix"; }' +#+end_src + +#+RESULTS: +: Nyx likes nix + +This looks cumberesome on first sight, but is for example useful when referencing flake inputs in the flake outputs, as it allows you to forego listing them all in the arguments (which can be a very long list!). More on that later. + +**** Language features +:PROPERTIES: +:CUSTOM_ID: h:5b660aa4-6dbb-4ca7-a006-c97319ae6b7e +:END: + +- The functions available to the nix language can be found in [[https://hydra.nixos.org/build/318491462/download/1/manual/language/builtins.html][the nix reference manual]]. Some explanations to common functions are provided in [[#h:ebe86cf4-fe65-464f-8f64-04b57870c5e9][Builtin functions]]. + +- Scopes are static in nix. That means that scoped variables are not inherited by the context of the evaluation (this needs to be kept in mind when e.g defining outputs using =flake-parts=) + - Variables can be created in an enclosing scope using =let in =. You actually saw me use this several times in the above function call. Note that usually, you will not be able to declare arbitrary variables; this is because usually we will be working within the confines of [[#h:8067f8fc-92d7-46af-b1be-e8a337d7a953][The module system]] - in that scope, you either need to resort to the =let ... in= construct or you need to declare things as module options (more on that later). In general nix however, this would be no problem. + - another way to add an expression to the scope is using the =rec= keyword: + - this adds all attributes of the enclosing attribute + - the following will error out: + +#+begin_src bash :tangle no :exports both :results output + swarsel-instantiate ' + { + a = true; + b = a; + } + ' +#+end_src + +#+RESULTS: +: error: undefined variable 'a' +: at «string»:4:9: +: 3| a = true; +: 4| b = a; +: | ^ +: 5| } +: [ Babel evaluation exited with code 1 ] + + - however, =rec= makes =a= available to the scope (also note that the order of expressions does /not/ matter): + +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + rec { + b = a; + a = true; + } + ' +#+end_src + +#+RESULTS: +: { a = true; b = true; } + + - the last way to something to the scope is by using =with=: + - this adds the passed set to the lexical scope of the enclosing expression: + - the following will error out: + +#+begin_src bash :tangle no :exports both :results output + swarsel-instantiate ' + let + functions = { + print = v: v; + }; + in + print "ok" + ' +#+end_src + +#+RESULTS: +: error: undefined variable 'print' +: at «string»:7:5: +: 6| in +: 7| print "ok" +: | ^ +: 8| +: [ Babel evaluation exited with code 1 ] + + - using =with= will make it work: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + let + functions = { + print = v: v; + }; + in + with functions; print "ok" + ' +#+end_src + +#+RESULTS: +: ok + + - generally, the values of the variable in the innermmost scope is given precedence in evaluation: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + let + scope = "outer"; + in + let + scope = "inner"; + in + scope + ' +#+end_src + +#+RESULTS: +: inner + + - the same holds true for =with=: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + with { scope = "outer"; }; with { scope = "inner"; }; scope + ' +#+end_src + +#+RESULTS: +: inner + +- you can copy variables from an outer scope by using =inherit () ;= + - This is syntactic sugar to achieve the same as = = .;= +- when not in a string, we can place parentheses around a nix expression to provide its return value to other expressions (remember the =builtins.toString (-1)= example from before. =-1= is indeed a nix expression.) +- you can write inline comments using =#= and multiline comments using =/* ... */= +- The following words are reserved keywords and cannot be used for variable names: =assert=, =else=, =if=, =in= =inherit=, =let=, =or=, =rec=, =then=, =with= + +**** Operators +:PROPERTIES: +:CUSTOM_ID: h:f995d46f-ff07-4a5e-9164-ed7aef369c42 +:END: + The following operators work as you know them from basically any other language: + - Any operator involving numbers (=+=, =-=, =*=, =/=) + /Sidenote:/ When negating a number in actual nix code you will need to wrap it in parentheses nearly every time due to function evaluation: +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + builtins.toString (-1) + ' +#+end_src + +#+RESULTS: +: -1 + +but: + +#+begin_src bash :tangle no :exports both :results output + swarsel-instantiate ' + builtins.toString -1 + ' +#+end_src + +#+RESULTS: +#+begin_example +error: + … while calling the 'sub' builtin + at «string»:2:21: + 1| let lib = import ; in + 2| builtins.toString -1 + | ^ + 3| + + … while evaluating the first argument of the subtraction + + error: expected an integer but found the built-in function 'toString': «primop toString» +[ Babel evaluation exited with code 1 ] +#+end_example + + - Logical operators (=!=, ====, =!==, =<=, =>=, =>==, =<==, =&&=, =||=) + - additionally, logical implications are available as =->= (defined as =!v1 || v2= aka. =if v1 then v2 else true=) + + The following operators are nix specific: + - Concatenations can be done between strings and paths (in any order) using =+= + - List Concatenations however use the =++= operator + - you can use the =?= operator on an attribue set to check whether it has an attribute: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + { exists = true; } ? exists + ' + swarsel-instantiate ' + { exists = true; } ? noExists + ' +#+end_src + +#+RESULTS: +| true | +| false | + + - Attribute selection is done using =.=: + +#+begin_src bash :tangle no :exports both + swarsel-instantiate ' + { v = 1; }.v + ' +#+end_src + + - Updates to attribute sets are done using [[#h:b1fe7a9a-661b-4446-aefa-98373108f8fd][The '//' operator]] + +#+RESULTS: +: 1 + +**** Flakes +:PROPERTIES: +:CUSTOM_ID: h:b44026d5-bf7e-4ab3-a31b-68b72292e908 +:END: + +Next, we shall talk about some concepts regarding nix flakes. + +What are flakes? The answer is surprisingly simple: Flakes can be thought of packages that contain nix expressions. They can include other flakes as inputs (acting as libraries if you want) and build on their functionality (in fact, this is exactly what happens when you are using [[#h:d62af55c-a7f3-47ba-b2a5-78cc60b03aef][NixOS]] with [[#h:f4e89635-e006-4c41-a545-d4b56c7ac293][nixpkgs]]). What exactly is provided by a flake is up to the owner entirely, although there are some "norms" that we will get to later. + +This historical note aside, a directory becomes a flake as soon as it contains a file called =flake.nix=, which contains an attribute set that has at least an attribute =outputs=, which must be a function returning an attribute set. Hence, the simplest possible flake is the following: + + #+begin_src nix-ts :tangle no + { + outputs = _: { }; + } +#+end_src + +What is very nice about flakes is that each flake generates a corresponding =flake.lock= file the first time a nix command is called targetting it. This file exactly specifies at which revision every flake input should be pulled in. This makes flakes very nice for setting up development environments that can then be committed to a project; this enables collaborators to do [[#h:c22f65c4-4c0d-4aea-9b70-d34c295764f0][nix develop]] which will setup a declarative environment, which is the same for every developer. This increases reproducibility greatly. + +That was a very dry introcution. Now, what we are usually interested in when pulling in a flake is any of the following: + +- modules to extend our configuration with +- packages that are not in =nixpkgs= +- flake templates +- devshells + + Where these things come from we will learn soon. + +***** Why flakes +:PROPERTIES: +:CUSTOM_ID: h:96d28671-8b9d-4fbc-8620-1a7d5fcec480 +:END: + +Flakes are currently an experimental feature in nix, although their interface has been quite stable for a while now - this is possible in part why nearly all bigger nix projects are using flakes nowadays. People like to (mistakenly) claim that it is because of flakes that we have version pinning in nix. In fact, people were doing this long before flakes were a thing using a construct that looked roughly like this: + +#+begin_src nix-ts :tangle no + let + let pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/0b20bf89e0035b6d62ad58f9db8fdbc99c2b01e8.tar.gz") {}; + in + pkgs.mkShell { + buildInputs = [ pkgs.cowsay ]; + } +#+end_src + +This approach simply imports a chunk of nix code that is fetched using =fetchTarball= - and the version is pinned by the git revision. However, imagine a bigger project; even if you were to import every input like this and pass the expressions to the modules that need them, updating the inputs manually would be a chore. And even though projects like [[https://github.com/nmattia/niv][niv]] and [[https://github.com/andir/npins][npins]] exist(/ed) to help with these problems, flakes provide a uniform interface and are reasonably easy to work with, which makes this this considerably more pleasant. + +That being said, some people decide to work without flakes. Depending on the reasoning, I think this is totally fair, as really for the code itself they dont automatically provide a lot of extra functionality that cannot be achieved with other tools). However, some people dismiss flakes for the only reason being "they are too complex" with really is not at all true (which is what I hope to prooe to you soon!). A simple project devshell for example could very reasonably be built only from the =fetchTarball= construct I showed above. + +****** Channels +:PROPERTIES: +:CUSTOM_ID: h:2cb443a5-9de2-4ae5-8a7e-d8e96748d599 +:END: + +Finally and for completeness sake, a historical note on nix channels. + +Nix channels are one of the official mechanisms to provide package sources. They are managed manually through the cli. A channel would be given a name (like =nixpkgs=) which could then in nix code be referenced for example like: + +#+begin_src nix-ts :tangle no + { pkgs ? import {} }: ... +#+end_src + +Really, this leaves it up to the caller to decide what =nixpkgs= really is, along with a few other considerations that I do not want to get into. This chapter is intentionally left short; I just want you to know how the pattern above (the ==) looks so that you know what you are working with if you encounter it in the wild. + +Channels are not a great tool for reproducability and I would advise against using them if possible. + +***** Flake arguments +:PROPERTIES: +:CUSTOM_ID: h:2dec2fcf-5923-4bae-9407-027dd9d917e1 +:END: + +Let us now talk about arguments that a flake (kind of) expects. + +****** Inputs +:PROPERTIES: +:CUSTOM_ID: h:e965faf8-5782-4720-aa44-d20d700a339c +:END: + +As we already learned, a flake must provide an attribute called =outputs= (whatever that might mean at this point!). However, unless we want to build everything that our flake provides from the ground up, we probably also want to pass some libraries or build tools to it. This is done using the attribute called =inputs=. An =inputs= definition might look something like this: + +#+begin_src nix-ts :tangle no + { + inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; + hydra = { + url = "github:nixos/hydra/nix-2.30"; + inputs.nix-eval-jobs.follows = "nix-eval-jobs"; + }; + nix-eval-jobs = { + url = "github:nix-community/nix-eval-jobs/v2.30.0"; + flake = false; + }; + toplevel.url = "./.."; + }; + } +#+end_src + +This is a surprisingly real-world example that we can use so explain most of the important =inputs= options: +- the most important attribute is =url=. It describes where the source should be fetched from. As you can see from the =toplevel= input, you can put as an input a path on the local filesystem. Note that by default, it is expected that every input is itself a flake - meaning, a =flake.nix= exists there which will be loaded into our flake along with its dependencies. You can also see that we set this to =false= for the =nix-eval-jobs= input. This is usually done when the target project is simply not a flake. Nix will then load in the contents of the repository as a raw filetree. + +However, if you go and check out [[https://github.com/nix-community/nix-eval-jobs][nix-eval-jobs]] you will see that this project /does/ have a =flake.nix=, and it is also very active; so what is going on here? In this particular case, this is done to avoid pulling in the dependencies that the =nix-eval-jobs= would bring along. Instead, the files provided by it must be then made to work using what is provided in the parent flake. This is not a common pattern, but something that is good to know. + +Another interesting option is the =hydra.inputs.nix-eval-jobs.follows=. Let us first talk about the general interface =.inputs.=: This structure allows to customize options for the *dependency* of a flake input. For example, we could have customized the source of the =nix-eval-jobs= dependency of =hydra= had we written =hydra.inputs.nix-eval-jobs.url = ...=. The =follows= option takes a similar approach by setting an input dependency to the same version as some other input of the parent flake (in our case, it is pinned to the =nix-eval-jobs= v2.30.0 version). + +When we are done defining our flake inputs, we can then decide what our flake shoud actually do. + +****** Outputs +:PROPERTIES: +:CUSTOM_ID: h:66a04339-dbc7-4d98-a763-5a9be9185f9b +:END: + +As we already learned earlier, =outputs= is a function returning an attribute set. In fact, it is usually a function that takes as an argument an attribute set containing our inputs, looking roughly like this: + +#+begin_src nix-ts :tangle no + { + inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; + outputs = inputs @ { self, nixpkgs, ... }: + let + inherit (self) outputs; + in + { + + nixosConfigurations = { + main = nixpkgs.lib.nixosSystem { + modules = [ "${self}/configuration.nix" ]; + specialArgs = { + inherit inputs outputs; + }; + }; + }; + }; + } +#+end_src + +With what we learned so far, you are already able to understand most of what is going on here: + - outputs is a function that expects =self= and =nixpkgs= but /tolerates/ other attributes in the set, and the attribute set =inputs= can be referenced. + - an attribute =outputs= is inherited from the outer scope =self=, meaning =outputs = self.outputs;= (what =self= is and why we can reference the =outputs= in the same place where we are defining them I will explain shortly) + - the resulting attribute set holds an attribute called =nixosConfigurations= that holds an attribute called =main=, which is set by calling a function that we now do not want to get into too much. + + You might (rightfully!) wonder where the =self= argument comes from. It is actually not a flake input. Instead, =self= is a special flake construct that refers to the root of the current flake, meaning all flake attributes can be accessed using =self.=. Moreover, when using =self= as a path, it refers to the path os the current flake in the nix store. This can be used to reference files within the repository without needing relative paths (this is exactly what happenss in =self.outputs.nixosConfigurations.main.modules=: it loads in =configuration.nix=, which resides in the same directory as the =flake.nix=. We could have also written =./configuration.nix= here instead). + + Another question might be what happens when we load in =self.outputs= while defining outputs. Normally that should not work? Remember that nix is a lazy language. When =outputs= is first created in the =let ... in= block, its value will be something along the lines of =outputs = ;=. That means, not all values are loaded upon instanciation, but only whenever they are needed. And if a value is needed, nix will then move on to compute that value. + + Of course that does mean that we need to be careful to not introtuce circular chains of dependency. Doing that would result in what is called an =infinite recursion error=, a type of error that can be very hard to wrap one's head around. Luckily, when writing a simple config, you are not very likely to encounter it in a shape that is not rather easily solved. + + Like we saw here with =nixosConfigurations= and some other outputs that we heard about earlier, there are some "standard" outputs that nix recognizes and is able to apply some defaults on. Let us quickly go over them: + +****** Standard flake outputs +:PROPERTIES: +:CUSTOM_ID: h:680b38c1-14de-4892-8840-2c82accd6827 +:END: + +I will now list output names and explain what they to; some outputs are what I call "system-scoped". That means they have toplevel attribute in the name of system architectures, e.g. =x86_64-linux=. I will mention when an output is system-scoped. This list is roughly ordered by importance to a NixOS beginner: + +- =nixosConfigurations=: A set of NixOS host configurations +- =devShells=: *system-scoped*. A set of devshells that can be used by calling [[#h:c22f65c4-4c0d-4aea-9b70-d34c295764f0][nix develop]]. Defaults to the devshell called =default=. +- =checks=: *system-scoped*. Flake checks to perform when calling [[#h:b07b1776-75cd-4a40-9e71-fb00eab65c6f][nix flake check]] +- =formatter=: *system-scoped*. Formatting package/config to use when calling [[#h:cf6cc5fa-cbaa-415a-b1d8-c6c5b7016700][nix fmt]] + +- =nixosModules=: A set of nixos [[#h:de7bc464-e04f-45d4-9d29-e46592535419][Modules]]. By default, when consuming one of these modules but not specifying which one, the overlay =default= will be chosen. +- =overlays=: A set of nixpkgs [[#h:7a059bd9-13f8-4005-b270-b41eeb6a4af2][Overlays]]. By default, when consuming one of these overlays but not specifying which one, the overlay =default= will be chosen. +- =packages=: *system-scoped*. Packages that can be directly built using [[#h:fa377c74-36e5-4aea-bc39-4d3def4b67d1][nix build]] +- =apps=: *system-scoped*. Packages that can be directly run using [[#h:9f55f619-892e-488c-936f-43962f90cde8][nix run]] +- =templates=: A set of templates that can be initialized using =nix flake init -t