Raspberry Pi NixOS Cluster

• Usage
  • Commands
  • Adding a new Pi node
  • Adding a new VPS
  • Adding the WSL node (Windows desktop)
    • Why tailscale rather than forwarding port 22 from Windows
  • Adding the macOS node (nix-darwin)
  • iMessage bridge — one-time GUI setup
  • Push notifications without Google (UnifiedPush via ntfy)
    • Three ways this fails on the phone
  • Matrix server administration (Ketesa)
  • Signal bridge
  • WhatsApp and Messenger bridges
  • Deploying changes
  • Headscale (first-time setup)
  • MagicDNS — names and IPv6 (2026-08)
    • Onboarding a new device (WSL desktop, macs, anything)
    • Laptop (le-fleur)
  • Editing secrets
• Design
  • Three layers
  • Services
  • Secrets
• Moving the Nix store to the external drive
• Meshtastic
  • Custom LoRa settings
  • Finding the serial port
  • Setting a static IP
• TODO
  • Infrastructure
  • Apps

NixOS configuration for a Raspberry Pi cluster using nixos-raspberrypi.

Usage

Commands

Command	Targets	Description
just deploy <node>	Pi + VPS	Deploy config to a running node
just install <node> <host>	VPS	First-time NixOS install via nixos-anywhere
just build <node>	Pi only	Build an SD card image
just flash <device> <node>	Pi only	Flash an image to an SD card
just add-node <node>	Pi only	Generate host key and set up SOPS for a new Pi
just grab-host-key <node> <host>	Pi + VPS	Grab SSH host key from a running machine
just edit-secrets <scope>	All	Edit an encrypted secrets file (common, wifi, or a node name)
just rotate-node <node>	All	Rotate a node's host key and rekey secrets
just shutdown	Pi only	Shut down all Pi nodes

Adding a new Pi node

1. Add an entry to nodes in flake.nix:

   rpi4-node2 = {
     platform = "raspberry-pi-4";
     # LAN is plain DHCP — no addresses here. After first tailnet join,
     # pin the headscale-assigned address:
     # tailscaleAddress = "100.64.0.x";
   };

2. Generate host key, update sops, and stage files:

   just add-node rpi4-node2

3. Build and flash:

   just build rpi4-node2
   just flash /dev/sdX rpi4-node2

Adding a new VPS

1. Add the node to flake.nix with model = "vps"
2. Grab or generate an SSH host key: just grab-host-key <name> <host>
3. Install: just install <name> <host>

Adding the WSL node (Windows desktop)

jokull is a NixOS userland inside WSL2 on the Windows desktop. It is the smallest kind of fleet member: Windows owns the hardware, the disk, the kernel and the uptime, so the config owns only identity, a user, sshd and tailscale (modules/platforms/wsl/configuration.nix). No image, no nixos-anywhere, no LAN address — WSL2 lives behind Windows' NAT on a 172.x address that is re-rolled on every wsl --shutdown, so the tailnet is the only stable way in.

Order matters here: the secrets must be rekeyed before the first switch, because apps.tailscale reads headscale/preauth-key out of secrets/common.yaml and sops-nix fails activation outright on a secret it cannot decrypt.

1. On Windows: install NixOS-WSL — download nixos.wsl from the latest release and double-click it (WSL ≥ 2.4.4), or:

   wsl --install --from-file nixos.wsl

2. On the laptop: the registry entry lives in flake.nix and the host key is already generated (keys/jokull/, via the same ritual as just add-node jokull). Register it as a sops recipient and rekey:

   just update-sops-yaml
   just rekey
   git commit -am "add jokull (WSL desktop)" && git push

3. Install the host key on the box. This is the one manual step: there is no SD card to inject into and no --extra-files to hand nixos-anywhere, but sops-nix still derives jokull's age identity from /etc/ssh/ssh_host_ed25519_key. On the laptop:

   age -d -i ~/.config/sops/age/keys.txt keys/jokull/ssh_host_ed25519_key.age

   then in the WSL shell, paste it and Ctrl-D:

   sudo install -d -m 755 /etc/ssh
   sudo sh -c 'umask 077; cat > /etc/ssh/ssh_host_ed25519_key'
   sudo sh -c 'ssh-keygen -y -f /etc/ssh/ssh_host_ed25519_key > /etc/ssh/ssh_host_ed25519_key.pub'

   Verify the box ended up with the identity the secrets were encrypted to — these two must print the same string:

   ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub   # on the WSL box
   just show-age-key jokull                          # on the laptop

4. First switch, run on the box (nothing can push to it yet — no tailnet, no route):

   git clone https://git.lundsaunet.no/blund/cluster.git ~/rpi-cluster
   sudo nixos-rebuild switch --flake ~/rpi-cluster#jokull \
     --option experimental-features 'nix-command flakes'

   The flags are only needed while the stock tarball's nix.conf is in charge; after the switch our own nix.settings carries them.

   Expect the login user to change. users.mutableUsers = false deletes the tarball's nixos user, and wsl.defaultUser is cluster (the fleet user, carrying the keys from modules/ssh-keys.nix). If a shell refuses to open afterwards, get back in with wsl -d NixOS --user root. To keep a personal account, declare it in the node entry alongside cluster.

5. Tailnet: it auto-registers with the shared preauth key on first boot. Read the address off the box and pin it in flake.nix (tailscaleAddress), same ritual as every other node:

   tailscale ip -4

   After that, from the laptop: ssh cluster@jokull.tail.kagayaki.no and just deploy jokull work like any other node.

6. Keep it alive across Windows reboots. The WSL VM only runs while a distro is running — no login, no sshd. Task Scheduler, trigger "at log on", action a one-line .vbs that launches it hidden:

   set object = createobject("wscript.shell")
   object.run "wsl.exe --distribution NixOS", 0

   Note wsl --shutdown (and Windows' idle timeout — see vmIdleTimeout in %USERPROFILE%\.wslconfig) takes the node off the tailnet instantly.

Why tailscale rather than forwarding port 22 from Windows

Reaching WSL from the LAN otherwise means netsh interface portproxy rules on the Windows side, re-pointed at a NAT address that changes on every WSL restart, plus a Windows Firewall rule — three moving parts maintained outside this repo, none of them declarative. networkingMode=mirrored in .wslconfig removes the NAT but then collides with Windows' own port 22 and is its own class of surprise. Tailscale gives one stable name that works off the home network too, using the same preauth-key path every other node already uses.

Adding the macOS node (nix-darwin)

The Mac (eve, a 2015 Intel Mac mini → x86_64-darwin) is managed with nix-darwin on top of upstream Nix (the official nixos.org installer — Determinate dropped x86_64-darwin in Nov 2025). It exists to do what Linux can't: an iMessage/Signal bridge and macOS build CI. It lives in the same flake as everything else, but under darwinConfigurations instead of nixosConfigurations, and it builds its own closure (the aarch64/x86_64 Linux builders can't cross-build Darwin).

Unlike the Pis there is no SD image and no static LAN IP — the Mac joins the tailnet and is reached at its tailscaleAddress.

First-time setup:

1. Install upstream Nix on the Mac (multi-user daemon install). Pinned to 2.28.x: current Nix installers require macOS ≥ 14, and the mini tops out at Monterey 12 — 2.28 matches the nixpkgs 25.05 pin:

   sh <(curl -L https://releases.nixos.org/nix/nix-2.28.7/install) --daemon --yes

   nix-darwin then manages the Nix installation itself (nix.enable default). Xcode Command Line Tools are also needed for git (headless: the softwareupdate --list CLT trick, or xcode-select --install at the machine).

2. Enable Remote Login (System Settings → General → Sharing) so the Mac has an SSH host key and is reachable.

3. Grab its host key so sops-nix can decrypt secrets on it:

   just grab-host-key eve <mac-ip>

   This encrypts the key, adds the Mac's age recipient to .sops.yaml, and rekeys the secrets files. Commit the result.

4. Bootstrap the first switch (clones this repo on the Mac, runs the initial nix run nix-darwin -- switch):

   just bootstrap-darwin eve <mac-ip>

5. The Mac auto-registers with Headscale using the shared headscale/preauth-key and comes up on the tailnet. Confirm its address and, if it differs, update tailscaleAddress in flake.nix:

   ssh cluster@<mac-ip> tailscale ip

Subsequent deploys (over the tailnet, builds on the Mac):

just deploy-darwin eve

Darwin service modules live in modules/services/darwin/ (kept separate from the Linux ones so the launchd-based modules never get loaded into a NixOS node). Today that's Tailscale and the iMessage bridge; a Forgejo macOS runner may get added later.

iMessage bridge — one-time GUI setup

The bridge (apps.imessageBridge on eve + apps.imessageRegistration on kumatetsu) deploys fully declaratively, but macOS gates the interesting parts behind GUI-only permission dialogs (TCC) and an Apple ID login. After the first deploy the bridge crash-loops on "unable to open database file: operation not permitted" until these one-time steps are done at the machine (or via Screen Sharing):

1. Auto-login: System Preferences → Users & Groups → Login Options → Automatic login: cluster. The bridge is a launchd agent in the GUI session (AppleScript needs Messages.app in an aqua session), so after a reboot nothing bridges until cluster is logged in at the console.
2. Sign into Messages.app with the Apple ID whose iMessages should be bridged.
3. Full Disk Access for the bridge binary: System Preferences → Security & Privacy → Privacy → Full Disk Access → + → ⌘⇧G and paste: /Users/cluster/Library/Application Support/mautrix-imessage/bin/mautrix-imessage This is a stable path — the agent copies the binary out of the nix store on every start precisely so TCC grants survive deploys.
4. The Automation (control Messages) and Contacts prompts appear on their own the first time the bridge sends / syncs — click OK.

5. Restart the agent (or just wait ~10s for KeepAlive):

   launchctl kickstart -k gui/501/org.nixos.mautrix-imessage

6. Watch it come up: tail -f ~/Library/Logs/mautrix-imessage.log on eve. Healthy startup logs Starting HTTP listener address=0.0.0.0:29331 (synapse pushes appservice transactions straight there over the tailnet) and then Startup sync complete.
7. From Matrix (@blund:lundsaunet.no), !im help in any portal room lists the commands — the prefix stops the message being relayed to iMessage. New iMessage conversations appear as invites from @imessage_<number>:lundsaunet.no ghosts.

Notes: e2be is on and encryptPortals is set, so the bridged chats are end-to-end encrypted on the Matrix side (the four that predate the flag were upgraded by hand — a one-way door per room, so any client pointed at them now needs full Olm/Megolm). personalSpace collects them in an "iMessage" space. The appservice tokens live in secrets/common.yaml (imessage/as-token / hs-token), and eve listens on :29331 for synapse's appservice transactions — reachable over the tailnet only in practice, since that is the only route to the Mac. FileVault is off, so remote reboots are safe. Details and the gotchas that cost hours: docs/imessage-bridge.org.

Push notifications without Google (UnifiedPush via ntfy)

Matrix clients normally reach your phone through Firebase. ntfy on kumatetsu replaces that with UnifiedPush. The chain:

Element  --registers-->  ntfy app (the UnifiedPush "distributor")
ntfy app --returns--->   https://ntfy.lundsaunet.no/upABC123
Element  --tells----->   synapse: push there, via gateway
                         https://ntfy.lundsaunet.no/_matrix/push/v1/notify
synapse  --POSTs---->    that gateway on every notifiable event
ntfy     --routes--->    topic upABC123 → the phone

No Sygnal in the middle: ntfy ships a Matrix push gateway itself and translates the Matrix push format into a plain ntfy publish.

Two settings are load-bearing. base-url must be the public URL — the gateway checks that the pushkey Element registered starts with it and rejects the push otherwise. And the ACL has to leave up* open for anonymous writes, because synapse pushes with no credentials at all; confidentiality rests on the topic name being unguessable.

Everything else is denied (auth-default-access: deny-all), so the server cannot be used as a free relay by whoever finds it. Anonymous reads are denied too — the phone authenticates as blund to subscribe. That user and its bcrypt hash live in secrets/kumatetsu.yaml under ntfy/auth-users and reach the service through an environment file, never the nix store.

Apps that subscribe (ink chat) should get their own access token rather than the password — same ACL, individually revocable:

ssh root@100.64.0.3 \
  'NTFY_AUTH_FILE=/var/lib/ntfy-sh/user.db ntfy token add --label <app> blund'
ssh root@100.64.0.3 \
  'NTFY_AUTH_FILE=/var/lib/ntfy-sh/user.db ntfy token list blund'   # or: token remove

Tokens are runtime state in user.db, not config: nothing to commit, nothing in sops. If the state directory is ever lost, tokens are regenerated, not restored. On the client they belong in the platform keystore (EncryptedSharedPreferences on Android), entered by hand — never in an app's source or repo.

Phone setup, once:

1. Install the ntfy app; add https://ntfy.lundsaunet.no as the server and log in as blund.
2. In Element → Settings → Notifications, pick ntfy as the UnifiedPush distributor.
3. Check it took: the gateway should answer curl https://ntfy.lundsaunet.no/_matrix/push/v1/notify with {"unifiedpush":{"gateway":"matrix"}}, and the pusher should show up in GET /_matrix/client/v3/pushers on synapse.

Note ntfy.lundsaunet.no needs its own A record → edge-vps. There is no wildcard on lundsaunet.no, and caddy cannot complete an ACME challenge for a name that does not resolve. Caddy automates the certificate, never the record — and having failed three times it does not retry for hours, so restart it once the name resolves.

Three ways this fails on the phone

All three were hit on 2026-08-12 and none announce themselves.

1. The topic must start with up. The ACL only lets anonymous clients publish to up*, and synapse pushes anonymously. A client that lets you name the topic something else (ink chat does) gets a silent 403 on every push. Test it directly:

   curl -d "probe" https://ntfy.lundsaunet.no/<topic>   # want 200

2. The ntfy app must be logged in. Anonymous reads are denied, on purpose, so a leaked topic name exposes nothing. The consequence is that the app cannot subscribe until you add the user under Settings → Manage users. Without it the app fails quietly and shows nothing.
3. A manual subscription is not a UnifiedPush registration. If ntfy itself displays the raw JSON payload, the topic was subscribed by hand and no app is being fed. In a real registration the distributor displays nothing: the app asks it for an endpoint (NEW_ENDPOINT), sends that to synapse as the pushkey, and gets the bytes handed to it (MESSAGE / PUSH_EVENT). Never type a topic in by hand — let the distributor generate it, which also guarantees the up prefix from (1).

Payloads are format: event_id_only, so the topic only ever carries an event id and room id — no message content, which is what makes the open up* write permission tolerable. The flip side is that the client must wake, fetch the event, and decrypt it (portals are encrypted) before it can show anything useful.

Matrix server administration (Ketesa)

Matrix separates what a user may do from what the operator may do, and Element only implements the first. Leaving a room sets your membership to leave and nothing more: the room, its state, history and media stay in Postgres forever. That is why the server accumulates rooms with zero joined members, which no client can even display.

Deleting them takes the Synapse admin API, and Ketesa (formerly Synapse Admin) is the UI for it — rooms, users, media, federation, plus a space hierarchy view.

Open http://100.64.0.3:8090 from any tailnet device and log in as @blund (a server admin — no separate credential). The homeserver field is pinned; the iMessage ghosts are filtered out of the user list.

Exposure is tailnet-only, and deliberately doubly so: caddy binds the socket to the tailnet address rather than 0.0.0.0, and the firewall opens the port on tailscale0 only. Anything that can load this page is one login away from deleting rooms — do not widen it.

Equivalent by hand, when a UI is overkill:

TOK=<admin access token>; HS=http://100.64.0.3:8008
curl -s -H "Authorization: Bearer $TOK" \
  "$HS/_synapse/admin/v1/rooms?order_by=joined_members&dir=f"
curl -s -X DELETE -H "Authorization: Bearer $TOK" \
  -d '{"purge":true,"block":false}' "$HS/_synapse/admin/v2/rooms/$ROOM"

Do not purge a bridge portal room without also clearing its row in the bridge's sqlite portal table, or the bridge keeps referring to a room id that no longer exists. Note too that leaving a portal does not unbridge anything — the bridge re-invites you on the next message.

Signal bridge

apps.signalBridge on kumatetsu. Unlike iMessage, nothing about it is exotic: nixpkgs' mautrix-signal next to synapse, loopback in both directions, shared postgres, registration auto-generated and auto-wired. Portals are encrypted from creation and the bot's command replies are encrypted too (bridgev2, not the legacy framework).

One-time login: DM @signalbot:lundsaunet.no, send login qr, scan with the phone (Signal → Settings → Linked devices). The bridge is a linked device — the phone stays primary and ~30 days without the phone online drops the link. History transfers on pairing (Signal's transfer archive), and double puppeting via apps.doublePuppet makes your own messages appear as @blund rather than a ghost.

Full architecture, verification steps and the postgres collation incident the first deploy uncovered: docs/signal-bridge.org.

WhatsApp and Messenger bridges

apps.whatsappBridge and apps.messengerBridge, both on kumatetsu, both the same architecture as Signal (loopback appservice, shared postgres, encrypted portals, double puppeting via apps.doublePuppet — the whole story is in docs/signal-bridge.org and applies here verbatim). Ports: whatsapp 29318, messenger 29319, signal 29328.

Logins (mute first — just notifications off <token>, and just sync-push-rules after the import settles):

• WhatsApp: DM @whatsappbot, login qr, scan fast (WhatsApp → Settings → Linked devices). Linked device; phone stays primary.
• Messenger: DM @messengerbot, login, follow the cookie flow (paste cookies from a logged-in browser session on messenger.com). Not a linked device. Meta sometimes challenges accounts that look automated — expect the occasional re-login, and know the risk exists before bridging an account you can't afford to lose.

Messenger runs on services.mautrix-meta (multi-instance; nixpkgs preconfigures facebook and instagram names but NOT messenger — our module defines the instance from scratch). Its serviceDependencies default includes the registration oneshot that also renders the config; a module restating that list must keep it, or the bridge boots configless.

On a deploy that adds a bridge, synapse may fail its first restart — it reads the new registration file before the bridge's first start has generated it. It self-heals on the systemd retry; a red "Failed to start matrix-synapse" during activation is only fatal if it stays red.

Deploying changes

Works for both Pi nodes and the VPS:

just deploy node1
just deploy edge-vps
just deploy-all

The Mac deploys separately (it's a darwin system, not NixOS):

just deploy-darwin eve

Headscale (first-time setup)

After deploying the edge node with headscale enabled:

1. Create a pre-auth key on the VPS:

   ssh root@37.27.248.157 "headscale preauthkeys create --user 1 --reusable --expiration 365d"

2. Add the key to SOPS under headscale/preauth-key (shared by all nodes → common scope):

   just edit-secrets common

3. Deploy all nodes (they auto-register):

   just deploy-all --skip=node2

MagicDNS — names and IPv6 (2026-08)

Every tailnet node has a stable dual-stack name, derived in the flake from its registry key (nodes.<name>.tailscaleName):

kumatetsu.tail.kagayaki.no  →  100.64.0.3   +  fd7a:115c:a1e0::3
wall-e.tail.kagayaki.no     →  100.64.0.10  +  fd7a:115c:a1e0::a
edge-vps.tail.kagayaki.no   →  100.64.0.1   +  fd7a:115c:a1e0::1

Two server-side pieces make this work (both in modules/services/headscale.nix):

1. The dns section: magic_dns on, base_domain (single knob: tailDomain in flake.nix), and global fallback nameservers — these are what make acceptDns safe (see the 2026-05 ACME-outage note in tailscale-client.nix).
2. The ACL policy granting the magicdns-aaaa node attribute. Without it, tailscaled silently strips peers' AAAA records on dual-stack nodes (tailscale/tailscale#1152) — A works, AAAA returns empty, no error anywhere. The policy also carries an explicit allow-all ACL rule: a headscale without a policy allows everything implicitly, but the moment a policy file exists, ACLs are explicit.

Client side, nodes opt in with apps.tailscale.acceptDns = true (kumatetsu, wall-e). Never set it on edge-vps: its resolv.conf would point at the tailscaled resolver whose configuration comes from the headscale it itself hosts.

Onboarding a new device (WSL desktop, macs, anything)

One command, full fleet by name, IPv4 + IPv6:

tailscale up --login-server https://hs.kagayaki.no \
             --auth-key <preauth-key> \
             --accept-dns

• The preauth key lives on the VPS: ssh root@edge-vps cat /var/lib/headscale/preauth-key (or mint a fresh one, see the headscale section above).
• --accept-dns rewrites the device's resolv.conf to the tailscale resolver with search tail.kagayaki.no — so bare ssh kumatetsu works too. Public DNS keeps working via the global nameservers headscale pushes.
• Verify: getent hosts wall-e.tail.kagayaki.no (expect the ULA first), then getent hosts cache.nixos.org (public DNS regression).

Laptop (le-fleur)

The laptop's tailscale lives in ~/config, not this repo. It is already on the tailnet but does not accept DNS yet — flip it there (declaratively services.tailscale.extraSetFlags = [ "--accept-dns" ] or one-shot tailscale set --accept-dns=true). Until then, tailnet names do not resolve on the laptop, which also blocks migrating the justfile deploy targets / /etc/nix/machines builder entry from IPs to names.

Editing secrets

EDITOR=vim just edit-secrets <scope>   # common, wifi, or a node name

Design

Three layers

Layer	Responsibility	Location
Node registry	Which node, what hardware+IPs	flake.nix (nodes)
Shared modules	Base infra: SSH, users, keys	modules/
Service modules	How to run an application	modules/services/

Services

Each node definition has two parts: metadata (platform, IPs, external drive) and app config (apps.* options). All service modules live in modules/services/ and are auto-discovered, but only activate when their apps.<service>.enable option is set. This keeps flake.nix a flat, readable declaration of what each node runs.

rpi4-node1 = {
  platform = "raspberry-pi-4";
  tailscaleAddress = "100.64.0.2";  # pinned after first tailnet join
  apps.adguard.enable = true;
  apps.navidrome.enable = true;
  apps.navidrome.musicFolder = "/mnt/storage/musikk";
};

Secrets

Secrets are managed with sops-nix. Each node gets a pre-generated SSH host key which sops-nix uses to decrypt secrets at boot.

Key delivery: the private host key exists only in two places — encrypted to the admin age key in keys/<node>/*.age, and on the node itself. It never enters the worktree, git, or any Nix store: just flash injects it onto the SD card's root partition after dd, and just install delivers it via nixos-anywhere --extra-files. Builds and deploys involve no secret material at all. Rotation: just rotate-node <node>.

Scoping: secrets files are split per recipient set, so a compromised node only exposes its own material. .sops.yaml is generated by just update-sops-yaml; new scope files are created with just new-secrets-file <scope>.

File	Recipients	Contents
secrets/common.yaml	all nodes	headscale/preauth-key
secrets/wifi.yaml	Pi nodes only	wifi/ssid, wifi/psk
secrets/<node>.yaml	admin + node	service secrets, WireGuard keys

Migration status: values still live in the legacy secrets/secrets.yaml and must be moved by hand (just edit-secrets <scope>), then the monolith and its .sops.yaml rule deleted. Routing:

• common ← headscale/preauth-key
• wifi ← wifi/ssid, wifi/psk
• node1 ← wireguard/node1-private
• kumatetsu ← wireguard/node3-private, forgejo/*, matrix-lundsaunet/*, samba/password, transmission/*, mullvad/private-key, vaultwarden/admin-token
• edge-vps ← wireguard/hetzner-private

Moving the Nix store to the external drive

Set externalDrive.hostsNixStore = true on a node and /nix is bind-mounted from the external drive by the initrd (stage 2 runs out of /nix, so it cannot wait for the normal mount). Worth doing on any Pi with a drive: the store is the most write-heavy thing on the system, and SD wear is what kills these boxes — kumatetsu is also the aarch64 builder, so it benefits twice.

This is not a plain deploy. Deploying the flag without copying the store across first produces an unbootable system. The order below matters: the --boot deploy writes the new generation into the SD card's /nix, and the rsync afterwards is what gets it onto the drive.

1. Stage the new generation (does not activate):

   just deploy kumatetsu --boot

2. Copy the store to the drive. -H is essential — the store is full of hardlinks (nix-store --optimise), and without it the copy balloons:

   ssh root@kumatetsu 'mkdir -p /mnt/storage/nix && rsync -aHAX --delete --info=progress2 /nix/ /mnt/storage/nix/'

   Don't build anything on the node while this runs — new paths would be missed. Re-run the same command if a build sneaks in; it is incremental.

3. Sanity check the two trees match:

   ssh root@kumatetsu 'du -sh /nix /mnt/storage/nix'

4. Reboot — this is the step that can fail:

   just reboot kumatetsu

   A screen on the node is useful here, but only to watch where the initrd stops. It is not how you recover — see below.

5. Verify the bind took:

   ssh root@kumatetsu 'findmnt /nix && df -h /nix'

6. Reclaim the SD space. The old store is still there, hidden underneath the bind mount, so it has to be reached through the underlying root:

   ssh root@kumatetsu 'mkdir -p /mnt/sdroot && mount --bind / /mnt/sdroot && rm -rf /mnt/sdroot/nix && umount /mnt/sdroot'

If it does not boot: these Pis use boot.loader.raspberry-pi.bootloader = "kernel" — the firmware boots a kernel straight from config.txt. There is no interactive boot menu, so no amount of screen-and-keyboard will let you pick a generation.

Recovery is offline instead, and is genuinely easy: power off, put the SD card in any machine (the FIRMWARE partition is FAT, readable everywhere) and point config.txt at the previous generation:

# /boot/firmware/nixos/ holds 84-default, 85-default, … plus default (active)
-os_prefix=nixos/default/
+os_prefix=nixos/86-default/

Boot, and you are back on the pre-migration generation with /nix on the SD — nothing has been deleted at that point. Step 6 is the only irreversible step, so do not run it until the node has rebooted cleanly at least once.

Consequence: the drive stops being optional. nofail is dropped for that node, because a missing drive means no /nix and therefore no boot either way; pretending otherwise would just hide the failure.

Meshtastic

Two Heltec V4 nodes on the local network (EU_868 region). Static IPs are configured on the devices themselves (not via NixOS).

Name	Device ID	Static IP
Kagayaki	!69851e60	192.168.86.121
Ooharu	!6983d42c	192.168.86.122

Node names in flake.nix map directly to hostnames (node1, node2, node3, edge-vps).

The meshtastic CLI is installed on rpi5-node3 and can reach them over TCP:

meshtastic --host 192.168.86.121 --info

Custom LoRa settings

The default LongFast preset uses 869.525 MHz with 250 kHz bandwidth, which overlaps with Norwegian smart meters (AMS) known to be active on that frequency. The Norwegian Meshtastic community uses shared custom settings to avoid this interference and ensure mutual compatibility across the mesh. Leave the default channel unnamed — Meshtastic will auto-name it "Custom" based on the LoRa config.

meshtastic --host 192.168.86.121 \
  --set lora.use_preset false \
  --set lora.bandwidth 62 \
  --set lora.spread_factor 8 \
  --set lora.coding_rate 5 \
  --set lora.channel_num 4

Channel 4 auto-selects 869.618 MHz. Both nodes must have matching settings to communicate.

Finding the serial port

When connected via USB, check which port the device is on:

dmesg | tail -20

Look for ttyACM* in the output. You can also list available ports:

ls /dev/ttyACM* /dev/ttyUSB*

Setting a static IP

The CLI requires IPs as little-endian 32-bit integers. To convert a dotted IP A.B.C.D:

A + (B × 256) + (C × 65536) + (D × 16777216)

For example 192.168.86.121: 192 + 43008 + 5636096 + 2030043136 = 2035722432

Common values for this network:

Address	Integer
192.168.86.121	2035722432
192.168.86.122	2052499648
192.168.86.1 (gw)	22456512
255.255.255.0	16777215
8.8.8.8 (dns)	134744072

Set via USB serial (replace port and IP integer as needed):

meshtastic --port /dev/ttyACM3 \
  --set network.address_mode STATIC \
  --set network.ipv4Config.ip 2035722432 \
  --set network.ipv4Config.gateway 22456512 \
  --set network.ipv4Config.subnet 16777215 \
  --set network.ipv4Config.dns 134744072

To reset back to DHCP:

meshtastic --port /dev/ttyACM3 --set network.address_mode DHCP

TODO

Infrastructure

• Move WiFi password to sops-nix
• Set up PostgreSQL on node3 (shared instance at /mnt/storage/postgresql)
• WireGuard VPN to Hetzner VPS (node1 + node3)
• Migrate Vaultwarden from SQLite to PostgreSQL (fresh start on kumatetsu after node1 SD wipe, 2026-08)
• Migrate Synapse from SQLite to PostgreSQL
• Add system.autoUpgrade to auto-pull config from git
• Set up one node as a binary cache (nix-serve-ng or harmonia)
• Configure distributed builds (node3 as remote builder)
• Add networking.extraHosts so nodes can refer to each other by name
• Look into deploy-rs or colmena for fleet deployment

Apps

• Deploy Kodi to kumatetsu (standalone GBM/KMS, see modules/services/kodi.nix)