Longhorn on Talos Linux

I am using Talos Linux as my main operating system for Kubernetes clusters. Talos is a modern, secure, and minimal operating system designed specifically for running Kubernetes. Since it does not have a package manager, adding additional packages a bit different procedure than traditional Linux distributions. In this blog post, I will walk you through how to setup Longhorn on Talos Linux so that your stateful workload can run happily.

Add Extensions to Talos Linux

Since we are deploying Longhorn, we need to add necessary packages to Talos Linux. iscsi-tools and util-linux-tools are the required packages for Longhorn to work properly on Talos Linux.

# extensions.yaml
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools

$ curl -X POST --data-binary @extensions.yml https://factory.talos.dev/schematics
{"id":"613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245"}

Add Extensions to Talos Linux

Do not forget to add --preserve flag.

talosctl upgrade --nodes 10.181.176.4 --image factory.talos.dev/metal-installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.11.1 --preserve
talosctl upgrade --nodes 10.181.176.5 --image factory.talos.dev/metal-installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.11.1 --preserve
talosctl upgrade --nodes 10.181.176.6 --image factory.talos.dev/metal-installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.11.1 --preserve
talosctl upgrade --nodes 10.181.176.7 --image factory.talos.dev/metal-installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.11.1 --preserve

Adding Block Disks to your Worker Nodes

Adding block disks to varies based on your virtualization platform or cloud provider. Since I am using incus for my lab , procedure as follows.

incus storage volume create pool-nvme-samsung-lvm lh-w0 size=200GiB  --type=block
incus storage volume create pool-nvme-samsung-lvm lh-w1 size=200GiB  --type=block
incus storage volume create pool-nvme-samsung-lvm lh-w2 size=200GiB  --type=block

incus config device add talos-w0 lh-w0 disk pool=pool-nvme-samsung-lvm source=lh-w0
incus config device add talos-w1 lh-w1 disk pool=pool-nvme-samsung-lvm source=lh-w1
incus config device add talos-w2 lh-w0 disk pool=pool-nvme-samsung-lvm source=lh-w2

Verify the Block Devices on Talos Linux

You can run the following command for each of nodes you want to add block devices for Longhorn usage.

root@debian-vm:~# talosctl get volumestatus -n 10.181.176.5
NODE           NAMESPACE   TYPE           ID                                  VERSION   TYPE        PHASE   LOCATION    SIZE
10.181.176.5   runtime     VolumeStatus   /dev/sdb-1                          2         partition   ready   /dev/sdb1   215 GB
...(omitted)

Patch the Machine Config

You need to patch the worker nodes with the following maching config in order to mount the block devices properly for Longhorn usage.

# patch.yml
machine:
  kubelet:
    extraMounts:
      - destination: /var/mnt/storage/longhorn
        type: bind
        source: /var/mnt/storage/longhorn
        options:
          - bind
          - rshared
          - rw
  disks:
      - device: /dev/sdb
        partitions:
          - mountpoint: /var/mnt/storage/longhorn

export TALOSCONFIG=~/talosconfig
talosctl patch mc --nodes 10.181.176.5  --patch @patch.yml  # do it for each worker nodes

root@debian-vm:~# talosctl get discoveredvolumes -n 10.181.176.5
NODE           NAMESPACE   TYPE               ID      VERSION   TYPE        SIZE     DISCOVERED   LABEL       PARTITIONLABEL
...(omitted)...
10.181.176.5   runtime     DiscoveredVolume   sdb     1         disk        215 GB   gpt
10.181.176.5   runtime     DiscoveredVolume   sdb1    1         partition   215 GB   xfs

root@debian-vm:~# talosctl get mountstatus -n 10.181.176.5
NODE           NAMESPACE   TYPE          ID                                  VERSION   SOURCE      TARGET                              FILESYSTEM   VOLUME
10.181.176.5   runtime     MountStatus   /dev/sdb-1                          2         /dev/sdb1   /var/mnt/storage/longhorn           xfs          /dev/sdb-1
...(omitted)...

Install Longhorn via Helm

You can install Longhorn via Helm as usual. With exception, you need to label longhorn-system namespace with privileged pod security policy.

apiVersion: v1
kind: Namespace
metadata:
  name: longhorn-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged

helm repo add longhorn https://charts.longhorn.io

helm repo update

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.defaultDataPath="/var/mnt/storage/longhorn" --version 1.10.0