Homelab

The goal: fault-tolerant infrastructure I can rebuild from a single nix build.

The homelab is a real production environment, just for one person. Proxmox cluster on bare metal, UniFi networking, Splunk indexers, Cribl Edge collectors, Home Assistant, a docker-host VM for the necessary evil of vendor-locked containers.

​

Hardware footprint

​

Buying used enterprise gear

Most of the rack is second-hand, so every arrival is audited against its listing before it is trusted. The BMC gives that away for free over Redfish and the vendor CLI, and the useful checks are rarely the advertised specs:

• Remaining write endurance on every SSD. Used enterprise drives are sold on the life left in them, and a listing almost never quotes it.
• Repeat counts in the system event log, not just whether an error is present. A recurring fault over months is a different machine from one that logged something once.
• The real-time clock. A BMC reporting a date near the epoch means a dead CMOS battery, whatever the seller claimed about replacing it.
• Line input per power supply. A Critical health rollup is very often one uncorded supply rather than a hardware fault.
• Prior-owner remnants. Hostnames and retained logs survive a drive-level secure erase, so their presence shows the erase was narrower than advertised.
• Firmware currency, before anything else. An old BMC doesn’t just lack niceties — it can drop entire management API surfaces: no memory or storage inventory, no virtual media, no remote power control. Treat a firmware update as required onboarding, not optional hygiene.
• The chassis’s CPU thermal ceiling, not the socket’s. Entry-tier 2U chassis often share a cooling design with smaller siblings and cap TDP well below what the socket supports. A bigger heatsink doesn’t raise that ceiling — it only changes which drive-count and ambient-temperature combinations are certified. Check the ceiling, and whether a newer CPU generation needs a BIOS update first, before buying an upgrade chip.
• Warranty matters most on drives and power supplies, least on CPUs. Server CPUs rarely fail once burned in, and datacenters retire previous-generation chips in bulk — that’s where the price-per-performance is. Put the warranty budget toward drives and PSUs instead, and burn in every drive before trusting it.

Firmware age has a second, sharper edge: the upgrade path itself can be long and mandatory, since a vendor often gates later versions behind a minimum earlier version. Read the release notes for the full prerequisite chain, not just the latest release, before scheduling the maintenance window — a single update can turn out to require several sequential ones first. Don’t trust a used server’s reported hardware inventory until POST has completed cleanly at least once. Hardware inventory is collected during POST, so a BMC that never sees a clean POST can keep serving a stale snapshot — identical across every management API and unchanged by a BMC reset or a cold power cycle — with nothing to flag it except its own collection timestamp. Check that timestamp before trusting any drive or memory listing from a used machine’s BMC; a stale one means everything reported about installed hardware is potentially fiction. A stalled POST is often diagnostic, not electrical: a single unhealthy component — a failing drive, a RAID controller flagging it in UEFI — can halt POST at an interactive prompt with no visible symptom on a headless machine, and firmware jobs queued behind that prompt never get a window to run. Modern BMCs can export a screenshot of the console over the management API, turning an unexplained hang into a one-line diagnosis — a capability the oldest firmware generations lack. Findings become tickets and follow-up tasks rather than remembered caveats, and the recorded inventory always reflects what the hardware reported — never what the listing claimed.

​

Network topology

The UniFi gateway sits at the centre of the LAN; the Proxmox cluster, personal devices, and the bare-metal LLM box all hang off it. WireGuard tunnels traverse the Internet → UniFi edge. Per-service VLANs encode workload tier. Diagram and the network-as-code that defines it: tofu-unifi.

​

Data flow

UniFi gear and host telemetry feed HAProxy → Cribl Edge → Splunk → AWS DR. Full log and NetFlow pipelines: Data pipelines.

​

Container philosophy

LXC is the default for production homelab services; Docker is the exception, fenced off to a dedicated docker-host VM whenever a vendor ships Docker-only images. The four-question decision tree: LXC vs Docker.

​

What runs where

Most workloads run as LXC on the Proxmox cluster — HAProxy, Cribl Edge, Home Assistant, Qdrant. Splunk Enterprise gets a bare-metal-ish VM for network volume. Docker is fenced off to a single docker-host VM. Local LLM inference runs bare-metal on NixOS to dodge passthrough overhead. Full per-workload inventory: Infrastructure overview.

​

Provisioning + configuration

tofu-proxmox builds VMs and LXCs. ansible-proxmox configures the host. ansible-proxmox-apps layers the apps on top. The macOS counterpart that runs the monitoring stack as Kubernetes is orbstack-kubernetes.

​

High availability

Core infrastructure services (DNS, secrets, ingress) run as redundant pairs across separate Proxmox nodes, managed by Proxmox HA: a node failure triggers automatic restart of the affected service elsewhere in the cluster, no manual intervention required.

​

DR plan

tofu-aws defines a cold AWS footprint sized to take a Splunk failover. Cribl Edge routes can be flipped to the AWS HEC endpoint via config change. Details: tofu-splunk-aws.