Homelab

The goal: fault-tolerant infrastructure that rebuilds from a single nix build.

The homelab is a real production environment, just for one person. Proxmox cluster on bare metal, UniFi networking, Splunk indexers, and Cribl Edge collectors keep the core running. Home Assistant and a docker-host VM cover the necessary evil of vendor-locked containers.

​

Hardware footprint

​

Buying used enterprise gear

Most of the rack is second-hand, so every arrival is audited against its listing before it is trusted. The BMC gives that away for free over Redfish and the vendor command-line tool, and the useful checks are rarely the advertised specs:

• Remaining write endurance on every SSD. Used enterprise drives are sold on the life left in them, and a listing almost never quotes it.
• Repeat counts in the system event log, not just whether an error is present. A recurring fault over months is a different machine from one that logged something once.
• The real-time clock. A BMC reporting a date near the epoch means a dead CMOS battery, whatever the seller claimed about replacing it.
• Line input per power supply. A Critical health rollup is very often one uncorded supply rather than a hardware fault.
• Prior-owner remnants. Hostnames and retained logs survive a drive-level secure erase, so their presence shows the erase was narrower than advertised.
• Firmware currency, before anything else. An old BMC doesn’t just lack niceties. It can drop entire management API surfaces: no memory or storage inventory, no virtual media, no remote power control. Treat a firmware update as required onboarding, not optional hygiene.
• The chassis’s CPU thermal ceiling, not the socket’s. Entry-tier 2U chassis often share a cooling design with smaller siblings and cap TDP well below what the socket supports. A bigger heatsink doesn’t raise that ceiling. It only changes which drive-count and ambient-temperature combinations are certified. Check the ceiling, and whether a newer CPU generation needs a BIOS update first, before buying an upgrade chip.
• Warranty matters most on drives and power supplies, least on CPUs. Server CPUs rarely fail once burned in, and datacenters retire previous-generation chips in bulk. That’s where the price-per-performance is. Put the warranty budget toward drives and PSUs instead, and burn in every drive before trusting it.

Firmware age has a second, sharper edge. The upgrade path itself can be long and mandatory, since a vendor often gates later versions behind a minimum earlier version. Read the release notes for the full prerequisite chain, not just the latest release, before scheduling the maintenance window. A single update can turn out to require several sequential ones first. Don’t trust a used server’s reported hardware inventory until POST has completed cleanly at least once. Hardware inventory is collected during POST. A BMC that never sees a clean POST can keep serving a stale snapshot. That snapshot looks identical across every management API and survives a BMC reset or a cold power cycle. Nothing flags it except its own collection timestamp. Check that timestamp before trusting any drive or memory listing from a used machine’s BMC. A stale one means everything reported about installed hardware is potentially fiction. A stalled POST is often diagnostic, not electrical. A single unhealthy component can cause it, such as a failing drive or a RAID controller flagging an issue in UEFI. That component can halt POST at an interactive prompt with no visible symptom on a headless machine. Firmware jobs queued behind that prompt never get a window to run. Modern BMCs can export a screenshot of the console over the management API, turning an unexplained hang into a one-line diagnosis. The oldest firmware generations lack that capability. Findings become tickets and follow-up tasks rather than remembered caveats. The recorded inventory always reflects what the hardware reported, never what the listing claimed.

​

Network topology

The UniFi gateway sits at the center of the LAN; the Proxmox cluster, personal devices, and the bare-metal LLM box all hang off it. WireGuard tunnels traverse the Internet → UniFi edge. Per-service VLANs encode workload tier. Diagram and the network-as-code that defines it: tofu-unifi.

​

Data flow

UniFi gear and host telemetry feed HAProxy → Cribl Edge → Splunk → AWS DR. See Data pipelines for the full log and NetFlow pipelines.

​

Container philosophy

LXC is the default for production homelab services; Docker is the exception, fenced off to a dedicated docker-host VM whenever a vendor ships Docker-only images. The four-question decision tree: LXC vs Docker.

​

What runs where

Most workloads run as LXC on the Proxmox cluster, including HAProxy, Cribl Edge, Home Assistant, and Qdrant. Splunk Enterprise gets a bare-metal-ish VM for network volume. Docker is fenced off to a single docker-host VM. Local LLM inference runs bare-metal on NixOS to dodge passthrough overhead. Full per-workload inventory: Infrastructure overview.

​

Provisioning + configuration

tofu-proxmox builds VMs and LXCs. ansible-proxmox configures the host. ansible-proxmox-apps layers the apps on top. The macOS counterpart that runs the monitoring stack as Kubernetes is orbstack-kubernetes.

​

High availability

Core infrastructure services (DNS, secrets, ingress) run as redundant pairs across separate Proxmox nodes, managed by Proxmox HA. A node failure triggers automatic restart of the affected service elsewhere in the cluster, with no manual intervention required.

​

DR plan

tofu-aws (private) defines a cold AWS footprint sized to take a Splunk failover. Cribl Edge routes can be flipped to the AWS HEC endpoint via config change. Details: tofu-splunk-aws.