Homelab Setup Finally Complete

Homelab Setup Finally Complete

Introduction

The phrase “Homelab Setup Finally Complete” is perhaps the greatest inside joke in the DevOps community - a beautiful oxymoron that perfectly captures the eternal nature of infrastructure refinement. If your significant other ever questions the growing pile of hardware or mysterious electricity bills, you’re now armed with the universal homelab defense: “It was only $2000 total, and most came from garage sales!”

For DevOps engineers and sysadmins, a homelab represents more than just hardware - it’s a sandbox for innovation, a risk-free environment for testing production-grade technologies, and crucially, a career accelerator. In this comprehensive guide, we’ll dissect a professional homelab implementation that balances enterprise-grade capabilities with garage-sale economics, covering:

1. Hardware selection strategies for maximum capability/minimum cost
2. Hypervisor and infrastructure-as-code implementation
3. Network architecture that mimics enterprise environments
4. Persistent storage solutions with proper redundancy
5. Automation frameworks for true DevOps workflows

Whether you’re preparing for certifications, testing deployment strategies, or building your private cloud, this guide delivers actionable insights from 15+ years of enterprise infrastructure experience compressed into a budget-conscious homelab context.

Understanding the Modern Homelab

What Exactly is a Homelab?

A homelab is a scaled-down enterprise infrastructure environment running in a residential setting, typically featuring:

• Virtualization hosts: Usually 2-3 physical nodes running Type 1 hypervisors
• Network infrastructure: Enterprise-grade switches, routers, and firewalls
• Storage systems: NAS devices or server-grade storage with RAID
• Management plane: Centralized configuration and monitoring tools

The Homelab Evolution

Homelabs have matured significantly from their origins as decommissioned workstations:

Era	Characteristics	Typical Components
2000-2010	Single-node, desktop hardware	Old PCs, consumer routers
2010-2018	Multi-node virtualization	Used servers, basic VLANs
2018-Present	Cloud-like architectures	Hyperconverged, Kubernetes, IaC

Why Professionals Invest in Homelabs

1. Technology Evaluation: Test drive new tools without enterprise bureaucracy
2. Certification Prep: Hands-on labs for RHCE, CKA, or AWS certifications
3. Disaster Recovery: Private backup destination for critical data
4. Career Development: Demonstrate tangible infrastructure skills to employers

The Garage-Scale Economics

The Reddit post’s “$2000 total” claim highlights the homelabber’s secret weapon: secondary markets. Consider these real-world price comparisons:

Component	New Retail	Used/EOL Price	Source
Dell R730xd	$5,000+	$600-800	eBay, local recyclers
Cisco SG350X-24	$1,200	$150-250	Facebook Marketplace
APC 2200VA UPS	$1,500	$200-300	Government auctions

Prerequisites

Hardware Requirements

A professional-grade homelab should handle enterprise workloads while fitting residential constraints:

Minimum Specifications:

• Hypervisor Nodes (2 required for HA):
  • CPU: 8 cores (Intel VT-d/AMD-V support required)
  • RAM: 64GB ECC DDR4
  • Storage: 2x 512GB SSD (RAID1) + HBA for expansion
  • NIC: Dual 1GbE (Intel i350 recommended)
• Network:
  • Managed switch with VLAN support
  • Separate firewall appliance (pfSense/OPNsense)
  • WAP supporting 802.11ac and VLAN tagging
• Shared Storage:
  • NAS with 4+ bays (TrueNAS compatible)
  • Dual 10GbE recommended for iSCSI/NFS

Software Requirements

Create an enterprise-grade foundation with these open-source solutions:

# Base Stack
Proxmox VE 8.x          # Hypervisor
TrueNAS Core 13.x       # Storage
OPNsense 23.x           # Firewall
Ubuntu Server 22.04 LTS # Template OS

# Management Plane
Terraform v1.5+         # Infrastructure as Code
Ansible Core 2.15+      # Configuration Management
Prometheus 2.47+        # Monitoring
Grafana 10.1+           # Visualization

Network Architecture

Implement proper segmentation from day one:

VLAN ID	Purpose	Subnet	Firewall Rules
10	Management	10.10.10.0/24	SSH/HTTPS only from trusted
20	Infrastructure	10.20.20.0/24	Internal services comms
30	Guest	10.30.30.0/24	Internet access only
40	IoT	10.40.40.0/24	Restricted outbound
50	Lab	10.50.50.0/24	No restrictions (test only)

Installation & Setup

Hypervisor Cluster Deployment

Proxmox VE forms our virtualization foundation. Install on all nodes identically:

# Download latest installer
wget https://enterprise.proxmox.com/iso/proxmox-ve_8.0-2.iso

# Create bootable USB (on Linux)
sudo dd if=proxmox-ve_8.0-2.iso of=/dev/sdX bs=4M status=progress conv=fdatasync

# Post-install configuration
apt update && apt full-upgrade -y
apt install -y git mc htop iftop jq

# Join nodes to cluster
pvecm create homelab-cluster -nodeid 100
pvecm add IP_OF_FIRST_NODE

Storage Configuration

Configure TrueNAS for shared storage:

# /etc/pve/storage.cfg (Proxmox side)
zfspool: local-zfs
	pool rpool
	content images,rootdir
	nodes node1,node2

nfs: truenas-nfs
	export /mnt/tank/proxmox
	path /mnt/pve/truenas-nfs
	server 10.10.10.10
	content backup,iso,vztmpl

Network Fabric Implementation

OPNsense provides enterprise-grade networking:

# Assign interfaces (via console)
1) WAN (igb0) -> ISP modem
2) LAN (igb1) -> Switch trunk port
3) LAB (igb2) -> Lab network direct

# Enable essential services
service dhcpd enable
service unbound enable
service sshd enable

# Configure VLAN trunking
ifconfig igb1 up
vlan create igb1 10
vlan create igb1 20
vlan create igb1 30

Configuration & Optimization

Infrastructure as Code Baseline

Initialize Terraform for declarative infrastructure:

# providers.tf
terraform {
  required_providers {
    proxmox = {
      source = "Telmate/proxmox"
      version = "2.9.14"
    }
  }
}

provider "proxmox" {
  pm_api_url = "https://10.10.10.2:8006/api2/json"
  pm_api_token_id = "terraform@pve!homelab_token"
  pm_api_token_secret = "xxxxx-xxxxx-xxxxx-xxxxx"
  pm_tls_insecure = true # For self-signed certs
}

Security Hardening

Implement defense-in-depth strategies:

1. Hypervisor Security

   # Disable root SSH
   sed -i 's/PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
      
   # Enable 2FA for Proxmox web UI
   pveum realm add radius --type radius --server 10.10.10.5 --port 1812 --secret radius_key
   

2. Network Hardening

   # OPNsense firewall rules for management VLAN
   block in on $MGMT_IF from ! 10.10.10.0/24
   pass in on $MGMT_IF proto tcp from 10.10.10.5 to any port {22 8006}
   

Performance Tuning

Optimize for residential power constraints:

# CPU Governor Settings (Proxmox)
echo 'GOVERNOR="performance"' > /etc/default/cpufrequtils

# ZFS ARC Size Limit (TrueNAS)
sysctl vfs.zfs.arc_max=8G

# NIC Offloading (Proxmox and OPNsense)
ethtool -K $ETH tx off rx off sg off tso off gso off gro off lro off

Usage & Operations

Day-to-Day Management

Essential commands for operational efficiency:

# Cluster status overview
pvecm status

# VM lifecycle management
qm start 100
qm stop 100 --skiplock
qm migrate 100 node2 --online

# Storage operations
zpool status -v
zfs list -t snapshot

# Network diagnostics
tcpdump -ni igb1 vlan 10
pfctl -sa

Monitoring Stack Implementation

Deploy Prometheus with homelab-optimized scraping:

# prometheus.yml
scrape_configs:
  - job_name: 'proxmox'
    static_configs:
      - targets: ['10.10.10.2:9221']
  - job_name: 'truenas'
    static_configs:
      - targets: ['10.10.10.10:9283']
  - job_name: 'opnsense'
    static_configs:
      - targets: ['10.10.10.1:9273']

Backup Strategy

Implement the 3-2-1 rule with Proxmox Backup Server:

# Weekly full, daily incremental schedule
proxmox-backup-client backup --repository backup01@pbs:homelab \
  --ns homelab \
  --exclude /mnt/media \
  --include-dev /etc \
  --include-dev /var/lib/important-data \
  --schedule "Mon 02:00"

Troubleshooting

Common Issues and Solutions

1. Cluster Communication Failures

   # Check corosync status
   corosync-cmapctl | grep members
      
   # Validate network routes
   mtr -rw 10.10.10.1
   

2. Storage Performance Degradation

   # Check ZFS ARC efficiency
   arc_summary.py
      
   # Identify disk bottlenecks
   zpool iostat -v 1
   

3. VM Network Connectivity Loss

   # Validate bridge configuration
   brctl show
      
   # Inspect VLAN tagging
   tcpdump -nei vmbr0
   

Debugging Methodology

1. Infrastructure Layer
   • Verify physical connections and link status
   • Check power and cooling systems
2. Network Layer

   # Spanning tree check
   tcpdump -ni igb1 stp
      
   # Firewall rule verification
   pfctl -sr | grep $RULE
   

3. Service Layer

   # Proxmox API health
   curl -k https://10.10.10.2:8006/api2/json/access/users \
     -H "Authorization: PVEAPIToken=user@realm!tokenid=secret"
   

Conclusion

While declaring a homelab “complete” remains the ultimate inside joke, this guide has demonstrated how to implement an enterprise-grade infrastructure within residential constraints. By combining used enterprise hardware with open-source software, we’ve created a platform capable of supporting:

• Private cloud environments
• CI/CD pipelines
• Kubernetes clusters
• Network security labs
• Enterprise storage solutions

The true value emerges not from the hardware itself, but from the operational experience gained managing this micro-datacenter. As you continue iterating (because we all know “completed” is temporary), consider exploring:

• GPU Passthrough: For AI/ML workloads
• 5G WAN Integration: Multi-homed internet
• Kubernetes Operators: Advanced orchestration
• Zero Trust Networking: Beyond VLAN segmentation

For continued learning, consult these essential resources:

• Proxmox VE Documentation
• TrueNAS Hardware Guide
• Terraform Proxmox Provider
• OPNsense Best Practices

Remember: In the world of homelabs, completion is not a destination but merely a checkpoint in the endless journey of infrastructure refinement. Now go forth and explain to your significant other why that 42U rack “was practically free on Craigslist.”