Post

Finally Understood Why Self-Hosting Felt Hard

Posted
By Usman Masood Ashraf
views 11 min read
Finally Understood Why Self-Hosting Felt Hard

Finally Understood Why Self-Hosting Felt Hard

Introduction

Self-hosting has become increasingly popular among DevOps engineers, sysadmins, and tech enthusiasts who want control over their data and services. Yet many people find themselves overwhelmed, frustrated, and ultimately giving up on their self-hosting journey. I was one of those people who struggled for years, trying to set up various services like Immich, Nextcloud, and Plex, only to face endless configuration issues, networking problems, and compatibility headaches.

The realization that changed everything for me wasn’t about mastering a specific tool or technology—it was understanding that the true complexity of self-hosting lies in the foundational infrastructure rather than the applications themselves. This comprehensive guide will walk you through the exact insights that transformed my approach to self-hosting, helping you build a robust, scalable, and maintainable homelab that actually works.

Whether you’re a seasoned DevOps professional or someone just starting their self-hosting journey, this guide will provide you with the architectural understanding and practical knowledge needed to overcome the common pitfalls that make self-hosting feel so difficult. We’ll cover everything from networking fundamentals to container orchestration, security best practices, and operational excellence.

Understanding the True Complexity of Self-Hosting

The initial attraction to self-hosting often comes from the desire to run specific applications like Immich for photo management, Nextcloud for file storage, or Home Assistant for smart home automation. However, this application-centric approach quickly reveals its limitations when you encounter issues like:

  • Services that work perfectly on your local network but fail when accessed remotely
  • Applications that refuse to communicate with each other despite being on the same machine
  • Security vulnerabilities that leave your data exposed
  • Performance degradation as you add more services
  • Configuration files that become increasingly complex and unmanageable

The fundamental issue is that self-hosting isn’t just about running applications—it’s about building and maintaining an entire infrastructure stack. This includes networking, security, storage, monitoring, backups, and orchestration, all of which need to work together seamlessly.

The Infrastructure-First Mindset

The shift from an application-centric to an infrastructure-first mindset is what finally made self-hosting manageable for me. Instead of trying to install Immich directly, I started by asking: “What infrastructure do I need to support this application and the ones that will follow?”

This approach led me to design a modular, scalable infrastructure that could support dozens of services without becoming unmanageable. The key insight was that once you have solid infrastructure foundations, adding new services becomes trivial—you’re essentially just deploying another container with minimal configuration.

Common Misconceptions About Self-Hosting

Many people approach self-hosting with misconceptions that lead to frustration:

  1. The “Easy Setup” Myth: Many applications advertise “one-click” or “simple setup” installations, but these rarely work in complex home network environments.

  2. The “I’ll Figure It Out” Approach: Diving into installations without understanding the underlying infrastructure leads to a patchwork of solutions that don’t work well together.

  3. The “Free Software” Expectation: While the software itself may be free, the time and expertise required to properly set up and maintain a self-hosted environment has significant costs.

  4. The “I Can Do It All Myself” Syndrome: Trying to handle every aspect of infrastructure, security, and operations without leveraging existing tools and best practices.

Understanding these misconceptions helps you approach self-hosting with realistic expectations and a proper strategy.

Prerequisites for Successful Self-Hosting

Before diving into specific installations, it’s crucial to establish the right foundation. Here are the prerequisites that will set you up for success:

Hardware Requirements

The hardware you choose will significantly impact your self-hosting experience. While you can start small, understanding your requirements upfront prevents painful upgrades later.

Minimum Viable Setup:

  • CPU: Modern multi-core processor (Intel i5 or AMD Ryzen 5 equivalent)
  • RAM: 8GB DDR4 (16GB recommended for multiple services)
  • Storage: 256GB SSD for OS and containers, plus additional storage for data
  • Network: Gigabit Ethernet connection
  • Power: Reliable UPS for power protection

Recommended Production Setup:

  • CPU: Intel i7 or AMD Ryzen 7+ with AES-NI support
  • RAM: 32GB DDR4 (64GB for heavy workloads)
  • Storage: 1TB NVMe SSD for OS/containers, multiple HDDs/SSDs for data storage (RAID configuration recommended)
  • Network: Gigabit Ethernet with dual ports for redundancy
  • Power: Enterprise-grade UPS with sufficient runtime

Software Stack Requirements

A successful self-hosting environment requires a carefully selected software stack:

Operating System:

  • Linux (Ubuntu Server LTS 22.04+ recommended for stability)
  • Docker Engine 20.10+ with Docker Compose v2.20+
  • Containerd as the container runtime
  • ZFS or Btrfs for storage management (optional but recommended)

Network Services:

  • WireGuard or OpenVPN for secure remote access
  • Traefik or Nginx for reverse proxying
  • DNS server (CoreDNS or dnsmasq)
  • Firewall management (UFW or iptables)

Monitoring and Management:

  • Prometheus + Grafana for monitoring
  • Portainer or Docker Compose UI for management
  • Automated backup solutions
  • Log aggregation (ELK stack or similar)

Network Infrastructure Considerations

Your network setup is critical for self-hosting success. Consider these factors:

Network Segmentation:

  • Separate VLANs for different service types (IoT, personal data, public services)
  • Proper firewall rules between segments
  • Guest network isolation for visitors

Static IP Addressing:

  • Reserve static IPs for your self-hosted services
  • Configure proper DNS records for internal resolution
  • Set up dynamic DNS for external access if needed

Port Forwarding and DMZ:

  • Carefully configure port forwarding for external access
  • Consider using a DMZ for services exposed to the internet
  • Implement proper firewall rules to limit exposure

Security Foundation

Security should be built into your infrastructure from the start, not added as an afterthought:

Authentication and Authorization:

  • Implement centralized authentication (LDAP or similar)
  • Use role-based access control (RBAC)
  • Enable two-factor authentication where possible

Encryption:

  • Use HTTPS everywhere with valid certificates (Let’s Encrypt)
  • Encrypt sensitive data at rest
  • Implement secure communication between services

Backup and Disaster Recovery:

  • Regular automated backups of all critical data
  • Off-site backup storage for disaster recovery
  • Documented recovery procedures

Building Your Self-Hosting Infrastructure

With the prerequisites established, let’s build a solid infrastructure foundation. This section covers the core components that will make adding new services straightforward.

Network Configuration with WireGuard

WireGuard has revolutionized secure networking for self-hosting. Here’s how to set it up properly:

1
2
3
4
5
6
7
8
9
10

# Install WireGuard on Ubuntu/Debian
sudo apt update
sudo apt install wireguard

# Generate server keys
wg genkey | sudo tee /etc/wireguard/server-private.key
wg genkey | sudo tee /etc/wireguard/server-public.key

# Create the WireGuard configuration
sudo nano /etc/wireguard/wg0.conf

1
2
3
4
5
6
7
8
9
10

[Interface]
Address = 10.0.0.1/24
ListenPort = 51820
PrivateKey = <server-private-key>
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <client-public-key>
AllowedIPs = 10.0.0.2/32

Key Benefits of WireGuard:

  • Simpler configuration than OpenVPN
  • Better performance and lower latency
  • Built-in roaming support
  • Strong cryptography with modern algorithms
  • Easy to set up on mobile devices

Reverse Proxy Configuration with Traefik

A reverse proxy is essential for managing multiple services and providing HTTPS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# docker-compose.yml for Traefik
version: '3.8'

services:
  traefik:
    image: traefik:v3.0
    container_name: traefik
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.yml:/traefik.yml
      - ./acme.json:/acme.json
    networks:
      - web

networks:
  web:
    external: true

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# traefik.yml
api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

certificatesResolvers:
  letsEncrypt:
    acme:
      email: your@email.com
      storage: acme.json
      httpChallenge:
        entryPoint: http

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    watch: true
    exposedByDefault: false

Docker Compose Structure

Organizing your Docker Compose files properly prevents configuration chaos:

1
2

mkdir -p /opt/docker
cd /opt/docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

/opt/docker/
├── traefik/
│   ├── docker-compose.yml
│   ├── traefik.yml
│   └── acme.json
├── services/
│   ├── immich/
│   │   ├── docker-compose.yml
│   │   └── .env
│   ├── nextcloud/
│   │   ├── docker-compose.yml
│   │   └── .env
│   └── portainer/
│       ├── docker-compose.yml
│       └── .env
└── shared/
    ├── nginx/
    │   ├── conf.d/
    │   └── logs/
    └── data/
        ├── immich/
        ├── nextcloud/
        └── portainer/

Storage Management with ZFS

Proper storage management prevents data loss and improves performance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Install ZFS
sudo apt install zfsutils-linux

# Create ZFS pool
sudo zpool create -O compression=lz4 -O atime=off data /dev/sdb

# Create datasets for different services
sudo zfs create -o mountpoint=/opt/docker/shared/data/immich data/immich
sudo zfs create -o mountpoint=/opt/docker/shared/data/nextcloud data/nextcloud
sudo zfs create -o mountpoint=/opt/docker/shared/data/portainer data/portainer

# Set up automatic snapshots
sudo apt install zfs-auto-snapshot
sudo dpkg-reconfigure zfs-auto-snapshot

Service Configuration and Deployment

With the infrastructure in place, deploying services becomes straightforward. Let’s configure a few common services to demonstrate the approach.

Immich Photo Management

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

# /opt/docker/services/immich/docker-compose.yml
version: '3.8'

services:
  db:
    image: postgres:15-alpine
    container_name: immich-db
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - /opt/docker/shared/data/immich/postgres:/var/lib/postgresql/data
    networks:
      - immich

  redis:
    image: redis:7-alpine
    container_name: immich-redis
    restart: unless-stopped
    command: redis-server --appendonly yes
    volumes:
      - /opt/docker/shared/data/immich/redis:/data
    networks:
      - immich

  app:
    image: altran1502/immich-server:latest
    container_name: immich-app
    restart: unless-stopped
    environment:
      - DATABASE_URL=postgresql://postgres:${DB_PASSWORD}@db:5432/${DB_NAME}
      - REDIS_URL=redis://redis:6379
      - UPLOAD_PATH=/app/upload
      - CACHE_PATH=/app/cache
      - THUMBNAIL_PATH=/app/thumbnail
      - DOCKER=true
      - PORT=3001
      - LOG_LEVEL=info
    volumes:
      - /opt/docker/shared/data/immich/upload:/app/upload
      - /opt/docker/shared/data/immich/cache:/app/cache
      - /opt/docker/shared/data/immich/thumbnail:/app/thumbnail
    networks:
      - immich
    depends_on:
      - db
      - redis

  worker:
    image: altran1502/immich-server:latest
    container_name: immich-worker
    restart: unless-stopped
    environment:
      - DATABASE_URL=postgresql://postgres:${DB_PASSWORD}@db:5432/${DB_NAME}
      - REDIS_URL=redis://redis:6379
      - UPLOAD_PATH=/app/upload
      - CACHE_PATH=/app/cache
      - THUMBNAIL_PATH=/app/thumbnail
      - DOCKER=true
      - PORT=3002
      - LOG_LEVEL=info
      - WORKER=true
    volumes:
      - /opt/docker/shared/data/immich/upload:/app/upload
      - /opt/docker/shared/data/immich/cache:/app/cache
      - /opt/docker/shared/data/immich/thumbnail:/app/thumbnail
    networks:
      - immich
    depends_on:
      - db
      - redis

  nginx:
    image: altran1502/immich-nginx:latest
    container_name: immich-nginx
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - IMMICH_SERVER_PORT=3001
    volumes:
      - /opt/docker/shared/data/immich/upload:/app/upload
      - /opt/docker/shared/data/immich/cache:/app/cache
      - /opt/docker/shared/data/immich/thumbnail:/app/thumbnail
    networks:
      - immich
    depends_on:
      - app
      - worker
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.immich.rule=Host(`photos.yourdomain.com`)"
      - "traefik.http.routers.immich.tls.certresolver=letsEncrypt"
      - "traefik.http.services.immich.loadbalancer.server.port=3000"

networks:
  immich:
    driver: bridge

Nextcloud File Storage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

# /opt/docker/services/nextcloud/docker-compose.yml
version: '3.8'

services:
  db:
    image: mariadb:10.11
    container_name: nextcloud-db
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: nextcloud
      MYSQL_USER: nextcloud
      MYSQL_PASSWORD: ${NEXTCLOUD_DB_PASSWORD}
    volumes:
      - /opt/docker/shared/data/nextcloud/mysql:/var/lib/mysql
    networks:
      - nextcloud

  app:
    image: nextcloud:25-fpm-alpine
    container_name: nextcloud-app
    restart: unless-stopped
    environment:
      - MYSQL_HOST=db
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=${NEXTCLOUD_DB_PASSWORD}
      - NEXTCLOUD_DATA_DIR=/var/www/html/data
      - NEXTCLOUD_TRASHBIN_RETENTION_OBLIGATION=false
      - NEXTCLOUD_UPDATE_AUTO=true
    volumes:
      - /opt/docker/shared/data/nextcloud/html:/var/www/html
      - /opt/docker/shared/data/nextcloud/data:/var/www/html/data
      - /opt/docker/shared/data/nextcloud/config:/var/www/html/config
      - /opt/docker/shared/data/nextcloud/themes:/var/www/html/themes
      - /opt/docker/shared/data/nextcloud/apps:/var/www/html/custom_apps
    networks:
      - nextcloud
    depends_on:
      - db
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud.rule=Host(`files.yourdomain.com`)"
      - "traefik.http.routers.nextcloud.tls.certresolver=letsEncrypt"
      - "traefik.http.services.nextcloud.loadbalancer.server.port=9000"

  cron:
    image: nextcloud:25-fpm-alpine
    container_name: nextcloud-cron
    restart: unless-stopped
    volumes:
      - /opt/docker/shared/data/nextcloud/html:/var/www/html
      - /opt/docker/shared/data/nextcloud/data:/var/www/html/data
      - /opt/docker/shared/data/nextcloud/config:/var/www/html/config
      - /opt/docker/shared/data/nextcloud/themes:/var/www/html/themes
      - /opt/docker/shared/data/nextcloud/apps:/var/www/html/custom_apps
    networks:
      - nextcloud
    command: /usr/local/bin/entrypoint.sh cron
    depends_on:
      - app

  nginx:
    image: nextcloud:25-fpm-alpine
    container_name: nextcloud-nginx
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /opt/docker/shared/data/nextcloud/html:/var/www/html
      - /opt/docker/shared/data/nextcloud/data:/var/www/html/data
      - /opt/docker/shared/data/nextcloud/config:/var/www/html/config
      - /opt/docker/shared/data/nextcloud/themes:/var/www/html/themes
      - /opt/docker/shared/data/nextcloud/apps:/var/www/html/custom_apps
    networks:
      - nextcloud
    depends_on:
      - app
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud-public.rule=Host(`files.yourdomain.com`)"
      - "traefik.http.routers.nextcloud-public.tls.certresolver=letsEncrypt"
      - "traefik.http.services.nextcloud-public.loadbalancer.server.port=8080"

networks:
  nextcloud:
    driver: bridge

Advanced Configuration and Optimization

Once you have basic services running, you can optimize your infrastructure for better performance, security, and manageability.

Security Hardening

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# Configure UFW firewall
sudo apt install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow from 10.0.0.0/24 to any port 22
sudo ufw enable

# Create a non-root user for Docker operations
sudo groupadd docker
sudo usermod -aG docker $USER

# Configure fail2ban for SSH protection
sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Performance Optimization

1
2

# Enhanced Docker daemon configuration
sudo nano /etc/docker/daemon.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 64000,
      "Soft": 64000
    }
  },
  "experimental": false
Open Source, Reddit Guides, Docker
homelab sysadmin how-to tools docker containers monitoring observability
This post is licensed under CC BY 4.0 by the author.