I Just Got Hacked Somehow

I Just Got Hacked Somehow: A DevOps Survival Guide for Container Compromises

Introduction

You’re monitoring your homelab server when htop reveals an unexpected process: xmrig (a cryptocurrency miner) running from a Docker overlay directory. Your heart skips a beat. A .js file lurks in another Docker layer. How did this happen? You never installed these. Welcome to every sysadmin’s nightmare - container compromise.

This scenario from a Reddit user exemplifies why modern infrastructure security requires constant vigilance. Homelabs and self-hosted environments are prime targets precisely because they often lack enterprise-grade security monitoring. Attackers exploit misconfigurations to hijack resources for cryptojacking, data exfiltration, or lateral movement.

In this comprehensive technical guide, we’ll dissect this breach vector and arm you with battle-tested DevOps strategies:

1. Anatomy of Docker-based intrusions
2. Forensic investigation techniques
3. Hardening container environments
4. Real-time threat detection
5. Recovery protocols

Whether you’re managing personal projects or production systems, these infrastructure security fundamentals apply universally. Let’s transform panic into actionable defense.

Understanding Container-Based Compromises

The Attack Vector: Docker as an Attack Surface

The Reddit user’s case shows classic container escape via writable overlay2 directories. Docker’s storage driver architecture creates potential attack surfaces:

/var/lib/docker/overlay2/<layer_hash>/diff

These writable container layers become ideal malware hideouts when:

1. Containers run with excessive privileges (--privileged, excessive capabilities)
2. Images contain vulnerabilities (unpatched CVEs)
3. Exposed Docker sockets allow API access
4. Default configurations remain unchanged

Why Cryptojacking Dominates

XMRig’s prevalence stems from economics - why steal data when you can monetize CPU cycles directly? The 2023 Sysdig Cloud Threat Report found cryptojacking in 95% of compromised environments. Containers provide perfect ephemeral execution environments.

Attack Chain Reconstruction

Based on the path locations, we can theorize the intrusion path:

1. Initial Access:
   • Exposed Docker API (TCP 2375/2376)
   • Compromised application vulnerability (Log4j, Spring4Shell etc.)
   • Malicious image pull from public registry
2. Execution:
   • Download payload via curl/wget in entrypoint
   • Mount attacker-controlled volumes
   • Privilege escalation to host via SYS_ADMIN capability
3. Persistence:
   • Hidden in overlay2 directory structure
   • Cron jobs or systemd timers
   • SSH backdoors in authorized_keys
4. Detection Evasion:
   • Process name masquerading (e.g., [kworker/u:0])
   • Minimal RAM usage (fileless execution)
   • Log deletion via shred

Why Traditional Security Fails

Containers break conventional security models:

• Ephemerality: Malware disappears with container stops
• Layered Filesystems: Hidden in writable container layers
• Shared Kernel: Container escapes impact host security

Prerequisites for Container Forensics

Before investigating, prepare these essentials:

Hardware Requirements

• Storage: Minimum 20GB free space for memory/core dumps
• RAM: Sufficient for live analysis without swapping
• Network: Isolated VLAN or physical disconnect

Software Toolkit

| Tool | Purpose | Installation | |——————-|———————————-|———————————-| | docker-explorer | Container forensic imaging | pip install docker-explorer | | foremost | File carving | apt install foremost | | yara | Malware pattern matching | apt install yara | | lsns | Namespace inspection | Built into util-linux | | checksec | Security property verification | apt install checksec |

Security Preparation

1. Isolate the Host:

   iptables -P INPUT DROP
   iptables -P OUTPUT DROP
   iptables -P FORWARD DROP
   systemctl disconnect-network.service
   

2. Preserve Volatility:

   systemctl stop docker containerd
   fsfreeze --freeze /var/lib/docker
   

3. Prepare Evidence Storage:

   mkdir /evidence
   mount /dev/sdb1 /evidence  # Use external media
   

Forensic Investigation: Step-by-Step

1. Process Analysis

First, capture running processes without touching disk:

# Capture process tree
ps auxf > /evidence/process_tree.txt

# List all namespace participants
lsns -p > /evidence/namespaces.txt

# Check for hidden processes via proc discrepancy
ls /proc | sort | uniq > /evidence/proc_list.txt
diff <(awk '{print $2}' /evidence/process_tree.txt) /evidence/proc_list.txt

2. Container Inspection

Identify compromised containers using Docker’s metadata:

# List all containers (including stopped)
docker ps -a --no-trunc --format 'table $CONTAINER_ID\t$CONTAINER_NAMES\t$CONTAINER_STATUS' > /evidence/containers.txt

# Inspect suspicious container
docker inspect $CONTAINER_ID > /evidence/${CONTAINER_ID}_inspect.json

# Export filesystem
docker export $CONTAINER_ID > /evidence/${CONTAINER_ID}_fs.tar

3. Overlay2 Analysis

Locate malicious files in writable container layers:

# Find suspicious paths from Reddit case
find /var/lib/docker/overlay2 -name xmrig -type f
find /var/lib/docker/overlay2 -name '*.js' -type f

# Carve deleted files
foremost -t all -i /var/lib/docker/overlay2 -o /evidence/foremost_output

4. Timeline Construction

Establish attack chronology:

# Collect MAC times from Docker objects
find /var/lib/docker -printf "%T+ %p\n" | sort > /evidence/docker_timeline.txt

# Correlate with system logs
journalctl --since "2023-07-01" --until "2023-07-30" > /evidence/journal.log

Hardening Docker Environments

Security-Enhanced Docker Daemon

/etc/docker/daemon.json:

{
  "userns-remap": "default",
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 65535,
      "Soft": 65535
    }
  },
  "live-restore": true,
  "icc": false,
  "userland-proxy": false,
  "no-new-privileges": true
}

Container Runtime Protection

Implement these podman/docker run flags:

docker run \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --read-only \
  --tmpfs /tmp:rw,size=64m \
  --memory 512m \
  --pids-limit 100 \
  --restart on-failure:5 \
  my-secure-app

Mandatory Access Control

AppArmor profile (/etc/apparmor.d/docker-hardened):

#include <tunables/global>

profile docker-hardened flags=(attach_disconnected,mediate_deleted) {
  # Deny write to all except whitelisted paths
  deny /var/lib/docker/** w,
  allow /var/lib/docker/volumes/** w,
  
  # Prevent privilege escalation
  deny capability sys_module,
  deny capability sys_admin,
  
  # Block dangerous syscalls
  deny @{PROC}/kallsyms r,
  deny mount,
}

Apply with:

apparmor_parser -r /etc/apparmor.d/docker-hardened
docker run --security-opt "apparmor=docker-hardened" ...

Real-Time Threat Detection

Falco Rules for Cryptojacking

/etc/falco/falco_rules.local.yaml:

- rule: Detect Crypto Miners
  desc: XMRig and similar miners
  condition: >
    spawned_process and
    (proc.name in ("xmrig", "minerd", "cpuminer") or
     proc.cmdline contains "pool.minexmr.com")
  output: "Cryptocurrency miner detected (user=%user.name proc=%proc.name cmdline=%proc.cmdline)"
  priority: CRITICAL
  tags: [process, crypto_mining]

eBPF-Based Anomaly Detection

Trace suspicious container behavior with bpftrace:

# Monitor unexpected process forks
bpftrace -e 'tracepoint:syscalls:sys_enter_clone 
  /comm == "docker-containerd" && args->flags & CLONE_NEWNS/ 
  { printf("Container escape attempt by PID %d\n", pid); }'

# Detect overlay2 writes
bpftrace -e 'kprobe:ovl_create_or_link 
  { @[comm] = count(); } interval:s:5 { print(@); clear(@); }'

Recovery and Remediation

Cleanup Protocol

1. Containment:

   docker kill $(docker ps -q)
   docker rm -f $(docker ps -aq)
   systemctl stop docker
   

2. Evidence Preservation:

   cp -rp /var/lib/docker /evidence/docker-state-$(date +%s)
   

3. Host Sanitization:

   # Nuke Docker artifacts
   rm -rf /var/lib/docker/*
      
   # Rebuild packages
   apt reinstall docker-ce containerd.io
      
   # Rotate all credentials
   for user in $(cut -d: -f1 /etc/passwd); do passwd -e $user; done
   

Post-Mortem Checklist

1. Attack Vector Analysis: How did initial access occur?
2. Impact Assessment: Data accessed? Systems compromised?
3. Credential Rotation: SSH keys, API tokens, passwords
4. CVE Review: Patch all exploited vulnerabilities
5. Monitoring Gaps: Why wasn’t this detected sooner?
6. Backup Verification: Ensure recovery points are clean

Conclusion

The “I just got hacked” moment is a harsh teacher. In this case, Docker’s flexibility became its vulnerability - writable overlay directories hosting unauthorized cryptocurrency miners. Through our forensic walkthrough, we’ve demonstrated:

1. Attackers increasingly target containers for resource hijacking
2. Default Docker configurations require immediate hardening
3. Real-time monitoring with eBPF/Falco provides critical visibility
4. Recovery demands systematic evidence preservation and analysis

For continued learning:

• Docker Security Best Practices
• CIS Docker Benchmark
• eBPF for Security Observability

Security isn’t a one-time checkbox but a continuous process. Implement layered defenses, assume breach, and practice incident response drills. Your next htop surprise might just become a masterclass in detection engineering rather than a crisis.