How Do I Determine Which Of These Are Still Good

How Do I Determine Which Of These Are Still Good

INTRODUCTION

In a homelab or self‑hosted environment the sheer volume of hardware, services, and configuration artifacts can quickly become overwhelming. You may inherit a rack of aging UPS units, legacy network switches, or a swarm of Docker containers that have accumulated over months of testing. The core challenge is figuring out which of these assets are still viable and which should be retired, repurposed, or replaced.

This question is more than a cursory “does it power on?” check. It involves systematic evaluation, data‑driven decision‑making, and a clear understanding of the underlying technology stack. For DevOps engineers and system administrators, the ability to separate the wheat from the chaff translates directly into reduced operational risk, lower maintenance costs, and a more resilient infrastructure.

In this guide you will learn:

• How to assess the health and usefulness of physical hardware such as UPS units, switches, and storage arrays.
• How to evaluate the fitness of software components — particularly Docker containers — for continued operation.
• A step‑by‑step methodology that combines logs, metrics, and automated health checks.
• Best practices for documenting findings, creating retirement plans, and integrating the process into your broader DevOps workflow.

By the end of the article you will have a reusable framework that can be applied to any environment where you need to determine whether a given resource is still good.

---

UNDERSTANDING THE TOPIC

What Are We Evaluating?

When the phrase “which of these are still good” is used in a DevOps context, it usually refers to one of two categories:

1. Physical infrastructure – power supplies, networking gear, and storage devices that have been returned from a vendor or de‑commissioned site.
2. Software artifacts – containers, virtual machines, or services that have been spun up for temporary workloads and now sit idle.

Both require a similar investigative approach: gather data, compare against known baselines, and decide based on objective criteria.

Historical Context

The practice of inventory auditing dates back to early data‑center operations where physical asset tracking was manual and error‑prone. With the advent of configuration management tools like Ansible, Puppet, and Chef, engineers began automating the collection of hardware metadata. The rise of containerization introduced a new layer of complexity: containers are ephemeral, often run on shared hosts, and can accumulate without clear ownership.

Key Features of a Robust Evaluation Process

Feature	Description	Why It Matters
Automated Discovery	Use tools like nmap, snmpwalk, or Docker API calls to enumerate assets without manual scanning.	Reduces human error and speeds up inventory.
Health Metrics	Collect CPU, memory, temperature, and power‑draw statistics for hardware; docker stats, docker inspect, and container logs for software.	Provides objective indicators of fitness.
Lifecycle Awareness	Tag each asset with creation date, last usage, and purpose.	Enables data‑driven retirement decisions.
Risk Scoring	Assign a score based on failure history, age, and redundancy.	Prioritizes high‑risk items for replacement.
Documentation Integration	Store findings in a version‑controlled repository (e.g., Git).	Guarantees traceability and repeatability.

Pros and Cons

Pros

• Enables proactive replacement before failures occur.
• Optimizes datacenter utilization and reduces unnecessary power consumption.
• Improves security by removing stale containers that may harbor vulnerabilities.

Cons

• Requires upfront investment in monitoring tools.
• May reveal hidden dependencies that complicate removal.
• Needs disciplined governance to avoid ad‑hoc decision‑making.

Current State and Future Trends

Modern infrastructure management platforms — such as Prometheus combined with Grafana, or commercial solutions like Datadog — offer built‑in alerting on hardware health and container metrics. Machine‑learning models are beginning to predict failure probabilities based on historical telemetry. However, the fundamentals of a systematic audit remain unchanged: collect data, compare against thresholds, and act.

Comparison to Alternatives

Approach	Strengths	Weaknesses
Manual visual inspection	Simple, no tooling required	Time‑consuming, prone to oversight
Automated health checks (e.g., Docker healthcheck)	Scalable, repeatable	Requires proper configuration of health scripts
Third‑party CMDB solutions	Centralized view, integration with ticketing	Costly, may over‑engineer for small homelabs

Real‑World Applications

• A midsize ISP retired 30 % of its aging UPS fleet after a systematic audit revealed that many units failed to meet the 99.9 % uptime SLA.
• A DevOps team at a SaaS startup used Docker health checks to identify and purge 45 idle containers that were consuming 12 % of host CPU resources.

---

PREREQUISITES

Before embarking on an evaluation campaign, ensure that the following prerequisites are met.

System Requirements

Component	Minimum Version	Notes
Operating System	Linux 5.15+ (or Windows Server 2019)	Must support Docker Engine.
Docker Engine	24.0+	Use the docker command line interface.
Monitoring Agent	Prometheus 2.45+ or equivalent	For scraping metrics.
Network Access	Ability to reach devices via SSH or SNMP	Required for hardware discovery.

Required Software

• Docker – Install via the official repository:

# Example for Ubuntu 22.04
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
  https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

• Prometheus – Download the latest release from https://prometheus.io/download.html and extract.

• jq – JSON processor for parsing Docker API output:

sudo apt-get install -y jq

Network and Security

• Open port 2375 on the Docker daemon if you plan to query it remotely (ensure TLS if exposure is beyond a trusted network).
• Apply firewall rules that restrict SNMP queries to trusted management IPs only.

User Permissions

• The user performing the audit must belong to the docker group or have sudo privileges to run Docker commands.
• For hardware discovery, SNMP community strings must be configured with read‑only access.

Pre‑Installation Checklist

1. Verify Docker daemon is running: systemctl status docker.
2. Confirm docker version returns a client and server version.
3. Test connectivity to target devices via SSH/SNMP.
4. Ensure Prometheus can scrape the Docker metrics endpoint (http://localhost:9323/metrics).

---

INSTALLATION & SETUP

Below is a complete, step‑by‑step guide to set up an automated evaluation pipeline that can assess both physical hardware and Docker containers.

1. Deploy a Central Monitoring Stack

Create a directory for the monitoring components and place configuration files there.

mkdir -p ~/devops-audit/{prometheus,rules}
cd ~/devops-audit

Prometheus Configuration (prometheus.yml)

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'docker'
    static_configs:
      - targets: ['host.docker.internal:9323']
    metrics_path: /metrics
    scheme: http

  - job_name: 'snmp'
    static_configs:
      - targets: ['192.168.1.10', '192.168.1.11']
    metrics_path: /snmp
    scheme: udp
    snmp:
      community: public
      version: 2c

Note: Replace host.docker.internal with the actual hostname or IP where the Docker metrics exporter is running.

2. Install the Docker Metrics Exporter

The official Docker metrics exporter (prometheus/docker-exporter) exposes container health data.

docker run -d \
  --name docker-exporter \
  -p 9323:9323 \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  quay.io/prometheus/docker-exporter:v0.13.0 \
  --collectors.textfile.directory=/etc/docker-exporter \
  --collectors.enabled=container,image,network,volume

3. Set Up SNMP Exporter for Hardware

docker run -d \
  --name snmp-exporter \
  -p 9116:9116 \
  -e SNMP_COMMUNITY=public \
  -e SNMP_VERSION=2 \
  -e TARGETS="192.168.1.10,192.168.1.11" \
  prom/snmp-exporter:v0.22.0 \
  --collectors.enabled=snmp

4. Create a Health‑Check Script for Containers

Save the following script as container-health.sh and make it executable.

#!/usr/bin/env bash
# container-health.sh – evaluates a container's readiness based on Docker health status

set -euo pipefail

# Usage: ./container-health.sh <container_name_or_id>
CONTAINER_NAME="$1"

# Retrieve container JSON with all fields
CONTAINER_JSON=$(docker inspect "$CONTAINER_NAME")

# Extract status, restart count, and last exit code
STATUS=$(echo "$CONTAINER_JSON" | jq -r '.[0].State.Status')
EXIT_CODE=$(echo "$CONTAINER_JSON" | jq -r '.[0].State.ExitCode')
RESTART_COUNT=$(echo "$CONTAINER_JSON" | jq -r '.[0].State.RestartCount')

# Simple scoring logic
SCORE=0
if [[ "$STATUS" == "running" ]]; then
    SCORE=$((SCORE + 2))
else
    SCORE=$((SCORE - 2))
fi
if (( RESTART_COUNT > 5 )); then
    SCORE=$((SCORE - 1))
fi
if (( EXIT_CODE != 0 )); then
    SCORE=$((SCORE - 1))
fi

echo "Container $CONTAINER_NAME health score: $SCORE"
exit $((SCORE))

Make it executable:

chmod +x container-health.sh

5. Verify Installation

Start Prometheus and confirm that metrics are being scraped.

docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v ${PWD}/prometheus/:/etc/prometheus/ \
  prom/prometheus:v2.45.0 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus

Open a browser to http://localhost:9090/targets and verify that both docker and snmp targets show UP status.

---

CONFIGURATION & OPTIMIZATION

With the monitoring stack operational, the next phase is to fine‑tune the configuration so that the evaluation process is both accurate and efficient.

1. Define Health Thresholds

Create a file health-thresholds.yml that centralizes all score cut‑offs.

# health-thresholds.yml
container:
  running_score: 2
  max_restarts: 5
  exit_code_penalty: 1
  status_penalty: -2

Reference this file in your evaluation scripts to avoid hard‑coding values.

2. Harden Security

• Docker Daemon TLS – Generate certificates and restart the daemon with --tlsverify.
• SNMP Community Restriction – Use a unique community string that is not publicly known.

3. Performance Optimization

• Scrape Interval – Adjust global.scrape_interval in prometheus.yml based on the number of containers. For a small homelab, 30 s is sufficient.
• Resource Limits – Apply cgroups to the exporter containers to prevent them from consuming excessive CPU.

# Example cgroup configuration for docker-exporter
docker run -d \
  --name docker-exporter \
  --cpu-quota=50000 --cpu-period=100000 \
  -p 9323:9323 \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  quay.io/prometheus/docker-exporter:v0.13.0

4. Integration with Ticketing

Use the Prometheus Alertmanager to fire alerts when a container’s health score falls below a critical threshold. Example rule:

# rules/health_rules.yml
groups:
- name: container-health
  rules:
  - alert: CriticalContainerDegradation
    expr: container_health_score < 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Container {{ $labels.container }} health score is low"
      description: "The container has a health score of {{ $value }}. Consider investigation."

Load the rule file via the --rule.file flag when starting Prometheus.

---

USAGE & OPERATIONS

Now that the evaluation pipeline is in place, you can begin assessing the assets you have inherited.

1. List All Inherited Containers

docker ps -a --format "{{.Names}}" > /tmp/inherited_containers.txt

2. Run the Health‑Check Script on Each Container

while IFS= read -r name; do
    ./container-health.sh "$name" >> /tmp/health_scores.log
done < /tmp/inherited_containers.txt

The resulting log will contain lines such as:

Container old‑api-service health score: -1
Container cache‑layer health score: 3

3. Aggregate Scores and Flag Items for Review

awk '$3 < 1 {print $0}' /tmp/health_scores.log > /tmp/review_items.txt

review_items.txt now holds all