Just Found Out We Had 200 Shadow Apis After Getting Pwned

Posted Sep 19, 2025

By Usman Masood Ashraf

views 6 min read

Just Found Out We Had 200 Shadow APIs After Getting Pwned

Introduction

The DevOps engineer stared at the forensic report in disbelief: 218 undocumented API endpoints in production. One unauthenticated /test-data-dump endpoint, created for debugging six months prior, had just leaked 14TB of customer data. The red team’s report read: “Attack path initiated via an unregistered GraphQL playground at internalapi.yourcompany.com/v3/explorer.”

This isn’t fiction - it’s the reality for organizations ignoring Shadow API risks. In this technical deep dive, we’ll dissect how undocumented endpoints become attack vectors and why traditional security tooling fails to detect them. We’ll build a battle-tested framework for:

Continuous API inventory management
Automated drift detection between deployed and documented APIs
Security controls that work in dynamic cloud environments

For DevOps teams managing Kubernetes clusters, serverless architectures, or hybrid cloud deployments, Shadow APIs represent the perfect storm of operational debt and security risk. They emerge from:

Developer shortcuts (temporary endpoints that become permanent)
Documentation drift (APIs updated without Swagger/OpenAPI syncs)
Third-party service integrations (webhooks, SaaS callbacks)
Legacy systems with forgotten entry points

What You’ll Learn:

How to implement real-time API discovery using service meshes and eBPF
Hardening techniques for API gateways (Kong, Traefik, Istio)
Automating OpenAPI specification enforcement in CI/CD pipelines
Detecting malicious API activity through behavioral analysis
Building a zero-trust API architecture with SPIFFE/SPIRE

Let’s turn your API sprawl from a security liability into a governed asset.

Understanding Shadow APIs

What Are Shadow APIs?

Shadow APIs are undocumented application programming interfaces that exist in production environments without security or operations team knowledge. They typically emerge from:

Developer experimentation (“I’ll just create a quick endpoint for testing”)
Documentation neglect (code changes without OpenAPI updates)
Zombie endpoints (deprecated but still deployed APIs)
Third-party service integrations (webhooks, callbacks, SaaS connections)

Why Traditional Security Tools Fail

The Reddit user’s “fancy API security scanner” likely failed because:

Traditional Scanner Workflow:
1. Import OpenAPI/Swagger documentation
2. Test documented endpoints for vulnerabilities
3. Generate compliance reports

Shadow API Blindspot:
- No documentation → No scanner coverage

Modern attackers use techniques like:

Parameter brute-forcing: POST /api/{version}/users → POST /v1/users, /v2/users, /beta/users
HTTP method cycling: Testing GET on POST endpoints, PUT on DELETE routes
GraphQL introspection: Exploiting __schema queries to map hidden fields

Real-World Impact Metrics

Source: 2023 API Security Trends Report, Salt Security

The Kubernetes Connection

In containerized environments, Shadow APIs proliferate due to:

Ephemeral containers bypassing API gateways
Ingress misconfigurations exposing internal services
Helm charts deploying undocumented endpoints

  
# Find Kubernetes services with unauthenticated ports
kubectl get svc -o json | jq '.items[] | select(.spec.ports[].name == "http-unauthenticated") | .metadata.name'

Prerequisites for Shadow API Management

System Requirements

Network Visibility:
- eBPF-enabled kernel (Linux 5.8+)
- Packet mirroring (SPAN ports) or CNI plugins (Cilium, Calico)
Service Mesh: Istio 1.16+, Linkerd 2.12+, or Consul 1.15+
API Gateway: Kong 3.4+, Traefik 3.0+, or Gloo Edge 1.14+

Security Considerations

Least Privilege Service Accounts:

  
# Kubernetes ServiceAccount with minimal permissions
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-auditor
automountServiceAccountToken: false

Encrypted Traffic Analysis:
- MITM decryption policies (with legal approval)
- TLS 1.3 with ephemeral keys
- Certificate transparency logs monitoring

Pre-Installation Checklist

Map all API ingress points (ALBs, NGINX, Cloudflare)
Inventory documented endpoints (Swagger/OpenAPI files)
Establish API change management policy
Deploy HTTP logging infrastructure (Elasticsearch, Grafana Loki)
Configure network time synchronization (NTP/PTP)

Installation & Real-Time API Discovery

Step 1: Deploy Cilium for eBPF Monitoring

  
helm install cilium cilium/cilium \
  --version 1.14.3 \
  --namespace kube-system \
  --set egressGateway.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Verify API endpoint discovery:

cilium status --all-endpoints

Step 2: Configure Kong API Gateway Enforcement

  
# kong.yaml
services:
- name: api-inventory
  url: http://api-auditor:7676
  routes:
  - name: enforce-documentation
    paths: ["/"]
    plugins:
    - name: openapi-validation
      config:
        specification: /etc/kong/openapi.yaml
        strict_validation: true

Step 3: Automated OpenAPI Sync

  
# openapi-sync.py
import requests
from pykson import Pykson

def sync_endpoints(gateway_url, openapi_path):
    live_apis = requests.get(f"{gateway_url}/metrics").json()
    documented_apis = Pykson().from_json(openapi_path)
    
    for endpoint in live_apis['endpoints']:
        if endpoint not in documented_apis['paths']:
            # Auto-quarantine undocumented endpoints
            requests.post(f"{gateway_url}/quarantine", 
                         json={"endpoint": endpoint})

Configuration & Zero-Trust Enforcement

API Gateway Hardening

Istio Authorization Policies

  
# shadow-api-blocker.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: block-undocumented
spec:
  action: DENY
  rules:
  - to:
    - operation:
        notHosts: ["*.company.com"]
        notPaths: ["/v1/*", "/v2/*"]

Automated Drift Detection

  
# Compare live APIs vs documentation
diff <(curl -s http://api-gateway/metrics | jq '.endpoints') <(jq '.paths' openapi.yaml)

Operational Monitoring & Threat Detection

Suspicious API Activity Indicators

High Entropy Endpoints:
- /v1/user/7xq91z (UUID-like patterns)
- /debug/pprof (debugging endpoints)
Unusual HTTP Methods:
- GET on POST-only routes
- HEAD requests to sensitive endpoints
Abnormal Response Sizes:
- 2MB responses from user profile endpoints

Grafana Alert Rule Example

  
{
  "alert": "ShadowAPIDetected",
  "expr": "sum(rate(http_requests_total{route=~\"undefined\"}[5m])) > 0",
  "for": "10m",
  "annotations": {
    "summary": "Undocumented API endpoint detected"
  }
}

Troubleshooting Guide

Common Issues & Solutions

Problem: False positives in API discovery
Fix: Refine detection filters

  
# cilium-config.yaml
api-discovery:
  ignore-patterns:
  - "^/healthz$"
  - "^/metrics$"

Problem: Legacy systems can’t generate OpenAPI specs
Fix: Deploy API transcription proxy

  
docker run -d --name api-transcriber \
  -v $(pwd)/specs:/specs \
  apitranscriber/v1.8:latest \
  --target http://legacy-app:8080 \
  --output /specs/legacy-openapi.yaml

Problem: Developers bypassing API gateway
Fix: NetworkPolicy enforcement

  
# block-direct-pod-access.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: api-gateway-only
spec:
  podSelector:
    matchLabels:
      app: backend-service
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway

Conclusion

The era of API sprawl demands a paradigm shift:

Inventory Before Protection: You can’t secure what you can’t see
Shift Left Documentation: OpenAPI specs as code review requirements
Runtime Enforcement: Gateways that block undocumented traffic
Behavioral Baselining: AI/ML detection of anomalous endpoints

Continue your API security journey with:

In the trenches of modern DevOps, Shadow APIs aren’t just technical debt—they’re live grenades in your infrastructure. Treat them accordingly.

Open Source, Reddit Guides, Kubernetes

This post is licensed under CC BY 4.0 by the author.