A production server is misbehaving at 3 AM. You SSH in. Now what? The engineers who stay calm here are the ones who already know the next ten commands by heart, because they have run this exact loop a hundred times before, just with the word “flag” instead of “incident.”

I spend my evenings doing Hack The Box challenges and CTF competitions. I have no plans to become a pentester. I like platform engineering. The reason I keep at it is that the skills carry straight into my day job, and the carryover is bigger than it sounds.

What does pwning a vulnerable web app have to do with running Kubernetes clusters? On the surface, nothing. Underneath, it is the same core skill: dropping into a system you do not fully understand and reconstructing what is actually happening, fast. That is understanding over trust, the thing I value most when something breaks. CTF just lets you practice it when the stakes are low and the only thing on fire is your free time.

Here is what transfers, layer by layer, starting at a single host and building up to a Kubernetes cluster.

The Mindset Shift

Drop into a CTF box and the first questions are always the same:

  • What’s running?
  • What’s vulnerable?
  • What can I access?
  • How do things connect?

Swap “vulnerable” for “broken” and that is incident response. Same questions when you SSH into a misbehaving server, when a pod keeps crashing, when you are reconstructing what an attacker touched.

The difference is reps. CTF players run this loop constantly, so the enumeration becomes muscle memory. When you do it for fun every week, you stop guessing under pressure and start gathering. The rest of this post is the toolkit that builds that habit, from filesystem to syscall to network to container.

File System Investigation

Finding Recently Modified Files

CTF use: Finding planted backdoors, modified configs, or attacker artifacts.

DevOps use: Finding what changed before a system broke, tracking config drift.

# Files modified in the last 24 hours
find / -type f -mtime -1 2>/dev/null

# Files modified in the last 10 minutes
find / -type f -mmin -10 2>/dev/null

# Files changed after a specific date
find / -type f -newermt "2026-02-01" 2>/dev/null

# Find files by size (useful for finding log bombs or large dumps)
find / -type f -size +100M 2>/dev/null

Real scenario: Deployment failed, app won’t start. What config files changed? find /etc -type f -mmin -30 shows someone edited the wrong config file 15 minutes ago.

Finding Files by Content

CTF use: Searching for passwords, keys, flags hidden in files.

DevOps use: Finding where a config value is set, locating hardcoded credentials that shouldn’t exist.

# Find files containing a string
grep -r "password" /etc 2>/dev/null
grep -r "API_KEY" /app 2>/dev/null

# Find files containing a pattern (like IP addresses)
grep -rE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /etc 2>/dev/null

# Find files with specific permissions (world-writable)
find / -perm -002 -type f 2>/dev/null

# Find SUID binaries (privilege escalation, but also config issues)
find / -perm -4000 -type f 2>/dev/null

Real scenario: “Where is that environment variable being set?” grep -r "DATABASE_URL" /etc /app ~/.* 2>/dev/null finds it in three different places with three different values.

File Metadata and Attributes

CTF use: Finding hidden files, checking file integrity, spotting tampering.

DevOps use: Debugging permission issues, understanding file states.

# Extended attributes
lsattr /etc/passwd
getfacl /var/log/secure

# File type and magic bytes
file /usr/bin/mystery-binary
file /var/log/weird.log

# Detailed stat info
stat /etc/shadow

# Immutable files (can't be modified even by root)
lsattr -a /etc/ | grep "\-i\-"

Real scenario: “Why can’t I delete this file even as root?” lsattr shows the immutable flag is set.

Process Investigation

What’s Running and Why

CTF use: Finding malicious processes, understanding what’s consuming resources, finding reverse shells.

DevOps use: Debugging resource issues, understanding application behavior, finding runaway processes.

# Detailed process tree
ps auxf
pstree -p

# What's using the most CPU/memory right now
top -bn1 | head -20
ps aux --sort=-%mem | head -10
ps aux --sort=-%cpu | head -10

# Process by name with full command line
ps aux | grep -E "[n]ginx|[a]pache"

# What process is using a specific port
lsof -i :8080
ss -tlnp | grep 8080
netstat -tlnp | grep 8080

# What files does a process have open
lsof -p <pid>
ls -la /proc/<pid>/fd/

# Process environment variables
cat /proc/<pid>/environ | tr '\0' '\n'

# Process working directory
ls -la /proc/<pid>/cwd

Real scenario: Java app is eating 100% CPU. ps aux --sort=-%cpu shows which PID. /proc/<pid>/fd/ shows it has 50,000 open file handles, which is a connection leak.

Tracing Process Activity

CTF use: Understanding what a binary does, finding hidden functionality.

DevOps use: Debugging why an app can’t connect, understanding syscall behavior.

# System calls made by a process
strace -p <pid>
strace -f -e trace=network <command>  # only network calls
strace -f -e trace=file <command>     # only file operations

# Library calls
ltrace -p <pid>

# Trace a specific command
strace -o output.txt -f ./mystery-app

# See what a process is doing in real-time
watch -n 1 "cat /proc/<pid>/io"

Real scenario: App says “connection refused” but the service is running. strace -e connect shows it’s connecting to the wrong IP, because an environment variable is misconfigured.

Network Investigation

Connection Analysis

CTF use: Finding C2 connections, exfiltration channels, pivoting opportunities.

DevOps use: Debugging connectivity issues, understanding traffic patterns, finding unauthorized connections.

# All network connections
ss -tunapl
netstat -tunapl

# Established connections only
ss -tnp state established

# Connections by process
ss -tnp | grep <pid>

# Listening services
ss -tlnp
netstat -tlnp

# Connections to a specific host
ss -tn dst 10.0.0.50

# DNS resolution trace
dig +trace example.com
host -v example.com
nslookup -debug example.com

Real scenario: Application times out connecting to database. ss -tn shows connection stuck in SYN_SENT, which points to a firewall issue or a wrong route.

Packet Capture and Analysis

CTF use: Capturing credentials, analyzing protocols, finding hidden data.

DevOps use: Debugging protocol issues, understanding what’s on the wire, verifying TLS works.

# Capture traffic on an interface
tcpdump -i eth0 -w capture.pcap

# Capture specific traffic
tcpdump -i eth0 port 443
tcpdump -i eth0 host 10.0.0.50
tcpdump -i eth0 'port 80 and host 10.0.0.50'

# Read capture with full packet content
tcpdump -r capture.pcap -A
tcpdump -r capture.pcap -X

# Quick HTTP debugging
tcpdump -i eth0 -A port 80 | grep -E "GET|POST|Host:"

# Check TLS handshake
openssl s_client -connect example.com:443 -servername example.com

Real scenario: HTTPS works from one server, not another. openssl s_client shows the broken server doesn’t trust the CA, because it is missing a root cert.

Log Analysis

Finding Patterns in Chaos

CTF use: Finding attack traces, correlating events, identifying anomalies.

DevOps use: Debugging issues, understanding error patterns, incident investigation.

# Follow logs in real-time
tail -f /var/log/syslog
journalctl -f -u myservice

# Logs since specific time
journalctl --since "2026-02-01 10:00:00"
journalctl --since "1 hour ago"

# Logs for specific service
journalctl -u nginx -n 100

# Filter by priority
journalctl -p err  # errors only
journalctl -p warning..err  # warnings and errors

# Count occurrences of patterns
grep -c "error" /var/log/app.log
awk '/error/ {count++} END {print count}' /var/log/app.log

# Extract timestamps and group by minute
awk '{print $1, $2, $3}' /var/log/app.log | cut -d: -f1,2 | uniq -c | sort -rn

# Find errors with context
grep -B5 -A5 "FATAL" /var/log/app.log

Real scenario: Intermittent 500 errors. grep -c "500" access.log per hour shows a pattern: a spike every hour at :00, which turns out to be a cron job causing issues.

Log Correlation

CTF use: Reconstructing attack timeline, understanding lateral movement.

DevOps use: Understanding cause and effect across services, incident timelines.

# Auth failures
grep "Failed password" /var/log/auth.log | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c

# Commands run by users (if auditd is enabled)
ausearch -k commands -i

# Correlate by timestamp across files
grep "2026-02-27T10:15" /var/log/*.log

# Parse structured logs (JSON)
cat app.log | jq 'select(.level == "ERROR")'
cat app.log | jq -r '[.timestamp, .message] | @tsv'

User and Authentication

User Investigation

CTF use: Finding compromised accounts, understanding privilege levels, lateral movement.

DevOps use: Auditing access, debugging permission issues, security reviews.

# Users with login shells
cat /etc/passwd | grep -v nologin | grep -v /bin/false

# Users with UID 0 (root equivalent)
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Last logins
last -n 20
lastlog

# Failed login attempts
lastb -n 20
faillog -a

# Who is logged in now
w
who

# User's groups and sudo rights
id username
sudo -l -U username

# SSH authorized keys
cat ~/.ssh/authorized_keys
find / -name "authorized_keys" 2>/dev/null

Real scenario: Unauthorized access suspected. last shows login from unusual IP. grep "Accepted" /var/log/auth.log shows they used SSH key, not password.

Disk and Storage

Disk Analysis

CTF use: Finding deleted files, understanding disk layout, recovering data.

DevOps use: Debugging disk full issues, understanding space usage.

# Disk usage by directory
du -sh /* 2>/dev/null | sort -h
du -sh /var/* 2>/dev/null | sort -h

# Find what's using space
ncdu /  # interactive

# Find large files
find / -type f -size +1G 2>/dev/null

# Deleted files still open (holding disk space)
lsof +L1

# Inode usage
df -i

# What's mounted
mount | column -t
cat /proc/mounts
findmnt

Real scenario: Disk 100% full but du says only 50% used. lsof +L1 shows deleted log files still held open by processes, so a restart frees the space.

Container-Specific Skills

Container Investigation

CTF use: Container escapes, understanding isolation, finding misconfigurations.

DevOps use: Debugging container issues, understanding container state.

# What's in a running container
docker exec -it <container> /bin/sh
kubectl exec -it <pod> -- /bin/sh

# Container processes from host
docker top <container>
ps aux | grep containerd

# Container resource usage
docker stats
kubectl top pods

# Container logs
docker logs -f <container>
kubectl logs -f <pod>

# Inspect container config
docker inspect <container>
docker inspect <container> | jq '.[0].Config.Env'
kubectl describe pod <pod>

# Container filesystem
docker diff <container>  # what changed
docker export <container> | tar -tf -  # list files

# Check capabilities
docker inspect <container> | jq '.[0].HostConfig.CapAdd'
kubectl auth can-i --list

Real scenario: Container crashes on startup. docker logs shows nothing. docker inspect reveals the wrong entrypoint set in an image override.

The Skills That Transfer

We started at a single host, walked up through processes, the network, logs, auth, disk, and finished inside a Kubernetes pod. That climb is the actual point. After months of CTF practice, the habits show up at work without me thinking about them:

  1. Systematic enumeration: I gather information before I touch anything. Guessing under pressure is how you turn one outage into two.
  2. Reading logs properly: I read the error message instead of retrying and hoping. The answer is usually right there.
  3. Understanding the full stack: I trace a problem from application to syscall to network, the same way I traced the path from web app to root on a box.
  4. Healthy paranoia: I notice what could go wrong and what is misconfigured, because a CTF trains you to look for exactly that.
  5. Speed with core tools: grep, find, awk, and sed stop being things I look up and become reflexes.

None of that requires a security job. It requires reps, and CTF is a cheap way to get them.

Getting Started

If you want to build the same toolkit, start at the shallow end:

  1. OverTheWire Bandit: Pure Linux command line practice, zero setup. Do this first.
  2. TryHackMe: Guided rooms with hints, gentler on-ramp.
  3. Hack The Box: Start with easy boxes and focus on the Linux machines.
  4. PicoCTF: Good forensics challenges when you want to go deeper.

You do not need to become a hacker. You need the investigative habit, and an evening a week is enough to build it. The next time production breaks at 3 AM, the loop will already be in your fingers.