I spend my evenings doing Hack The Box challenges and CTF competitions. Not because I want to become a pentester — I’m happy in platform engineering. But because the skills I learn there make me significantly better at my day job.

This isn’t obvious at first. What does pwning a vulnerable web app have to do with running Kubernetes clusters? More than you’d think.

Forensics and offensive security train you to think about systems differently. You learn to investigate, to trace, to understand what’s actually happening rather than what should be happening. And that mindset — plus the tooling — is exactly what you need when debugging production issues at 3 AM.

Let me show you what I mean.

The Mindset Shift

In CTF challenges, you’re dropped into an unknown system and need to figure out:

  • What’s running?
  • What’s vulnerable?
  • What can I access?
  • How do things connect?

Sound familiar? That’s exactly what you do when you SSH into a server that’s misbehaving. Or when you’re trying to understand why a pod keeps crashing. Or when you’re investigating a security incident.

The difference is that CTF players practice this constantly. They build muscle memory for investigation.

File System Investigation

Finding Recently Modified Files

CTF use: Finding planted backdoors, modified configs, or attacker artifacts.

DevOps use: Finding what changed before a system broke, tracking config drift.

# Files modified in the last 24 hours
find / -type f -mtime -1 2>/dev/null

# Files modified in the last 10 minutes
find / -type f -mmin -10 2>/dev/null

# Files changed after a specific date
find / -type f -newermt "2026-02-01" 2>/dev/null

# Find files by size (useful for finding log bombs or large dumps)
find / -type f -size +100M 2>/dev/null

Real scenario: Deployment failed, app won’t start. What config files changed? find /etc -type f -mmin -30 shows someone edited the wrong config file 15 minutes ago.

Finding Files by Content

CTF use: Searching for passwords, keys, flags hidden in files.

DevOps use: Finding where a config value is set, locating hardcoded credentials that shouldn’t exist.

# Find files containing a string
grep -r "password" /etc 2>/dev/null
grep -r "API_KEY" /app 2>/dev/null

# Find files containing a pattern (like IP addresses)
grep -rE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /etc 2>/dev/null

# Find files with specific permissions (world-writable)
find / -perm -002 -type f 2>/dev/null

# Find SUID binaries (privilege escalation, but also config issues)
find / -perm -4000 -type f 2>/dev/null

Real scenario: “Where is that environment variable being set?” grep -r "DATABASE_URL" /etc /app ~/.* 2>/dev/null finds it in three different places with three different values.

File Metadata and Attributes

CTF use: Finding hidden files, checking file integrity, spotting tampering.

DevOps use: Debugging permission issues, understanding file states.

# Extended attributes
lsattr /etc/passwd
getfacl /var/log/secure

# File type and magic bytes
file /usr/bin/mystery-binary
file /var/log/weird.log

# Detailed stat info
stat /etc/shadow

# Immutable files (can't be modified even by root)
lsattr -a /etc/ | grep "\-i\-"

Real scenario: “Why can’t I delete this file even as root?” lsattr shows the immutable flag is set.

Process Investigation

What’s Running and Why

CTF use: Finding malicious processes, understanding what’s consuming resources, finding reverse shells.

DevOps use: Debugging resource issues, understanding application behavior, finding runaway processes.

# Detailed process tree
ps auxf
pstree -p

# What's using the most CPU/memory right now
top -bn1 | head -20
ps aux --sort=-%mem | head -10
ps aux --sort=-%cpu | head -10

# Process by name with full command line
ps aux | grep -E "[n]ginx|[a]pache"

# What process is using a specific port
lsof -i :8080
ss -tlnp | grep 8080
netstat -tlnp | grep 8080

# What files does a process have open
lsof -p <pid>
ls -la /proc/<pid>/fd/

# Process environment variables
cat /proc/<pid>/environ | tr '\0' '\n'

# Process working directory
ls -la /proc/<pid>/cwd

Real scenario: Java app is eating 100% CPU. ps aux --sort=-%cpu shows which PID. /proc/<pid>/fd/ shows it has 50,000 open file handles — connection leak.

Tracing Process Activity

CTF use: Understanding what a binary does, finding hidden functionality.

DevOps use: Debugging why an app can’t connect, understanding syscall behavior.

# System calls made by a process
strace -p <pid>
strace -f -e trace=network <command>  # only network calls
strace -f -e trace=file <command>     # only file operations

# Library calls
ltrace -p <pid>

# Trace a specific command
strace -o output.txt -f ./mystery-app

# See what a process is doing in real-time
watch -n 1 "cat /proc/<pid>/io"

Real scenario: App says “connection refused” but the service is running. strace -e connect shows it’s connecting to the wrong IP — environment variable misconfigured.

Network Investigation

Connection Analysis

CTF use: Finding C2 connections, exfiltration channels, pivoting opportunities.

DevOps use: Debugging connectivity issues, understanding traffic patterns, finding unauthorized connections.

# All network connections
ss -tunapl
netstat -tunapl

# Established connections only
ss -tnp state established

# Connections by process
ss -tnp | grep <pid>

# Listening services
ss -tlnp
netstat -tlnp

# Connections to a specific host
ss -tn dst 10.0.0.50

# DNS resolution trace
dig +trace example.com
host -v example.com
nslookup -debug example.com

Real scenario: Application times out connecting to database. ss -tn shows connection stuck in SYN_SENT — firewall issue or wrong route.

Packet Capture and Analysis

CTF use: Capturing credentials, analyzing protocols, finding hidden data.

DevOps use: Debugging protocol issues, understanding what’s on the wire, verifying TLS works.

# Capture traffic on an interface
tcpdump -i eth0 -w capture.pcap

# Capture specific traffic
tcpdump -i eth0 port 443
tcpdump -i eth0 host 10.0.0.50
tcpdump -i eth0 'port 80 and host 10.0.0.50'

# Read capture with full packet content
tcpdump -r capture.pcap -A
tcpdump -r capture.pcap -X

# Quick HTTP debugging
tcpdump -i eth0 -A port 80 | grep -E "GET|POST|Host:"

# Check TLS handshake
openssl s_client -connect example.com:443 -servername example.com

Real scenario: HTTPS works from one server, not another. openssl s_client shows the broken server doesn’t trust the CA — missing root cert.

Log Analysis

Finding Patterns in Chaos

CTF use: Finding attack traces, correlating events, identifying anomalies.

DevOps use: Debugging issues, understanding error patterns, incident investigation.

# Follow logs in real-time
tail -f /var/log/syslog
journalctl -f -u myservice

# Logs since specific time
journalctl --since "2026-02-01 10:00:00"
journalctl --since "1 hour ago"

# Logs for specific service
journalctl -u nginx -n 100

# Filter by priority
journalctl -p err  # errors only
journalctl -p warning..err  # warnings and errors

# Count occurrences of patterns
grep -c "error" /var/log/app.log
awk '/error/ {count++} END {print count}' /var/log/app.log

# Extract timestamps and group by minute
awk '{print $1, $2, $3}' /var/log/app.log | cut -d: -f1,2 | uniq -c | sort -rn

# Find errors with context
grep -B5 -A5 "FATAL" /var/log/app.log

Real scenario: Intermittent 500 errors. grep -c "500" access.log per hour shows pattern — spike every hour at :00 — cron job causing issues.

Log Correlation

CTF use: Reconstructing attack timeline, understanding lateral movement.

DevOps use: Understanding cause and effect across services, incident timelines.

# Auth failures
grep "Failed password" /var/log/auth.log | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c

# Commands run by users (if auditd is enabled)
ausearch -k commands -i

# Correlate by timestamp across files
grep "2026-02-27T10:15" /var/log/*.log

# Parse structured logs (JSON)
cat app.log | jq 'select(.level == "ERROR")'
cat app.log | jq -r '[.timestamp, .message] | @tsv'

User and Authentication

User Investigation

CTF use: Finding compromised accounts, understanding privilege levels, lateral movement.

DevOps use: Auditing access, debugging permission issues, security reviews.

# Users with login shells
cat /etc/passwd | grep -v nologin | grep -v /bin/false

# Users with UID 0 (root equivalent)
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Last logins
last -n 20
lastlog

# Failed login attempts
lastb -n 20
faillog -a

# Who is logged in now
w
who

# User's groups and sudo rights
id username
sudo -l -U username

# SSH authorized keys
cat ~/.ssh/authorized_keys
find / -name "authorized_keys" 2>/dev/null

Real scenario: Unauthorized access suspected. last shows login from unusual IP. grep "Accepted" /var/log/auth.log shows they used SSH key, not password.

Disk and Storage

Disk Analysis

CTF use: Finding deleted files, understanding disk layout, recovering data.

DevOps use: Debugging disk full issues, understanding space usage.

# Disk usage by directory
du -sh /* 2>/dev/null | sort -h
du -sh /var/* 2>/dev/null | sort -h

# Find what's using space
ncdu /  # interactive

# Find large files
find / -type f -size +1G 2>/dev/null

# Deleted files still open (holding disk space)
lsof +L1

# Inode usage
df -i

# What's mounted
mount | column -t
cat /proc/mounts
findmnt

Real scenario: Disk 100% full but du says only 50% used. lsof +L1 shows deleted log files still held open by processes — restart needed.

Container-Specific Skills

Container Investigation

CTF use: Container escapes, understanding isolation, finding misconfigurations.

DevOps use: Debugging container issues, understanding container state.

# What's in a running container
docker exec -it <container> /bin/sh
kubectl exec -it <pod> -- /bin/sh

# Container processes from host
docker top <container>
ps aux | grep containerd

# Container resource usage
docker stats
kubectl top pods

# Container logs
docker logs -f <container>
kubectl logs -f <pod>

# Inspect container config
docker inspect <container>
docker inspect <container> | jq '.[0].Config.Env'
kubectl describe pod <pod>

# Container filesystem
docker diff <container>  # what changed
docker export <container> | tar -tf -  # list files

# Check capabilities
docker inspect <container> | jq '.[0].HostConfig.CapAdd'
kubectl auth can-i --list

Real scenario: Container crashes on startup. docker logs shows nothing. docker inspect reveals wrong entrypoint set in image override.

The Skills That Transfer

After months of CTF practice, I notice these habits carrying over to my day job:

  1. Systematic enumeration: I don’t guess, I gather information first
  2. Reading logs properly: I actually read error messages instead of just retrying
  3. Understanding the full stack: I trace problems from application to syscall to network
  4. Healthy paranoia: I think about what could go wrong, what’s misconfigured
  5. Speed with core tools: grep, find, awk, sed become second nature

Getting Started

If you want to develop these skills:

  1. Hack The Box: Start with easy boxes, focus on Linux machines
  2. TryHackMe: More guided, better for beginners
  3. OverTheWire Bandit: Pure Linux command line practice
  4. PicoCTF: Good for forensics challenges

You don’t need to become a hacker. But thinking like one will make you a better engineer.

Next time your production system breaks at 3 AM, you’ll be grateful you practiced these skills when the stakes were low.