Runtime Security with Falco: Detect Suspicious Behavior in Your Cluster

TL;DR Trivy and Kyverno check an image before it runs. Falco watches what the container does after that, syscall by syscall, and alerts when the behaviour doesn’t match a rule. This post goes from the one thing Falco does to the config I run, with the noise problem in between.

I scanned my images with Trivy. I enforced policies with Kyverno. My workloads got cryptographic identity through SPIFFE. Three layers of prevention, all green, and for a while that felt like enough.

Then I asked myself what happens after a pod is running. My scanner checked the image that went in. My admission controller checked the spec at deploy time. Neither of them is watching once the process is executing. A zero-day that lands in a running container arrives after every one of those controls has finished its job.

That bothered me more than I’d like to admit. I can’t claim to understand what my cluster does if I’ve no idea what the processes inside it are doing. Prevention tells me what should be true. I wanted something that tells me what is true, right now.

Falco is that something. It watches the system calls every container makes, file access, network connections, processes spawning, and alerts when the behaviour doesn’t match the rules I gave it. Here’s how I learned it. The one thing it does, then a rule, then my own rules, then what I run and how much of that is tuning.

The gap prevention leaves

I think of cluster security as three moments in time. Build time, where Trivy scans the image for known vulnerabilities. Deploy time, where Kyverno and the other admission controllers check the spec. And runtime, where something has to watch what the workload does. Falco is the third moment.

Most setups stop after the first two, and mine did for a long time. They’re cheaper, they fit in the pipeline, and they make a dashboard go green. The trouble is that an attacker who’s already inside a running container doesn’t replay your pipeline. They work with what’s executing.

Here’s the kind of thing the first two moments never see:

A container suddenly executing /bin/bash in production, where nothing should ever spawn a shell
A process reading /etc/shadow
An outbound connection to a crypto mining pool
Something writing into /etc/
Kubernetes service account tokens getting read from a pod that has no business touching them

None of that shows up in a scan or an admission check, because none of it is configuration. It’s behaviour, and behaviour is the only thing Falco looks at.

One thing, before any rules

Falco sits below your containers, reads their system calls, and matches each one against a set of conditions. eBPF (or a kernel module) is how it gets that seat:

Every syscall flows through the rules engine. A match becomes an alert. That’s the whole model. The rest of Falco is detail on top of it, and once this picture is in your head the detail reads easily.

Installing it

I start with the Helm chart and the eBPF driver. eBPF means no kernel module on every node, which matters on Talos and other locked-down hosts:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true

I don’t run helm install by hand, though. Everything in my cluster goes through Git, so the real version is an ArgoCD Application pointing at the same chart:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: falco
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://falcosecurity.github.io/charts
    chart: falco
    targetRevision: 4.0.0
    helm:
      values: |
        driver:
          kind: ebpf
        falcosidekick:
          enabled: true
          config:
            slack:
              webhookurl: "https://hooks.slack.com/services/xxx"
        tty: true
  destination:
    server: https://kubernetes.default.svc
    namespace: falco
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Reading a rule

Falco is as good as its rules and no better. A rule is a small block of YAML, and once you can read one you can read all of them. Here’s the canonical example, a shell spawning inside a container:

- rule: Terminal shell in container
  desc: Detect shell being spawned in a container
  condition: >
    spawned_process and
    container and
    shell_procs and
    proc.tty != 0
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name shell=%proc.name
     parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

Four parts do the work. The condition decides when to fire, in Falco’s filter language. The output is what lands in the alert, with the % fields filled in at runtime. The priority is the severity, DEBUG up to CRITICAL. The tags group and filter, here mapping to MITRE technique categories.

Read the condition out loud and it’s nearly English. A process was spawned, inside a container, the process is a shell, and it has a TTY attached. Every other rule reuses that grammar.

The rules that ship with it

You don’t start from a blank file. Falco ships a large set of community rules that already cover the obvious attacks:

# View loaded rules
kubectl exec -n falco -it falco-xxx -- falco --list

Container escapes, privilege escalation, sensitive file reads, suspicious network activity, Kubernetes API abuse. A few of them, so you can see they read the same way as the one above:

# Detect container escape via mount
- rule: Launch Sensitive Mount Container
  condition: >
    spawned_process and container and
    sensitive_mount

# Detect reading sensitive files
- rule: Read sensitive file untrusted
  condition: >
    open_read and sensitive_files and
    not trusted_containers

# Detect crypto mining
- rule: Detect outbound connections to crypto mining pools
  condition: >
    outbound and
    fd.sip.name in (cryptomining_pool_domains)

Rules that know your app

The defaults catch generic attacks. They’ve no idea what your application is supposed to do. That knowledge is yours, and writing it down is where Falco gets interesting:

# custom-rules.yaml
customRules:
  rules-custom.yaml: |-
    # Alert when our API server spawns unexpected processes
    - rule: Unexpected process in api-server
      desc: Detect processes other than the main app in api-server containers
      condition: >
        spawned_process and
        container.image.repository contains "api-server" and
        not proc.name in (api-server, node, npm)
      output: >
        Unexpected process in api-server
        (command=%proc.cmdline container=%container.name image=%container.image.repository)
      priority: WARNING

    # Alert on database connection from unexpected pods
    - rule: Database connection from non-backend pod
      desc: Detect connections to PostgreSQL from pods that shouldn't connect
      condition: >
        outbound and
        fd.sport = 5432 and
        not k8s.pod.label.app in (api-server, worker, migration-job)
      output: >
        Unexpected database connection
        (pod=%k8s.pod.name namespace=%k8s.ns.name dest=%fd.sip)
      priority: ERROR

Custom rules go in through Helm values:

# values.yaml
customRules:
  rules-custom.yaml: |-
    - rule: My custom rule
      ...

The filter language

I don’t like copying YAML I can’t read, and I’d rather you didn’t either. There’s a handful of field classes, plus macros and lists to glue them together. Know those and every rule above stops being a black box.

Fields

# Process fields
proc.name         # Process name
proc.cmdline      # Full command line
proc.pname        # Parent process name
proc.exepath      # Executable path

# Container fields
container.name    # Container name
container.id      # Container ID
container.image.repository  # Image name

# File fields
fd.name           # File descriptor name (file path)
fd.directory      # Directory of the file

# Network fields
fd.sip            # Server IP
fd.sport          # Server port
fd.cip            # Client IP

# Kubernetes fields
k8s.pod.name      # Pod name
k8s.ns.name       # Namespace
k8s.pod.label.app # Pod label

Macros

Macros are named condition fragments you reuse. Every rule so far leaned on these without saying so:

- macro: container
  condition: container.id != host

- macro: shell_procs
  condition: proc.name in (bash, sh, zsh, dash, ksh)

- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

- macro: open_read
  condition: evt.type in (open, openat) and evt.is_open_read = true

Lists

Lists are named sets of values, referenced instead of repeated:

- list: sensitive_files
  items: [/etc/shadow, /etc/sudoers, /etc/pam.d, /root/.ssh]

- list: package_managers
  items: [apt, apt-get, yum, dnf, apk, pip, npm]

- list: shell_binaries
  items: [bash, sh, zsh, csh, tcsh, ksh, dash]

Three rules worth their alert

With the grammar in hand, here are the three detections I find worth the noise. They map to what an attacker does once they’re in.

Reverse shells

- rule: Reverse shell detected
  desc: Detect reverse shells via common patterns
  condition: >
    spawned_process and
    container and
    ((proc.name = bash and proc.args contains ">&" and proc.args contains "/dev/tcp") or
     (proc.name = nc and proc.args contains "-e") or
     (proc.name = python and proc.args contains "socket" and proc.args contains "subprocess"))
  output: >
    Reverse shell detected (user=%user.name command=%proc.cmdline container=%container.name)
  priority: CRITICAL
  tags: [mitre_execution, reverse_shell]

Kubernetes secret access

- rule: K8s secret accessed from unexpected namespace
  desc: Detect when secrets are read from non-standard locations
  condition: >
    open_read and
    fd.name startswith "/var/run/secrets/kubernetes.io" and
    not k8s.ns.name in (kube-system, monitoring, vault)
  output: >
    K8s secret accessed (file=%fd.name pod=%k8s.pod.name namespace=%k8s.ns.name)
  priority: WARNING

Package installs at runtime

- rule: Package manager in container
  desc: Detect package installations in running containers
  condition: >
    spawned_process and
    container and
    proc.name in (apt, apt-get, yum, dnf, apk, pip, npm) and
    not container.image.repository in (allowed_builder_images)
  output: >
    Package manager run in container
    (command=%proc.cmdline container=%container.name image=%container.image.repository)
  priority: ERROR
  tags: [mitre_persistence, package_install]

A package manager starting up in a running container means someone is building a foothold, or at best a developer debugging in production. Either way I want to know, so it’s an ERROR and not a warning.

Getting the alerts somewhere you’ll see them

Detecting something is pointless if the alert dies in a pod’s stdout, and by default that’s where Falco writes. Falcosidekick fans the alerts out to wherever you look:

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/xxx"
      minimumpriority: warning

    prometheus:
      enabled: true

    elasticsearch:
      hostport: "elasticsearch.logging:9200"
      index: "falco"

    alertmanager:
      hostport: "http://alertmanager.monitoring:9093"

    webhook:
      address: "http://security-responder.security:8080/falco"

That config sends warnings and up to Slack, every alert into Prometheus as a metric, everything into Elasticsearch for history, criticals to Alertmanager for paging, and the full stream to a webhook for automated response. One detection, and each destination gets the severity it cares about.

Folding Falco into Prometheus

I already run Prometheus and Grafana for everything else, so I want security events where I watch the rest of the cluster. Falcosidekick exposes metrics, and a ServiceMonitor scrapes them:

# ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: falco-sidekick
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: falcosidekick
  endpoints:
    - port: http

From there a PrometheusRule turns a Falco detection into the same kind of alert as a full disk or a hot CPU:

# PrometheusRule
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: falco-alerts
spec:
  groups:
    - name: falco
      rules:
        - alert: FalcoCriticalAlert
          expr: increase(falco_events{priority="Critical"}[5m]) > 0
          for: 0m
          labels:
            severity: critical
          annotations:
            summary: "Falco critical security event detected"
            description: "{{ $labels.rule }} triggered in {{ $labels.k8s_ns_name }}/{{ $labels.k8s_pod_name }}"

The first week is noise

Here’s the trade-off nobody puts on the landing page. Out of the box, the default rules page you for things that are normal in your cluster. The first week I ran Falco, my own config-reload sidecar woke me up. A tool you’ve learned to ignore is a tool you’ve switched off without noticing, so tuning isn’t a step after the install. It’s most of the work.

Two moves. Switch off rules that don’t fit, and narrow the macros the rest depend on.

Disabling and overriding rules

# values.yaml
falco:
  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml  # Overrides
    - /etc/falco/rules.d  # Custom rules

customRules:
  falco_rules.local.yaml: |-
    # Disable noisy rule
    - rule: Terminal shell in container
      enabled: false

    # Modify existing rule to exclude certain containers
    - rule: Read sensitive file untrusted
      condition: >
        open_read and sensitive_files and
        not trusted_containers and
        not container.image.repository in (my-trusted-app)

Tuning macros

customRules:
  falco_rules.local.yaml: |-
    # Add to trusted containers
    - macro: trusted_containers
      condition: >
        (container.image.repository in (
          my-company/trusted-app,
          my-company/another-app
        ))

    # Exclude namespaces from monitoring
    - macro: user_namespace
      condition: >
        k8s.ns.name not in (kube-system, falco, monitoring)

Editing trusted_containers beats disabling the rule, because the rule still protects every other container. You’re teaching Falco what normal looks like here instead of blinding it.

What I run

Everything stacked together. The eBPF driver, JSON output, tiered alerting, and two rules that say what my own workloads may do. Compare it with the helm install near the top and you can see how much of it is tuning:

driver:
  kind: ebpf
  ebpf:
    hostNetwork: true

falco:
  grpc:
    enabled: true
  grpc_output:
    enabled: true
  json_output: true
  log_level: info

  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml
    - /etc/falco/rules.d

falcosidekick:
  enabled: true
  replicaCount: 2
  config:
    slack:
      webhookurl: "${SLACK_WEBHOOK}"
      minimumpriority: error

    prometheus:
      enabled: true

    alertmanager:
      hostport: "http://alertmanager.monitoring:9093"
      minimumpriority: critical

customRules:
  rules-custom.yaml: |-
    # Our application-specific rules
    - rule: Unexpected outbound connection from backend
      desc: Backend pods should only connect to known services
      condition: >
        outbound and
        k8s.pod.label.app = "backend" and
        not fd.sip.name in (postgres.db, redis.cache, api.internal)
      output: >
        Backend made unexpected outbound connection
        (pod=%k8s.pod.name dest=%fd.sip:%fd.sport)
      priority: WARNING

  falco_rules.local.yaml: |-
    # Tune default rules for our environment
    - macro: user_known_write_etc_conditions
      condition: >
        (container.image.repository = "my-company/config-manager")

Why each choice. eBPF over the kernel module because it performs better and keeps the host cleaner. JSON output because downstream tools parse it without regex. Tiered alerting so an error nudges Slack and a critical pages someone. And custom rules because only I know what my backend should be talking to.

Automated response

Falco only detects. What happens next is your call. For most rules I want a human to look. For a CRITICAL I’m fine letting the cluster defend itself, and the webhook receiver from the routing config is where that lives:

# security-responder deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: security-responder
spec:
  template:
    spec:
      containers:
        - name: responder
          image: my-company/security-responder
          env:
            - name: SLACK_WEBHOOK
              valueFrom:
                secretKeyRef:
                  name: security-responder
                  key: slack-webhook

The receiver can be tiny. Kill the pod, let the Deployment recreate it, and post to Slack so a human knows:

@app.route('/falco', methods=['POST'])
def handle_falco_alert():
    alert = request.json

    if alert['priority'] == 'Critical':
        # Kill the pod immediately
        pod = alert['output_fields']['k8s.pod.name']
        namespace = alert['output_fields']['k8s.ns.name']

        # Delete pod (Deployment will recreate it)
        v1.delete_namespaced_pod(pod, namespace)

        # Notify
        send_slack(f"Killed pod {namespace}/{pod} due to: {alert['rule']}")

    return 'ok'

Killing a pod off a single alert is a sharp tool and I’m careful with it. Wire a noisy rule to it and you’ve built something that takes down your own cluster. So automation only goes on rules I’ve watched in production long enough to trust. Everything else pages a person.

Should and is

Step back from the YAML and the reason I run Falco is the reason I self-host the rest of my stack. I want to know what my infrastructure is doing, not assume it. Trivy and Kyverno tell me what my cluster should be. Falco tells me what it is. Without it, my honest answer to “what’s running in your cluster right now” was “whatever the image said three weeks ago, probably.”

That gap between should and is, between the spec and the running process, is where an attacker works. The three layers I opened with are still green. Falco is the one that’s had something to say since the pods started, and for the first week that was mostly about my own sidecar.