Runtime Security with Falco: Detect Suspicious Behavior in Your Cluster

I scanned my images with Trivy. I enforced policies with Kyverno. My workloads got cryptographic identity through SPIFFE. Three layers of prevention, all green, and for a while that felt like enough.

Then I started asking the uncomfortable question. What happens after a pod is running? My scanners checked the image that went in. My admission controller checked the spec at deploy time. Neither of them is watching once the process is actually executing. If a container gets popped by a zero-day at 3am, every one of those controls has already done its job and gone home.

That gap bothered me. I can’t claim to understand what my cluster is doing if I have no idea what the processes inside it are actually doing. Prevention tells me what should be true. I wanted something that tells me what is true, right now, syscall by syscall.

That something is Falco. It watches the system calls every container makes - file access, network connections, processes spawning - and shouts when the behaviour does not match the rules I gave it. This post builds it up the way I learned it: the smallest useful version first, then real rules, then the production config I actually run.

The gap prevention leaves

I think of cluster security as three moments in time:

Build time - scan images for known vulnerabilities (Trivy)
Deploy time - enforce policies on the spec (Kyverno, admission controllers)
Runtime - watch what the workload actually does (Falco)

Most setups, mine included for a long time, stop after the first two. They are cheaper, they fit neatly into the pipeline, and they make a dashboard go green. The trouble is that an attacker who has already landed in a running container does not replay your pipeline. They work with what is executing.

Here is the kind of thing the first two layers never see:

A container suddenly executing /bin/bash in production, where nothing should ever spawn a shell
A process reading /etc/shadow
An outbound connection to a crypto mining pool
Something writing into /etc/
Kubernetes service account tokens getting read from a pod that has no business touching them

None of that shows up in a scan or an admission check, because none of it is about configuration. It is about behaviour. Falco catches it because behaviour is the only thing it looks at.

The simplest version: watching syscalls

Before any rules, understand the one thing Falco does. It sits below your containers and reads their system calls, then matches each one against a set of conditions. eBPF (or a kernel module) is how it gets a seat in that position:

flowchart TD
    subgraph userspace["User Space"]
        A["Container A"]
        B["Container B"]
        C["Container C"]
    end

    subgraph kernel["Kernel Space"]
        EBPF["eBPF / Kernel Module<br/>(intercepts syscalls)"]
    end

    A --> EBPF
    B --> EBPF
    C --> EBPF

    EBPF --> FALCO["Falco<br/>(rules engine)"]
    FALCO --> ALERTS["Alerts<br/>(stdout, webhook, Kafka...)"]

Every syscall flows through the rules engine. A match becomes an alert. That is the whole model, and once it clicks the rest of Falco is just detail layered on top.

Installing Falco

I start with the Helm chart and the eBPF driver. eBPF avoids loading a kernel module into every node, which matters on Talos and other locked-down hosts:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true

I do not actually run helm install by hand though. Everything in my cluster goes through Git, so the real version is an ArgoCD Application that points at the same chart:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: falco
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://falcosecurity.github.io/charts
    chart: falco
    targetRevision: 4.0.0
    helm:
      values: |
        driver:
          kind: ebpf
        falcosidekick:
          enabled: true
          config:
            slack:
              webhookurl: "https://hooks.slack.com/services/xxx"
        tty: true
  destination:
    server: https://kubernetes.default.svc
    namespace: falco
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Layer one: reading a rule

Falco is only as good as the rules it runs. A rule is a small block of YAML, and once you can read one you can read all of them. Here is the canonical example, a shell spawning inside a container:

- rule: Terminal shell in container
  desc: Detect shell being spawned in a container
  condition: >
    spawned_process and
    container and
    shell_procs and
    proc.tty != 0
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name shell=%proc.name
     parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

Four parts do all the work:

condition decides when to fire, written in Falco’s filter language
output is what lands in the alert, with %-prefixed fields filled in at runtime
priority is the severity, from DEBUG up to CRITICAL
tags let you group and filter, here mapping to MITRE technique categories

Read that condition out loud and it is almost English: a process was spawned, inside a container, the process is a shell, and it has a TTY attached. That is the grammar every other rule reuses.

Built-in rules

You do not start from a blank file. Falco ships a large set of community rules that already cover most of the obvious attacks:

# View loaded rules
kubectl exec -n falco -it falco-xxx -- falco --list

They cover the categories you would expect: container escape attempts, privilege escalation, sensitive file access, suspicious network activity, Kubernetes API abuse. A few examples, so you can see they read the same way as the one above:

# Detect container escape via mount
- rule: Launch Sensitive Mount Container
  condition: >
    spawned_process and container and
    sensitive_mount

# Detect reading sensitive files
- rule: Read sensitive file untrusted
  condition: >
    open_read and sensitive_files and
    not trusted_containers

# Detect crypto mining
- rule: Detect outbound connections to crypto mining pools
  condition: >
    outbound and
    fd.sip.name in (cryptomining_pool_domains)

Layer two: rules that know your app

The defaults catch generic attacks. They have no idea what your application is supposed to do. That knowledge is yours, and writing it down as a rule is where Falco starts earning its keep:

# custom-rules.yaml
customRules:
  rules-custom.yaml: |-
    # Alert when our API server spawns unexpected processes
    - rule: Unexpected process in api-server
      desc: Detect processes other than the main app in api-server containers
      condition: >
        spawned_process and
        container.image.repository contains "api-server" and
        not proc.name in (api-server, node, npm)
      output: >
        Unexpected process in api-server
        (command=%proc.cmdline container=%container.name image=%container.image.repository)
      priority: WARNING

    # Alert on database connection from unexpected pods
    - rule: Database connection from non-backend pod
      desc: Detect connections to PostgreSQL from pods that shouldn't connect
      condition: >
        outbound and
        fd.sport = 5432 and
        not k8s.pod.label.app in (api-server, worker, migration-job)
      output: >
        Unexpected database connection
        (pod=%k8s.pod.name namespace=%k8s.ns.name dest=%fd.sip)
      priority: ERROR

Deploy custom rules via Helm values:

# values.yaml
customRules:
  rules-custom.yaml: |-
    - rule: My custom rule
      ...

The filter language, so the rules stop being magic

I do not like copying YAML I cannot read, and neither should you. Once you know the handful of field classes and how macros and lists glue them together, every rule above stops being a black box.

Fields

# Process fields
proc.name         # Process name
proc.cmdline      # Full command line
proc.pname        # Parent process name
proc.exepath      # Executable path

# Container fields
container.name    # Container name
container.id      # Container ID
container.image.repository  # Image name

# File fields
fd.name           # File descriptor name (file path)
fd.directory      # Directory of the file

# Network fields
fd.sip            # Server IP
fd.sport          # Server port
fd.cip            # Client IP

# Kubernetes fields
k8s.pod.name      # Pod name
k8s.ns.name       # Namespace
k8s.pod.label.app # Pod label

Macros

Macros are named, reusable condition fragments. Every rule earlier leaned on these without you noticing:

- macro: container
  condition: container.id != host

- macro: shell_procs
  condition: proc.name in (bash, sh, zsh, dash, ksh)

- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

- macro: open_read
  condition: evt.type in (open, openat) and evt.is_open_read = true

Lists

Lists are named sets of values you reference instead of repeating:

- list: sensitive_files
  items: [/etc/shadow, /etc/sudoers, /etc/pam.d, /root/.ssh]

- list: package_managers
  items: [apt, apt-get, yum, dnf, apk, pip, npm]

- list: shell_binaries
  items: [bash, sh, zsh, csh, tcsh, ksh, dash]

Putting it together: rules that earn their alert

With the grammar in hand, here are three detections I find genuinely worth the noise. These are the ones that map to how an attacker actually behaves once they are inside.

Reverse shells

- rule: Reverse shell detected
  desc: Detect reverse shells via common patterns
  condition: >
    spawned_process and
    container and
    ((proc.name = bash and proc.args contains ">&" and proc.args contains "/dev/tcp") or
     (proc.name = nc and proc.args contains "-e") or
     (proc.name = python and proc.args contains "socket" and proc.args contains "subprocess"))
  output: >
    Reverse shell detected (user=%user.name command=%proc.cmdline container=%container.name)
  priority: CRITICAL
  tags: [mitre_execution, reverse_shell]

Kubernetes secret access

- rule: K8s secret accessed from unexpected namespace
  desc: Detect when secrets are read from non-standard locations
  condition: >
    open_read and
    fd.name startswith "/var/run/secrets/kubernetes.io" and
    not k8s.ns.name in (kube-system, monitoring, vault)
  output: >
    K8s secret accessed (file=%fd.name pod=%k8s.pod.name namespace=%k8s.ns.name)
  priority: WARNING

Package installs at runtime

- rule: Package manager in container
  desc: Detect package installations in running containers
  condition: >
    spawned_process and
    container and
    proc.name in (apt, apt-get, yum, dnf, apk, pip, npm) and
    not container.image.repository in (allowed_builder_images)
  output: >
    Package manager run in container
    (command=%proc.cmdline container=%container.name image=%container.image.repository)
  priority: ERROR
  tags: [mitre_persistence, package_install]

A package manager firing up in a running container is a strong signal that someone is building a foothold, so I treat it as an ERROR rather than a warning.

Getting alerts somewhere you will see them

Detecting something is useless if the alert dies in a pod’s stdout. By default that is exactly where Falco writes. Falcosidekick is the piece that fans those alerts out to wherever you actually look:

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/xxx"
      minimumpriority: warning

    prometheus:
      enabled: true

    elasticsearch:
      hostport: "elasticsearch.logging:9200"
      index: "falco"

    alertmanager:
      hostport: "http://alertmanager.monitoring:9093"

    webhook:
      address: "http://security-responder.security:8080/falco"

That config sends warnings and up to Slack, every alert into Prometheus as a metric, everything into Elasticsearch so I can search history, criticals to Alertmanager for paging, and the full stream to a custom webhook I use for automated response. One detection, several audiences, each filtered to the severity it cares about.

Folding Falco into Prometheus

I already run Prometheus and Grafana for everything else, so I want security events in the same place I watch the rest of the cluster. Falcosidekick exposes metrics, and a ServiceMonitor is enough to scrape them:

# ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: falco-sidekick
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: falcosidekick
  endpoints:
    - port: http

From there a PrometheusRule turns a Falco detection into the same kind of alert as a failing disk or a hot CPU:

# PrometheusRule
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: falco-alerts
spec:
  groups:
    - name: falco
      rules:
        - alert: FalcoCriticalAlert
          expr: increase(falco_events{priority="Critical"}[5m]) > 0
          for: 0m
          labels:
            severity: critical
          annotations:
            summary: "Falco critical security event detected"
            description: "{{ $labels.rule }} triggered in {{ $labels.k8s_ns_name }}/{{ $labels.k8s_pod_name }}"

The part nobody warns you about: noise

Here is the honest trade-off. Out of the box, the default rules will page you for things that are completely normal in your cluster. The first week I ran Falco I got woken up by my own config-reload sidecar. An alerting tool you have learned to ignore is worse than no tool at all, so tuning is not optional, it is the actual work.

There are two moves: switch off rules that do not fit, and narrow the macros they depend on.

Disabling and overriding rules

# values.yaml
falco:
  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml  # Overrides
    - /etc/falco/rules.d  # Custom rules

customRules:
  falco_rules.local.yaml: |-
    # Disable noisy rule
    - rule: Terminal shell in container
      enabled: false

    # Modify existing rule to exclude certain containers
    - rule: Read sensitive file untrusted
      condition: >
        open_read and sensitive_files and
        not trusted_containers and
        not container.image.repository in (my-trusted-app)

Tuning macros

customRules:
  falco_rules.local.yaml: |-
    # Add to trusted containers
    - macro: trusted_containers
      condition: >
        (container.image.repository in (
          my-company/trusted-app,
          my-company/another-app
        ))

    # Exclude namespaces from monitoring
    - macro: user_namespace
      condition: >
        k8s.ns.name not in (kube-system, falco, monitoring)

Editing the trusted_containers macro is better than disabling a rule wholesale, because the rule still protects every other container. You are teaching Falco what normal looks like in your environment rather than blinding it.

The full picture: my production config

This is everything stacked together: the eBPF driver, JSON output, tiered alerting, and a couple of rules that encode what my own workloads are allowed to do. Compare it to the one-line helm install near the top and you can see the whole staircase you just climbed.

driver:
  kind: ebpf
  ebpf:
    hostNetwork: true

falco:
  grpc:
    enabled: true
  grpc_output:
    enabled: true
  json_output: true
  log_level: info

  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml
    - /etc/falco/rules.d

falcosidekick:
  enabled: true
  replicaCount: 2
  config:
    slack:
      webhookurl: "${SLACK_WEBHOOK}"
      minimumpriority: error

    prometheus:
      enabled: true

    alertmanager:
      hostport: "http://alertmanager.monitoring:9093"
      minimumpriority: critical

customRules:
  rules-custom.yaml: |-
    # Our application-specific rules
    - rule: Unexpected outbound connection from backend
      desc: Backend pods should only connect to known services
      condition: >
        outbound and
        k8s.pod.label.app = "backend" and
        not fd.sip.name in (postgres.db, redis.cache, api.internal)
      output: >
        Backend made unexpected outbound connection
        (pod=%k8s.pod.name dest=%fd.sip:%fd.sport)
      priority: WARNING

  falco_rules.local.yaml: |-
    # Tune default rules for our environment
    - macro: user_known_write_etc_conditions
      condition: >
        (container.image.repository = "my-company/config-manager")

The reasoning behind each choice: eBPF over the kernel module because it performs better and keeps the host cleaner, JSON output because downstream tools parse it without regex gymnastics, tiered alerting so errors nudge Slack while criticals page someone awake, and custom rules because only I know what my backend should and should not talk to.

Closing the loop with automated response

Falco only detects. What happens next is your call. For most rules I want a human to look, but for a CRITICAL I am comfortable letting the cluster defend itself. The webhook receiver from the routing config is where that lives:

# security-responder deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: security-responder
spec:
  template:
    spec:
      containers:
        - name: responder
          image: my-company/security-responder
          env:
            - name: SLACK_WEBHOOK
              valueFrom:
                secretKeyRef:
                  name: security-responder
                  key: slack-webhook

The receiver itself can be tiny. Kill the offending pod, let the Deployment recreate it, and post to Slack so a human knows it happened:

@app.route('/falco', methods=['POST'])
def handle_falco_alert():
    alert = request.json

    if alert['priority'] == 'Critical':
        # Kill the pod immediately
        pod = alert['output_fields']['k8s.pod.name']
        namespace = alert['output_fields']['k8s.ns.name']

        # Delete pod (Deployment will recreate it)
        v1.delete_namespaced_pod(pod, namespace)

        # Notify
        send_slack(f"Killed pod {namespace}/{pod} due to: {alert['rule']}")

    return 'ok'

Why I bother with all of this

Killing a pod automatically off a single alert is a sharp tool, and I am cautious with it. A noisy rule plus an automated response is how you DoS yourself. So I only wire automation to rules I have watched in production long enough to trust, and everything else just pages a person.

Step back from the YAML and the reason I run Falco is the same reason I self-host the rest of my stack. I want to understand what my infrastructure is doing, not assume it. Trivy and Kyverno tell me what my cluster should be. Falco tells me what it is. Without that last layer, the honest answer to “what is running in your cluster right now” is “whatever the image said three weeks ago, probably.”

That gap between should and is, between the spec and the running process, is exactly where attackers operate. Closing it does not mean my prevention failed. It means I stopped pretending prevention is the whole story.

The gap prevention leaves#

The simplest version: watching syscalls#

Installing Falco#

Layer one: reading a rule#

Built-in rules#

Layer two: rules that know your app#

The filter language, so the rules stop being magic#

Fields#

Macros#

Lists#

Putting it together: rules that earn their alert#

Reverse shells#

Kubernetes secret access#

Package installs at runtime#

Getting alerts somewhere you will see them#

Folding Falco into Prometheus#

The part nobody warns you about: noise#

Disabling and overriding rules#

Tuning macros#

The full picture: my production config#

Closing the loop with automated response#

Why I bother with all of this#