Falco Runtime Security for Containers

Falco is a cloud-native runtime security tool that detects anomalous behavior in containers and Kubernetes by monitoring system calls, alerting on policy violations in real time. This guide covers deploying Falco on Linux and Kubernetes, customizing detection rules, configuring Slack and webhook alerts, and building incident response workflows.

Prerequisites

  • Ubuntu 22.04/Debian 12 or CentOS/Rocky 9
  • Linux kernel 5.4+ (for eBPF driver) or kernel headers installed (for kmod driver)
  • Root access (Falco requires kernel-level access)
  • Docker or Kubernetes for container monitoring
  • For Kubernetes: helm installed

Install Falco on Linux

# Ubuntu/Debian
curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] \
  https://download.falco.org/packages/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/falcosecurity.list

sudo apt update && sudo apt install -y falco

# CentOS/Rocky
sudo tee /etc/yum.repos.d/falcosecurity.repo << 'EOF'
[falcosecurity]
name=falcosecurity repository
baseurl=https://download.falco.org/packages/rpm/stable/$basearch/
enabled=1
gpgcheck=1
gpgkey=https://falco.org/repo/falcosecurity-packages.asc
EOF
sudo dnf install -y falco

# The installer will ask which driver to use:
# - Modern eBPF (recommended for kernel 5.8+)
# - eBPF (kernel 4.14+ with eBPF support)
# - Kernel module (kmod) - fallback for older kernels
# Enable and start Falco
sudo systemctl enable --now falco

# Verify Falco is running
sudo systemctl status falco
sudo falco --version

Install Falco in Kubernetes

Deploy Falco using Helm:

# Add the Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco with modern eBPF driver (recommended)
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set tty=true

# Install with custom values file
cat > falco-values.yaml << 'EOF'
driver:
  kind: modern_ebpf

falco:
  grpc:
    enabled: true
  grpc_output:
    enabled: true
  json_output: true
  log_stderr: true
  log_syslog: false
  loglevel: info

falcosidekick:
  enabled: true
  webui:
    enabled: true

tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
EOF

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  -f falco-values.yaml

# Check deployment
kubectl -n falco get pods
kubectl -n falco logs -l app.kubernetes.io/name=falco --tail=50

Understanding and Customizing Rules

Falco rules define what constitutes suspicious behavior. Default rules are in /etc/falco/falco_rules.yaml.

View existing rules:

# List all loaded rules
sudo falco --list

# Check what files Falco loads
sudo falco --config /etc/falco/falco.yaml --validate /etc/falco/falco_rules.yaml

Create custom rules in /etc/falco/falco_rules.local.yaml (override or add to defaults):

# /etc/falco/falco_rules.local.yaml

# Override the default rule to reduce noise (append to the condition)
- rule: Terminal shell in container
  append: true
  condition: and not container.image.repository = "my-debug-image"

# Custom rule: Alert on sensitive file reads
- rule: Read sensitive files in container
  desc: Detect reads of sensitive files inside containers
  condition: >
    open_read and container and
    (fd.name startswith /etc/shadow or
     fd.name startswith /etc/sudoers or
     fd.name startswith /root/.ssh or
     fd.name startswith /etc/kubernetes/admin.conf)
  output: >
    Sensitive file opened for reading in container
    (user=%user.name user_uid=%user.uid command=%proc.cmdline
     file=%fd.name container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [container, filesystem, mitre_credential_access]

# Custom rule: Alert on outbound connections from database containers
- rule: Unexpected outbound connection from database
  desc: Database container making outbound network connection
  condition: >
    outbound and container and
    container.image.repository contains "postgres" and
    not fd.sip in (allowed_db_destinations)
  output: >
    Unexpected network connection from database container
    (user=%user.name command=%proc.cmdline connection=%fd.name
     container_id=%container.id image=%container.image.repository)
  priority: CRITICAL
  tags: [network, database]

# Define a macro for reuse
- macro: allowed_db_destinations
  condition: (fd.sip="10.0.0.100" or fd.sip="10.0.0.101")

Reload rules without restarting:

sudo kill -1 $(pidof falco)   # Send SIGHUP to reload
# Or:
sudo systemctl reload falco

Configure Alerts and Notifications

Falco outputs alerts to multiple destinations. Configure in /etc/falco/falco.yaml:

sudo nano /etc/falco/falco.yaml
# Output channels in falco.yaml

# Standard output (stdout)
stdout_output:
  enabled: true

# Syslog output
syslog_output:
  enabled: true

# File output (JSON format for log aggregation)
file_output:
  enabled: true
  keep_alive: false
  filename: /var/log/falco/falco_events.log

# JSON output format
json_output: true
json_include_output_property: true

# Program output (pipe to a script or falcosidekick)
program_output:
  enabled: true
  keep_alive: false
  program: "jq '{summary: .output, priority: .priority, time: .time}' | curl -s -X POST -H 'Content-Type: application/json' -d @- https://webhook.example.com/falco"

Slack and Webhook Integration

Use falcosidekick for rich notification routing:

# Install falcosidekick
curl -L https://github.com/falcosecurity/falcosidekick/releases/latest/download/falcosidekick_linux_amd64.tar.gz \
  -o /tmp/falcosidekick.tar.gz
tar xvf /tmp/falcosidekick.tar.gz -C /usr/local/bin/ falcosidekick

# Configure falcosidekick
sudo tee /etc/falcosidekick/config.yaml << 'EOF'
listenaddress: "0.0.0.0"
listenport: 2801
debug: false

slack:
  webhookurl: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
  channel: "#security-alerts"
  username: "Falco"
  icon: "https://falco.org/img/brand/falco-logo.png"
  minimumpriority: "warning"
  messageformat: "Rule: *{{.Rule}}* triggered by {{.OutputFields.container_image}} (Priority: *{{.Priority}}*)"

# PagerDuty for critical alerts
pagerduty:
  apikey: "YOUR_PAGERDUTY_API_KEY"
  routingkey: "YOUR_ROUTING_KEY"
  minimumpriority: "critical"

# Generic webhook
webhook:
  address: "https://webhook.example.com/falco-events"
  method: "POST"
  minimumpriority: "notice"
EOF

# Create systemd service for falcosidekick
sudo tee /etc/systemd/system/falcosidekick.service << 'EOF'
[Unit]
Description=Falco Sidekick notification forwarder
After=network.target

[Service]
ExecStart=/usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now falcosidekick

Configure Falco to send events to falcosidekick:

# In falco.yaml, add:
program_output:
  enabled: true
  keep_alive: true
  program: "curl -d @- localhost:2801/"

Testing Falco Rules

Generate test events to verify rules are working:

# Test rule: shell spawned inside a container
docker run -it --rm ubuntu:22.04 bash -c "echo 'testing falco'"

# Test rule: sensitive file access
docker run --rm ubuntu:22.04 cat /etc/shadow

# Test rule: privileged container started
docker run --rm --privileged alpine id

# Test rule: package manager run in container
docker run --rm ubuntu:22.04 apt-get install -y curl

# Watch Falco alerts in real-time
sudo journalctl -fu falco
# Or:
sudo tail -f /var/log/falco/falco_events.log | jq '.'

Use event-generator to simulate security events:

# Install the Falco event generator
docker run -it --rm falcosecurity/event-generator run syscall

# Run a specific test
docker run -it --rm falcosecurity/event-generator run syscall.ReadSensitiveFileUntrusted

Incident Response Workflows

Automate responses to specific Falco alerts:

# Automated response script - isolate a compromised container
cat > /usr/local/bin/falco-respond.sh << 'EOF'
#!/bin/bash
# Called by Falco's program_output or falcosidekick

# Read event from stdin
EVENT=$(cat)
RULE=$(echo "${EVENT}" | jq -r '.rule')
PRIORITY=$(echo "${EVENT}" | jq -r '.priority')
CONTAINER_ID=$(echo "${EVENT}" | jq -r '.output_fields.container_id')

# Log the event
echo "$(date): Falco alert - Rule: ${RULE}, Priority: ${PRIORITY}, Container: ${CONTAINER_ID}" \
  >> /var/log/falco-response.log

# Respond to critical events
if [[ "${PRIORITY}" == "CRITICAL" ]] && [[ -n "${CONTAINER_ID}" ]]; then
    # Pause the container (stops new processes, preserves state for forensics)
    docker pause "${CONTAINER_ID}" 2>/dev/null
    echo "$(date): Paused container ${CONTAINER_ID} due to critical Falco alert: ${RULE}" \
      >> /var/log/falco-response.log

    # Capture container state for forensics
    docker inspect "${CONTAINER_ID}" > "/var/log/forensics-${CONTAINER_ID}.json" 2>/dev/null
fi
EOF

chmod +x /usr/local/bin/falco-respond.sh

For Kubernetes, use falcosidekick with kubeless or a webhook to automatically quarantine pods:

# falcosidekick Kubernetes config
kubeless:
  namespace: "default"
  function: "isolate-pod"
  minimumpriority: "critical"

# Or use a Kubernetes NetworkPolicy to isolate the pod

Troubleshooting

Falco not detecting events:

# Check Falco is loaded and driver is working
sudo systemctl status falco
sudo falco --list | head -20

# Verify the kernel driver is loaded
lsmod | grep falco

# Test with a known-bad action
docker run --rm ubuntu:22.04 cat /etc/shadow
sudo journalctl -u falco -n 20 --no-pager

Rule syntax errors:

# Validate rule files
sudo falco --config /etc/falco/falco.yaml \
  --validate /etc/falco/falco_rules.local.yaml

# Check for YAML syntax errors
python3 -c "import yaml; yaml.safe_load(open('/etc/falco/falco_rules.local.yaml'))"

High CPU usage:

# Check which rules are generating the most events
sudo falco --stats_interval=10 2>&1 | grep -i "rule"

# Suppress noisy rules by adding conditions
# Example: ignore specific containers from a noisy rule
- rule: Noisy Rule
  append: true
  condition: and not container.image.repository = "known-noisy-image"

Falco not starting after kernel update:

# Rebuild the kernel module for the new kernel
sudo dkms install -m falco -v $(falco --version | grep -o '[0-9.]*')
# Or switch to eBPF driver (no recompilation needed)
sudo apt install -y falco-driver-loader
sudo falco-driver-loader modern_ebpf

Conclusion

Falco provides real-time threat detection for containers and Kubernetes by monitoring system calls at the kernel level, catching shell spawns, privilege escalations, and sensitive file accesses that static scanning misses. Custom rules let you tailor detection to your specific environment, while falcosidekick routes alerts to Slack, PagerDuty, or automated response scripts. The key to effective Falco deployment is starting with the default rules, suppressing false positives incrementally, and building automated responses for the highest-priority alerts.