runtime-security

Detect and respond to threats inside running containers using Falco.

Interactive Wizard (fires when no arguments are provided)

When invoked with no arguments, ask before proceeding:

Q1 — Mode?

What do you need?
  1. install  — deploy Falco on Kubernetes (EKS or GKE) with eBPF driver
  2. rules    — write and test a custom Falco rule
  3. alerts   — configure Falcosidekick to route alerts (Slack, webhook, etc.)
  4. debug    — diagnose why a Falco rule is not firing
  5. harden   — map runtime Falco alerts to Kyverno admission policies

  Alerts on behaviour a container never showed while learning (profile-based,
  no rule writing), or Kubescape already runs → /platform-skills:kubescape-operator runtime

Enter 1–5 or mode name:

Before install, check for another eBPF agent: kubectl get ds -A | grep -iE 'node-agent|tetragon'. Kubescape's node-agent with runtimeDetection overlaps with Falco on the same nodes; pick one per node pool, or disable Kubescape's runtimeDetection and keep its posture and CVE capabilities.

Q2 — Context (after mode selected, one at a time):

• install: Which cloud platform? EKS / GKE / other (specify node OS if known):
• rules: Describe the behaviour to detect (e.g. "shell spawned in a container", "unexpected outbound connection"):
• alerts: Which output destination? Slack / webhook / PagerDuty / other:
• debug: Describe the symptom — which rule is not firing, and what event should trigger it?
• harden: Which Falco alert or rule output should block re-admission? (paste the alert or rule name):

Then proceed into the relevant mode below.

Mode: install

Deploy Falco on Kubernetes (EKS or GKE) using the eBPF driver via Helm.

Prerequisites:

• Helm 3.x
• kubectl access to the target cluster
• Node OS: Amazon Linux 2/2023 (EKS) or Container-Optimized OS (GKE) — both support eBPF

Steps:

1. Add the Falco Helm repository:

   helm repo add falcosecurity https://falcosecurity.github.io/charts
   helm repo update

2. Install Falco with eBPF driver (never kernel module on managed K8s):

   helm install falco falcosecurity/falco \
     --namespace falco \
     --create-namespace \
     -f examples/runtime-security/falco-values.yaml

3. Verify DaemonSet is running on all nodes:

   kubectl rollout status daemonset/falco -n falco
   kubectl get pods -n falco -o wide

4. Confirm Falco is receiving events:

   kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=20

   Expected: lines like Notice A shell was spawned... (from the default ruleset test events)

EKS-specific note: if using Bottlerocket nodes, set driver.kind: modern_ebpf in Helm values — Bottlerocket does not ship kernel headers for the classic eBPF probe.

GKE-specific note: COS nodes require driver.kind: modern_ebpf. GKE Autopilot does not support Falco (no DaemonSet scheduling).

Mode: rules

Write and test custom Falco rules.

Steps:

1. Explain the rule structure:

   - rule: Shell spawned in container
     desc: A shell was spawned inside a container — potential interactive intrusion
     condition: >
       spawned_process
       and container
       and shell_procs
       and not container.image.repository in (allowed_shell_images)
     output: >
       Shell spawned in container
       (user=%user.name user_id=%user.uid
        container=%container.name image=%container.image.repository
        shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
     priority: WARNING
     tags: [container, shell, mitre_execution]

2. Key condition fields:
   • spawned_process — a new process was exec'd
   • container — event is inside a container (not on the host)
   • proc.name — process name
   • container.image.repository — image name without tag
   • fd.net — network file descriptor (for connection events)
3. Test with falco-event-generator:

   kubectl run event-generator \
     --image=falcosecurity/event-generator \
     --restart=Never \
     --rm -it -- run syscall

   Then check logs: kubectl logs -n falco -l app.kubernetes.io/name=falco | grep WARNING
4. Load custom rules via Helm values:

   customRules:
     custom-rules.yaml: |-
       - rule: Shell spawned in container
         ...

Mode: alerts

Configure Falcosidekick to route Falco alerts to Slack, webhooks, or other outputs.

Steps:

1. Install Falcosidekick alongside Falco:

   helm upgrade --install falco falcosecurity/falco \
     --namespace falco \
     --create-namespace \
     --set falcosidekick.enabled=true \
     --set falcosidekick.webui.enabled=true \
     -f examples/runtime-security/falcosidekick-values.yaml

2. Configure Slack output in Falcosidekick values:

   falcosidekick:
     config:
       slack:
         webhookurl: "https://hooks.slack.com/services/<token>"
         minimumpriority: warning
         messageformat: >
           Alert: *{{ .Rule }}* (Priority: {{ .Priority }})
           Container: `{{ index .OutputFields "container.name" }}`
           Image: `{{ index .OutputFields "container.image.repository" }}`

3. Verify alerts are routing:

   kubectl port-forward -n falco svc/falco-falcosidekick-ui 2802:2802
   # Open http://localhost:2802 — events appear in the UI

4. Deduplication: set slack.minimumpriority: warning to suppress DEBUG/INFO noise

Mode: debug

If you need deeper context on any Falco component, load references/runtime-security.md.

Diagnose why a Falco rule is not firing.

Checklist (work through in order):

1. Is Falco running?

   kubectl get pods -n falco -o wide
   kubectl logs -n falco -l app.kubernetes.io/name=falco | tail -20

2. Is the rule loaded?

   kubectl logs -n falco -l app.kubernetes.io/name=falco | grep -i "Loading rules"
   kubectl exec -n falco daemonset/falco -- cat /etc/falco/rules.d/custom-rules.yaml 2>/dev/null \
     || kubectl exec -n falco daemonset/falco -- ls /etc/falco/

3. Is the condition correct? Test the event type:

   kubectl exec -n falco daemonset/falco -- falco --validate /etc/falco/rules.d/custom-rules.yaml

4. Is the event being generated? Use event-generator for syscall rules:

   kubectl run test --image=falcosecurity/event-generator --rm -it -- run syscall

5. Is the priority too low? If falcosidekick.config.slack.minimumpriority: error, rules with priority: WARNING are silently dropped. Adjust priority or threshold.
6. Is there a macro override? Check if a built-in macro like never_true is shadowing your condition:

   kubectl exec -n falco daemonset/falco -- cat /etc/falco/falco_rules.yaml | grep -A3 "macro: <macro name>"
   kubectl exec -n falco daemonset/falco -- ls /etc/falco/rules.d/

Mode: harden

Map Falco runtime alerts to Kyverno admission policies to prevent redeployment of flagged workloads.

Steps:

1. Explain the bridge pattern: Falco detects a runtime violation → alert metadata is added as a label/annotation to the offending Pod or Deployment → Kyverno admission policy blocks future admission of workloads with that label
2. Label the offending workload (from an alert webhook handler or manual response):

   kubectl label deployment <name> -n <namespace> \
     security.platform/falco-alert=critical

3. Apply the Kyverno policy to block re-admission:

   apiVersion: policies.kyverno.io/v1
   kind: ValidatingPolicy
   metadata:
     name: block-falco-flagged-workloads
   spec:
     validationActions: [Deny]
     matchConstraints:
       resourceRules:
       - apiGroups: ["apps"]
         apiVersions: ["v1"]
         operations: ["CREATE", "UPDATE"]
         resources: ["deployments"]
     validations:
     - expression: >
         !has(object.metadata.labels) ||
         !('security.platform/falco-alert' in object.metadata.labels) ||
         object.metadata.labels['security.platform/falco-alert'] != 'critical'
       message: "Deployment is flagged by a Falco critical alert and cannot be re-admitted. Remove the flag after remediation."

4. Important: Kyverno blocks new pod admissions only — it does not terminate existing running pods that violate the policy. After applying the policy, manually identify and terminate flagged pods:

# Find pods matching the label that triggered the alert
kubectl get pods -n <namespace> -l <label-key>=<label-value>
# Terminate with grace period to allow in-flight requests to drain
kubectl delete pod <name> -n <namespace> --grace-period=30

5. Remediation workflow: fix the workload → remove the label → re-admit

Recovery: If a Kyverno policy blocks a legitimate workload, remove the triggering label immediately:

kubectl label deployment <name> -n <namespace> security.platform/falco-alert-

The workload will be re-admitted on next rollout. Fix the underlying security issue, then re-apply the label to re-admit under policy enforcement.

After completing this task, log errors and learnings via /platform-skills:self-improve log.