Coding Agent Security & Evaluation (Effective Trust)

This skill turns Google's "Vibe Coding Agent Security and Evaluation" whitepaper (Kartakis, Eidelman, Bakkali, Subasioglu — May 2026) into a work-time procedure: how to contain a coding agent (the 7-Pillar Agent Security Architecture under a continuous Effective Trust model), which threat classes are specific to agent-generated code, and which evaluation gates must pass before agent-written code ships. Core framing: security tells you whether the agent stayed inside the boundary; evaluation tells you whether what happened inside that boundary is worth shipping. An agent can pass every security check and still misread intent, ignore project conventions, or silently degrade UX — you must run both halves.

When to use

• You are about to execute, review, merge, or deploy code generated by an AI coding agent (including your own output).
• You are designing or hardening the harness, sandbox, identity, or CI/CD pipeline around a coding agent.
• You are adding a tool/MCP server, a new dependency, or pasted third-party context (repos, forum posts, RAG data) to an agent workflow.
• You are building or running an eval suite to decide whether a coding agent's output is deployable.
• Do NOT use for: subjective evaluation of non-verifiable outputs (quality defined by user/enterprise preference, not ground truth) or the feedback loop from user corrections back into model training — the source explicitly scopes both out. Not a general security guide for human-written code.

Rules

Trust model

1. Treat trust as continuous, never binary. Effective Trust is "a continuous metric evaluated across an agent's supply chain, identity, runtime behaviour, and contextual associations" (whitepaper, Security section). Trust is not a gate passed once at deployment; it is earned, verified, and enforced at runtime.
2. Shift from Identity-as-a-Perimeter (static RBAC) to Context-as-a-Perimeter. A raw model is not an agent; it becomes one inside a harness (state, tool execution, feedback loops, constraints). Secure the harness, not the code syntax — assume the model itself can fail or be compromised.

The 7-Pillar Agent Security Architecture (containment baseline)

3. Pillar 1 — Infrastructure & Networking: run all agent-generated code in ephemeral, kernel-level, network-isolated sandboxes (e.g. gVisor) that block raw host access and fully reset state between runs. Route egress only through authorised offline caches or explicit internal proxies.
4. Pillar 2 — Data: encrypt context at rest with Customer-Managed Encryption Keys (CMEK), in transit with mutual TLS (mTLS); scope data access to least privilege. Enforce strict tenant partitioning in vector databases to prevent Cross-Tenant Vector Poisoning.
5. Pillar 3 — Model: the prompt and "Instructions and Rule Files" are the new source code. Treat system instructions and prompt templates as highly sensitive, cryptographically attested artifacts.
6. Pillar 4 — Application & Runtime: deploy LLM firewalls for dynamic prompt/response filtering, deterministic hooks at lifecycle points (before a tool call, after a file edit), and a Centralised Agent Gateway governing Agent-to-Agent (A2A) orchestration to block lateral movement.
7. Pillar 5 — IAM: assign every agent a unique cryptographic identity (e.g. SPIFFE ID) tagged as agentic — never the human's delegated credentials (that ambient access enables the Confused Deputy). Use Attribute-Based Access Control (ABAC) + Just-In-Time (JIT) token downscoping enforcing an Intent × User × Time permissions matrix; tokens expire the moment the task concludes.
8. Pillar 6 — Observability & Security Ops: instrument with OpenTelemetry and Agent Behavioural Analytics (ABA); run an autonomous Red/Blue/Green SecOps triad (rule 14).
9. Pillar 7 — Governance: comply with the EU AI Act, run Algorithmic Impact Assessments for high-risk agents, keep an immutable audit trail attributing every action to a specific agent and the human who deployed/approved it, replace approval buttons with plain-language "Logic Reviews," and bind digital signatures to outputs via Risk-Stratified Attestation.

Threat classes in agent-generated code — and the counter for each

10. Hallucinated packages / slopsquatting: models invent package names; attackers pre-publish malware under those exact names (Wiz calls this "slopsquatting"). Counter: source dependencies only from vetted providers or internal registries, enforce cryptographic version pinning, and have CI/CD verify SBOM entries and digital signatures before promotion — Binary Authorisation as the definitive gate.
11. Predictable application flaws: generated apps fail two ways — they trust the browser too much (API keys, password validation, session flags dumped client-side, readable/manipulable via dev tools) and leave the backend wide open (default-deny access controls not enabled, row-level database security skipped, staging environments exposed). Check both explicitly; compiling and running proves neither.
12. MCP spoofing: a forged or compromised server poses as a legitimate MCP tool to inject payloads or demand excessive privileges. Counter: runtime LLM firewall in front of the agent plus Contextual Authorisation at the Centralised Agent Gateway — verify each tool call aligns with the developer's original intent.
13. Invisible payloads / repository poisoning: zero-width Unicode characters and homoglyphs inserted into repos bypass human review, and agents replicate a hidden payload across hundreds of files in minutes (per Knostic). Counter: content-scan all retrieved code. Also treat any pasted open-source/forum context as a prompt-injection vector (Confused Deputy trigger); an allowlist of domains cannot stop indirect prompt injection hidden in web pages — restrict agents to non-interactive internet access via offline caches or pre-sanitised crawling services.
14. Runtime drift and resource attacks: monitor for Intent Drift (chain of thought pursuing sub-goals diverging from the original intent — e.g. "optimise the database query" drifting into downloading an unauthorised indexing library), Trust Decay, and Denial of Wallet (DoW) — adversarially induced infinite, expensive API loops. An HTTP 200 can mask a hallucination loop.

Runtime defence and high-stakes actions

15. Run the agentic SecOps triad continuously: Red Team (Agent Attacker) injects "Adversarial Vibes" — roleplay jailbreaks and malicious instructions buried in RAG context or dummy forum posts; Blue Team (Agent Defender) uses ABA (not UEBA — ineffective for non-deterministic AI) to baseline execution paths and watch the Runtime Agent Bill of Materials (AgBOM), a live inventory of tools/models/data in use; Green Team (Agent Fixer) executes a "Stateful Quarantine" via SOAR playbooks — revoke tool access, preserve short-term memory for forensics — then auto-refactors the insecure code and presents the fix in the IDE.
16. Enforce small batch sizes. Use a test-driven loop where the system blocks the agent from modifying tests and implementation code simultaneously, keeping tests an objective baseline.
17. Before any codebase modification, create a version-control checkpoint. Feed ABA's evaluation of the Vibe Trajectory (the full trace from user prompt to compiled Abstract Syntax Tree) into a dynamic Agent Trust Score; if it drops below the pre-defined threshold, trip a circuit breaker: roll back to checkpoint, revoke tool access, freeze execution without corrupting connected APIs.
18. Gate high-stakes actions (production DB changes, financial transfers, IAM changes) with structured context-aware elicitation plus two boundaries: Cryptographic Hardware MFA (e.g. physical USB security key touch) and the "Vibe Diff" — an Evaluator Quorum translates the generated code into a plain-English summary mapping the user's fuzzy intent to the proposed execution steps, before cryptographic consent. Plain approve/deny buttons cause confirmation fatigue and blind authorisation.

Evaluation gates before shipping

19. Know why this evaluation is different: (a) the Underspecification Gap — "make the dashboard load faster" is not a test case; first judge whether the agent reconstructed the right unstated spec; (b) the user often cannot validate 600 lines of output, so the gap between "agent thinks it succeeded" and "code is correct" is the widest of any agent category; (c) the session is iterative and the codebase is state — bad early decisions compound.
20. Score all seven dimensions. User-facing: (1) intent satisfaction, (2) functional correctness — the floor, not the ceiling, and easy to game by deleting/mocking tests, (3) visual and behavioural correctness — for UI agents the artifact is the rendered output, not the code, (4) cost and efficiency — token spend, wall-clock latency, tool-call count, iteration count (1-turn convergence vs 8 corrections is a different product). Internal: (5) code quality and convention matching — a diff that passes tests but violates codebase style is a failure, (6) trajectory quality — correct output from bad reasoning is fragile success, (7) self-repair behaviour. Safety and responsible AI is transversal — evaluate it alongside every dimension (vulnerabilities, refusals, content safety, IP exposure), not as a separate gate.
21. Map methods to dimensions: automated functional testing (pytest, jest, eslint, mypy in CI) → dims 2, 5; static security scanning (Snyk, Semgrep; git-secrets for credential leaks) + scripted red-team suites → cross-cutting safety; LLM-as-judge / Agent-as-judge → dims 1, 5, 6; browser-based testing (Playwright scripts, screenshot comparison) → dim 3; trajectory inspection over OpenTelemetry span data → dims 6, 7; human review → dims 1 and 5, mainly to calibrate the other methods (it doesn't scale); online evaluation — sample live traffic against the same rubrics, biased toward high-cost, many-correction, and abandoned sessions (a flat 1% misses the long tail).
22. Use benchmarks for cognitive calibration only, never as the ship gate: Vibe Code Bench (zero-to-one web app generation), SWE-bench Verified (code changes on real GitHub repos), LiveCodeBench (contamination-resistant signal), and Kaggle Standardised Agent Exams (SAE) — zero-setup evaluation via a SKILL.md API integration where the agent registers, fetches exam questions, runs them in its own sandbox, and publishes to a public leaderboard. A high SWE-bench score gives zero guarantee of production intent satisfaction; beware benchmark overfitting.
23. Make observability the prerequisite: capture agent.session spans (whole task), agent.think spans (reasoning before action), agent.tool spans (arguments, latencies). Use tail-based dynamic sampling — keep traces with errors or excessive self-repair loops, drop routine successes.
24. Applied tips: (a) treat the first 1–2 user messages as the intent rubric — derive 3–5 acceptance criteria from the session prefix (the source uses gemini-3-pro), then score every turn 1–5 against them; (b) judge the rendered artifact, not the code — a multimodal judge scores screenshots on layout_match, styling, interactive_correctness (1–5 each), paired with Playwright assertions for interactivity; (c) evaluate session convergence, not turn-level accuracy — abandoned sessions are the most informative failures; track converged / turns_to_converge / abandoned / cost_to_converge per trace (Cloud Trace via ADK and Agent Engine); (d) mine user corrections as labeled failure data — embed them (text-embedding-005), cluster (e.g. KMeans, n_clusters=8), and treat the clusters as the prioritised failure-mode list.

Checklist

Harness / before execution:

• Generated code runs only in an ephemeral, kernel-level, network-isolated sandbox; state resets between runs
• Dependencies from vetted/internal registries; versions cryptographically pinned; CI/CD verifies SBOM + signatures (Binary Authorisation)
• Egress non-interactive: offline caches or pre-sanitised crawl services only
• Agent has a dedicated agentic identity (SPIFFE ID), JIT-downscoped ephemeral tokens — never the developer's ambient credentials
• File-tree allowlist, deny-by-default on secrets, build scripts, production manifests
• Hooks fire before tool calls / after file edits; LLM firewall + Agent Gateway with Contextual Authorisation on all tool/A2A calls
• Version-control checkpoint created before any codebase modification; Trust Score circuit breaker armed
• High-stakes actions require elicitation + hardware MFA + a Vibe Diff

Ship gates on the generated code:

• Build, test suite, linters pass in CI (pytest / jest / eslint / mypy)
• SAST + SCA run as deterministic CI/CD checks (IDE linters are advisory only)
• Snyk/Semgrep scan clean; git-secrets finds no credential leaks
• Retrieved/pasted code content-scanned; no zero-width Unicode or homoglyphs
• No API keys, password validation, or session flags client-side; backend default-deny and row-level security enabled
• Test files unmodified by the agent (diff the test directory); no deleted or mocked-out tests
• UI output verified on the rendered artifact: Playwright assertions + multimodal judge on screenshots
• LLM-as-judge score against acceptance criteria derived from the session prefix
• Trajectory inspected; token cost, latency, and iteration count within budget
• Diff is a small, reviewable batch; human Logic Review (plain language) for high-impact changes

Anti-patterns

• Trusting code because it compiles and runs — the "It Works, Ship It" fallacy; compilation says nothing about bypassed backend controls.
• Simple approve/deny buttons on high-risk actions — confirmation fatigue makes developers blindly authorise code they don't understand.
• Hard-blocking insecure prompts in the IDE — easily bypassed and high-friction; advise in the IDE, enforce deterministically in CI/CD.
• A domain allowlist as the only egress control — cannot stop indirect prompt injection inside allowed pages.
• Shared, long-lived service identities, or running the agent under the user's delegated credentials.
• Killing a compromised agent's container mid-thought — corrupts connected APIs; use a Stateful Quarantine instead.
• Static SBOMs as the runtime inventory — instantly stale for agents; monitor the live AgBOM.
• Reading HTTP 200 / passing unit tests as success — may mask hallucination loops or gamed tests.
• Letting the agent edit tests and implementation in the same change; deleting or mocking tests to turn red green.
• Benchmark overfitting — top scores on static benchmarks with no evaluation of real user intent.
• Flat-rate (e.g. 1%) sampling of production sessions — misses the informative long tail.
• Using UEBA for agent monitoring — built for humans, ineffective on non-deterministic agents.

As the whitepaper's conclusion puts it: "Generation is largely a solved problem. Verification, security, and architectural judgment are the new craft."

Source

• Vibe Coding Agent Security and Evaluation — Sokratis Kartakis, Aron Eidelman, Wafae Bakkali, Meltem Subasioglu (Google; Kaggle whitepaper series) — https://www.kaggle.com/whitepaper-vibe-coding-agent-security-and-evaluation — May 2026 (document date; published with the June 2026 Kaggle course materials).

Distilled from the official document(s) above on 2026-08-12. If this skill and the source disagree, trust the source.