From Vibing Chaos to Reliable Agents
Harness Engineering, Loop Engineering, and Spec-Driven Development in the Google Ecosystem
Why Unconstrained "Vibe Coding" Collapses on Monday Morning
- Hallucination City — Invented APIs, deprecated SDKs, and phantom packages
- Review Theatre & "LGTM" Fatigue — Bleary-eyed rubber-stamping of sprawling diffs
- Unmaintainable Tech Debt on Day 1 — Incoherent architecture with zero documentation
- Insecure by Default — Hardcoded secrets, wide-open permissions, and suppressed warnings
- The Human Clipboard Furnace — Manually copying terminal errors back into a bloating chat window
Wait... What Do All These Terms Actually Mean?
Click any term below — are they competing religions, or interlocking parts of a single engineering machine?
Agent = Model + Harness.
What "Shift-Left" Means Here: Stop Debugging on the Right
🔴 Vibe Coding: Intervening on the RIGHT (Downstream)
Breezy chat prompt → unconstrained AI dumps 500 lines → human acts as an exhausted manual filter debugging diffs and hallucinations after failure occurs.
🟢 Harness Engineering: Intervening on the LEFT (Upstream)
Engineer the environment before the agent runs: GEMINI.md rules, living specs, container sandboxes, and automated evaluators heal errors inside the loop.
Agent = Model + Harness — Waiting for a Bigger Model Won't Save You
⚠️ The Model Upgrade Fallacy
- The Knee-Jerk Instinct: When an agent fails in production, teams immediately blame the model and switch to a massive 2-trillion-parameter frontier model at 4x the token cost.
- High-Priced Slop: If an agent lacks repo context, can't run tests, and has no verification loop, a bigger model just generates more articulate, harder-to-spot slop at four times the price!
🛡️ Model-Agnostic Reliability
- Build a Good Harness → Get Reliable Results: A fast, cost-effective workhorse like
Gemini 3.8-Flashin a well-engineered harness beats a giant frontier model in a weak harness. - Insulated From Model Variance: A strong harness insulates your engineering quality from individual model quirks, inconsistencies, and version changes.
- Commercial Truth (GDE Jaroslav Pantsjoha): The model is a rented commodity; the harness is your proprietary IP.
The Model is the Driver. The Harness is the Entire Race Car.
🏎️ THE DRIVER = AI MODEL
Probabilistic reasoning & reflexes (e.g. Gemini 3.8-Flash). Without a car, he just does doughnuts on an airfield.
🛡️ CHASSIS & BRAKES = STRUCTURAL HARNESS
GEMINI.md rules, MCP tools, skills, sandboxes, and evaluation gates stopping 180 mph crashes.
⚙️ POWERTRAIN = LOOP ENGINEERING
The dynamic runtime engine inside the harness — iterative intake, combustion, telemetry, and self-correction.
The Deterministic Components That Make Up the Harness
Our Talk Touchpoints
- Hierarchical Context (
GEMINI.md/AGENTS.md) - Tools, MCP Servers & Progressive Skills
- Living State & File Memory (
SPEC.md,plan.md,LIVING_MEMORY.md) - Evaluation Gates & Loop Engine (
RUBRIC.md) - Execution Sandboxing (Blast Radius Control)
- Observability & Trajectory Monitoring
Real Example: Inside My Production GEMINI.md
# Agent Execution Guardrails & Standards (~/.gemini/GEMINI.md) ## Python Guidance & Tooling - Environment Source of Truth: Respect `pyproject.toml` (default `python 3.13`). Manage environments and dependencies exclusively via `uv`. - Avoid Inline `python -c` Commands: Do NOT run inline `python -c "..."` snippets (they break prefix-matched approvals). Write one-off scripts to `scratch/` and execute cleanly via `uv run python scratch/helper.py`. - Standards & Quality: Include top-of-module docstrings, built-in type hints (`list`, `dict`), and `rich` for CLI output. - Strict TDD: Write the failing test first, verify failure, implement, verify pass. - Mandatory Post-Edit Gate: Run from project root after modifying any `.py` file: uvx codespell@latest -s # check spelling & show summary uvx ruff@latest check --fix . # static analysis & auto-fix ## Agent & AI Development - NEVER downgrade a Gemini model (e.g. NEVER change `gemini-3.8-flash` to `gemini-3-flash`). - Build agents using Google ADK (`google-adk`) and Google GenAI SDK (`google-genai`). Avoid deprecated `google-generativeai`. - NEVER push to Git without explicit human sign-off.
Why This Eliminates Slop
- Layered Context Scopes: Global (
~/.gemini/GEMINI.md), team/domain, and repo-specific rules (workspace/GEMINI.mdorAGENTS.md). - Deterministic Tooling: Mandates
uvand strict TDD before writing implementation code. - Prefix-Match Ergonomics: Banning
python -cstops approval prompts on every turn. - Self-Cleaning Hygiene: Every edit triggers
codespellandruffautomatically. - Blast Radius Control: Strict prohibition on unapproved Git pushes keeps humans steering.
Whistle-Stop Tour of Agent Skills (They Come Up Everywhere!)
# ~/.gemini/config/skills/loopy/SKILL.md --- name: loopy description: Discover, audit, craft, and run repeatable AI-agent loops with clear checks and stopping conditions. --- # Step-by-Step Playbook (Loaded ONLY when triggered!) 1. Inspect target repo & verify deterministic test runner... 2. Set clear stopping conditions (100% pass or max-turn limit)...
Progressive Disclosure = Reduced Context Bloat
- Tool & Skills Sprawl Traps: Connecting 20 MCP servers (150+ schemas) or stuffing 40 guides into
GEMINI.mdburns thousands of tokens every turn and confuses the model. - Progressive Disclosure Levels:
• Level 1 (Metadata): Only the 2-line YAML frontmatter (<100 tokens) loads at startup.
• Level 2 (Instructions): FullSKILL.mdplaybook loads on demand when a matching task begins.
• Level 3 (Resources): Deepscripts/, schemas & templates load only when needed.
Plugins & My Curated Production Toolkit
- Plugins = Packaging Vehicle: Not a separate harness component — a distribution bundle uniting Rules + MCP Servers + Agent Skills (e.g.
google-cloud-developerwith livedeveloper-knowledgeMCP). - Core Production Skills:
code-review-and-quality,Google Agents CLI (ADK),agent-platform-eval-flywheel,find-skills/organise-agent-skills,maintaining-core-documentation,spec-driven-development&loopy.
The Harness Must Be Automated & Loop
The Closed-Loop Cadence
- 1. ACT → 2. CHECK → 3. DIAGNOSE → 4. ADJUST → 5. RETRY or STOP/ESCALATE: Converge autonomously to 100% pass, or trip a circuit breaker.
- Human-in-the-Loop ≠ Human-in-Every-Turn: Replace continuous babysitting with strategic governance at predefined escalation points.
- Dev Workbench vs Runtime (Spec Spend vs Run Spend): The loop is your development-time forge — not a self-modifying
whileloop shipped into customer-facing production.
How to Construct the Loop (And Avoid a Token Bonfire!)
Core Loop Building Blocks
- Context & Goal — Strictly Decoupled:
Context (GEMINI.md+ Skills) = reusable "how we build".
Goal (spec.md) = specific "what we build today", restated every turn. - Sandboxed Tools & TDD + Evals: Safe file/test runners + failing unit tests written first + deterministic linters + semantic Critic forming our Rubric (preventing The Premature Exit).
- Structured Iteration Memory (
LIVING_MEMORY.md): Saving score deltas, root causes, and "Do NOT Repeat" notes after each turn. - Iteration Discipline (The One-Change Rule): Modify one semantic behaviour at a time to isolate regressions (while batching mechanical lint/typo fixes in one pass).
- Escalation Protocol & Stopping Conditions:
✓ 100% Rubric Pass • 🛑 Max Turn Limit (e.g. 5 turns) • ⚠️ Escalate to Human if Score Plateaus or Hits Protected Boundaries
The Ratchet Principle: Locking In Institutional Memory
Loop Bug vs Harness Bug Litmus Test
- Loop Bug (Refine the Loop): Fixed by sharpening the goal, refining the prompt, updating
RUBRIC.mdchecks, or adjusting the stopping condition. - Harness Bug (Upgrade the Harness): Fixed by changing tool definitions, adjusting sandbox permissions, modifying timeouts, adding a
SKILL.md/GEMINI.mdrule, or intercepting runtime errors. - Permanent Immunity (The Ratchet): Once encoded into the harness, that entire class of failure is extinct across all future sessions.
Great Harness + Bad Prompt = Slop. The Spec Is Now the IP.
Why Our Race Car Needs High-Octane Fuel
- We've locked down HOW we build in
GEMINI.md, MCPs, Skills, and TDD rubrics. - But if you feed a vague 2-line prompt into that harness, you still get slop — just well-linted, type-checked slop that builds the wrong thing!
- We need a persistent specification defining WHAT we build.
🔥 Controversial Message: Code is Disposable
- Code Is No Longer What We Care About: Your harnessed agent can spit out verified implementation code in 90 seconds, over and over again.
- Specifications Are the Permanent Source of Truth: When requirements change, don't untangle code — update the spec and regenerate!
Updating the Spec With Each Loop: The Logical Conclusion of Agile
Zero Compounding Specification Drift
- 1. Scope & Edge Discovery (The WHAT): Quick dialogue pinning down intent, non-goals, and edge cases.
- 2. Interface Contracts & Boundaries (The HOW): Schemas + Always Do / Ask First / Never Do boundaries.
- 3. Acceptance Criteria (The PROOF): Gherkin scenarios & failing test stubs.
- 4. 90-Second Sandbox Build & Verify: Inner loop converges to 100% green tests.
- 5. Inspect & Evolve the Spec: Update the spec on disk for the next iteration — documentation never drifts!
What Goes in a Spec? It's Version-Controlled Markdown
# Implementation Spec: Cloud Spanner Audit Event Logger ## 1. Architectural Scope & Contracts - Component: `AuditEventLogger` service in `services/audit_logger.py` - Persistence: Cloud Spanner `audit_events (event_id, timestamp, actor, action, payload JSON)` - Transport: Async gRPC client via `google-cloud-spanner` Python SDK ## 2. Operational Boundaries (Always / Ask / Never) - ALWAYS DO: Enforce type hints; Spanner Mutation batches; author failing test in `tests/` first; run `ruff` & `codespell`. - ASK FIRST: Adding dependencies to `pyproject.toml`; altering Spanner DDL. - NEVER DO: Hardcode credentials (use ADC); run inline `python -c`; swallow generic `Exception`. ## 3. Phased Vertical Slices & Verification Gates - [ ] Phase 1: Domain Models & Failing Test Stub (`pytest` -> fails `NotImplementedError`) - [ ] Phase 2: Batch Mutations & Backoff Retries (`pytest` -> 100% pass) - [ ] Phase 3: Static Hygiene (`uvx ruff check --fix .` & `codespell -s` -> 0 warnings)
tasks/plan.md (Implementation Plan & Boundaries)
Why Structured Markdown + YAML + Gherkin Works Best
- Two Spec Layers:
spec.md(user-facing WHAT & non-goals) +plan.md(codebase-aware HOW & file impact map). - Hybrid Markdown + Conditional YAML: Use clean Markdown headers and bullets for narrative, and flat, compact YAML for structured schemas, configs, and data contracts (especially when nesting depth > 3) instead of verbose JSON or XML.
# Declarative Gherkin Acceptance Contract Scenario: Booking a table with active dietary preferences Given a customer with dietary preference "gluten-free" in profile When they request a reservation for 2 people at 19:00 Then verify table availability via Cloud SQL reservation tool And append "gluten-free" to booking notes payload before committing
spec.md (Acceptance Contract)
Practical Tools to Build & Govern Your Specifications
1. spec-driven-development Skill ↗
By Addy Osmani. Conversational gate (Scope Check → Specify → Plan → Tasks → Implement). Pauses to surface explicit "ASSUMPTIONS I'M MAKING" and outputs tasks/plan.md.
2. GitHub spec-kit ↗
CLI & slash workflow (/speckit.specify → .clarify → .plan → .tasks → .implement) governed by .specify/memory/constitution.md, plus lightweight Bug Fixing & Idea Assessment workflows.
3. Google Conductor Extension ↗
Now natively integrated in Google Antigravity! Interviews you in the IDE sidebar to maintain product.md, tech-stack.md, spec.md, and plan.md before unlocking code gen.
Doing It Super Easily with Google Antigravity
What Antigravity Coordinates Out of the Box
Developer Surfaces + SDK: Antigravity App • Antigravity IDE • IDE Extension (VS Code, JetBrains, Visual Studio, Zed) • Antigravity CLI • Python SDK.
Built-In Plumbing: Auto-binds GEMINI.md, progressive Skills, MCP, container sandboxes, and isolated Subagents.
Combining Formal SDD on Disk + Interactive Slash Commands
- 1. Draft Persistent Spec on Disk (Formal SDD): Native
/plan&/goalalone are ephemeral in chat — use Conductor,spec-kit, orspec-driven-developmentto committasks/plan.mdto Git. - 2. Stress-Test the Spec (
/grill-me): Run/grill-meagainsttasks/plan.mdas a sceptical peer reviewer to expose edge cases before writing code. - 3. Tactical Slice Execution (
/plan+/goal+/boost): Feed one slice fromtasks/plan.mdinto/goal(with/planfor file edits and/boostfor adversarial Builder/Critic review). - 4. Harden the Harness (
/learn): Crystallise runtime discoveries into.agents/rules/orGEMINI.md!
Deterministic Tests + Binary LLM-as-a-Judge Rubrics
LLM: "Why yes, Dazbo! So hunky!"
0 / 1 binary assertion graded by an independent, clean-slate Critic agent.
| Category | Strict Binary Assertion (1 = Pass, 0 = Fail) |
How It's Checked |
|---|---|---|
| Deterministic | All unit tests in tests/ pass (100%) AND ruff / codespell report zero errors? |
pytest, ruff & codespell |
| Deterministic | Built-in type hints used AND zero eval(), exec(), or unparameterised queries? |
Static Code Scanner (ruff / bandit) |
| Deterministic | Deterministic Tool Trajectory: Only allowlisted tools invoked with valid schemas? | Agent Tool Execution Trace |
| Deterministic | Live API & UI Wiring: FastAPI starts cleanly & frontend buttons call real /api/game/* endpoints? |
FastAPI TestClient & HTML Check |
| Semantic Critic | Goal Fidelity, Scope Discipline & Clean Architecture: Delivers spec capabilities, zero hallucinated files, and direct tool pathways (no convoluted detours)? | Independent Critic Agent (LLM Judge) |
| Semantic Critic | Remediation Clarity, Design Intent & UX Ergonomics: Actionable error messages, sound design rationale, and responsive interactive gameplay? | Independent Critic Agent (LLM Judge) |
Side-by-Side Workbench: Vibe Coding vs The Engineered Harness
What Both Tracks Are Building: Cosmic Conquest
- Same Spec (
cosmic_conquest_spec.md): Full-stack FastAPI backend + directed sector graph + turn-based combat + interactive browser star map. - Left Track (Unharnessed Vibe): Bare ADK
Agent— single pass, noharness_context.md, no skills, no loop. - Right Track (Harnessed Loop): ADK
LoopAgent(harnessed_builder_agent+RubricEvaluationGateAgent) + 3 Skills (api-and-interface-design,gemini-api-dev,test-driven-development) +LIVING_MEMORY.md+ 14-point binary rubric.
The Amazing Conclusion: 4x Faster, 85% Cheaper, & Actually Works!
| Metric | Unharnessed (Vibe) | Harnessed (Spec + Loop) | Harness Advantage |
|---|---|---|---|
| Execution Time | 90–120+ seconds | 23–35 seconds | 3–4x Faster |
| Combined Tokens | 18,000–45,000 (~16k out) | ~6,700 (~1,000 out) | ~80% Fewer (~15x less out) |
| Estimated API Cost | ~$0.010–$0.017 | ~$0.0014–$0.0015 | ~85% Cheaper (7–8x less) |
| 14-Point Rubric | 6–11 / 14 (Always fails) | 14 / 14 (100% pass) | Deterministic Quality |
| Functional Game? | ❌ Broken / Dead UI | ✅ Playable & Verified | Only Harnessed Works! |
How Can a Multi-Turn Loop Be Faster & Cheaper Than 1 Prompt?
1. Output Discipline (code to disk, 150-word chat summary → 15x fewer output tokens) •
2. Externalised Reasoning (ThinkingLevel.LOW + millisecond pytest/ruff checks) •
3. Context Pruning (passing only LIVING_MEMORY.md between turns).
Inside Every Iteration: Fresh Context & Output Discipline
1. Why Single-Thread Chat Loops Fail
- Runaway Context Bloat: Every failed draft and 200-line stack trace stays in the chat history.
- Going in Circles: By Turn 4, the model retries a fix that failed on Turn 1 because the error is buried in noise.
- Broken Edits & Drift: Confuses old code drafts in chat with the actual files on disk.
2. Fresh Context per Pass (Clean Subagents)
- Clean Slate Every Turn: Delegate each loop pass to a fresh subagent with a clean context window.
- Receives Only 3 Things:
1. The Target Spec Slice (PLAN.md)
2. Current Workspace Files on Disk
3. The CompactLIVING_MEMORY.mdReceipt - Subagent edits, runs tests, writes receipt, and finishes!
3. Strict Output Discipline (~15–20x Token Reduction)
- Output Tokens Drive Latency & Cost: They are the biggest driver of both how long a loop takes and how much it costs!
- Keep Code in Files, Not Chat: Write code directly to disk via tools; ban full-file dumps in chat replies.
- 150–250 Word Turn Summary: Reply only with what was tackled, root cause of any failure, and files updated.
Persistent LIVING_MEMORY.md Across Iterations (Real Example)
# Living Memory: Iteration Ledger (LIVING_MEMORY.md) ## Turn 1: UserSchema Migration (FAILED) - Changes Made: `app/schemas/user.py` (lines 12–48): replaced `class Config` with `model_config = ConfigDict(from_attributes=True)`. - Verifier Output: `uv run pytest tests/test_user.py`: 16 passed, 2 failed. - Error: `PydanticUserError: '@validator' is deprecated and no longer supports 'each_item=True' in v2 (app/schemas/user.py:34)`. - Root Cause: Left legacy Pydantic v1 `@validator('roles', each_item=True)`. - Do NOT Repeat: Do not use `@validator` in Pydantic v2 models. ## Turn 2: UserSchema Migration (FAILED) - Changes Made: `app/schemas/user.py` (lines 34–58): migrated to `@field_validator('email', 'roles')`. - Verifier Output: `pytest`: 17 passed, 1 failed (`test_invalid_email_code`). `ruff`: 0 errors. - Rubric Delta: 13 / 14 passed (Failed Check 1.2: API error `'invalid_email'`). - Root Cause: Plain `ValueError` sets error `type` to `'value_error'`, breaking v1 API contract. - Next Hypothesis (Turn 3): Raise `PydanticCustomError('invalid_email', ...)` from `pydantic_core` inside `validate_email`.
Why LIVING_MEMORY.md Works
- Bridges Fresh Subagents: Gives Turn 3 complete institutional memory of Turns 1 & 2 in ~200 tokens instead of 20,000 tokens of raw logs.
- Explicit "Do NOT Repeat" Guardrail: Stops the agent oscillating between two broken approaches.
- Tracks Score Progress: Shows clearly whether each turn is improving the rubric score or stuck in a rut.
- Is a Single Rubric Pass Enough? 1 pass suffices for deterministic code gates (
pytest,ruff); when building an agent, prompt, or skill, require multi-trial consecutive passes (e.g. 3/3 runs).
Ways You Can Add a Loop to Any New or Existing Repo Today
| Mechanism | What It Supplies | Setup Time |
|---|---|---|
| Antigravity Slash Commands ( /grill-me, /plan, /goal, /boost, /learn) |
Interactive stress-testing, hands-off loop execution, multi-agent critique, and ratcheting. | 0 mins (Built-in) |
Loopy ↗ (loopy skill + LOOPS.md + Loop Library) |
Ready-made or Q&A-crafted loop prompts, verification gates, stop rules, and a shared repo loop catalogue. | 1–2 mins |
| Custom Markdown ( AGENTS.md, RUBRIC.md, LIVING_MEMORY.md) |
Standing engineering rules, binary pass/fail criteria, and persistent cross-turn memory. | 5–10 mins |
| Antigravity CLI & SDK | Headless CI/CD execution (-p, --output-format json), Python @hooks.stop gates, and budget caps using Antigravity's built-in coding harness. |
5–15 mins |
| Google ADK ↗ ( google-adk LoopAgent & Workflow Graphs) |
Custom multi-agent topologies, bespoke evaluation gates, and conditional graph routing. | 15–30 mins |
Self-Improving Harnesses — & Implementing the Ratchet Yourself
The Outer Meta-Loop Concept (Harness²)
- Keep the Model Fixed, Upgrade the Harness: Instead of only tweaking prompts, an outer meta-loop (
Propose → Probe → Compose) evolves the entire harness:System Prompt,Guardrails,Memory,Tool Guidance,Subagents,Skills,Scripts, andPlugins. - Cross-Task Memory (
EXPERIENCE.md): Keeps the base harness clean while recording operational lessons across tasks so future runs never start from scratch.
How You Implement This Ratchet in 2 Minutes Today
- Step 1 (Inspect): Whenever a loop took >1 turn to pass
RUBRIC.md, inspectLIVING_MEMORY.md. - Step 2 (Log to
EXPERIENCE.md&/learn): RecordCue → Harness Upgrade → Evidencein.agents/experience/EXPERIENCE.md(instead of bloatingGEMINI.md) and update the relevantSKILL.md, script, orRUBRIC.md. - Step 3 (Apply): Apply the harness upgrade so future tasks pass first time.
Recap: Everything That Makes Up Our Complete Harness
Model-Agnostic Reliability
A strong harness insulates your engineering quality from individual model quirks, inconsistencies, and version changes — delivering reliable results regardless of the underlying model.
GEMINI.md & Progressive Skills
Hierarchical mechanical standards (uv, TDD, Output Discipline) + on-demand SKILL.md playbooks without Skills Sprawl.
MCP Tools & Container Sandbox
Live truth via MCP + isolated container execution controlling blast radius.
Persistent Living Specs
Code is disposable; the spec is permanent IP. spec.md & tasks/plan.md with Always/Ask/Never boundaries & Gherkin contracts.
Fresh Subagents + Binary Rubric
Fresh subagent per pass + LIVING_MEMORY.md receipts + Two-Tier 0 / 1 RUBRIC.md (Deterministic pytest/ruff + Semantic Critic) + Circuit Breakers.
The Ratchet (EXPERIENCE.md & /learn)
Every multi-turn stumble logs cross-task lessons in EXPERIENCE.md and upgrades GEMINI.md, LOOPS.md, or RUBRIC.md so the entire class of failure is permanently extinct.
Your Practical Harness Engineering Checklist
GEMINI.md / AGENTS.md: Enforce toolchain (uv), TDD, and post-edit ruff + codespell.developer-knowledge MCP, spec-driven-development, and loopy (no Skills Sprawl)./grill-me): Author a persistent tasks/plan.md with Always Do / Ask First / Never Do boundaries.RUBRIC.md (0 / 1): Combine deterministic pytest/ruff checks with an independent Semantic Critic.LIVING_MEMORY.md & Set Stopping Conditions: Log root causes each turn, and halt if stuck or hitting a turn limit (e.g. 5 turns).EXPERIENCE.md & /learn): Log cross-task lessons in EXPERIENCE.md and turn every multi-turn stumble into a permanent harness upgrade!Links, Code & 4-Part Series
🌐 This Interactive Deck: harness-engineering.dazbo.co.uk
📖 4-Part Blog Series (Part 1): Dev.to (GDE) & Medium (Google Cloud)
💻 Workbench Repo: github.com/derailed-dash/harness-engineering-demo