Why Your AI Agent Keeps Failing: A Deep Dive into Harness Engineering

OpenAI Shipped 1 Million Lines of Production Code with Codex in Five Months. Zero Lines Written by Humans. Here’s the Engineering behind It.

In February 2026, OpenAI published a blog post describing how their engineering team used Codex to write 100% of the code across 1,500 pull requests over five months. One line in the post stuck with me:

The primary job of the engineering team became to enable the agents to do useful work.

Around the same time, LangChain engineers ran a controlled experiment: same model (gpt-5.2-codex), a different system around it. Terminal Bench 2.0 score jumped from 52.8% to 66.5% — rank 30 to rank 5.

That 13.7-percentage-point improvement came entirely from changes to the Harness, not the model.

Agent = Model + Harness. The model sets the ceiling. The Harness determines how close you get to it.

This article isn’t about what Harness Engineering is. It’s about why agent systems fail in specific, predictable ways — and how each Harness component is engineered to counter those failures.

The Four Structural Failure Modes

These aren’t bugs. They’re structural properties of LLMs.

Failure 1: The state doesn’t survive across sessions. LLMs have no persistent memory. Each session starts with a clean context window. In a complex project, the next session has no idea what the previous one did, how far it got, or what problems it hit. The agent either starts over from scratch or repeats work already done.

Failure 2: One-shot Greed. Agents tend to try completing the entire goal in one shot. For complex projects, this means the context window runs out mid-task, leaving a codebase in an unpredictable half-finished state. The next session inherits a mess.

Failure 3: Premature Completion. Agents claim tasks are done when they aren’t. This isn’t deception — it’s an unreliable self-evaluation mechanism. The agent sees the code is written and concludes the feature works. These are different things.

Failure 4: The Doom Loop. When an agent hits a hard problem, it retries the same approach with slight variations. LLM reasoning, under difficulty, tends to stay within a similar solution space rather than stepping back to reassess. Ten identical failing attempts is a common pattern.

Each Harness component addresses one or more of these failures at the system level.

Component 1: Readable Environment

All states must be externalized. LLMs are stateless, so the environment has to carry it. OpenAI’s team initially put everything into one massive AGENTS.md — and ran into three problems fast:

1. Context pollution: The larger the file, the lower the signal-to-noise ratio for any given task
2. Staleness velocity: Big files become authoritative-but-wrong faster than anyone can maintain them
3. No progressive disclosure: An agent planning a small task doesn’t need the full system spec, but a monolithic file forces it to process everything

The fix is information architecture, not information accumulation. AGENTS.md becomes a table of contents pointing to sub-documents:

AGENTS.md              ← lightweight entry point, rarely changes 
├── product-specs/     ← user stories + acceptance criteria (split by feature) 
├── design-docs/       ← architecture decisions + ADRs 
├── exec-plans/        ← current execution plan, updates frequently 
├── db-schema/         ← database schema, auto-generated preferred 
└── security/          ← security rules, manually maintained

The entry point stays stable — AGENTS.md itself rarely changes, so each session can always find current sub-documents without loading the entire document tree. Each sub-document has a single responsibility: one thing per file, loaded on demand. LangChain implements this as LocalContextMiddleware, which scans the directory structure on session start and injects only the context relevant to the current task.

AgentsMesh’s 52-day practice took this to its logical conclusion: the repository itself is the most important context — no separate RAG system needed. The prerequisite is that the documentation is actually maintained, not decorative.

Component 2: Task State Machine

State loss, one-shot greed, premature completion — three failures at once. OpenAI’s solution is externalizing task management as a JSON state machine:

{ 
  "id": "auth-001", 
  "title": "Email login", 
  "spec": "Email + password login. Success returns JWT, failure returns 401.", 
  "acceptance_criteria": [ 
    "POST /auth/login accepts email and password", 
    "Wrong password returns { error: 'invalid_credentials' }", 
    "Token expires in 24 hours, stored as httpOnly cookie." 
  ], 
  "status": "fail" 
}

status defaults to fail, not pending. That's a deliberate choice. pending is neutral — "hasn't been done yet." fail is negative — "hasn't passed yet." The distinction matters at the cognitive level. The agent isn't completing tasks; it's proving tasks pass. OpenAI broke the entire project into 200+ tasks, all of which failed by default. The agent's job is turning them into pass.

The acceptance_criteria field is a machine-readable contract, not human documentation. Every criterion should be programmatically verifiable — if you can write it as a test, it should run as a test. This is the minimum viable Spec unit inside a Harness.

And Feature List + git log together form the current state snapshot. Each new Coding Agent session reads both to reconstruct the project state in under 30 seconds, without depending on anything the previous agent left implicitly. This is what breaks the state loss cycle.

The two-layer agent split matters here:

• Initializer Agent: writes no business code. Builds the inheritable environment — Feature List JSON, init.sh (dev server startup), PROGRESS.txt (progress summary), initial git commit
• Coding Agent: each session reads state → picks highest-priority fail task → implements → verifies → marks pass → commits → updates PROGRESS.txt. Session must end in a clean state — hard constraint, not suggestion

Component 3: Verification Loop

The premature completion bias is structural, not a prompt engineering problem. After writing code, the agent self-evaluates inside the same context window in which it just wrote code. That context is full of “code is done” signals. The prior probability of “task complete” is already skewed before evaluation starts.

LangChain’s solution is PreCompletionChecklistMiddleware:

class PreCompletionChecklistMiddleware(AgentMiddleware): 
    def before_complete(self, state: AgentState) -> AgentState: 
        if not state.get("verification_done"): 
            state.inject(SystemMessage( 
                "Before marking complete, run the full verification checklist: " 
                "1) All acceptance_criteria tests pass " 
                "2) No regressions in existing tests " 
                "3) End-to-end flow verified" 
            )) 
            state.set("verification_done", False) 
        return state

This exploits a known LLM behavior: when you explicitly tell the agent its work will be evaluated by programmatic tests, self-verification behavior changes significantly. The system message is an anchor. The middleware forces the verification flow before exit.

OpenAI took a different approach — Chrome DevTools Protocol integrated directly into the agent runtime:

# Agent verification workflow 
agent.reproduce_bug(dom_snapshot=True, screenshot=True) 
agent.implement_fix() 
agent.validate_fix(record_video=True)  # record "after" video 
agent.submit_pr(evidence=[before_video, after_video])

PRs include video evidence. Reviewers don’t need to rebuild the environment. Verification evidence is a replayable video, not a text description. Video doesn’t lie.

AgentsMesh implemented a four-layer feedback loop:

Compile → Unit tests (700+) → E2E → CI 
  ↓ hot reload  ↓ real-time    ↓ full flow  ↓ multi-platform

Every failure feeds back to the agent immediately. Strong typing (Go compiler, TypeScript, Protobuf) catches a large class of errors at compile time, before anything reaches E2E.

One data point: LangChain tested different reasoning budget allocations — maximum reasoning for planning, medium for implementation, maximum again for verification. They called this the “reasoning sandwich.” Result: 63.6% vs 53.9%. In the verification phase, reasoning quality matters as much as planning. Don’t cut it just because the code is written.

Component 4: Architecture Enforcement

The AgentsMesh author’s summary: agents copy every pattern in the codebase, including the bad ones. That’s not a bug — it’s the core LLM mechanism. Pattern matching and replication. In a codebase with technical debt, the debt spreads at the rate of agent-generated code.

The fix is encoding architectural constraints into tooling, not documentation.

OpenAI’s implementation: domain-based repo structure, unidirectional dependency flow, no circular dependencies, enforced through custom linter + structural tests on every git pre-commit:

# .git/hooks/pre-commit 
#!/bin/sh 
# Check dependency direction 
npx check-deps --config .dep-rules.json || exit 1 
# Check naming conventions 
npx lint-names --strict || exit 1 
# Check architecture boundaries 
go test ./cmd/check-arch/... || exit 1

Violations don’t produce warnings. They block the commit.

Strong typing is free architectural enforcement. The Go compiler, TypeScript’s type checker, and Protobuf schema definitions — errors caught at compile time don’t reach E2E testing. AgentsMesh pushed as many constraints as possible to compile time to reduce the cost of the verification loop.

Architectural discipline is day-one work, not day-100 work. Traditional teams delay architecture boundaries until scale forces the issue, because engineers have intuition — they sense “this doesn’t feel right.” Agents don’t have that intuition. They only follow rules encoded in tooling. Waiting until the codebase has significant technical debt before adding constraints costs far more than starting from day one.

Component 5: Loop Detection

When an agent hits a hard problem, it retries similar approaches with increasing effort. Not a random walk — directed. Each attempt is “harder” than the last, but the direction is wrong. Ten identical failing attempts is a common pattern, token consumption is linear, and the problem remains unsolved.

LangChain’s LoopDetectionMiddleware:

class LoopDetectionMiddleware(AgentMiddleware): 
    def __init__(self, threshold: int = 5): 
        self.file_edit_counts: Dict[str, int] = {} 
        self.threshold = threshold 
  
    def after_edit(self, file: str) -> Optional[Intervention]: 
        self.file_edit_counts[file] = self.file_edit_counts.get(file, 0) + 1 
        if self.file_edit_counts[file] > self.threshold: 
            return Intervention( 
                f"You've edited {file} {self.file_edit_counts[file]} times. " 
                "Consider an entirely different approach or ask for help." 
            ) 
        return None

Track edit counts per file. Intervene when the threshold is crossed. The keyword is intervene — not informing the agent after it finishes, but breaking the reasoning inertia while the loop is happening.

The Ralph Loop pattern: for long-running tasks across sessions, LangChain reinjects the goal prompt into a clean context window at the start of each session. This prevents task objective drift over many sessions. Implementation is simple: at the start of the session, reinject the highest-priority incomplete task from the Feature List.

The cognitive bandwidth ceiling: AgentsMesh puts the human decision ceiling at 50,000 lines per day. Beyond that, manual review loses meaning — not from lack of skill, but from cognitive bandwidth exhaustion. Past that point, decisions need to be delegated to a higher-level coordinating agent, not forced on humans doing impossible review work.

How the Five Components Work Together

You can’t just add the Verification Loop and discover it depends on the acceptance_criteria field, which lives in the Task State Machine. The five components are a system. The coupling is specific.

A readable environment is the foundation for everything else. The Feature List, the architecture rules, and the acceptance criteria — all stored in the repository- are only useful to the agent if the documentation structure is real and maintained.

Task State Machine’s acceptance_criteria field is the Verification Loop's input. PreCompletionChecklistMiddleware checks against these criteria. The two components are coupled through that single field.

Architecture Enforcement’s pre-commit hook is part of the Verification Loop — each commit triggers the architecture check, one step in the four-layer feedback cycle.

Loop Detection protects Task State Machine throughput. Without it, token consumption on a single stuck task can stall the entire Feature List.

Together, the five components answer: how do you make a stateless, greedy, self-evaluation-impaired, loop-prone LLM work reliably on a complex, long-running project?

Open Source References

LangChain DeepAgents (github.com/langchain-ai/deepagents) All middleware mentioned above has open-source implementations. write_todos, LocalContextMiddleware, PreCompletionChecklistMiddleware, LoopDetectionMiddleware — use them directly or as reference. The virtual filesystem backend is pluggable.

AgentsMesh (V2EX thread) 52 days, 960K lines of throughput, 350K lines in production. The DDD layered architecture (domain / service / handler) clearly shows how architectural boundaries indicate where the agent should add code. Complete engineering implementation of the four-layer feedback loop.

DeerFlow (ByteDance open source) The deerflow-harness package decouples the agent engineering layer from business logic — a reference for "Harness as a pluggable layer."

Community list: github.com/walkinglabs/awesome-harness-engineering

Summary

The model isn’t the bottleneck. LangChain proved that, with a 13.7-percentage-point jump using the same model, a different Harness.

The four structural failure modes (state loss, One-shot Greed, Premature Completion, Doom Loop) come from the LLM architecture. Better prompts don’t fix them.

The five Harness components do: externalize state as an information architecture rather than a monolithic file; use status: fail by default and machine-readable acceptance_criteria; build verification loops with real evidence (middleware, video, reasoning sandwich); encode architecture constraints into tooling before the codebase accumulates debt; and break loops in real time rather than post-hoc.

They’re a system, not a checklist. The coupling relationships are the point.

References

• OpenAI, Harness Engineering: Leveraging Codex in an Agent-First World (Feb 2026)
• LangChain, The Anatomy of an Agent Harness (Mar 2026)
• LangChain, Improving Deep Agents with Harness Engineering (Mar 2026)
• AgentsMesh author, 52 days of solo Harness Engineering practice on V2EX (Mar 2026)
• LangChain DeepAgents documentation (Mar 2026)