Inside Webflow's MCP eval harness

A green checkmark in your agent doesn't tell you an agent will do the right thing with your tools. It tells you the tools returned valid JSON. Those are different claims, and the gap between them is where MCP servers actually fail in front of a real user, with a real agent making its own decisions. And sometimes those decisions are pretty bad when customers rely on their brand experience to be powered by a product. In this post, we'll talk about how we build more confidence in how our MCP server behaves in agents we don't control.

The problem with testing an MCP server

Webflow's Model Context Protocol (MCP) server hands AI agents, whether Claude, ChatGPT, Cursor, or whatever a person connects with, a set of tools for building pages, managing CMS content, and publishing sites on someone's behalf. Traditional tests can establish that each MCP tool is correct in isolation: it accepts the right inputs, enforces permissions, mutates the expected state, and handles failures properly. But they cannot answer a different question: when an autonomous agent sees the full tool surface and receives a plain-English task, will it discover and combine those capabilities correctly?

You only get an answer by watching an agent behave. So that's what our eval harness does. It drives real agent hosts, headlessly, against real, disposable Webflow sites, with plain-English tasks. It scores what the agent chooses to do, not just whether the tools it called returned valid responses.

What it actually does

The harness runs stories: multi-step agentic tasks written as plain prompts, each with assertions about what should happen. A story might be "create a CMS collection with three fields and one entry," or "build a full SaaS landing page headlessly and publish it."

For each story, the harness provisions a fresh site and runs a real agent against it turn by turn, currently Claude Code or OpenAI’s Codex CLI. Deterministic checks verify expected and forbidden tool use and, where relevant, verify that a claimed write actually took effect. An LLM judge reads the transcript and final answer for task completion, tool selection, and correctness. Page-building stories add a visual judge over the rendered result, while structural signals can inspect what the screenshot cannot.

Every trace, score, finding, and screenshot ships to Datadog so runs are queryable and comparable over time. That's the skeleton. The interesting part is what we learned once a single pass/fail number stopped being enough.

Did the agent actually do the right thing?

We separate outcome correctness from execution-path diagnostics. First, did the intended state change actually happen? Second, did the agent violate any genuine constraint — for example, using a destructive tool or a capability unavailable in that host? Finally, did it take an unexpectedly difficult path that suggests the tool surface could be clearer? A different path is not necessarily wrong, but it is often worth understanding.

Every run leaves a punch list, not just a score

The same LLM judge call that scores a story also produces findings: concrete, actionable problems it noticed in the transcript whether or not the story passed. A tool call failed. The agent worked around a confusing capability. A missing capability had to be faked. A story can pass every assertion and still surface findings, because "the agent got there in the end" and "the tools made that easy" are different questions.

A few findings pulled straight from real runs:

• data_whtml_builder rejects multiple root elements per call, forcing the agent to split three sections into separate inserts and reverse-order them; the tool should support multi-root fragments.
• data_style_tool returns responses over 71 KB that exceed token limits, forcing the agent to shell out to Bash to parse them.
• Webflow auto-renames CSS classes on collision (.hero becomes .hero-1-2-3), silently breaking the agent's intended class name with no warning from the tool.

None of those are pass/fail bugs. Every story still completed. But each is specific product feedback surfaced by an agent actually hitting the papercut, not by someone guessing what might go wrong. Those findings live in the trace alongside the score, so a green checkmark no longer erases the path the agent had to take.

What does a single run hide?

A single successful run can hide meaningful variation, so the harness can execute the same story repeatedly with a fresh site and agent session each time. This is initially a diagnostic tool, not a claim of statistical reliability. In one early three-run diagnostic, deterministic assertions passed in all three runs, while design-quality scores were 40%, 80%, and 85%. Three observations are not enough to estimate the underlying success distribution, but they immediately revealed something the original summary concealed: tool execution was stable while output quality was not.

The original console summary reported mean=1.00 and stdev=0.00 because the aggregation only saw the deterministic score. That result was mathematically correct and operationally misleading. We now report those dimensions separately and preserve the raw run-level results: deterministic checks passed 1.00 on all three runs; design quality scored 40%, 80%, and 85%. As we increase sample sizes, the same infrastructure can support more defensible reliability and regression analysis.

The obvious signal isn't the only signal

An LLM judge scoring a screenshot only sees the screenshot. Two pages can render pixel-identical while one is built from meaningful structure (<nav>, <section>, <button>) and the other is div-soup all the way down. That is a real accessibility and maintainability difference, and it is invisible to an image.

We added semantic_html_ratio, a deterministic, no-LLM signal computed from the rendered HTML already captured for the screenshot: the fraction of semantic tags versus generic wrappers, combined across pages by markup volume rather than naively averaged. It is not a pass/fail gate. Wrapper divs are normal, and Webflow's own visual builder is naturally div-heavy, but it gives us a second, independent signal for comparing runs and watching trends over time.

Agent-host coverage is part of eval coverage

This is the one that mattered most in practice. OpenAI rejected one of our ChatGPT app submissions after an agent tried to use a Designer-canvas tool that requires a live Webflow Designer tab, something a ChatGPT session can never have. We ran the exact rejected prompts through our harness repeatedly and never reproduced the failure. Not once.

The reason was uncomfortably simple: the harness had only ever been driven by Claude, and Claude essentially never took the bait from that piece of static tool guidance. The production rejection came from a GPT-family model. We had built a real eval harness and never once run it against the agent that was actually failing in production.

So we added a second driver that shells out to OpenAI's Codex CLI the same way the original shells out to Claude Code. Both share one orchestration core: provisioning, scoring, judges, screenshots, retries, and upload are identical. Only "spawn the agent and parse its output" differs.

Then we ran a controlled comparison: the original prompt versus one hardened with an explicit "don't use Designer-canvas tools" instruction, five independent repeats each, on both drivers.

The same task and MCP server produced materially different behavior across Claude Code and Codex CLI. This experiment doesn't isolate the underlying model from the surrounding host orchestration — and operationally, it doesn't need to. Users experience the combined system: model, system instructions, tool presentation, context management, and execution policy. The lesson wasn't that one model was better. It was that validating against a single agent host gave us confidence that didn't transfer to another environment. Every confidence-building feature above, including repeat testing, semantic signals, and transcript judging, now runs identically against either host because those capabilities live in the shared core rather than in Claude-specific code. Whatever we check going forward, we check on both.

Why this matters going forward

Chasing a green checkmark was never the point. Before we ship, we can say something specific and true: we ran this exact task, multiple times, against multiple real agents, and here's what actually happened. That's a different kind of confidence than tests passing.

• Every tool-schema change runs through the story suite before it ships. The same stories can point at a locally running MCP server, so regressions can be caught on a pull request rather than after deploy.
• Every ChatGPT app resubmission is checked against both drivers first, so a fix is checked against the model family that actually failed, not merely against the one that happened to pass.
• A flaky story shows up as flaky, with variance, instead of getting lucky in CI and surprising someone later.

The suite now runs nightly, turning drift into a number that moves every day rather than a production complaint someone has to trace backward weeks later. 

The nightly suite gives us continuous coverage, but the harness is just as useful in day-to-day product and engineering work. An engineer changing a tool, or someone on the product team validating a new capability, can manually run exactly the stories and agent coverage they care about before anything ships.

We turned the operational knowledge into a Claude Code skill, /eval-harness-runbook. Someone can describe what they want to test, and the skill handles the setup, relevant coverage, experiment naming, and post-run findings. That turns the harness into a self-serve testing surface for engineering and product.

Running it has gone from knowing the incantation to asking, and getting an answer.

Where it's headed

The near-term work is operational: automatically flag tools with no story coverage, and turn each nightly run into a report of regressions, new findings, and notable variance that lands somewhere people already look.

The harder question is what a score actually means. Today, every story’s brief, including what counts as “good”, comes from whoever wrote it. Useful, but invented. We want a gold-reference ceiling: hand-build an ideal version of a flagship story, score it through the same judge, and express future runs relative to that ceiling instead of treating a bare 0-to-4 score as self-explanatory.

And the story set itself should not stay frozen at launch. Right now it reflects what we imagined users might ask. Over time, we want new stories grounded in real usage patterns and analytics, not raw user data, so what actually happens in the world teaches the suite what to test next. That's the version of this harness worth building toward: not just catching what we thought to check for, but learning what we did not.