mcptest docs GitHub

Web Bot Auth and MCP

Web Bot Auth is the IETF effort to give legitimate bots, scrapers, and now agents a cryptographically verifiable identity at the HTTP layer. It is built on the HTTP Message Signatures spec (RFC 9421) plus a small stack of related drafts that pin down directory discovery, signature agents, and the HTTP architecture itself. The drafts are on a fast track toward Best Current Practice publication, with the chairs publicly targeting roughly August 2026 for BCP status.

mcptest has a direct interest in this work. MCP servers increasingly sit on the receiving end of agent traffic, and once Web Bot Auth lands as a BCP the same servers will be expected to verify those signatures or be flagged by upstream infrastructure (Cloudflare, Fastly, large origin platforms, search and crawl operators) that have already begun shipping support. The conformance suite is going to be load-bearing, and we want it to live where mcptest's existing compliance corpus already lives.

This post explains what Web Bot Auth is, where the WG charter draws the line between human-driven bots and agent-to-agent traffic, what we plan to do about that gap, and what we are doing right now (Phase 1, v1.0 timeframe) to be useful to the WG before we have a single line of code in the suite.

Client-side signing is now functional. mcptest can present a Web Bot Auth identity when it connects to a URL server: it signs each request with an Ed25519 or RSA-PSS (rsa-pss-sha512) key per RFC 9421 and attaches Signature, Signature-Input, and Signature-Agent. The YAML shape lives under auth.web_bot_auth and is documented in docs/auth.md. Both Ed25519 and RSA-PSS are supported, mcptest doctor validates either key type offline (and confirms the key directory is reachable with --probe), and mcptest web-bot-auth directory emits the public-key directory document a server operator publishes so verifiers find the bot's public key. This is distinct from the server-side conformance suite below, which is still Phase 2.

# Publish the public-key directory for a signing key (public key only).
mcptest web-bot-auth directory --key bot.ed25519
# RSA-PSS keys: add --algorithm rsa-pss.

See examples/auth-fixtures/ for a full local round trip: sign with mcptest, publish the directory, and verify the signature with a small Node script.

What Web Bot Auth actually is

Web Bot Auth is a set of IETF drafts that, taken together, define how a non-human HTTP client proves who it is to an origin server. The pieces:

The combined target is a BCP RFC track. The chairs have stated August 2026 as the target window. That date moves, but it is close enough that the conformance work needs to start now if mcptest wants to ship a credible suite by the time servers are being asked to comply.

Why agent-to-agent matters and is currently out of scope

The WG charter, as written today, scopes Web Bot Auth to "bots" in the classical sense: scrapers, crawlers, RSS readers, well-behaved automated clients that fetch content on behalf of a human operator. The intent is to give Cloudflare, Fastly, and origins like Reddit, Stack Overflow, and the New York Times a way to tell a legitimate Bingbot from an unsigned scraper.

That is not the only flavor of automated traffic in 2026. Agent-to-agent calls (an AI agent calling an MCP server, an MCP server calling another MCP server on the user's behalf, an agent fanning out to multiple MCP servers in parallel) are economically larger and growing faster than the classical bot surface the charter targets. The same cryptographic primitive (RFC 9421 signatures with key directory discovery) applies. The threat model is different (agents need delegation semantics; "on behalf of" is a first-class concept) and so the wire format will need extensions.

The WG's position, as of mid-2026, is that agent-to-agent semantics are out of scope for the current charter. That is the right call for the charter: the BCP target date depends on a narrow scope. But it leaves a hole that the MCP ecosystem will have to fill.

mcptest's plan is to fill that hole on a delay: ship a conformance suite for the bot-side of the spec at a future release, then layer agent-to-agent extensions on top as the relevant drafts coalesce. The conformance suite for the classical bot surface is roughly the same shape we already maintain for the MCP spec itself (compliance/registry.yml, the per-version checks under compliance/v2024-11-05/, with standardized cross-tool rule IDs). The agent-to-agent surface will hang off the same scaffolding once the WG (or a successor effort) charters it.

Our position

We are publishing this post for two reasons.

First, we want bot operators and origin platforms that already track this work to know that mcptest intends to host the conformance suite at a future release. That commitment reduces the risk that the conformance work scatters across five different testing tools. A single suite, in a tool that already does MCP compliance, is the path of least resistance.

Second, we want the IETF working group to know that the MCP ecosystem is paying attention. Comments on the drafts during the WG-LC window carry more weight than comments after publication. We will be commenting from the MCP angle, in coordination with the SBproxy team, who have a parallel interest from the proxy and gateway side.

Near-term actions (Phase 1, v1.0 timeframe)

Phase 1 is everything we can do without writing code, in the v1.0 release window.

  1. Subscribe to the WG mailing list and the relevant Datatracker pages. The mailing list (httpbis-bot-auth) and the per-draft Datatracker pages are where chair calls and WG-LC announcements land. Our maintainers subscribe individually so the WG signal does not depend on a shared inbox.
  2. Comment on the drafts during the WG-LC windows. Each draft will have a working group last call window before it advances. We file substantive comments from the MCP perspective during those windows, not after publication. Comments cover (a) where the bot-auth surface intersects MCP's own auth model, (b) where the directory format intersects MCP's discovery surface, and (c) what extensions an agent-to-agent profile would need.
  3. Coordinate with the SBproxy team. SBproxy implements the server-side of Web Bot Auth verification at the proxy layer. mcptest tests servers from the outside. The two efforts share a wire-format reading and benefit from a shared interpretation of the drafts. Coordination happens through scheduled syncs.
  4. Publish this position post. This document. It exists so search, AI assistants, and prospective customers can find a clear statement of what mcptest is doing about Web Bot Auth, what the BCP timeline looks like, and where the agent-to-agent gap is. We refresh it whenever a draft advances or the WG charter changes.

Phase 1 is deliberately small. There is no Rust code in this phase, no rules in compliance/registry.yml, no new schema version. v1.0 ships without the conformance suite, on time, with this engagement plan as the public commitment.

Phase 2 (a future release)

Phase 2 is the conformance suite itself. The draft rule list lives in docs/compliance/web-bot-auth-corpus.md and the strategic plan is settled. Phase 2 is out of scope for this post and for the v1.0 release. We mention it here so readers understand that "we have not shipped the suite yet" does not mean "we are not going to ship the suite."

References