Runtime fault taxonomy coverage

A passing test suite tells you what your MCP server does right. It does not tell you which failure modes you actually checked for. MCP servers fail at runtime in recurring families, slips in the protocol lifecycle, tool-invocation crashes, schema gaps, state leaks, provider-integration breaks, security lapses, timeouts, and unenforced configuration, and a suite can look green while never having probed half of them.

mcptest fault-coverage answers one question: which runtime fault families did this suite exercise? It loads a versioned fault taxonomy, detects the probes your suite uses, and marks each family covered, partially covered, uncovered, or not applicable, with an optional coverage threshold you can gate CI on.

This complements, rather than duplicates, the other quality lanes. compliance and conformance check spec adherence; security checks adversarial inputs; fault-coverage reports, across all of them, which fault families your run touched.

The eight fault families

The taxonomy is grounded in A Taxonomy of Runtime Faults in Model Context Protocol Servers. Each family has a stable category id; each leaf fault carries a FAULT-<CATEGORY>-<NAME> id.

Category	Family	Example leaf fault
`PROTO`	Protocol interaction	request processed before initialize completes
`TOOL`	Tool invocation	tool crashes instead of returning an error result
`SCHEMA`	Schema enforcement	structuredContent violates the declared outputSchema
`STATE`	State management	state leaks across tests or sessions
`PROVIDER`	Model-provider integration	provider rejects the served tool schema
`SECURITY`	Security validation	tool-description injection followed
`TIMEOUT`	Timeout and cancellation	tool hangs without a clean timeout
`CONFIG`	Configuration not enforced	read-only hint not enforced

PROVIDER faults apply only to suites with agents:; a server-only suite reports that family as n/a rather than uncovered.

How coverage is computed

Each leaf fault is mapped to the mcptest probes that exercise it (a probe is a subcommand or a suite feature). A leaf is exercised when any of its probes ran. A family is then:

covered: every leaf was exercised,
partial: some leaves were exercised,
uncovered: the family applies but no leaf was exercised,
n/a: the family does not apply to this suite shape.

The command detects these probes from the suite YAML automatically:

Suite feature	Probe token
a `compliance:` block	`compliance`, `conformance`
a `security:` block or a `trust_boundary:`	`security`, `trust-boundary`
`faults:` / agent `recovery:` / `inject:`	`fault-recovery`
a `negative_path:` check	`negative-path`
an `output_schema:`	`output-schema`
`run_options.restart_policy`	`test-isolation`
`agents:` (with `trajectory`/`golden_path`/`distractors`/`equal_function`)	`agent`, `tool-selection`

Probes you run out of band (a separate mcptest fuzz or mcptest schema-lint pass) are folded in with --probe.

A worked example

# Which fault families does this suite exercise, plus a fuzz pass run separately?
mcptest fault-coverage examples/negative-path.yml --probe fuzz

Runtime fault family coverage

  partial   Protocol interaction         2/3 faults  [PROTO]
  covered   Tool invocation              2/2 faults  [TOOL]
  partial   Schema enforcement           2/4 faults  [SCHEMA]
  uncovered State management             0/2 faults  [STATE]
  n/a       Model-provider integration   0/2 faults  [PROVIDER]
  uncovered Security validation          0/2 faults  [SECURITY]
  uncovered Timeout and cancellation     0/2 faults  [TIMEOUT]
  uncovered Configuration not enforced   0/1 faults  [CONFIG]

  38% of applicable faults exercised; 1 of 7 families fully covered

Gate CI on a floor with --threshold; the command exits 6 when coverage is below it. Use --format json for a machine-readable report, and --no-suite to print the bare taxonomy (combine with --probe to score an out-of-band pipeline).

Fixtures

Every family ships a runnable fixture under examples/, so you can see each one exercised in isolation: examples/negative-path.yml (PROTO/SCHEMA), examples/fuzz/ (TOOL), examples/output-schema-conformance.yml (SCHEMA), examples/test-isolation.yml (STATE), examples/agent-weather.yml and examples/distractor-tools/ (PROVIDER), examples/security/ and examples/trust-boundary/ (SECURITY), examples/fault-injection-recovery.yml (TIMEOUT), and examples/config-enforcement/ (CONFIG).

Extending the taxonomy

The taxonomy lives in crates/mcptest-core/src/runtime_faults/registry.yml and is embedded in the binary, so --registry <path> can point at a customized copy. To add a fault: pick a category, give the leaf a FAULT-<CATEGORY>-<NAME> id, list the probes that exercise it, point fixture at an example, and run cargo test -p mcptest-core runtime_faults to confirm it loads.