mcptest docs GitHub

Getting started with mcptest

Install mcptest and get your first passing test in under five minutes.

This guide gives you three install tracks. Pick the one that matches how you already work, run the canonical five-step workflow at the end of the track, and stop reading. Every track ends at the same place: mcptest run passing against a fixture MCP server with three green tests.

File issues on GitHub when you find gaps.

Prefer a real server to poke at? examples/reference-server/ is a small, runnable MCP server (five tools, two resources) with a smoke suite. Clone the repo, install it, and run the bundled tests:

cd examples/reference-server && npm install && cd -
mcptest run --config examples/reference-server/tests/smoke.yml

Or let an LLM write the first suite. Hand a model your tool list and the prompt at Generate tests with an LLM, then run mcptest validate on what it produces.

Pick a track

After any track lands you on the same mcptest run output, jump to the canonical workflow and the what is next pointers.

Track 1: direct install (macOS, Linux, Windows)

Pick whichever package manager you already trust.

Homebrew

brew install soapbucket/tap/mcptest
mcptest --version

The tap publishes a signed bottle for Intel and Apple Silicon Macs and an x86_64 Linux build. The formula has no native dependencies.

Cargo

cargo install mcptest
mcptest --version

Cargo compiles from source. Use this when you want to track a specific git ref (cargo install --git https://github.com/soapbucket/mcptest --tag v1.0.0) or you need a binary on an architecture the prebuilt releases do not cover.

Install script (curl)

curl -fsSL https://download.mcptest.sh/install.sh | sh
mcptest --version

The script downloads the matching prebuilt release for your platform, verifies the SHA256 against the signed manifest, and drops the mcptest binary on your $PATH. macOS, Linux, and Windows (under Git Bash or WSL) are all covered. The script never asks for root unless $PATH requires it.

Direct binary download

Every release tag publishes signed tarballs and a SHA256SUMS file at github.com/soapbucket/mcptest/releases. Pick the archive for your platform, extract mcptest onto your $PATH, and you are done. The release pipeline cross-compiles for Linux x86_64 and aarch64, macOS x86_64 and aarch64, and Windows x86_64.

Verify the install

mcptest --version
mcptest --help

mcptest --version should print the release tag. mcptest --help lists every subcommand. Per-subcommand detail lives at mcptest <subcommand> --help.

Now run the canonical workflow.

Track 2: Docker

docker run --rm \
  -v "$PWD:/work" -w /work \
  soapbucket/mcptest:latest --version

The image is a static binary on gcr.io/distroless/static. It is small, it has no shell, and it expects your test YAML to be mounted at the working directory. Replace --version with any other subcommand, exactly as you would on the host. Mount ~/.config or ~/.aws read-only if your tests need host credentials.

For day-to-day use, alias the long incantation:

alias mcptest='docker run --rm -v "$PWD:/work" -w /work soapbucket/mcptest:latest'
mcptest --version

With the alias in place, the rest of this guide reads the same as Track 1.

Now run the canonical workflow.

Track 3: GitHub Actions (CI)

Install the CLI with the install script in a workflow step, then run it. Add .github/workflows/mcptest.yml to your repository:

name: mcptest
on:
  pull_request:
  push:
    branches: [main]

jobs:
  mcptest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install mcptest
        run: curl -fsSL https://download.mcptest.sh/install.sh | MCPTEST_VERSION=v1.0.0 sh
      - run: mcptest doctor
      - run: mcptest run --reporter junit --output mcptest.junit.xml
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: mcptest-results
          path: mcptest.junit.xml

The install step puts the pinned mcptest binary on $PATH. The remaining steps are the same calls you would run locally. The JUnit output rolls up cleanly in the GitHub pull-request UI.

For GitLab CI or CircleCI, run the Docker image directly:

# .gitlab-ci.yml
mcptest:
  image: soapbucket/mcptest:latest
  script:
    - mcptest doctor
    - mcptest run --reporter junit --output mcptest.junit.xml
  artifacts:
    when: always
    reports:
      junit: mcptest.junit.xml

Now run the canonical workflow on your laptop to build the test suite the CI job will execute.

The canonical workflow

Every track ends with the same five-step loop. Run it once after install and you should see three passing tests in well under five minutes.

1. Install

You already did this. Verify with mcptest --version.

2. mcptest doctor

mcptest doctor

doctor checks your environment and prints a one-screen report:

The report exits non-zero if a hard prerequisite is missing. Fix what it asks for before moving on.

3. mcptest init

mcptest init

init scaffolds a new test suite in the current directory:

.
├── mcptest.yml
└── tests/
    └── example.yaml

mcptest.yml holds project defaults (reporter, parallelism, timeout). It is not a test suite, so you do not point mcptest run at it. tests/example.yaml is the hello-world suite: it exercises the well-known @modelcontextprotocol/server-filesystem stdio server, so mcptest run --config tests/example.yaml passes out of the box if npx is on your $PATH. Swap the server command: for your own server when you are ready. Two flags adjust the scaffold:

init refuses to overwrite existing files; pass --force to replace them.

4. mcptest run

Point run at the suite init scaffolded:

mcptest run --config tests/example.yaml

Sample output from the pretty reporter:

mcptest 1.0.0 run

  [PASS] search returns at least one result  (124 ms)

Summary: 1 passed, 0 failed, 0 skipped in 124 ms

The default pretty reporter prints one line per test plus a summary, and on a failure prints the expectation, the actual value, and a hint inline under the failing test. For a compact one-line count (ran 1 tests: 1 passed, ...), pass --reporter minimal.

With no --config, mcptest run looks for mcptest.yaml (note the .yaml spelling) in the current directory and prints no config if it does not find one. The mcptest.yml that init writes is the project-defaults file, not a suite, so pass the suite path explicitly until you rename your suite to mcptest.yaml.

Useful flags on the first run (all take the same --config tests/example.yaml):

Exit codes:

run can also exit 5 (cost cap exceeded, or --update-snapshots refused under CI=true), 6 (coverage below --coverage-threshold), and 7 (no tests selected). The full table is in the CLI reference.

5. Read the output

The pretty reporter is the default because it is what you read most often. The summary line at the bottom is the one you scan in CI. Open a failing test and the reporter prints:

Use --reporter junit --output run.xml in CI to produce a machine-readable artifact your CI system renders as a test report; the run still prints progress to stderr. For the saved-run formats (sarif, markdown, html), re-render the record with mcptest report --format <fmt>.

What's next

You have a passing suite. The reading order from here depends on what you want to do next.

If something here is wrong, ambiguous, or missing, open an issue on GitHub and we will pick it up.