> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-claude-local-claude-code-native-syb1dt.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CI / CD

> Run MCP health checks, conformance suites, and evals in GitHub Actions, GitLab CI, and other CI environments

Run `mcpjam` in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.

## GitHub Actions

### Authentication

There are three ways to authenticate in CI, depending on your server setup.

#### Option 1: Headless OAuth login

Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: OAuth login (headless)
        run: |
          set -euo pipefail
          npx -y @mcpjam/cli@latest oauth login \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --protocol-version 2025-11-25 \
            --registration dcr \
            --auth-mode headless \
            --format json > /tmp/oauth-result.json
          TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
          rm -f /tmp/oauth-result.json
          if [ -z "$TOKEN" ]; then
            echo "::error::OAuth login did not return an access token"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "MCP_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json
```

#### Option 2: Refresh token

Best when you already have a refresh token from a previous `oauth login`. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.

**Secrets needed:**

| Secret              | Description                                         |
| ------------------- | --------------------------------------------------- |
| `MCP_SERVER_URL`    | Your MCP server URL                                 |
| `MCP_REFRESH_TOKEN` | OAuth refresh token from a previous login           |
| `MCP_CLIENT_ID`     | OAuth client ID (required with refresh tokens)      |
| `MCP_CLIENT_SECRET` | OAuth client secret (if the client is confidential) |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: |
          npx -y @mcpjam/cli@latest server doctor \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --refresh-token ${{ secrets.MCP_REFRESH_TOKEN }} \
            --client-id ${{ secrets.MCP_CLIENT_ID }} \
            --client-secret ${{ secrets.MCP_CLIENT_SECRET }} \
            --format json
```

<Tip>
  To get a refresh token, run `mcpjam oauth login` locally with `--format json` and grab `.credentials.refreshToken` from the output.
</Tip>

#### Option 3: Static API key

Best when your server uses a non-expiring API key instead of OAuth.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |
| `MCP_API_KEY`    | Static API key      |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token ${{ secrets.MCP_API_KEY }} --format json
```

#### Option 4: No auth

Some servers don't require authentication at all.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --format json
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).

```yaml theme={"theme":"css-variables"}
      - name: Snapshot before
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > before.json

      # your deploy step here

      - name: Snapshot after
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > after.json

      - name: Diff
        run: diff <(jq -S . before.json) <(jq -S . after.json)
```

### OAuth conformance suite

Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.

```yaml theme={"theme":"css-variables"}
      - name: OAuth conformance
        run: |
          npx -y @mcpjam/cli@latest oauth conformance-suite \
            --config ./oauth-matrix.json \
            --reporter junit-xml > report.xml

      - name: Upload test report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: oauth-conformance
          path: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

### Protocol conformance suite

Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.

```yaml theme={"theme":"css-variables"}
      - name: Protocol conformance
        run: |
          npx -y @mcpjam/cli@latest protocol conformance-suite \
            --config ./protocol-conformance.json \
            --reporter junit-xml > protocol-report.xml

      - name: Upload protocol report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: protocol-conformance
          path: protocol-report.xml
```

### MCP Apps conformance suite

Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.

```yaml theme={"theme":"css-variables"}
      - name: MCP Apps conformance
        run: |
          npx -y @mcpjam/cli@latest apps conformance-suite \
            --config ./apps-conformance.json \
            --reporter junit-xml > apps-report.xml

      - name: Upload apps report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: apps-conformance
          path: apps-report.xml
```

Single-run `protocol conformance`, `oauth conformance`, and `apps conformance` also accept `--reporter junit-xml` when you only need one target/check selection instead of a suite config file.

***

## GitLab CI

The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and `.gitlab-ci.yml` syntax.

### Authentication

#### Headless OAuth login

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
  script:
    - |
      npx -y @mcpjam/cli@latest oauth login \
        --url "$MCP_SERVER_URL" \
        --protocol-version 2025-11-25 \
        --registration dcr \
        --auth-mode headless \
        --format json > /tmp/oauth-result.json
      TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
      rm -f /tmp/oauth-result.json
      if [ -z "$TOKEN" ]; then
        echo "OAuth login did not return an access token"
        exit 1
      fi
      export MCP_TOKEN="$TOKEN"
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Refresh token

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_REFRESH_TOKEN: $MCP_REFRESH_TOKEN
    MCP_CLIENT_ID: $MCP_CLIENT_ID
    MCP_CLIENT_SECRET: $MCP_CLIENT_SECRET
  script:
    - |
      npx -y @mcpjam/cli@latest server doctor \
        --url "$MCP_SERVER_URL" \
        --refresh-token "$MCP_REFRESH_TOKEN" \
        --client-id "$MCP_CLIENT_ID" \
        --client-secret "$MCP_CLIENT_SECRET" \
        --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Static API key

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_API_KEY: $MCP_API_KEY
  script:
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_API_KEY" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes.

```yaml theme={"theme":"css-variables"}
mcp-tool-diff:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_TOKEN: $MCP_TOKEN
  script:
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > before.json
    # your deploy step here
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > after.json
    - jq -S . before.json > /tmp/before-sorted.json
    - jq -S . after.json > /tmp/after-sorted.json
    - diff /tmp/before-sorted.json /tmp/after-sorted.json
    - rm -f /tmp/before-sorted.json /tmp/after-sorted.json
```

### OAuth conformance suite

```yaml theme={"theme":"css-variables"}
mcp-oauth-conformance:
  image: node:20
  script:
    - |
      npx -y @mcpjam/cli@latest oauth conformance-suite \
        --config ./oauth-matrix.json \
        --reporter junit-xml > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

***

## Evals in CI

There are two ways to wire MCPJam evals into a pipeline: trigger a **hosted eval run** with the CLI, or run evals **locally with the SDK** and upload the results. Both authenticate with an MCPJam API key (`sk_…`) from **Settings → API keys**.

### Trigger a hosted eval suite

`mcpjam cloud eval run` starts an asynchronous run of a suite that lives in your MCPJam project. Without `--wait`, it prints a launch receipt and returns immediately. In CI, add `--wait` and `--out` to write a structured JSON report after every launched run reaches a terminal state.

**Secrets needed:**

| Secret           | Description             |
| ---------------- | ----------------------- |
| `MCPJAM_API_KEY` | MCPJam API key (`sk_…`) |

```yaml theme={"theme":"css-variables"}
      - name: Run hosted eval
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          npx -y @mcpjam/cli@latest cloud eval run \
            --suite "Nightly regression" \
            --project "My project" \
            --wait \
            --out eval-report.json \
            --format json > eval-result.json
          echo "Completed run $(jq -r '.runs[0].id' eval-result.json)"

      - name: Gate and write JUnit
        # Always run this step, even if the run above exited 1 on a failed
        # verdict: a default `if:` is `success()`, and skipping this step
        # would lose the gate's configurable policy and its JUnit report
        # exactly when the run failed. The job still ends up failed either
        # way — the run step above already exited nonzero.
        if: always()
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          RUN_ID=$(jq -r '.runs[0].id' eval-result.json)
          npx -y @mcpjam/cli@latest cloud eval gate \
            --run "$RUN_ID" \
            --project "My project" \
            --wait \
            --min-pass-rate-percent 100 \
            --reporter junit-xml \
            --out eval-report.xml

      - name: Upload eval reports
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: hosted-eval
          path: |
            eval-report.json
            eval-report.xml
```

In human format, `eval run` prints a `View:` line after the payload so you can open the run directly from the terminal:

```text theme={"theme":"css-variables"}
View: https://app.mcpjam.com/evals/suite/<suiteId>/runs/<runId>?project=<projectId>
```

This line is only emitted in human format — `--format json` output is unchanged, so scripts that parse the JSON stream are unaffected.

Use `--wait-timeout <ms>` to replace the 10-minute default. `--out` defaults to the structured JSON format; add `--reporter junit-xml` to write JUnit XML instead, or `--reporter html` for a self-contained HTML page (decision summary + failures only — traces, parity, and history are paid tiers not included here). When `--reporter` is present, the same report is also written to stdout.

<Warning>
  **`eval run --wait` sets a verdict-based exit code.** `0` pass, `1` a completed run's verdict failed (the ONLY condition that produces `1`), `2` usage error or an invalid suite file, `3` auth failed (no credential, or the platform rejected it, at launch or mid-wait), `4` a connection/setup failure this CLI itself observed before evaluation ran (or a local `--out` write failure), `5` no valid verdict — `inconclusive`, a null/unrecognized result, a run status of failed/cancelled/timed-out, or a wait that hit its deadline. A multi-target launch merges these worst-of across every waited run, in the order `1 > 3 > 4 > 5 > 0`.

  Retry guidance: `4` and `5` mean infrastructure, or an absence of observation — nothing here says the server is wrong, so retrying the CI job is reasonable. But a bare re-run is not automatically safe: `eval run --suite` only dedupes against an in-flight or already-completed launch when you pass a **stable** `--idempotency-key`, and exit `5` can mean the run is still running (a wait that hit its deadline) — without that key, a retry can start a second paid run alongside the first rather than resuming it. Pass `--idempotency-key` (or poll/resume the run id already in the receipt) before retrying on `4` or `5`. `3` means fix the credential first; it poisons every other observation in the same launch. Never retry blindly on `1` — that code is reserved for a run the platform actually graded as failed.

  Either `eval run --wait` or `eval gate` already fails the job on its own — the `eval gate` step above adds a configurable pass/fail *policy* (thresholds, per-scorer gates) and baseline comparison on top of the same verdict, so keep it when you want more than "did this run's own verdict pass".
</Warning>

<Note>
  `eval gate` sets a verdict-based exit code, and writes its report before doing so: `0` passed **or waived**, `1` an eval verdict failed, `2` usage error, `3` incomplete or non-gateable. Infrastructure conditions never map to `1`, so retrying on `3` is safe. This is a **different, four-code contract from `eval run --wait`** above — `gate`'s `3` means "incomplete", not the six-code scheme's `3` ("auth failed"), and the two are deliberately not unified (see the [CLI reference](/cli/reference#cloud-eval-gate) for why). `eval status` also prints a `View:` line in human format, identical to the one `eval run` prints.
</Note>

#### Waiving a gate

A run whose gate failed can be overridden by an authorized user until an expiry
they name, so a release is not blocked while a known regression is being fixed:

```bash theme={"theme":"css-variables"}
mcpjam cloud eval gate waive --run "$RUN_ID" --reason "hotfix ships today; tracked in ENG-4821" --expires-in 3d
```

`eval gate` then exits `0` and reports the outcome as `waived`. It is **not**
reported as a pass: the run keeps its failed result, the failing verdicts stay
in the report, and the waiver — who granted it, why, and until when — is named
in every artifact the command writes, including the JUnit XML your CI job
uploads (as a `<skipped>` element, so it neither fails the build nor renders as
a clean green row).

Only a real verdict failure is waivable. A cancelled run, a `--wait` timeout, or
a network failure still exits `3` with a waiver in place — those established
nothing, and a waiver granted for a regression is not consent to ship on an
infrastructure failure.

Waivers expire, and expiry is enforced on both sides: the platform republishes
the GitHub Check Run when the waiver lapses, and the CLI re-derives the expiry
itself rather than trusting the platform's answer. `mcpjam cloud eval gate
unwaive --run "$RUN_ID"` ends one early.

<Warning>
  The waiver reason is stored **unredacted** and readable by anyone who can see
  the suite, for as long as the suite exists. Never put secrets, tokens, or
  customer data in it.
</Warning>

#### Decision summary

`eval run --wait`, `eval status`, `eval gate` and `eval compare` all read one versioned object — the **run decision summary** — and every output format restates it. Where each command puts it:

| Command           | `--format json`                          | `--format human`                           | `--out` / `--reporter`          |
| ----------------- | ---------------------------------------- | ------------------------------------------ | ------------------------------- |
| `eval run --wait` | `decisionSummary` on the stdout receipt  | block on stdout, after the receipt         | `decisionSummary` on the report |
| `eval status`     | `decisionSummary` on the stdout document | block on stdout, above the `View:` line    | —                               |
| `eval gate`       | `decisionSummary` beside `gate`          | block on **stderr**, under the gate report | `decisionSummary` on the report |
| `eval compare`    | `decisionSummary` beside `compare`       | block on **stderr**, under the gate report | `decisionSummary` on the report |

Two scoping rules that are easy to miss. `eval run --wait` attaches a summary only when the invocation launched **one** run: a fan-out has several, and labelling a receipt about N runs with the decision of one would be a false claim rather than a partial one. `eval compare` reports the **compare side's** decision only — the baseline's failures are a different run's diagnostics, and printing them here would read as this run's.

```text theme={"theme":"css-variables"}
Decision summary: failed (verdict policy v2) — 2/3 case variants passed, 1 failed
  Why: a case did not meet its pass threshold
  Diagnostics: 2 non-passing of 9 trials examined (the complete set)
  First break: Tool call — the call arguments did not match what the case expects (1 of 2 measured trials)
  Fetch order (c_orders, iteration 2) — failed
    First failed stage: Tool call — the call arguments did not match what the case expects
    Failure category: call arguments
    Expected tool calls: fetch_order
    Observed failure: server rejected arguments
    Evidence at Tool call: span ids span-call-2; reasons order_id must be a string
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_2/trace
    Next action: review the authored arguments against the tool input schema
  Setup abort (c_setup, iteration 5) — failed
    First failed stage: none was established — the run never reached the server's stages
    Failure category: setup
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_5/trace
    Next action: check the server connection and environment configuration
```

Four things about it are worth knowing before you script against it.

**The counts carry the population they count.** `measurementUnit` is `caseVariant` under verdict policy v2 — one case under one provider/model, with repetitions as *trials inside it* — and `trial` on a legacy percent-threshold run. A 3-case suite with 5 repetitions is legitimately "3" under one unit and "15" under the other, so a count quoted without its unit is not a fact.

**The summary explains the verdict; it never re-decides it.** Under policy v2 the run's own decision is the authority for the verdict, the rates, the validity phase and the per-case aggregation, and it is carried through on `decision`. The per-trial diagnostics sit *underneath* that: a case can pass with a failing trial in it, so tallying the diagnostics gives a different answer than the platform reached.

**`notEstablished` is not a failure.** It is a fourth verdict meaning no verdict exists at all — the run is unfinished, it stopped before finishing, or its decision could not be read. `undecided.reason` says which. It is also not `inconclusive`, which *is* a decision: the validity phase ran and withheld a verdict because the run did not measure the server well enough.

**A page of diagnostics says whether it is the whole story.** `diagnostics.complete` is true only when the listed trials are the run's entire non-passing set, and `scannedIterations` says how many were examined — so an empty list from a complete page ("nothing failed") is distinguishable from an empty list from a partial one ("we did not look").

Evidence is scoped to the claim it supports: for a measured failure the span ids, prompt indexes and reasons come from the first failed stage's row alone, and a setup abort or evaluator error keeps a stage-less pointer rather than naming a stage nothing established.

`eval status` prints the block only when a terminal run did not pass — a clean pass has nothing to diagnose. `--format json` stays exactly one parseable document in every case: the summary rides *inside* it, never as a second block appended after it. If the summary cannot be fetched, it is omitted rather than failing the command.

**The human block leads with where the chain broke.** Under the diagnostics headline, before any per-trial detail, a non-passing run gets one line naming the earliest stage at which a readable trial stopped, why, and how many trials stopped there:

```text theme={"theme":"css-variables"}
  First break: Tool call — the call arguments did not match what the case expects (2 of 3 measured trials)
```

"First" means earliest in chain order — `connection → discovery → selection → call → response → userValue` — never "most common", so the count beside it is what tells you whether the run had one problem or several. When the breaks are spread the line says so (`earliest of 3 stages that broke`), and when some chains could not be read it says that too (`1 more had no readable chain`), because otherwise the denominator quietly shrinks to the trials that happened to validate. A run that reached no stage at all — a setup abort, an evaluator error — names its bucket instead of inventing a location.

The line is **not** a diagnosis. A first failed stage is a location and a failure category is a bucket; neither on its own says what to change.

`eval status --stages` expands each failing trial to all six chain rows with their states and reasons. It is off by default: six rows per trial is a lot of terminal on a run with twenty failures, and the first-break line above already carries the answer. Human output is not a stable contract — script against `--format json`, where the enums travel as enums.

Hosted runs execute LLM iterations on the platform and consume your organization's credits or configured provider keys. See the [`cloud eval` command reference](/cli/reference#cloud-eval-commands) for the full surface, including `cloud eval judge` (request LLM-as-judge grading on a finished run), `cloud eval validate` (offline suite-file validation), `cloud eval export` (write a hosted suite to a local file), `cloud eval checks list/connect` (GitHub Checks integration), and more.

### Upload SDK eval results

If you instead run evals inside your own CI job with [`@mcpjam/sdk`](/sdk/concepts/running-evals) (`EvalTest` / `EvalSuite`), set `MCPJAM_API_KEY` and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):

```yaml theme={"theme":"css-variables"}
      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx vitest run evals/
```

See [Save Results to MCPJam](/sdk/concepts/saving-results) for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON).