05 / 06 MCP 除錯與評估MCP debugging & eval

ProbeLion

MCP 視覺化除錯工作台與評估沙盒

A visual debugging workbench and eval sandbox for MCP

看見模型為什麼選錯工具、為什麼傳錯參數。

See why the model picked the wrong tool — and why the arguments were wrong.

即將開源 · Request accessComing to open source · Request access即將開源 · Request accessOpen soon · Request access Repo 目前為私有Repo currently private
ProbeLion icon
執行環境Runtime
Node.js ≥ 22.5 · 本機Node.js ≥ 22.5 · local
傳輸Transports
stdio · Streamable HTTP · HTTP+SSEstdio · Streamable HTTP · HTTP+SSE
模型Models
OpenAI · Anthropic · Gemini · 腳本模型OpenAI · Anthropic · Gemini · scripted
技術Built with
  • MCP
  • Hono
  • WebSocket
  • SQLite
  • Vue 3
  • OpenTelemetry export

(01) 為什麼Why

寫 MCP Server 時，開發者看不到模型為什麼選錯工具或傳錯參數；換了模型或改了工具描述，也沒有回歸測試可以確認準確率。ProbeLion 不打算取代官方 MCP Inspector，而是補上「模型 × 工具」互動層的可觀測性與評估。

When you write an MCP server you cannot see why a model picks the wrong tool or passes the wrong arguments — and after a model or description change there is no regression test. ProbeLion does not replace the official MCP Inspector; it adds observability and evaluation for the model × tool layer.

(02) 06 項items

它做什麼What it does

  1. 01

    連線管理Connections

    stdio、Streamable HTTP 與舊版 HTTP+SSE；可同時掛載多個 Server，可匯入 Claude Desktop / Cursor 的 mcpServers 設定（執行前需確認指令）。

    stdio, Streamable HTTP and legacy HTTP+SSE; mount several servers at once and import Claude Desktop / Cursor mcpServers configs (commands are confirmed before running).

  2. 02

    Agent PlaygroundAgent playground

    OpenAI、Anthropic、Gemini 與相容端點（BYOK），加上 3 個內建腳本模型；可在呼叫工具前人工核准，並讓 2～3 個模型並排比較。

    OpenAI, Anthropic, Gemini and compatible endpoints (BYOK), plus 3 built-in scripted models; approve tool calls by hand and compare 2–3 models side by side.

  3. 03

    Trace 時間軸與流程圖Trace timeline & flow graph

    自動標示 invalid_args、tool_error、loop、max_steps；用新模型重播舊 Trace 找出分歧點，並匯出 OpenTelemetry。

    Flags invalid_args, tool_error, loop and max_steps; replay an old trace with a new model to find where it diverges; export to OpenTelemetry.

  4. 04

    批次 EvalBatch eval

    YAML 題組（預期工具、參數比對、禁止工具、最終回答），關鍵字或 LLM-as-Judge、Tool Mock、紅隊 Prompt Injection、兩次執行回歸比較。

    YAML suites (expected tools, argument matching, forbidden tools, final answer), keyword or LLM-as-judge, tool mocks, red-team prompt injection and run-to-run regression compare.

  5. 05

    工具描述 LinterTool-description linter

    檢查描述缺漏、名稱籠統、跨 Server 工具衝突與 Token 成本，並提供改寫建議。

    Catches missing descriptions, vague names, cross-server tool clashes and token cost, with rewrite suggestions.

  6. 06

    工具瀏覽器Tool browser

    依 inputSchema 產生表單，支援 Tools / Resources / Prompts 與原始 JSON-RPC 紀錄。

    Forms generated from inputSchema, with Tools / Resources / Prompts and the raw JSON-RPC log.

(03) 怎麼運作How it works

一次除錯迴圈One debugging loop

  1. 01

    掛載Mount

    接上 stdio 或 HTTP 的 MCP Server。

    Attach a stdio or HTTP MCP server.

  2. 02

    對話Converse

    真實或腳本模型多輪呼叫工具。

    Real or scripted models call tools over many turns.

  3. 03

    追蹤Trace

    每一步的選擇、驗證、耗時與成本。

    Every choice, validation, latency and cost.

  4. 04

    評估Evaluate

    題目 × 模型 × 重複次數。

    Questions × models × repeats.

  5. 05

    守門Gate

    在 CI 以通過率門檻與回歸比較擋下退步。

    Pass-rate thresholds and regression compare in CI.

(04) README

快速開始Quick start

預設自動掛載範例 weather-mcp；內建 probelion-* 腳本模型只模擬天氣情境，成本為 0。

The sample weather-mcp is mounted by default; built-in probelion-* scripted models only simulate the weather scenario and cost nothing.

本機（Node.js ≥ 22.5）Local (Node.js ≥ 22.5) 
npm install
npm run build
npx probelion
# prints http://127.0.0.1:6274?token=...
CLICLI 
probelion eval evals/weather.yaml --models probelion-smart-v1,claude-haiku-5-5 --repeat 3 --min-pass-rate 0.85
probelion lint --server "node examples/weather-mcp-server.mjs"

(05) 實測Measured

真實模型實測Real-model run

2026-11-02 · evals/weather.yaml · 自有 20 題 × 1 次2026-11-02 · evals/weather.yaml · own 20 cases × 1 run

模型Model通過率Pass rateP95 延遲P95 latency20 題總成本Total cost (20)
claude-haiku-5-580%5.06 s約 US$0.008≈ US$0.008
claude-sonnet-5-575%7.87 s約 US$0.166≈ US$0.166

(06) 安全Security

安全Security

  • 除 GET /api/health 外，所有 API 與 /ws 都需要 Token。Every API and /ws needs a token, except GET /api/health.
  • 只接受 localhost 與允許清單內的 Host，防 DNS Rebinding。Only localhost and allow-listed hosts are accepted, against DNS rebinding.
  • Server 的 env 與 headers 以 AES-256-GCM 加密存入 SQLite，API 回應一律遮罩。Server env and headers are AES-256-GCM encrypted in SQLite and always masked in responses.
  • Eval Mock 預設嚴格模式：沒有對應 mock 的工具呼叫直接失敗。Eval mocks are strict by default: a tool call without a matching mock fails.

目前狀態Status

  • Repo 目前為私有，準備開源中。Repo currently private — coming to open source.
  • README 宣告 MIT 授權。README declares the MIT licence.

想先試用 ProbeLion？Want to try ProbeLion early?

所有 repo 目前都是私有的，準備開源中。想搶先試用，請透過 GitHub 個人頁聯絡。All repos are private for now and on their way to open source. To try one early, get in touch via the GitHub profile.

即將開源 · Request accessComing to open source · Request access即將開源 · Request accessOpen soon · Request access Repo 目前為私有Repo currently private