servicetitan/arcade-eval
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: _TEMPLATE + all-10 criteria-section stubs (verbatim criteria)

Browse Source
This commit is contained in:
iztaylor
2026-06-18 10:10:17 -04:00
parent 29c5b2c8be
commit 593e1e63b6
13 changed files with 510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/cat1-functional/criteria-section-1.md
+43
View File
@@ -0,0 +1,43 @@
# Category 1 — Functional MCP Gateway Capability (weight 8)
> Verbatim criteria / gates / questions from the criteria Google Doc. Fill Score / Evidence /
> Findings / Answers locally; **the human pastes** into the Google Doc. 1–5 scale; anchors at 1/3/5.
## Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Implements MCP protocol correctly — tool listing, tool invocation, error responses. | | |
| 2 | Gateway tool curation — ability to expose a subset of tools from underlying servers to a given doorway. | | |
| 3 | Per-user tool scoping — different users see different tool lists based on their explicit grants. | | |
| 4 | Supports all required MCP clients without custom adapters (Claude Code, Cursor, LangGraph, internal agent frameworks). | | |
| 5 | Tool execution isolation — one user's tool call cannot access another user's tokens or context. | | |
| 6 | Supports mixing prebuilt (global catalog) and custom (self-hosted) servers behind a single gateway URL. | | |
| 7 | Gateway is pure metadata — adding or removing tools does not require server redeployment. | | |
| 8 | Dynamic tool registration — new tools become available without gateway restart. | | |
**Average:** ___ **Category score:** ___
## Score anchors
- **1** — Basic MCP server, no per-user scoping or curation
- **3** — Gateway curation works; per-user scoping requires workarounds
- **5** — Full per-user tool scoping, mixed-server gateways, zero-config for MCP clients
## Benchmark questions
| # | Question (verbatim) | Answer | Evidence |
|---|---|---|---|
| 1 | Can a Claude Code client connect to the gateway and see only the tools granted to the current user? | | |
| 2 | Can the same gateway URL serve two different users with different tool lists? | | |
| 3 | Can we add a tool to the gateway without restarting any server or the Engine? | | |
| 4 | Can we expose tools from both a prebuilt connector and a custom self-hosted server through one gateway endpoint? | | |
| 5 | What happens when a client requests a tool the user has not been granted? | | |
## Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| MCP protocol compliance | Any compliant MCP client connects without custom adapters | | |
| Tool curation | Gateway tool list matches exactly the configured allow-list | | |
| Per-user isolation | User A cannot see or invoke tools granted only to User B | | |
| Mixed server gateway | Prebuilt and custom server tools coexist behind one gateway URL | | |
## Findings
-