cat1: live probes — protocol/curation PASS, per-user list gateway-wide (finding), ungranted rejected
This commit is contained in:
+3
-1
@@ -44,6 +44,8 @@ Self-hosted on `backstage-wus2-v4` via Flux; vendor Helm chart **1.8.8**
|
||||
`Authorization: Bearer <ARCADE_API_KEY>` + `Arcade-User-ID: <user_id>`. The user_id is any stable
|
||||
string (an email works); this mode is for clients without browser auth / token refresh. Self-hosted
|
||||
gateway URL: `https://api.arcade.st.dev/mcp/<slug>`. (Source: docs.arcade.dev call-tool-client.)
|
||||
- **Baseline gateway:** _slug + tool allow-list (Task 1.2)_
|
||||
- **Baseline gateway:** `zeb-gateway-test` — auth mode **Arcade Headers** (API key + `Arcade-User-ID`);
|
||||
7 main-catalog tools (Slack ×2, GoogleDocs ×4, Brightdata ×1). See `config/targets.yaml`.
|
||||
Confirmed live 2026-06-18: tool list is gateway-wide (same for all `Arcade-User-ID`s).
|
||||
- **Shared reference server:** _name + tools echo/whoami/add (Task 1.4)_
|
||||
- **`whoami` identity field:** _exact field the server reads (Task 1.4 / 2.4)_
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# Lane notes — Category 1 (Functional MCP Gateway Capability)
|
||||
|
||||
- **Owner:** ztaylor
|
||||
- **Last live-state check:** 2026-06-18 (dashboard 200; gateway `zeb-gateway-test` live)
|
||||
- **Fixtures used:** gateway `zeb-gateway-test` (7 main-catalog tools, Arcade-Headers auth); users A/B from `config/targets.yaml`. Raw evidence in `tests/probes.md`.
|
||||
|
||||
## Log
|
||||
- 2026-06-18 — lib client connects live; protocol/curation/per-user-visibility/ungranted probes done.
|
||||
- PASS: protocol compliance (connect/list/invoke/structured error), tool curation (7 listed == 7 selected).
|
||||
- FINDING: per-user tool *list* scoping not differentiated via Arcade-Headers on one gateway (A==B). Needs cat-3 Contextual Access or separate gateways / User Source.
|
||||
- Q5: ungranted tool → `McpError: tool not enabled for this gateway`.
|
||||
|
||||
## Remaining for cat-1 scoring
|
||||
- [ ] 2.2 — connect a **second real MCP client (Claude Code)** to the gateway (no-adapter evidence).
|
||||
- [ ] 2.5 — **dynamic registration**: add/remove a tool on the gateway (dashboard or API), re-list, confirm no restart. (needs a gateway edit)
|
||||
- [ ] 2.7 — **mixed prebuilt + custom**: compose a gateway with a `main` tool + a `lib/mcp_server` tool. (needs reference server → `arcade login`/`arcade deploy`)
|
||||
- [ ] 2.4 — **`whoami` execution proof** that calls run as the calling user. (needs reference server)
|
||||
- [ ] 2.8 — finalize scores once the above land.
|
||||
@@ -2,18 +2,20 @@
|
||||
|
||||
> Verbatim criteria / gates / questions from the criteria Google Doc. Fill Score / Evidence /
|
||||
> Findings / Answers locally; **the human pastes** into the Google Doc. 1–5 scale; anchors at 1/3/5.
|
||||
> Status: **in progress** — scores held until the remaining tests (2.2 Claude Code, 2.5 dynamic
|
||||
> reg, 2.7 mixed, 2.4 whoami) land. Raw evidence: `tests/probes.md`.
|
||||
|
||||
## Scores
|
||||
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|
||||
|---|---|---|---|
|
||||
| 1 | Implements MCP protocol correctly — tool listing, tool invocation, error responses. | | |
|
||||
| 2 | Gateway tool curation — ability to expose a subset of tools from underlying servers to a given doorway. | | |
|
||||
| 3 | Per-user tool scoping — different users see different tool lists based on their explicit grants. | | |
|
||||
| 4 | Supports all required MCP clients without custom adapters (Claude Code, Cursor, LangGraph, internal agent frameworks). | | |
|
||||
| 5 | Tool execution isolation — one user's tool call cannot access another user's tokens or context. | | |
|
||||
| 6 | Supports mixing prebuilt (global catalog) and custom (self-hosted) servers behind a single gateway URL. | | |
|
||||
| 7 | Gateway is pure metadata — adding or removing tools does not require server redeployment. | | |
|
||||
| 8 | Dynamic tool registration — new tools become available without gateway restart. | | |
|
||||
| 1 | Implements MCP protocol correctly — tool listing, tool invocation, error responses. | | PASS (live) — lib `mcp` SDK client connected, initialized, listed 7 tools, invoked, got structured `isError` result + JSON-RPC error. Minor: 202 on session close. |
|
||||
| 2 | Gateway tool curation — ability to expose a subset of tools from underlying servers to a given doorway. | | PASS — 7 tools listed == the 7-tool allow-list selected (Slack×2, GoogleDocs×4, Brightdata×1). |
|
||||
| 3 | Per-user tool scoping — different users see different tool lists based on their explicit grants. | | **FINDING** — User A and User B see the **identical 7 tools** on one gateway (Arcade-Headers). List is gateway-wide, not per-user. Per-user differentiation needs cat-3 Contextual Access or separate gateways / User Source. |
|
||||
| 4 | Supports all required MCP clients without custom adapters (Claude Code, Cursor, LangGraph, internal agent frameworks). | | PARTIAL — custom `mcp`-SDK client connected with no adapter ✓. Claude Code connect = 2.2; Cursor = teammate test. |
|
||||
| 5 | Tool execution isolation — one user's tool call cannot access another user's tokens or context. | | PENDING — vault is per-`user_id` by design; direct proof via reference-server `whoami` (2.4). |
|
||||
| 6 | Supports mixing prebuilt (global catalog) and custom (self-hosted) servers behind a single gateway URL. | | PENDING — needs reference server (2.7). |
|
||||
| 7 | Gateway is pure metadata — adding or removing tools does not require server redeployment. | | PARTIAL — gateway is metadata (per Arcade model); add/remove-without-restart probe = 2.5. |
|
||||
| 8 | Dynamic tool registration — new tools become available without gateway restart. | | PENDING — 2.5 (needs a gateway edit). |
|
||||
|
||||
**Average:** ___ **Category score:** ___
|
||||
|
||||
@@ -25,19 +27,22 @@
|
||||
## Benchmark questions
|
||||
| # | Question (verbatim) | Answer | Evidence |
|
||||
|---|---|---|---|
|
||||
| 1 | Can a Claude Code client connect to the gateway and see only the tools granted to the current user? | | |
|
||||
| 2 | Can the same gateway URL serve two different users with different tool lists? | | |
|
||||
| 3 | Can we add a tool to the gateway without restarting any server or the Engine? | | |
|
||||
| 4 | Can we expose tools from both a prebuilt connector and a custom self-hosted server through one gateway endpoint? | | |
|
||||
| 5 | What happens when a client requests a tool the user has not been granted? | | |
|
||||
| 1 | Can a Claude Code client connect to the gateway and see only the tools granted to the current user? | Connect: lib client ✓; Claude Code pending (2.2). "Only granted tools": N/A — no per-user grants on this gateway (list is gateway-wide). | probes.md |
|
||||
| 2 | Can the same gateway URL serve two different users with different tool lists? | **No** — A and B see identical 7 tools. | probes.md (A==B) |
|
||||
| 3 | Can we add a tool to the gateway without restarting any server or the Engine? | Pending (2.5). | |
|
||||
| 4 | Can we expose tools from both a prebuilt connector and a custom self-hosted server through one gateway endpoint? | Pending reference server (2.7). | |
|
||||
| 5 | What happens when a client requests a tool the user has not been granted? | `McpError: tool not enabled for this gateway` — clean rejection at the Engine, no leak/execution. | probes.md |
|
||||
|
||||
## Suggested pass/fail gates
|
||||
| Gate | Pass condition (verbatim) | Result | Evidence |
|
||||
|---|---|---|---|
|
||||
| MCP protocol compliance | Any compliant MCP client connects without custom adapters | | |
|
||||
| Tool curation | Gateway tool list matches exactly the configured allow-list | | |
|
||||
| Per-user isolation | User A cannot see or invoke tools granted only to User B | | |
|
||||
| Mixed server gateway | Prebuilt and custom server tools coexist behind one gateway URL | | |
|
||||
| MCP protocol compliance | Any compliant MCP client connects without custom adapters | PASS (lib client; Claude Code to add in 2.2) | probes.md |
|
||||
| Tool curation | Gateway tool list matches exactly the configured allow-list | PASS | probes.md |
|
||||
| Per-user isolation | User A cannot see or invoke tools granted only to User B | Not demonstrable on this gateway — no per-user grants (both see all 7). Needs cat-3 / separate gateways / User Source. **(finding)** | probes.md |
|
||||
| Mixed server gateway | Prebuilt and custom server tools coexist behind one gateway URL | Pending (2.7) | |
|
||||
|
||||
## Findings
|
||||
-
|
||||
- **Per-user tool-list scoping is gateway-wide, not per-user, in Arcade-Headers mode** (A==B identical). Differentiation requires Contextual Access (cat 3) or separate gateways / a User Source. Signals the score-3 anchor ("per-user scoping requires workarounds") unless cat-3 lifts it.
|
||||
- **Invocation routes through the Engine and fails cleanly** when an OAuth provider/secret isn't configured (`Slack_WhoAmI` → "unsupported authorization provider type ID '' (providerID 'slack')") — no silent fallback to a shared credential.
|
||||
- **Ungranted tool** → `tool not enabled for this gateway` (clean rejection).
|
||||
- Minor protocol nit: client logs `Session termination failed: 202` on session DELETE (benign).
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Cat-1 live probes — raw evidence
|
||||
|
||||
Target: `https://api.arcade.st.dev/mcp/zeb-gateway-test` (auth mode: Arcade Headers — API key + `Arcade-User-ID`).
|
||||
Client: `lib/mcp_client.py` (official `mcp` SDK, streamable HTTP). Date: 2026-06-18.
|
||||
|
||||
## Connect + list (as User A) — protocol compliance
|
||||
`connect_and_list` initialized an MCP session and returned 7 tools, no custom adapter:
|
||||
```
|
||||
Brightdata_ScrapeAsMarkdown
|
||||
GoogleDocs_CreateBlankDocument
|
||||
GoogleDocs_CreateDocumentFromText
|
||||
GoogleDocs_GetDocumentAsDocmd
|
||||
GoogleDocs_GetDocumentById
|
||||
Slack_SendMessage
|
||||
Slack_WhoAmI
|
||||
```
|
||||
Minor: client logs `Session termination failed: 202` on session close (server returns 202 to the
|
||||
streamable-HTTP session DELETE; benign — listing/calls succeed). Note as a protocol nit.
|
||||
|
||||
## Per-user visibility (User A vs User B) — criterion 3
|
||||
Listed as `ARCADE_USER_A` and `ARCADE_USER_B`:
|
||||
```
|
||||
A count: 7 | B count: 7
|
||||
A == B set?: True
|
||||
only in A: set() | only in B: set()
|
||||
```
|
||||
**Finding:** on a single gateway in Arcade-Headers mode the curated tool list is **gateway-wide,
|
||||
identical for every `Arcade-User-ID`** — not per-user. Per-user list differentiation would need
|
||||
Contextual Access (cat 3) or separate gateways / a User Source. (Token vault is still per-user_id.)
|
||||
|
||||
## Ungranted / unknown tool — benchmark Q5
|
||||
Calling a tool not exposed by the gateway, as User A:
|
||||
```
|
||||
call_tool("Github_CreateIssue", {}) -> McpError: tool not enabled for this gateway
|
||||
```
|
||||
Clean rejection at the Engine; no leak, no execution.
|
||||
|
||||
## Invocation of an auth-requiring tool (Slack_WhoAmI) as User A — error responses
|
||||
```
|
||||
isError: True
|
||||
content: "unsupported authorization provider type ID '' (providerID 'slack')"
|
||||
```
|
||||
The Engine processed the call and failed cleanly because the Slack OAuth provider isn't configured
|
||||
(the gateway's "Missing Requirements"). Demonstrates: invocation routes through the Engine, returns a
|
||||
structured error result, and does **not** fall back to a shared credential. Clean per-call invocation
|
||||
without per-user OAuth + a project secret (e.g. serp_api_key) is deferred to the reference server (`echo`).
|
||||
+13
-1
@@ -9,7 +9,19 @@ endpoints:
|
||||
mcp_url_pattern: https://api.arcade.st.dev/mcp/{slug}
|
||||
|
||||
# slug -> {tools: [...], servers: [...], created_by, notes} (filled in Phase 1)
|
||||
gateways: {}
|
||||
gateways:
|
||||
zeb-gateway-test:
|
||||
auth_mode: arcade-headers # API key + Arcade-User-ID (headless)
|
||||
created_by: ztaylor
|
||||
tools:
|
||||
- Brightdata_ScrapeAsMarkdown
|
||||
- GoogleDocs_CreateBlankDocument
|
||||
- GoogleDocs_CreateDocumentFromText
|
||||
- GoogleDocs_GetDocumentAsDocmd
|
||||
- GoogleDocs_GetDocumentById
|
||||
- Slack_SendMessage
|
||||
- Slack_WhoAmI
|
||||
notes: baseline cat-1 gateway from main catalog (Slack, GoogleDocs, Brightdata)
|
||||
|
||||
# name -> {kind: hosted|self-hosted, tools: [...], created_by, notes} (filled in Task 1.4)
|
||||
servers: {}
|
||||
|
||||
Reference in New Issue
Block a user