From a0d9bcf2fc7182322d3c370529668c859bd00f1b Mon Sep 17 00:00:00 2001 From: iztaylor Date: Thu, 18 Jun 2026 11:06:20 -0400 Subject: [PATCH] cat1: dynamic registration PASS (saved edit -Brightdata +Youtube reflected live, no restart) --- categories/cat1-functional/NOTES.md | 2 +- categories/cat1-functional/criteria-section-1.md | 7 ++++--- categories/cat1-functional/tests/probes.md | 12 ++++++++++++ config/targets.yaml | 4 ++-- 4 files changed, 19 insertions(+), 6 deletions(-) diff --git a/categories/cat1-functional/NOTES.md b/categories/cat1-functional/NOTES.md index 7d1098b..65d5578 100644 --- a/categories/cat1-functional/NOTES.md +++ b/categories/cat1-functional/NOTES.md @@ -12,7 +12,7 @@ ## Remaining for cat-1 scoring - [ ] 2.2 — connect a **second real MCP client (Claude Code)** to the gateway (no-adapter evidence). -- [ ] 2.5 — **dynamic registration**: add/remove a tool on the gateway (dashboard or API), re-list, confirm no restart. (needs a gateway edit) +- [x] 2.5 — **dynamic registration**: PASS — saved add/remove (−Brightdata, +Youtube) reflected on next list, no restart; draft didn't propagate until Save. - [ ] 2.7 — **mixed prebuilt + custom**: compose a gateway with a `main` tool + a `lib/mcp_server` tool. (needs reference server → `arcade login`/`arcade deploy`) - [ ] 2.4 — **`whoami` execution proof** that calls run as the calling user. (needs reference server) - [ ] 2.8 — finalize scores once the above land. diff --git a/categories/cat1-functional/criteria-section-1.md b/categories/cat1-functional/criteria-section-1.md index 96c4c87..2939405 100644 --- a/categories/cat1-functional/criteria-section-1.md +++ b/categories/cat1-functional/criteria-section-1.md @@ -14,8 +14,8 @@ | 4 | Supports all required MCP clients without custom adapters (Claude Code, Cursor, LangGraph, internal agent frameworks). | | PARTIAL — custom `mcp`-SDK client connected with no adapter ✓. Claude Code connect = 2.2; Cursor = teammate test. | | 5 | Tool execution isolation — one user's tool call cannot access another user's tokens or context. | | PENDING — vault is per-`user_id` by design; direct proof via reference-server `whoami` (2.4). | | 6 | Supports mixing prebuilt (global catalog) and custom (self-hosted) servers behind a single gateway URL. | | PENDING — needs reference server (2.7). | -| 7 | Gateway is pure metadata — adding or removing tools does not require server redeployment. | | PARTIAL — gateway is metadata (per Arcade model); add/remove-without-restart probe = 2.5. | -| 8 | Dynamic tool registration — new tools become available without gateway restart. | | PENDING — 2.5 (needs a gateway edit). | +| 7 | Gateway is pure metadata — adding or removing tools does not require server redeployment. | | PASS — saved edit (remove Brightdata, add Youtube_SearchForVideos) reflected on next `tools/list`, no restart. | +| 8 | Dynamic tool registration — new tools become available without gateway restart. | | PASS — new tool appeared immediately after Save; no engine/server restart. | **Average:** ___ **Category score:** ___ @@ -29,7 +29,7 @@ |---|---|---|---| | 1 | Can a Claude Code client connect to the gateway and see only the tools granted to the current user? | Connect: lib client ✓; Claude Code pending (2.2). "Only granted tools": N/A — no per-user grants on this gateway (list is gateway-wide). | probes.md | | 2 | Can the same gateway URL serve two different users with different tool lists? | **No** — A and B see identical 7 tools. | probes.md (A==B) | -| 3 | Can we add a tool to the gateway without restarting any server or the Engine? | Pending (2.5). | | +| 3 | Can we add a tool to the gateway without restarting any server or the Engine? | **Yes** — saved add/remove appeared on the next `tools/list`, no restart. (Draft edit did NOT propagate until Save — expected.) | probes.md | | 4 | Can we expose tools from both a prebuilt connector and a custom self-hosted server through one gateway endpoint? | Pending reference server (2.7). | | | 5 | What happens when a client requests a tool the user has not been granted? | `McpError: tool not enabled for this gateway` — clean rejection at the Engine, no leak/execution. | probes.md | @@ -45,4 +45,5 @@ - **Per-user tool-list scoping is gateway-wide, not per-user, in Arcade-Headers mode** (A==B identical). Differentiation requires Contextual Access (cat 3) or separate gateways / a User Source. Signals the score-3 anchor ("per-user scoping requires workarounds") unless cat-3 lifts it. - **Invocation routes through the Engine and fails cleanly** when an OAuth provider/secret isn't configured (`Slack_WhoAmI` → "unsupported authorization provider type ID '' (providerID 'slack')") — no silent fallback to a shared credential. - **Ungranted tool** → `tool not enabled for this gateway` (clean rejection). +- **Dynamic registration works**: a saved gateway edit (add + remove tools) takes effect on the next `tools/list` with no engine/server restart — gateway is pure metadata. Edits only apply after **Save** (drafts don't propagate). - Minor protocol nit: client logs `Session termination failed: 202` on session DELETE (benign). diff --git a/categories/cat1-functional/tests/probes.md b/categories/cat1-functional/tests/probes.md index 5a51e43..f018b37 100644 --- a/categories/cat1-functional/tests/probes.md +++ b/categories/cat1-functional/tests/probes.md @@ -44,3 +44,15 @@ The Engine processed the call and failed cleanly because the Slack OAuth provide (the gateway's "Missing Requirements"). Demonstrates: invocation routes through the Engine, returns a structured error result, and does **not** fall back to a shared credential. Clean per-call invocation without per-user OAuth + a project secret (e.g. serp_api_key) is deferred to the reference server (`echo`). + +## Dynamic registration (2.5) — criteria 7 & 8, benchmark Q3 +Edited `zeb-gateway-test` in the dashboard: removed `Brightdata_ScrapeAsMarkdown`, added +`Youtube_SearchForVideos`. A fresh `tools/list`: +``` +added since first probe: ['Youtube_SearchForVideos'] +removed since first probe: ['Brightdata_ScrapeAsMarkdown'] +``` +**PASS** — change took effect on the next list with **no engine/server restart** (gateway = pure +metadata). Corollary: the edit did **not** propagate while unsaved (draft); it appeared only after +**Save** — correct/expected, not a defect. Propagation was effectively immediate (next poll). + diff --git a/config/targets.yaml b/config/targets.yaml index 46916b1..0ab856d 100644 --- a/config/targets.yaml +++ b/config/targets.yaml @@ -14,14 +14,14 @@ gateways: auth_mode: arcade-headers # API key + Arcade-User-ID (headless) created_by: ztaylor tools: - - Brightdata_ScrapeAsMarkdown - GoogleDocs_CreateBlankDocument - GoogleDocs_CreateDocumentFromText - GoogleDocs_GetDocumentAsDocmd - GoogleDocs_GetDocumentById - Slack_SendMessage - Slack_WhoAmI - notes: baseline cat-1 gateway from main catalog (Slack, GoogleDocs, Brightdata) + - Youtube_SearchForVideos + notes: baseline cat-1 gateway from main catalog (Slack, GoogleDocs, Youtube). Edited live 2026-06-18 (-Brightdata, +Youtube) to test dynamic registration. # name -> {kind: hosted|self-hosted, tools: [...], created_by, notes} (filled in Task 1.4) servers: {}