Files
arcade-eval/categories/cat1-functional/NOTES.md
T

1.7 KiB
Raw Blame History

Lane notes — Category 1 (Functional MCP Gateway Capability)

  • Owner: ztaylor
  • Last live-state check: 2026-06-18 (dashboard 200; gateway zeb-gateway-test live)
  • Fixtures used: gateway zeb-gateway-test (7 main-catalog tools, Arcade-Headers auth); users A/B from config/targets.yaml. Raw evidence in tests/probes.md.

Log

  • 2026-06-18 — lib client connects live; protocol/curation/per-user-visibility/ungranted probes done.
    • PASS: protocol compliance (connect/list/invoke/structured error), tool curation (7 listed == 7 selected).
    • FINDING: per-user tool list scoping not differentiated via Arcade-Headers on one gateway (A==B). Needs cat-3 Contextual Access or separate gateways / User Source.
    • Q5: ungranted tool → McpError: tool not enabled for this gateway.

Remaining for cat-1 scoring

  • 2.2 — connect a second real MCP client (Claude Code) to the gateway (no-adapter evidence).
  • 2.5 — dynamic registration: PASS — saved add/remove (Brightdata, +Youtube) reflected on next list, no restart; draft didn't propagate until Save.
  • Reference server built at lib/mcp_server (echo/add/whoami); locally validated by arcade deploy (3 tools, 0 secrets). arcade deploy is cloud-only (finding) — see LIVE-POC.
  • 2.7 — mixed prebuilt + custom: needs the ref server behind the self-hosted Engine via the register path (run server.py --transport http + cloudflared tunnel + dashboard Add Server), then compose a gateway (a main tool + echo). Doubles as cat-9 Stage-2.
  • 2.4 — whoami execution proof: once registered, call whoami as A vs B (expect A→A, B→B).
  • 2.8 — finalize scores once the above land.