Files
arcade-eval/categories/cat1-functional/NOTES.md
T

2.4 KiB
Raw Blame History

Lane notes — Category 1 (Functional MCP Gateway Capability)

  • Owner: ztaylor
  • Last live-state check: 2026-06-18 (dashboard 200; gateway zeb-gateway-test live)
  • Fixtures used: gateway zeb-gateway-test (7 main-catalog tools, Arcade-Headers auth); users A/B from config/targets.yaml. Raw evidence in tests/probes.md.

Log

  • 2026-06-18 — lib client connects live; protocol/curation/per-user-visibility/ungranted probes done.
    • PASS: protocol compliance (connect/list/invoke/structured error), tool curation (7 listed == 7 selected).
    • FINDING: per-user tool list scoping not differentiated via Arcade-Headers on one gateway (A==B). Needs cat-3 Contextual Access or separate gateways / User Source.
    • Q5: ungranted tool → McpError: tool not enabled for this gateway.

Remaining for cat-1 scoring

  • 2.2 (Claude Code) — connected with NO adapter in both modes: Arcade-Headers (claude mcp add) AND Entra User-Source OAuth (/mcp login → tools loaded in-session; echo/whoami ran). Key kept as ${ARCADE_API_KEY} ref (not persisted).
  • [~] 2.2 (Cursor/LangGraph/internal) — not exercised this round; no adapter expected (same transport). Cursor config currently empty.
  • 2.8 — scorecard FINALIZED (draft 4/5) in criteria-section-1.md; awaiting user paste into Google Doc.

Side evidence generated (handed to other lanes)

  • cat-2: Entra IdP login works; identity = opaque sub; downstream OAuth consent-bind mismatch (see LIVE-POC).
  • cat-4/8/9: arcade deploy is cloud-only → self-hosted servers use the register path.
  • cat-9: full tunnel-registration chain validated end-to-end (client→gateway→Engine→tunnel→local server).
  • 2.5 — dynamic registration: PASS — saved add/remove (Brightdata, +Youtube) reflected on next list, no restart; draft didn't propagate until Save.
  • Reference server built at lib/mcp_server (echo/add/whoami); locally validated by arcade deploy (3 tools, 0 secrets). arcade deploy is cloud-only (finding) — see LIVE-POC.
  • 2.7 — mixed prebuilt + custom: PASS — gateway lists 7 prebuilt + 3 custom (ArcadeEvalRef_*, self-hosted via cloudflared tunnel) in one flat list; echo invokes. Full chain validated (also cat-9 Stage-2).
  • 2.4 — whoami execution proof: PASS — whoami A→user-a, B→user-b (calls execute as caller).
  • 2.8 — finalize scores once the above land.