3.8 KiB
3.8 KiB
Category 9 — Developer Experience (weight 5)
Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; the human pastes. 1–5 scale; anchors at 1/3/5.
Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Local development loop is productive — stdio mode enables tool development without cloud infrastructure (Stage 1: code runs locally, MCP client spawns the server directly). | ||
| 2 | Tunnel-based development loop is supported — a developer can expose their locally running MCP server through a tunnel (Cloudflare, ngrok) and register it against a shared dev Arcade instance to exercise the full request chain (gateway → Engine → tunnel → local server) without deploying to Kubernetes. This is the primary development pattern for custom server authors. | ||
| 3 | A shared dev Arcade instance is available for ServiceTitan developers to register tunnel endpoints against — no need to provision a personal Arcade org for every developer. | ||
| 4 | Cloudflare tunnel (or equivalent) is the standardized proxy mechanism — documented, with a permanent named-tunnel option so the registered server URL does not change on every session restart. | ||
| 5 | SDK documentation is complete, accurate, and has working examples. | ||
| 6 | Error messages are actionable — auth failures, misconfigurations, and policy blocks identify the root cause. | ||
| 7 | MCP client integration requires no custom adapters or wrappers. | ||
| 8 | Gateway and server management are automatable via API. |
Average: ___ Category score: ___
Score anchors
- 1 — No tunnel support; local development requires full Kubernetes deployment to test the gateway chain
- 3 — Tunnel registration works but is underdocumented; no shared dev instance; engineers figure it out individually
- 5 — Tunnel loop is documented with a standard Cloudflare recipe; shared dev instance available; engineers are productive without platform hand-holding
Benchmark tests
| # | Test (verbatim) | Result | Evidence |
|---|---|---|---|
| 1 | Stage 1 — local stdio: Time an engineer from SDK install to first successful local tool call (stdio mode, no Arcade infrastructure). Target: under 2 hours. | ||
| 2 | Stage 2 — tunnel registration (the key test): Developer runs a local MCP server in HTTP mode, opens a Cloudflare tunnel, registers the tunnel URL as a self-hosted server in the dev Arcade instance, and makes a tool call that flows: Claude Code → gateway → Engine → Cloudflare tunnel → local server. Verify the full chain works including auth and Contextual Access. Measure time from working local server to first successful gateway-mediated call. Target: under 1 day. | ||
| 3 | Verify Cloudflare named tunnel (permanent hostname) — confirm the registered URL survives session restarts without re-editing the server registration. | ||
| 4 | Intentionally misconfigure an OAuth provider. Measure how quickly the error message identifies the root cause. | ||
| 5 | Integrate with Claude Code from scratch — time from gateway URL to working tool invocation in Claude Code. Target: under 30 minutes. |
Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| Tunnel loop | Full gateway → Engine → Cloudflare tunnel → local server chain works end-to-end | ||
| Permanent tunnel URL | Named tunnel hostname persists across session restarts without re-registration | ||
| Shared dev instance | ServiceTitan developers can register local servers against a shared dev Arcade org without individual account provisioning | ||
| Time to first call | Engineer reaches a working gateway-mediated tool call in under 1 day from scratch |