3.2 KiB
3.2 KiB
Category 4 — Connector Coverage and Custom Server Development (weight 10)
Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; the human pastes. 1–5 scale; anchors at 1/3/5.
Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Prebuilt catalog covers required systems (GitHub, Salesforce, Atlassian/Jira). | ||
| 2 | Python SDK (arcade-mcp) supports building custom servers with minimal boilerplate. | ||
| 3 | Tool schema is auto-derived from Python type annotations — no manual schema authoring. | ||
| 4 | Local development loop works without cloud infrastructure (stdio mode). | ||
| 5 | Custom servers can be registered as self-hosted (HTTPS endpoint) and routed by the Engine. | ||
| 6 | Custom OAuth provider registration — Engine brokers per-user tokens for custom systems. | ||
| 7 | Custom servers can be versioned and updated without gateway downtime. |
Average: ___ Category score: ___
Score anchors
- 1 — No SDK; custom integration requires raw HTTP server and manual schema
- 3 — SDK works for basic cases; custom OAuth is underdocumented; some systems blocked
- 5 — SDK is productive, custom OAuth providers are documented and straightforward, all six systems have a working path
Coverage of required systems (verbatim)
| System | Prebuilt? | Path |
|---|---|---|
| GitHub | Yes | Prebuilt (global catalog) |
| Salesforce | Yes | Prebuilt (global catalog) |
| Atlassian / Jira | Partial | Prebuilt; confirm Confluence coverage |
| HubSpot | Yes | Prebuilt (global catalog) |
| QuickBooks | No | Custom server + custom OAuth provider |
| Sage | No | Custom server + custom OAuth provider |
| Snowflake | No | Custom server + custom OAuth provider |
| Workday | No | Custom server + custom OAuth provider |
| TenantTalk | No (internal) | Custom server + Okta-backed OAuth |
Benchmark tests
| # | Test (verbatim) | Result | Evidence |
|---|---|---|---|
| 1 | Build a minimal custom server for one internal API using the arcade-mcp SDK. Measure time from schema to first successful local tool call. Target: under 2 hours. | ||
| 2 | Register the custom server as self-hosted (HTTPS endpoint). Verify Engine routing works and tool calls reach the server. | ||
| 3 | Configure Snowflake (or equivalent) as a custom OAuth provider. Complete per-user token flow end-to-end. | ||
| 4 | Verify tool schema is auto-derived from Python type annotations. Confirm no manual JSON schema authoring is required. | ||
| 5 | Update a custom tool's implementation. Verify the change takes effect without restarting the gateway. |
Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| SDK productivity | Custom server from scratch to first local tool call in under 2 hours | ||
| Self-hosted registration | HTTPS endpoint registers and routes correctly through the Engine | ||
| Custom OAuth | At least one non-standard OAuth provider configured end-to-end | ||
| Schema derivation | Tool schema is auto-derived; no manual JSON schema authoring required | ||
| All systems covered | A working path (prebuilt or custom) exists for all six required systems |