servicetitan/arcade-eval
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: _TEMPLATE + all-10 criteria-section stubs (verbatim criteria)

Browse Source
This commit is contained in:
iztaylor
2026-06-18 10:10:17 -04:00
parent 29c5b2c8be
commit 593e1e63b6
13 changed files with 510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/cat4-connectors/criteria-section-4.md
+56
View File
@@ -0,0 +1,56 @@
# Category 4 — Connector Coverage and Custom Server Development (weight 10)
> Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; **the human
> pastes**. 1–5 scale; anchors at 1/3/5.
## Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Prebuilt catalog covers required systems (GitHub, Salesforce, Atlassian/Jira). | | |
| 2 | Python SDK (arcade-mcp) supports building custom servers with minimal boilerplate. | | |
| 3 | Tool schema is auto-derived from Python type annotations — no manual schema authoring. | | |
| 4 | Local development loop works without cloud infrastructure (stdio mode). | | |
| 5 | Custom servers can be registered as self-hosted (HTTPS endpoint) and routed by the Engine. | | |
| 6 | Custom OAuth provider registration — Engine brokers per-user tokens for custom systems. | | |
| 7 | Custom servers can be versioned and updated without gateway downtime. | | |
**Average:** ___ **Category score:** ___
## Score anchors
- **1** — No SDK; custom integration requires raw HTTP server and manual schema
- **3** — SDK works for basic cases; custom OAuth is underdocumented; some systems blocked
- **5** — SDK is productive, custom OAuth providers are documented and straightforward, all six systems have a working path
## Coverage of required systems (verbatim)
| System | Prebuilt? | Path |
|---|---|---|
| GitHub | Yes | Prebuilt (global catalog) |
| Salesforce | Yes | Prebuilt (global catalog) |
| Atlassian / Jira | Partial | Prebuilt; confirm Confluence coverage |
| HubSpot | Yes | Prebuilt (global catalog) |
| QuickBooks | No | Custom server + custom OAuth provider |
| Sage | No | Custom server + custom OAuth provider |
| Snowflake | No | Custom server + custom OAuth provider |
| Workday | No | Custom server + custom OAuth provider |
| TenantTalk | No (internal) | Custom server + Okta-backed OAuth |
## Benchmark tests
| # | Test (verbatim) | Result | Evidence |
|---|---|---|---|
| 1 | Build a minimal custom server for one internal API using the arcade-mcp SDK. Measure time from schema to first successful local tool call. Target: under 2 hours. | | |
| 2 | Register the custom server as self-hosted (HTTPS endpoint). Verify Engine routing works and tool calls reach the server. | | |
| 3 | Configure Snowflake (or equivalent) as a custom OAuth provider. Complete per-user token flow end-to-end. | | |
| 4 | Verify tool schema is auto-derived from Python type annotations. Confirm no manual JSON schema authoring is required. | | |
| 5 | Update a custom tool's implementation. Verify the change takes effect without restarting the gateway. | | |
## Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| SDK productivity | Custom server from scratch to first local tool call in under 2 hours | | |
| Self-hosted registration | HTTPS endpoint registers and routes correctly through the Engine | | |
| Custom OAuth | At least one non-standard OAuth provider configured end-to-end | | |
| Schema derivation | Tool schema is auto-derived; no manual JSON schema authoring required | | |
| All systems covered | A working path (prebuilt or custom) exists for all six required systems | | |
## Findings
-