servicetitan/arcade-eval
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: _TEMPLATE + all-10 criteria-section stubs (verbatim criteria)

Browse Source
This commit is contained in:
iztaylor
2026-06-18 10:10:17 -04:00
parent 29c5b2c8be
commit 593e1e63b6
13 changed files with 510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/cat10-product-fit/criteria-section-10.md
+49
View File
@@ -0,0 +1,49 @@
# Category 10 — Product Fit — Tools Catalog and Multi-Tenancy (weight 5)
> *Scored only if the engineering team proceeds to evaluate Arcade as the MCP gateway layer for
> ServiceTitan's customer-facing tools catalog.* Verbatim criteria/gates from the criteria Google
> Doc. Fill Score/Evidence locally; **the human pastes**. 1–5 scale; anchors at 1/3/5.
**The multi-tenancy problem (verbatim):** ServiceTitan is a multi-tenant SaaS serving tens of
thousands of business tenants. Creating one Arcade project per tenant is not a viable architecture.
The requirement is a single shared Arcade deployment where tenant isolation is enforced within it:
Tenant A's users cannot access Tenant B's tokens, tool grants, or data. Arcade's native isolation
boundary is the **project**; within a project, isolation is at the `user_id` level.
## Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Native multi-tenant isolation within a single project — Tenant A's tokens, tool grants, and policy are fully isolated from Tenant B's without separate projects. | | |
| 2 | Per-tenant tool access policies — different tenants can have different tool allowlists and Contextual Access rules. | | |
| 3 | Per-tenant quota and rate limits — one tenant's usage cannot degrade another's. | | |
| 4 | Cross-tenant token isolation — provably no path for Tenant A's token to be served on a Tenant B tool call. | | |
| 5 | New tenants can be provisioned programmatically via API — no manual steps, no UI clicks. | | |
| 6 | Gateway configuration is API-driven to support programmatic tenant onboarding at scale. | | |
| 7 | Custom servers built for internal use can be reused for the product use case without re-architecting. | | |
**Average:** ___ **Category score:** ___
## Score anchors
- **1** — No multi-tenant model; one project per tenant is the only isolation path — does not scale
- **3** — user_id-level token isolation works within a project; tenant-level policy and quota require significant custom work
- **5** — Native multi-tenant model within a single deployment — per-tenant isolation, policy, quota, and API-driven onboarding all supported
## Benchmark questions
| # | Question (verbatim) | Answer | Evidence |
|---|---|---|---|
| 1 | Does Arcade have a native multi-tenancy model within a single project, or does tenant isolation require one project per tenant? | | |
| 2 | If `tenant_id:user_id` is used as the user_id, does Arcade enforce any tenant-level policy or quota boundaries, or is it purely token isolation? | | |
| 3 | Can per-tenant tool access policies (different tool lists per tenant) be managed via API? | | |
| 4 | Can a new tenant be onboarded — token vault initialized, tool grants set, gateway access configured — entirely via API with no manual steps? | | |
| 5 | What is the recommended architecture for serving tens of thousands of tenants from a single Arcade deployment? | | |
## Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| Multi-tenant isolation | Tenant A's tokens and tool grants are provably inaccessible to Tenant B within a single deployment | | |
| No per-tenant project | Tenant isolation does not require one Arcade project per tenant | | |
| API-driven onboarding | A new tenant can be fully provisioned via API with no manual steps | | |
| Per-tenant policy | Different tenants can have different tool allowlists managed programmatically | | |
## Findings
-