servicetitan/arcade-eval
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: _TEMPLATE + all-10 criteria-section stubs (verbatim criteria)

Browse Source
This commit is contained in:
iztaylor
2026-06-18 10:10:17 -04:00
parent 29c5b2c8be
commit 593e1e63b6
13 changed files with 510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/cat7-performance/criteria-section-7.md
+42
View File
@@ -0,0 +1,42 @@
# Category 7 — Performance and Availability (weight 8)
> Because every gateway-mediated tool call routes through the Arcade Engine — even when the custom
> server is self-hosted — Engine latency and availability are a floor on the entire agent stack.
> Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; **the human
> pastes**. 1–5 scale; anchors at 1/3/5.
## Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Engine-added latency per tool call is within acceptable bounds for interactive agent use. | | |
| 2 | Engine SLA — defined uptime guarantees with incident response process. | | |
| 3 | Failure behavior when Engine is unavailable: fail-closed with a clean, catchable error. | | |
| 4 | Self-hosted server HA — multi-replica, pod failure handling, no dropped calls on restart. | | |
| 5 | Multi-region failover design — documented and validated. | | |
| 6 | Engine geographic placement and round-trip latency from ServiceTitan's primary region. | | |
**Average:** ___ **Category score:** ___
## Score anchors
- **1** — Engine SLA undocumented; failure behavior is a hang or silent failure; no HA guidance
- **3** — SLA documented; HA works with manual configuration; failure behavior is known but requires client-side handling
- **5** — SLA with incident response in writing; HA is the documented default; failure behavior is clean and observable
## Benchmark tests
| # | Test (verbatim) | Result | Evidence |
|---|---|---|---|
| 1 | Make 100 tool calls through the Engine to a self-hosted server. Measure P50, P95, P99 round-trip latency. Compare against a direct server call (bypassing the Engine) to isolate Engine-added overhead. | | |
| 2 | Simulate Engine unavailability (block the Engine endpoint). Confirm tool calls fail with a clean, catchable error — not a hang or silent failure. | | |
| 3 | Deploy the custom server with multiple replicas. Kill one pod. Confirm tool calls continue without dropped requests. | | |
| 4 | Confirm Engine SLA documentation: uptime percentage, response time commitment, and P0 escalation path. | | |
## Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| Engine overhead | P95 Engine-added latency is under 500ms for standard (non-streaming) tool calls | | |
| SLA documented | Engine uptime SLA and incident response process confirmed in writing | | |
| HA | Self-hosted server survives pod failure; no tool calls dropped during pod restart | | |
| Fail behavior | Engine outage produces a clean, catchable error to the agent — no hangs | | |
## Findings
-