servicetitan/arcade-eval
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: _TEMPLATE + all-10 criteria-section stubs (verbatim criteria)

Browse Source
This commit is contained in:
iztaylor
2026-06-18 10:10:17 -04:00
parent 29c5b2c8be
commit 593e1e63b6
13 changed files with 510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/cat8-deployment-ops/criteria-section-8.md
+41
View File
@@ -0,0 +1,41 @@
# Category 8 — Deployment and Operations (weight 7)
> Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; **the human
> pastes**. 1–5 scale; anchors at 1/3/5.
## Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Helm chart available and documented for self-hosted server deployment in Kubernetes. | | |
| 2 | Zero-downtime configuration updates — gateway and policy changes do not interrupt in-flight calls. | | |
| 3 | GitOps-compatible — gateway, server, and policy configuration is expressible as code. | | |
| 4 | Upgrade and rollback process is documented and tested. | | |
| 5 | Runbooks for common failure scenarios. | | |
| 6 | Vendor support model during the pilot: dedicated solutions engineer, response SLA. | | |
| 7 | P0/P1 escalation path after the pilot, in production. | | |
**Average:** ___ **Category score:** ___
## Score anchors
- **1** — No Helm chart; manual deployment only; no dedicated support
- **3** — Helm chart works with gaps; zero-downtime config updates unverified; support exists but is not dedicated
- **5** — Helm-native, GitOps-compatible, zero-downtime config, dedicated SE, documented escalation path
## Benchmark tests
| # | Test (verbatim) | Result | Evidence |
|---|---|---|---|
| 1 | Deploy a self-hosted custom server to a Kubernetes namespace via Helm chart. Measure time from clean namespace to first successful tool call. Target: under 1 day. | | |
| 2 | Update a gateway configuration (add a tool). Verify in-flight calls are not dropped. | | |
| 3 | Simulate a configuration rollback. Verify the rollback completes cleanly and the prior configuration is restored. | | |
| 4 | Stage a K8s namespace and confirm the Helm deployment matches the architecture recommended for production. | | |
## Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| Helm deployment | Full Kubernetes deployment via Helm chart in under 1 day | | |
| Config safety | Gateway configuration changes are zero-downtime | | |
| Rollback | Prior configuration can be restored cleanly | | |
| Dedicated SE | A dedicated solutions engineer is available and responsive during the pilot | | |
## Findings
- Note: a live deployment already exists (`k8s-backstage-v2/apps/arcade`, chart 1.8.8, Flux/GitOps) — a head start for this category's evidence.