servicetitan/arcade-eval
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: _TEMPLATE + all-10 criteria-section stubs (verbatim criteria)

Browse Source
This commit is contained in:
iztaylor
2026-06-18 10:10:17 -04:00
parent 29c5b2c8be
commit 593e1e63b6
13 changed files with 510 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/cat3-access-policy/criteria-section-3.md
+44
View File
@@ -0,0 +1,44 @@
# Category 3 — Tool-Level Access Control and Policy (weight 15)
> Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; **the human
> pastes**. 1–5 scale; anchors at 1/3/5.
## Scores
| # | Criterion (verbatim) | Score (1–5) | Evidence / note |
|---|---|---|---|
| 1 | Tool-level allow-list per user — a user can only call tools explicitly granted to them; the gateway enforces this, not the client. | | |
| 2 | Contextual Access rules — per-user tool visibility and invocation policy layered on top of the gateway allow-list. | | |
| 3 | Input filtering — ability to block or rewrite tool inputs based on policy before execution reaches the server. | | |
| 4 | Output redaction — ability to mask or strip sensitive fields from tool outputs before they reach the agent. | | |
| 5 | Policy is enforced at the Engine, not the client — a malicious or compromised client cannot bypass it. | | |
| 6 | All policy decisions (allow, block, redact) are logged. | | |
| 7 | Per-user tool grants can be updated without restarting the gateway or any server. | | |
| 8 | Gateway scopes map to Okta groups — access managed in Okta, not a separate system. | | |
**Average:** ___ **Category score:** ___
## Score anchors
- **1** — Gateway-level tool list only; no per-user scoping or input/output policy
- **3** — Per-user grants work; Contextual Access input/output rules require significant manual work
- **5** — Full per-user policy, Contextual Access input/output rules, Okta-managed scopes, all decisions audited
## Benchmark tests
| # | Test (verbatim) | Result | Evidence |
|---|---|---|---|
| 1 | Grant User A access to GitHub tools and User B access to Atlassian tools. Verify User A cannot invoke Atlassian tools even if they know the tool name. | | |
| 2 | Write a Contextual Access rule that blocks inputs containing a specific pattern (e.g., a mock SSN). Send a matching input — verify it is blocked before execution and logged. | | |
| 3 | Write a Contextual Access rule that redacts a field from tool outputs. Verify the field is absent from the agent's response. | | |
| 4 | Update User A's tool grants (add a new tool). Verify the change takes effect without restarting anything. | | |
| 5 | Confirm policy enforcement point: attempt to bypass Contextual Access by calling the server directly (bypassing the Engine). Confirm this is architecturally prevented or explicitly documented as a known boundary. | | |
## Suggested pass/fail gates
| Gate | Pass condition (verbatim) | Result | Evidence |
|---|---|---|---|
| Tool isolation | Cross-user tool calls are rejected at the Engine regardless of client behavior | | |
| Input policy | Blocked inputs are rejected before execution, not after | | |
| Output policy | Redacted fields are absent from the agent's response | | |
| Audit | Every policy decision (allow/block/redact) produces a retrievable log entry | | |
| Dynamic grants | Tool grant updates take effect without service restart | | |
## Findings
-