arcade-eval/categories/cat3-access-policy/criteria-section-3.md

Category 3 — Tool-Level Access Control and Policy (weight 15)

Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; the human pastes. 1–5 scale; anchors at 1/3/5.

Scores

#	Criterion (verbatim)	Score (1–5)	Evidence / note
1	Tool-level allow-list per user — a user can only call tools explicitly granted to them; the gateway enforces this, not the client.
2	Contextual Access rules — per-user tool visibility and invocation policy layered on top of the gateway allow-list.
3	Input filtering — ability to block or rewrite tool inputs based on policy before execution reaches the server.
4	Output redaction — ability to mask or strip sensitive fields from tool outputs before they reach the agent.
5	Policy is enforced at the Engine, not the client — a malicious or compromised client cannot bypass it.
6	All policy decisions (allow, block, redact) are logged.
7	Per-user tool grants can be updated without restarting the gateway or any server.
8	Gateway scopes map to Okta groups — access managed in Okta, not a separate system.

Average: ___ Category score: ___

Score anchors

• 1 — Gateway-level tool list only; no per-user scoping or input/output policy
• 3 — Per-user grants work; Contextual Access input/output rules require significant manual work
• 5 — Full per-user policy, Contextual Access input/output rules, Okta-managed scopes, all decisions audited

Benchmark tests

#	Test (verbatim)	Result	Evidence
1	Grant User A access to GitHub tools and User B access to Atlassian tools. Verify User A cannot invoke Atlassian tools even if they know the tool name.
2	Write a Contextual Access rule that blocks inputs containing a specific pattern (e.g., a mock SSN). Send a matching input — verify it is blocked before execution and logged.
3	Write a Contextual Access rule that redacts a field from tool outputs. Verify the field is absent from the agent's response.
4	Update User A's tool grants (add a new tool). Verify the change takes effect without restarting anything.
5	Confirm policy enforcement point: attempt to bypass Contextual Access by calling the server directly (bypassing the Engine). Confirm this is architecturally prevented or explicitly documented as a known boundary.

Suggested pass/fail gates

Gate	Pass condition (verbatim)	Result	Evidence
Tool isolation	Cross-user tool calls are rejected at the Engine regardless of client behavior
Input policy	Blocked inputs are rejected before execution, not after
Output policy	Redacted fields are absent from the agent's response
Audit	Every policy decision (allow/block/redact) produces a retrievable log entry
Dynamic grants	Tool grant updates take effect without service restart

Findings