arcade-eval/categories/cat2-delegated-authz/criteria-section-2.md

Category 2 — Delegated Authorization and Identity (weight 20)

The load-bearing category: every tool call executes as the calling user, using that user's own credentials, and the agent code never sees the token. Verbatim criteria/gates from the criteria Google Doc. Fill Score/Evidence locally; the human pastes. 1–5 scale; anchors at 1/3/5.

Scores

#	Criterion (verbatim)	Score (1–5)	Evidence / note
1	Per-user OAuth token vault — tokens are stored and refreshed per user, per service, per scope.
2	Tool calls execute as the calling user — not a shared service account or bot credential.
3	Okta (OIDC/SAML) integration as the primary IDP for gateway access.
4	Custom OAuth provider support — ability to register non-standard OAuth providers (Snowflake, Workday, TenantTalk via Okta).
5	Token refresh is handled automatically without requiring user re-authentication on every call.
6	The LLM and agent code never see raw tokens — token injection happens server-side in the Engine.
7	Token vault is project-scoped — no cross-project token leakage.
8	Admin consent — ability for an admin to pre-authorize a scope on behalf of a class of users.
9	Admin-initiated token revocation — an admin can invalidate all vault tokens for a specific user directly in Arcade, without touching any downstream provider. Primary use case: employee offboarding or security incident response.

Average: ___ Category score: ___

Score anchors

• 1 — Shared API keys or service accounts only; no per-user identity
• 3 — Per-user OAuth works for prebuilt connectors; custom providers require undocumented manual steps; revocation requires going to each provider individually
• 5 — Full per-user vault, Okta integration, custom OAuth providers documented and working, token refresh transparent, admin-initiated revocation works from one place

Benchmark tests

#	Test (verbatim)	Result	Evidence
1	Call a tool as User A. Verify it executes with User A's credentials by checking the downstream system's own audit log (e.g., GitHub shows the call as User A, not a service account).
2	Revoke User A's OAuth token in the provider. Verify the next tool call triggers a consent/re-auth flow rather than silently failing or falling back to a shared credential.
3	Configure a custom OAuth provider (Snowflake or Workday). Complete a full per-user token flow end-to-end: authorize → vault stores token → tool call executes as that user.
4	Configure TenantTalk authentication via Okta as a custom OAuth provider. Verify the Engine brokers the token correctly.
5	Verify token refresh: let a token expire. Confirm the next call either refreshes transparently or returns a clear re-auth prompt.
6	Admin-initiated revocation: as an admin, invalidate all vault tokens for User A in Arcade directly (no downstream provider action). Verify User A's next tool call fails or triggers re-auth, across all connected systems simultaneously.

Suggested pass/fail gates

Gate	Pass condition (verbatim)	Result	Evidence
Per-user execution	Tool calls provably execute as the calling user (verifiable in the downstream system's own logs)
No shared credentials	No service account or shared token is used in any tool call path
Okta integration	Gateway access works end-to-end through Okta OIDC/SAML
Custom OAuth	At least one custom provider (Snowflake or Workday) configured and functional
Token isolation	No user's token is accessible by, or executed as, another user
Downstream revocation	Revoking a token at the provider level triggers re-auth on the next call — no silent fallback
Admin-initiated revocation	An admin can invalidate all of a specific user's vault tokens in Arcade directly, taking effect across all connected systems without touching each provider individually

Findings

• Note (deployment): live POC upstream IdP is Entra ID, not Okta yet — score criterion 3 against that gap.