Conformance Testing - Agent Identity Protocol

Overview

The AIP Conformance Test Suite provides test vectors for validating implementations of the Agent Identity Protocol. Any implementation claiming AIP conformance must pass all tests at their claimed conformance level.

Conformance tests ensure interoperability across different AIP implementations and guarantee adherence to the specification.

Conformance Levels

AIP defines five conformance levels, each building on the previous:

Level	Description	Test Files	API Version
Basic	Minimum viable implementation	`basic/*.yaml`	v1alpha1+
Full	Complete feature support	Basic + `full/*.yaml`	v1alpha1+
Extended	Human-in-the-loop support	Full + `extended/*.yaml`	v1alpha1+
Identity	Token lifecycle management	Extended + `identity/*.yaml`	v1alpha2+
Server	HTTP validation endpoints	Identity + `server/*.yaml`	v1alpha2+

Level Details

Basic

Tool allowlists, blocking, error codes, and default deny behavior

Full

Argument validation, DLP scanning, rate limiting, and Unicode normalization

Extended

Human approval dialogs (ask action) and timeout handling

Identity

Agent Authentication Tokens (AAT), session binding, and token rotation

Server

HTTP validation endpoints, TLS, and distributed policy enforcement

Test Categories

Basic Conformance

basic/authorization.yaml

What it tests:

Tool allowlist enforcement
Tool blocking via tool_rules
Default deny behavior (fail-closed)
Monitor mode vs. enforce mode

Key test cases:

Tools in allowed_tools are allowed
Tools NOT in allowed_tools are blocked
Blocked tools remain blocked even if in allowlist
Monitor mode logs violations but allows requests

basic/methods.yaml

What it tests:

Method-level authorization
Method allowlist and denylist
Default allowed methods

basic/errors.yaml

What it tests:

Correct error codes (-32001 for permission denied)
Error message format
Error response structure

Full Conformance

full/arguments.yaml

What it tests:

Regex validation on tool arguments
Strict args mode (unexpected arguments)
Type coercion and validation

Example test:

tool_rules:
  - tool: write_file
    args:
      path:
        pattern: "^/tmp/.*"
# Only allows writing to /tmp/

full/normalization.yaml

What it tests:

Unicode NFKC normalization for security
Case-insensitive matching
Whitespace handling

Why it matters: Prevents bypass attacks using Unicode variants

full/dlp.yaml

What it tests:

Pattern matching for sensitive data
Redaction format (e.g., AKIA**** for AWS keys)
DLP in both requests and responses

Example patterns:

AWS keys: AKIA[A-Z0-9]{16}
GitHub tokens: ghp_[a-zA-Z0-9]{36}

Extended Conformance

extended/ask.yaml

What it tests:

Human-in-the-loop behavior (action: ask)
Approval dialog triggering
Timeout handling for user responses

Decision flow:

Agent calls sensitive tool
Proxy returns ASK decision
User approves or denies via OS dialog
Request proceeds or is blocked

Identity Conformance (v1alpha2)

identity/tokens.yaml

What it tests:

Agent Authentication Token (AAT) generation
Token structure validation (JWT format)
Token expiration and rotation
Cryptographic signature verification

identity/validation.yaml

What it tests:

Token validation success/failure
Replay attack detection
Policy change detection via hash
Revocation list checking

Server Conformance (v1alpha2)

server/endpoints.yaml

What it tests:

/validate HTTP endpoint request/response
/health endpoint
/metrics endpoint format (Prometheus)

server/authentication.yaml

What it tests:

Bearer token authentication
Token requirement enforcement
HTTP 401/403 error responses

Running Conformance Tests

Install the conformance validator

# Install the official validator
go install github.com/openagentidentityprotocol/agentidentityprotocol/tools/aip-conformance@latest

Run tests against your implementation

# Test at Basic level
aip-conformance --impl "your-binary" --level basic

# Test at Full level
aip-conformance --impl "your-binary" --level full

# Test all levels
aip-conformance --impl "your-binary" --level server

View results

The validator outputs:

✅ Passed tests
❌ Failed tests with details
Summary of conformance level achieved

Test Vector Format

Each test file contains YAML test cases:

name: "Basic Authorization"
description: "Tests for tool-level authorization decisions"
spec_version: "aip.io/v1alpha1"

tests:
  - id: "auth-001"
    description: "Tool in allowed_tools list should be allowed"
    policy: |
      apiVersion: aip.io/v1alpha1
      kind: AgentPolicy
      metadata:
        name: test-policy
      spec:
        allowed_tools:
          - read_file
    input:
      method: "tools/call"
      tool: "read_file"
      args:
        path: "/tmp/test.txt"
    expected:
      decision: "ALLOW"
      error_code: null
      violation: false

Matching Rules

decision

Exact string match: ALLOW, BLOCK, ASK, RATE_LIMITED, PROTECTED_PATH

error_code

Exact match: -32001 (permission denied), -32002 (rate limited), or null

violation

Boolean: true if policy violation detected, false otherwise

DLP tests

Verify redaction occurred in response (e.g., AKIA****)

Example: Running a Basic Test

Test Case

- id: "auth-002"
  description: "Tool NOT in allowed_tools list should be blocked"
  policy: |
    apiVersion: aip.io/v1alpha1
    kind: AgentPolicy
    metadata:
      name: test-policy
    spec:
      allowed_tools:
        - read_file
  input:
    method: "tools/call"
    tool: "delete_file"
    args:
      path: "/tmp/test.txt"
  expected:
    decision: "BLOCK"
    error_code: -32001
    violation: true

Test Execution

Load the policy

The validator loads the inline policy YAML into your implementation

Submit the input request

Sends tools/call with tool delete_file

Verify the output

Checks that your implementation returns:

Decision: BLOCK
Error code: -32001
Violation flag: true

Current Implementation Conformance

Implementation	Language	Conformance Level	Maintainer
go-proxy	Go	Full + Extended	@ArangoGutierrez

v1alpha2 Support: Identity and Server conformance levels are currently in development. The Go proxy targets Full + Extended for v1alpha1.

Implementation Guidance

Recommended Implementation Order

Start with Basic

Implement core authorization:

Tool allowlists (allowed_tools)
Tool blocking (action: block)
Default deny behavior
Error code -32001

Add Full support

Implement advanced features:

Argument validation (regex patterns)
DLP scanning
Unicode NFKC normalization
Rate limiting

Add Extended for production

Implement human-in-the-loop:

action: ask support
OS approval dialogs (platform-specific)
Timeout handling

Add Identity (optional)

Implement token management (v1alpha2):

AAT generation and validation
Session binding
Token rotation

Add Server (optional)

Implement HTTP endpoints for distributed deployments:

/validate endpoint
TLS requirement
Bearer token auth

Schema Validation

Validate policy files against the JSON Schema before testing:

# Using ajv-cli (v1alpha2)
npm install -g ajv-cli
ajv validate -s spec/schema/agent-policy-v1alpha2.schema.json -d your-policy.yaml

# Using Python jsonschema (v1alpha2)
pip install jsonschema pyyaml
python -c "
import yaml, jsonschema, json
schema = json.load(open('spec/schema/agent-policy-v1alpha2.schema.json'))
policy = yaml.safe_load(open('your-policy.yaml'))
jsonschema.validate(policy, schema)
print('Valid!')
"

Contributing Tests

When adding new conformance tests:

Positive AND Negative

Include both success and failure cases

Edge Cases

Test empty strings, Unicode, null values

Documentation

Explain WHY the expected result is correct

Deterministic

Ensure tests produce consistent results

Test Contribution Process

Open an issue

Describe the test scenario you want to add

Add test vectors

Write YAML test cases following the format

Submit PR

Include test file and update this documentation

Verify against reference implementation

Ensure Go proxy passes your new tests

Versioning

Test vectors are versioned alongside the specification:

v1alpha1 - Basic, Full, Extended
v1alpha2 - Identity, Server (backward compatible with v1alpha1)

Backward Compatibility: Implementations supporting v1alpha2 MUST also pass all v1alpha1 tests.

Next Steps

Go Implementation

Reference implementation and SDK

Specification

Read the formal protocol definition

Policy Reference

Complete YAML schema guide

Contributing

Build your own AIP implementation

Resources

Conformance Suite: spec/conformance/
JSON Schemas: spec/schema/
Reference Implementation: Go Proxy
Submit Issues: GitHub Issues

Specifications

API Reference

SDKs & Implementations

​Overview

​Conformance Levels

​Level Details

Basic

Full

Extended

Identity

Server

​Test Categories

​Basic Conformance

​Full Conformance

​Extended Conformance

​Identity Conformance (v1alpha2)

​Server Conformance (v1alpha2)

​Running Conformance Tests

​Test Vector Format

​Matching Rules

decision

error_code

violation

DLP tests

​Example: Running a Basic Test

​Test Case

​Test Execution

​Current Implementation Conformance

​Implementation Guidance

​Recommended Implementation Order

​Schema Validation

​Contributing Tests

Positive AND Negative

Edge Cases

Documentation

Deterministic

​Test Contribution Process

​Versioning

​Next Steps

Go Implementation

Specification

Policy Reference

Contributing

​Resources

Build docs developers (and LLMs) love

Overview

Conformance Levels

Level Details

Test Categories

Basic Conformance

Full Conformance

Extended Conformance

Identity Conformance (v1alpha2)

Server Conformance (v1alpha2)

Running Conformance Tests

Test Vector Format

Matching Rules

Example: Running a Basic Test

Test Case

Test Execution

Current Implementation Conformance

Implementation Guidance

Recommended Implementation Order

Schema Validation

Contributing Tests

Test Contribution Process

Versioning

Next Steps

Resources