✅ Schema Validation: Ensures your passports and decisions conform to OAP JSON schemas
✅ Policy Evaluation: Tests your policy logic with various contexts and limits
✅ Signature Verification: Validates your Ed25519 signatures over JCS-canonicalized payloads
✅ API Testing: Tests your OAP endpoints for compliance
✅ Report Generation: Produces detailed PASS/FAIL reports for certification

Quick Start

1. Get the OAP Specification

# Clone the OAP spec repository
git clone https://github.com/aporthq/oap-spec.git
cd oap-spec

2. Install Dependencies

# Navigate to conformance directory
cd spec/conformance

# Install dependencies
pnpm install

3. Test Your OAP Implementation

# Test your OAP API endpoint
pnpm test --endpoint https://your-oap-api.com

# Test with your passport file
pnpm test --passport-file /path/to/your-passport.json

# Test with your decision file
pnpm test --decision-file /path/to/your-decision.json

# Test all components
pnpm test --endpoint https://your-oap-api.com --passport-file passport.json --decision-file decision.json

4. Expected Output

🔍 OAP Conformance Test Runner v1.0.0

Testing your OAP implementation...
✅ Passport validation: PASS
✅ Decision validation: PASS
✅ Policy evaluation: PASS
✅ Signature verification: PASS
✅ API compliance: PASS

📊 Conformance Test Results
✅ Passed: 5
❌ Failed: 0
📈 Success Rate: 100.0%

🎯 Your implementation is OAP compliant!

CLI Commands

Testing Your Implementation

# Test your OAP API endpoint
pnpm test --endpoint https://your-oap-api.com

# Test with your passport file
pnpm test --passport-file /path/to/your-passport.json

# Test with your decision file
pnpm test --decision-file /path/to/your-decision.json

# Test all components together
pnpm test --endpoint https://your-oap-api.com --passport-file passport.json --decision-file decision.json

Policy Pack Testing

# Test specific policy pack against your implementation
pnpm test --endpoint https://your-oap-api.com --pack finance.payment.refund.v1

# Test with verbose output
pnpm test --endpoint https://your-oap-api.com --pack data.export.create.v1 --verbose

Reporting

# Generate detailed JSON report
pnpm test --endpoint https://your-oap-api.com --report

# Verbose output for debugging
pnpm test --endpoint https://your-oap-api.com --verbose

Development

# Watch mode for development
pnpm dev

# Build TypeScript
pnpm build

# Run simple JavaScript version (fallback)
pnpm run test:simple

Understanding Test Results

What "PASS" Means

✅ PASS: Implementation correctly enforces OAP policies
✅ PASS: Valid operations are allowed
✅ PASS: Invalid operations are properly denied

What "FAIL" Means

❌ FAIL: Implementation incorrectly allows/denies operations
❌ FAIL: Schema validation errors
❌ FAIL: Policy logic errors

Example Output

🔍 OAP Conformance Test Runner v1.0.0

✅ Loaded 5 test cases
Running data.export.create.v1:allow_users...
  ✅ PASS
Running data.export.create.v1:deny_pii...
  ❌ FAIL: Policy evaluation failed: PII export not allowed
Running finance.payment.refund.v1:allow_50usd...
  ✅ PASS
Running finance.payment.refund.v1:deny_150usd...
  ❌ FAIL: Policy evaluation failed: Amount 15000 exceeds max per transaction 5000
Running finance.payment.refund.v1:deny_currency...
  ✅ PASS

📊 Test Results
✅ Passed: 3
❌ Failed: 2
📈 Success Rate: 60.0%

❌ Failed Tests:
  • data.export.create.v1:deny_pii
    - Policy evaluation failed: PII export not allowed
  • finance.payment.refund.v1:deny_150usd
    - Policy evaluation failed: Amount 15000 exceeds max per transaction 5000

🎯 Conformance testing complete!

Note: The "failures" above are actually correct behavior - the system is properly denying operations that should be denied!

Certification Process

For OAP Implementers

Test Your Implementation: Run pnpm test --endpoint https://your-oap-api.com
Validate All Components: Test passports, decisions, and API endpoints
Achieve 100% Pass Rate: All tests must pass for certification
Review Detailed Report: Use --report flag for comprehensive results
Document Compliance: Use results for OAP certification claims

What Gets Tested

The conformance runner tests your implementation against:

Passport Creation: Does your API create valid OAP passports?
Decision Making: Does your policy engine make correct allow/deny decisions?
Schema Compliance: Do your JSON responses match OAP schemas?
Signature Generation: Do you generate valid Ed25519 signatures?
Error Handling: Do you return proper OAP error codes?
API Endpoints: Do your endpoints follow OAP patterns?

Integration with CI/CD

# Example GitHub Actions workflow
name: OAP Conformance Tests
on: [push, pull_request]
jobs:
  conformance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'
      - name: Install pnpm
        run: npm install -g pnpm
      - name: Run OAP Conformance Tests
        run: |
          cd spec/conformance
          pnpm install
          pnpm test --report

Test Structure

spec/conformance/
├── README.md              # This file
├── package.json           # Dependencies and scripts
├── tsconfig.json          # TypeScript configuration
├── test-runner.js         # Simple JavaScript fallback
├── src/                   # TypeScript source code
│   ├── runner.ts          # Main test runner
│   ├── validators.ts      # Schema and signature validators
│   ├── jcs.ts            # JCS canonicalization
│   ├── ed25519.ts        # Ed25519 signature verification
│   └── cases.ts          # Test case definitions
├── cases/                 # Test cases by policy pack
│   ├── finance.payment.refund.v1/       # Refunds policy pack tests
│   │   ├── passports/     # Test passport templates
│   │   ├── contexts/      # Test contexts
│   │   └── expected/      # Expected decisions
│   ├── data.export.create.v1/   # Data export policy pack tests
│   └── repo.release.publish.v1/ # Repository release tests
└── reports/              # Generated test reports

Test Cases

Each policy pack includes:

passports/ - Passport examples (template and instance)
contexts/ - Policy evaluation contexts
expected/ - Expected decision outputs
receipts/ - Decision receipts for signature verification

What Gets Tested

1. Passport Validation

Required fields present
Correct data types
Valid UUIDs and timestamps
Proper assurance levels (L0-L4FIN)
Valid capability structures

2. Policy Evaluation

Refunds: Amount limits, currency support, reason codes
Data Export: PII restrictions, collection limits, row limits
Repository Release: Branch restrictions, artifact signing

3. Decision Validation

Correct allow/deny logic
Proper reason codes
Valid signatures and digests
Correct TTL handling

4. Signature Verification

Ed25519 signature format validation
JCS canonicalization verification
Key resolution and validation

Reports

Test results are saved to reports/ with:

conformance-{timestamp}.json - Complete test results
Summary statistics and detailed per-case results
Signature verification results

Troubleshooting

Common Issues

"spawn /bin/zsh ENOENT"

# Use the simple JavaScript version
pnpm run test:simple

TypeScript compilation errors

# Install dependencies first
pnpm install

# Then run tests
pnpm test

Permission denied

# Make sure the test runner is executable
chmod +x test-runner.js

Getting Help

Check the OAP Specification for detailed requirements
Review test cases in the cases/ directory
Use --verbose flag for detailed debugging output