A test suite that shows its work.
SAMLscope doesn't ask an operator to decide PASS or FAIL. It drives real protocol endpoints, records what actually happened, and reports what it could and could not verify.
Test Peer, not an admin API
SAMLscope acts as a real SAML counterpart: it issues its own IdP or SP Test Peer metadata, exchanges standard SAML and metadata messages with the target, and records a redacted Transcript of everything that crossed the wire. It never calls a vendor's administration API and never asks an operator to self-certify a result it could instead observe directly.
Every case result carries its own evidence basis
Results are never hidden for being weak. A result missing evidence is reported as NOT_VERIFIED, not silently excluded or treated as a target failure.
Externally verified
StrongestDerived from Suite-observed SAML, browser, metadata, or Transcript evidence.
Self-attested
Operator-reportedBased on operator-supplied evidence for behavior that standard external interfaces cannot establish on their own.
Not verified
No evidenceRequired evidence or configuration was unavailable or inconclusive. It stays in the denominator; it is never silently excluded.
Separately, every Run also reports conformance andcompleteness as two independent axes, always shown together. A Run can have zero failures and still beINDETERMINATE if a required obligation was never resolved: displaying only one axis is exactly the kind of contradiction SAMLscope's report schema is built to prevent.
Two signed gates before a case can score anything
A test that maps to the wrong requirement, or that passes without detecting a real violation, is worse than no test. Both gates require sign-off from someone other than the author.
Requirement catalog
Every requirement in the source specification is read to the end of its section and decomposed into obligations, stored in tests/{specs,coverage,predicates}.yaml.
Independent approval
Someone other than the author compares the source text against the catalog directly and signs tests/approvals/g1.yaml. The catalog cannot be edited afterward without invalidating the approval.
Test design review
Obligations are assigned to test cases with required variants and both a positive and a negative control. Each case needs a counterexample: an implementation that would pass without truly satisfying the obligation. An independent reviewer signs the design.
Local files and Hosted Runs mean different things
A tool anyone can self-host produces results anyone could hand-write. Only one trust level is treated as verified execution.
A third tier, uploading self-hosted results to become a shareable URL, was considered and rejected: forged JSON could be uploaded, which would make even genuine Hosted Run results look self-declared. Self-hosted users distributereport.html directly instead.