Analyzer failure layers

The stig source-code-analysis pipeline has three distinct layers where a check can stop the build or surface a problem. They're not interchangeable — each layer answers a different question. Putting a check in the wrong layer either masks a real bug or makes the failure unrecoverable when it should have been a warning.

The three layers

1. Analyzer (C++)

Where: stig-code-analyzer — Pipeline.cpp, EmitAnalysisResults.cpp, main.cpp.

Question it answers: Can I even produce a sane JSON?

Hard-fails with std::runtime_error/ process exit 1. No JSON is written. Use for conditions that make the output meaningless or that indicate a toolchain / input bug the analyzer would only be guessing around:

Unreadable bitcode, missing entry points.
Solver/emit incompatibilities (--emit-feasibility without --solver=ide-xtaint).
A --repo flag that names a non-existent path.
Internal invariants the analyzer relies on (e.g. an edge endpoint referencing a segment id we never created).

Not the place for "this looks weird" signals — the analyzer's job is to produce data, not judge it. If the analyzer can finish, it should.

2. Typed schema and semantic validator (Kotlin)

Where: stig-engine/.../diagram/AnalysisResult.kt — AnalysisResult.validate(), called by active P7 diagram generation through AnalysisResultJson.parseAndValidate(...).

Question it answers: Is this JSON internally consistent, and can every consumer trust the analyzer's own evidence model?

Throws IllegalStateException at JSON parse time. The active stigAnalysisP7GenerateDiagrams task converts that into a CRITICAL postcondition and calls StigProblemReporter.fail(), so the task stops unconditionally. The configured failAt threshold is not consulted because there is nothing useful the renderer can do with internally contradictory analyzer evidence.

Use for shape/cross-reference invariants the type system can't catch:

Type-prefix conformance (instruction-…, segment-…, dagSegmentEdge-…, etc.).
Every feasibleFacts[].factId, segment instruction ID, and DAG-edge endpoint points at a real entry of the right type and scope.
sourceRepo (when non-empty) references a declared repo in repos[].
Feasibility presence is consistent across sinks when pathFeasibility != NO_CHECK.
isSink == true ⇔ matching entry in top-level sinks[].
For every configured sink, sink.leaked is equivalent to valid source-to-sink evidence in its selected schema-v7 components (or the per-sink DAG edges of an archived v5/v6 artifact).
When Z3 feasibility is enabled, every feasible source-to-sink segment path implies sink.leaked = true; the reverse is not required because Z3 is post-solve evidence and may leave structural leaks infeasible or unknown.
Positive leak verdicts have matching control-flow endpoints in the same run.

Not the place for domain rules (asil expectations, leak-count policies). Those are layer 3. See Analyzer output validation for the complete current validator list.

3. Postcondition rules (Kotlin)

Where:

stig-gradle-plugin/.../analysis/StigAnalysisP6RunAnalyzerTask — validateAnalysisJsonContent(), for immediate per-run analyzer checks.
stig-gradle-plugin/.../analysis/StigAnalysisP7GenerateDiagramsTask, for diagram/report-generation completeness and typed validation failures.

Question it answers: Does this JSON match what we expected operationally for this pass?

Emits ProblemSeverity-tagged problems. Domain problems at HIGH or lower use the threshold-controlled reporting path: problems below the resolved failAt threshold are recorded and processing continues, while problems at or above it throw. Unusable or contradictory generated evidence is different: it is a CRITICAL hard failure and stops through StigProblemReporter.fail() independently of failAt.

Use for domain rules whose verdict depends on the intended pass semantics, not on the JSON's structural integrity:

analysisKind=data-and-control-flow ⇒ a per-run zero-reached verdict is a diagnostic (MODERATE) because precise per-sink runs may legitimately be unreached. An aggregate "nothing reached anywhere" check should be modeled separately.
analysisKind=cascading-faults ⇒ expects 0 reached sinks. Any reached ⇒ HIGH (a low-integrity source taints a higher-ASIL sink).
Analyzer JSON semantic validation failures surfaced during P7 diagram generation ⇒ CRITICAL.
Future: feasibility distribution policies, fact-count gating, etc.

Not the place for shape checks — those belong in the validator.

Picking the right layer

When adding a new check, decide as follows:

Can the analyzer detect this from the inputs it already has, before writing JSON? → Layer 1.
Is this a property of the JSON itself (refers, prefixes, presence consistency) that any consumer would need to assume? → Layer 2.
Is this an operational expectation specific to one pass kind, where the verdict could legitimately be "warn but proceed" vs "fail"? → Layer 3.

If you find yourself wanting to put the same check in two layers, that's a sign the check is misclassified — pick the highest layer that catches it.

23 October 2026