Testing overview

STIG organizes verification into three levels, presented top-down from the complete software to individual components:

Verification tests / Functional tests / qualification tests.
Integration tests.
Component tests / Unit tests.

These correspond, in this order, to the ISO 26262-6 objectives of testing of the embedded software, software integration and verification, and software unit verification. STIG is a development tool, so its consumer-facing functional tests apply the complete-software testing objective within STIG's software scope; they do not replace testing of the user's embedded software. The mapping alone is not a compliance claim. Executed results and traceability provide the verification evidence.

A test kind identifies a runner, implementation language and test subject. Several kinds can belong to one verification level. In particular, PhASAR CTest, STIG native C++ CTest and Kotlin analyzer integration tests are three kinds used for native-analysis verification, not three ISO levels. Neither CTest nor the name unittests proves that every case isolates one software unit: PhASAR also has cases combining its solver, alias analysis and call graph.

Verification levels

Verification level	Test kind and scope	Test and fixture paths, relative to the STIG checkout	Focused task	Test count
Verification tests / Functional tests / qualification tests	Gradle TestKit functional tests: consumer builds check the public plugin, model import, report generation, pipeline failure handling and generated HTML/JSON/SARIF.	`stig-gradle-plugin/src/functionalTest/`	`./gradlew :stig-gradle-plugin:functionalTest`	16 cases in the 2 focused model/SARIF and Dokka suites; full inventory not recounted
Verification tests / Functional tests / qualification tests	Playwright viewer tests: browser rendering and interaction with report viewers.	`stig-gradle-plugin/js-vendor/test/`	`./gradlew :stig-gradle-plugin:stigViewerRenderTest`	41 Chromium cases
Integration tests	Kotlin analyzer integration: compile physical C++ fixtures, invoke the production native analyzer, and check schema, source/sink semantics, provenance, feasibility, metrics and installation. `TaintMatrixTest` produces the percentage table.	Tests: `stig-gradle-plugin/src/integrationTest/kotlin/com/stigway/gradle/integration/`; fixtures: `stig-gradle-plugin/src/integrationTest/resources/cases/`	`./gradlew :stig-gradle-plugin:integrationTest`	1,231 cases in 23 suites: 350 taint-matrix, 770 semantic-matrix and 111 other cases
Component tests / Unit tests	PhASAR GoogleTest/CTest: solvers, alias analysis, call graphs, virtual tables and path construction, including internal integration cases and complete solves on small C/C++ fixtures.	Tests: `../phasar/unittests/`; fixtures: `../phasar/test/llvm_test_code/`	Through `:stig-code-analyzer:stigBuildAnalyzer`	1,635 GoogleTest cases in 64 CTest executables
Component tests / Unit tests	STIG native C++ GoogleTest/CTest: selectors, evidence handling, sink reachability, serialization and monitoring.	Tests: `stig-code-analyzer/src/test/cpp/`; fixtures: `stig-code-analyzer/src/test/resources/native/`	Through `:stig-code-analyzer:stigBuildAnalyzer`	50 individually discovered CTest cases
Component tests / Unit tests	Kotlin module tests: wire models, rules, diagrams, LLVM helpers, DSL/tasks and IDEA inspections.	`<module>/src/test/`; shared fixtures: `stig-core/src/testFixtures/`	`./gradlew test` or `./gradlew :<module>:test`	1,225 core, 524 engine, 685 Gradle-plugin, 51 IDEA-plugin, and 9 Codex-plugin cases; other modules not recounted

Native-analysis counts are the local inventory from September 16, 2026; the JVM model/report and browser counts are from September 18, 2026. Both include uncommitted changes. The focused TestKit count covers StigCheckModelSarifFunctionalTest and StigDokkaModelFunctionalTest, not the entire functional-test source set. PhASAR's total includes 2 skipped and 21 disabled cases; its CTest entries count executables, while STIG native CTest counts individual cases. The Kotlin integration total includes the expected-failure contracts listed below. An unrecorded count does not mean zero tests.

Automatic model-rule validation uses 1,028 distinct scenarios covering all 94 executable rules. The shared catalog and physical Kotlin fixtures run through core rule evaluation, the real stigCheckModel batch task, and IntelliJ IDEA PSI inspections. Scenarios exercise valid and invalid outcomes, nullable values, enum alternatives, ownership, and zero/one/many relationship cardinalities where applicable, including bypassed Analysis levels, all Severity.S0 through Severity.S10 constants and their Float constructors, tolerance boundaries, and exact diagnostic text. This is extensive partition-based validation, not a claim of exhaustive JVM branch coverage. The 1,225 core tests include these scenarios plus catalog, identity, and other model tests; executing the catalog through three routes does not multiply the number of distinct scenarios. IDEA and TestKit each execute the catalog inside a matrix test, so their JUnit case counts do not individually enumerate its 1,028 scenarios.

The integration task depends on :stig-code-analyzer:stigBuildAnalyzer. The native build has a toolchain-availability guard; integration tests have an executable-availability guard. A skipped task is not a passing verification run.

Test frameworks by level

The IDEA plugin does not use Gradle TestKit. Its tests run in IntelliJ Platform light fixtures; TestKit is used by the Gradle plugin's functional-test source set.

Verification tests / functional tests / qualification tests

Framework or tool	Used for	One-line example
Gradle TestKit (`GradleRunner`)	Running STIG in physical temporary consumer builds through its public Gradle plugin API.	Apply STIG to a fixture project, run `stigGenerateHtml`, and assert that the expected report is produced.
Kotest `StringSpec` on JUnit Platform	Defining and running functional specifications and assertions.	Run a consumer build and assert that plugin application succeeds without leaking implementation dependencies.
Jsoup, Jackson, and shared `stig-core` test fixtures	Checking generated HTML, JSON, and SARIF against realistic STIG models.	Parse generated HTML with Jsoup and verify the model tree and ASIL badges.

Integration tests

Framework or tool	Used for	One-line example
Kotest `FunSpec` on JUnit Platform	Defining analyzer integration scenarios and assertions.	Run a focused taint case and assert the expected source-to-sink result.
Production `stig-code-analyzer`, `FixtureBitcodeCompiler`, and pinned Clang/LLVM	Compiling physical C++ fixtures to LLVM bitcode and invoking the real native analyzer.	Compile a C++ source/sink snippet, analyze it, and assert the expected data-flow DAG.
Jackson and schema validation	Reading and validating analyzer JSON output before semantic assertions.	Validate the analyzer result against its schema and then check that unrelated subobjects do not propagate taint.

The QM/ASIL B C++ fixtures specify independent source-to-sink dependencies: a selected QM source must reach its intended output and leave the separate ASIL B output clean. The native solver follows configured source/sink markers; function names do not supply ASIL annotations. ActivityCallbackSourcesTest additionally checks both source_qm → sink_qm and source_asilb → sink_asilb, with no cross-object or clean-field flows, source-provenance checks and a 30-second subprocess limit. The matching PhASAR callback cases cover strong/weak updates with a 60-second CTest limit.

These tests stop at native analyzer JSON. They do not verify Gradle's ASIL-based source selection or the full stigGenerateHtml chain. Existing TestKit tests check real model reports (StigExplicitPackageHtmlFunctionalTest, StigModelLibraryFunctionalTest), prepared producer outputs (AnalysisPartialResultsTest), or a fake analyzer (StigAnalyzerExecutionCoordinatorFunctionalTest). Real C++ expected-flow coverage through P5 source selection, P6, P7/model JSON and final HTML is planned in STIG workplan issue 63.

Component tests / unit tests

Framework or tool	Used for	One-line example
GoogleTest and CTest	Testing PhASAR and STIG native C++ behavior; each project registers its own tests.	Analyze the callback fixture and check exact positive/negative sinks, or test evidence normalization on a small graph.
Kotest `FunSpec`/`StringSpec`, Kotest assertions, MockK, and `kotlinx-coroutines-test` on JUnit Platform	Testing module-local Kotlin logic with controlled dependencies and coroutine execution.	Pass analysis JSON to diagram logic and assert the resulting graph nodes and edges.
Gradle `ProjectBuilder`	Testing Gradle task and DSL behavior without launching a separate consumer build.	Create a task, supply fixture inputs, invoke its action, and assert its output files.
IntelliJ Platform Test Framework (`BasePlatformTestCase`), light PSI fixtures, K2 Analysis API, and MockK	Testing IDEA PSI materialization, inspections, highlights, and quick fixes.	Load `MissingName.analysis.kt` as PSI and assert that the unnamed `Failure` receives the expected highlight and quick fix.
JUnit Vintage on JUnit Platform	Discovering the JUnit 3/4-style tests required by IntelliJ's platform fixtures.	Run a `BasePlatformTestCase` inspection test through the same Gradle `test` task as the other JVM unit tests.

IDEA archive and compatibility checks are separate from its unit-test framework. The root ideaPluginVerification gate also runs archive, plugin-structure, project-configuration, and JetBrains Plugin Verifier checks.

Fixtures and other test families

stig-core/src/testFixtures contains the shared rule expectation catalog and physical Kotlin model projects. It is support data, not an independently executed suite. Gradle-plugin functional tests and IDEA-plugin unit tests consume it as described in Test fixtures and generated sources.
stig-gradle-plugin/src/integrationTest/resources contains the small C++ programs and model inputs compiled or loaded by analyzer integration tests. They are test inputs, not production analyzer sources.
stig-gradle-plugin/js-vendor/test contains Playwright viewer tests. Run them with ./gradlew :stig-gradle-plugin:stigViewerRenderTest; the task requires the Chromium installation provided by the repository setup.
stig-code-analyzer/src/test/cpp/sanity_test.cpp contains four GoogleTest selector checks. CMake includes it in stig-code-analyzer-native-tests, alongside the evidence, reachability and runtime-monitor tests. The STIG source-checkout build passes --run-analyzer-tests, enables BUILD_TESTING, and executes the discovered CTest cases.

Known failures and unexecuted cases — September 16, 2026

The latest complete three-kind analyzer run has zero unexpected failures. This does not mean zero known defects. KnownMatrixFailures.kt records 98 expected-failure cells across 30 distinct scenarios. They execute and count as successful Kotlin assertions when the observed failure matches its declared mode and sink set; an unexpected pass or changed failure shape requires review and fails the test. None of these 98 cells is a tolerated timeout/no-report case in this snapshot. No new expected failures were added for the September recovery changes.

The five columns below cover all observed expected-failure cells. — means the scenario passes in that combination. IDE is ide-xtaint; FS is ifds-fieldsens-taint; IFDS is ifds-taint; UF is union-find; CFL is cflanders. IDE/UF is the production default. FN means a missing expected flow; FP means an unexpected flow to a clean sink.

Scenario (`resources/cases/<name>.cpp`)	IDE/UF	IDE/CFL	FS/UF	FS/CFL	IFDS/UF
`arithmetic_merge`	FN	FN	—	—	—
`member_function_pointer`	FN	FN	FN	FN	FN
`fptr_multi_targets`	FN	FN	FN	FN	FN
`virtual_operator_apply`	FP	FP	FP	FP	FP
`virtual_inheritance_converter`	FP	FP	FP	FP	FP
`constant_branch_path_disambiguation`	FP	FP	FP	FP	FP
`validator_returns_comparison`	—	—	FN	FN	FN
`validator_returns_passthrough`	—	—	FN	FN	FN
`validator_writes_via_pointer`	—	—	FN	FN	FN
`nested_class_via_helper_function`	—	FP	FN	FN	—
`activity_local_class_two_struct_members`	—	—	FN	FN	FP
`activity_local_class_via_local_buffers`	—	—	FN	FN	FP
`activity_member_struct_pointer_indirection`	—	—	FN	FN	FP
`activity_member_struct_uint_widths`	—	—	FN	FN	FP
`activity_member_struct_array_element`	—	—	FN	FN	FP
`activity_member_struct_external_kernel`	—	FP	FN	FN	FP
`activity_three_sibling_fields`	—	—	FN	FN	FP
`activity_cross_method_member`	—	—	FN	FN	FP
`activity_cross_method_setter_getter`	—	—	FN	FN	FN
`activity_two_boundary_global_source`	—	FP	FN	FN	FP
`class_methods_modify_members`	—	—	FN	FN	FN
`activity_kernel_param_passing`	—	—	FP	FP	FP
`activity_member_struct_external_kernel_offset`	—	FP	FN	FN	FN
`struct_copy_memcpy`	—	—	FN	FN	FN
`offset0_alias_unification`	—	—	FP	FP	FP
`std_function_sret_callback`	—	FN	FN	FN	FN
`harness_sinkleaf_template`	—	—	FP	FP	FP
`value_combine_or`	—	FP	—	—	FP
`helper_ctx_sensitive`	—	—	—	FP	—
`harness_sibling_fields_in_struct`	—	—	—	—	FP
Expected-failure cells	6	12	26	27	27

The FN sink is sink_qm, except the three validator_* cases (sink_qm_via_bool) and fptr_multi_targets (sink_propagate_a). The FP sink is sink_asilb, except the three virtual/constant-branch cases (sink_drop_b), activity_three_sibling_fields (sink_asilb1 and sink_asilb2), and harness_sinkleaf_template (_ZN7harness8SinkLeafIiNS_4tags11asilb_fieldEEEvPKv). The IFDS/UF offset0_alias_unification case additionally reaches sink_drop. Exact expectations live in KnownMatrixFailures.kt; observed outcomes live in the ranking report.

Production relevance and the percentage table

The six production failures can affect an activity if its analyzed LLVM contains the corresponding pattern:

Failure	Potential effect
`arithmetic_merge`	Missed propagation from raw source-call return values through arithmetic. The retained diagnosis concerns the source fact's access path; loaded-field arithmetic is a different tested shape.
`member_function_pointer`, `fptr_multi_targets`	Missed flows through C++ pointers to member functions, including a shared dispatcher with multiple possible methods.
`virtual_operator_apply`, `virtual_inheritance_converter`	Extra flows across virtual calls through a shared helper. A virtual-inheritance fixture is coverage, not evidence that virtual inheritance caused a particular activity failure.
`constant_branch_path_disambiguation`	A flow reported through an infeasible branch, because structural analysis does not eliminate the branch.

These are correctness limitations, not demonstrated causes of the unresolved long-running activity solve. The 24 additional scenarios fail only in other combinations in this snapshot; their current production-default cells pass. Do not switch to a less precise solver to make an activity finish.

For IDE/UF, the ranking reports leak accuracy 96.2% (TP 76, FN 3, TN 75, FP 3), source-provenance accuracy 95.5% (TP 75, FN 4, TN 75, FP 3), Z3 feasibility accuracy 90.8% (TP 69, FN 1, TN 0, FP 6), and CFG reachability 100.0% (TP 241, FN 0, TN 1, FP 0). Each percentage counts observations on its own axis, not passed Kotlin cases or completed activity sources. In particular, an inconclusive Z3 result can lower that accuracy without being an unexpected Kotlin test failure; the aggregate provenance axis is not itself the per-cell build gate.

PhASAR skipped and disabled cases

The two runtime skips are IDETaintAnalysisTest.XTaint19 (marked as duplicating XTaint17) and PathTracingTest.Handle_Inter_10 (Need globals support). The latter leaves a gap in that path-tracing test; it does not establish that every global-variable flow is unsupported.

The count covers the configured LLVM 18/Z3 build with PHASAR_USE_SVF=OFF and PHASAR_BUILD_OPENSSL_TS_UNITTESTS=OFF. Three CTest executables have no runtime GoogleTest cases (InterMonoFullConstantPropagationTest, InterMonoTaintAnalysisTest, LLVMIRToSrcTest); CompilationTests is a separate compile-check executable.

These 21 cases are disabled in the current compiled definitions and were not executed:

Suite	Disabled case names, including their actual `DISABLED_` prefix
`LLVMBasedICFGExportTest`	`DISABLED_ExportICFGSource01`, `DISABLED_ExportICFGSource02`, `DISABLED_ExportICFGSource03`, `DISABLED_ExportCFG01`
`LLVMBasedICFGGlobCtorDtorTest`	`DISABLED_LCATest4`
`IFDSConstAnalysisTest`	`DISABLED_HandlePointerTest_03`, `DISABLED_HandleGlobalTest_04`, `DISABLED_HandleCallParamTest_04`, `DISABLED_HandleCallParamTest_05`, `DISABLED_HandleArrayTest_04`, `DISABLED_HandleArrayTest_07`, `DISABLED_HandleSTLArrayTest_04`, `DISABLED_HandleCStringTest_02`
`IDETaintAnalysisTest`	`DISABLED_XTaint07`, `DISABLED_XTaint08`, `DISABLED_XTaint10`, `DISABLED_XTaint11`, `DISABLED_XTaint13`, `DISABLED_XTaint14`, `DISABLED_XTaint15`
`IDETSAnalysisFileIOTest`	`DISABLED_HandleTypeState_06`

The extended-taint definitions document unsupported variadic-parameter configuration (XTaint07/08), existing sanitizer false positives (XTaint13/14), and source call-site discovery without call-graph information (XTaint15). These are relevant coverage gaps if those exact mechanisms occur in an activity. Disabled cases are not current passing evidence, and their failure on the current candidate has not been established by this run. The separate constant-analysis and file-typestate suites are not STIG's production IDE-XTaint analysis.

Regression evidence for termination repairs

Keep three claims separate: a fixture resembles the affected C++ mechanism, it completes with the correct current result, and it has demonstrated the earlier defect. The callback/object fixture satisfies all three for an old false negative: the retained before/after run misses both expected flows before the repair and reaches exactly the two expected sinks afterward. It has not reproduced the original activity timeout.

The caller-traversal tests separately cover convergence/cycles with bounded expansion counts; explicit-graph tests cover seed identity and repeated seeds. Such component regressions help localize growth without claiming that a complete activity now terminates. The reduced original-timeout fixture and full activity completion remain pending.

Run normal regressions only against the current implementation. When practical, execute a new reproducer once with the relevant fix absent, retaining input/binary identity, failure output and a timeout or expansion bound. An isolated temporary build is sufficient; do not add permanent dual-version builds. Existing runtime counterexamples plus a focused current regression can be retained when reconstructing the old build would add little evidence. Compatibility tests and instrumentation tests do not each need an old-version failure: some deliberately pass both versions.

Aggregate task wiring

stig-gradle-plugin:check includes its unit, integration, functional, and viewer checks. The root releaseCoreVerification task builds every subproject, so it reaches those suites plus the other modules' unit tests. preReleaseCheck adds cleanup/static analysis, documentation, and IDEA-plugin verification around that core gate. quickCheck deliberately compiles only main source sets and runs no tests.

Use focused tasks while iterating and the aggregate gate required by the relevant change path before committing.

Workflow	PhASAR CTest	STIG native C++ CTest	Kotlin analyzer integration / percentage table
Path 2 submission	Required through native build in `preReleaseCheck`; a tested pinned installation may be reused.	Required through the source-checkout native build; unchanged Gradle outputs may be reused.	`preReleaseCheck` → `releaseCoreVerification` → module `build`/`check` → `integrationTest`.
Path 3 submission	Same as Path 2.	Same as Path 2.	Same as Path 2; a release marker does not replace tests.
GitLab CI	Runs when the pinned dependency is built into the toolchain image; later jobs may reuse its tested installation.	`build_and_test` → `releaseCoreVerification` → `stigBuildAnalyzer`.	`build_and_test` → `releaseCoreVerification`. This is configured wiring; remote execution of this local candidate is unverified.
First installation, PhASAR absent	`PHASAR_BUILD_UNITTESTS=ON`, fixture generation and CTest before installation.	Source-checkout builds pass `--run-analyzer-tests`. Published-JAR lazy installs omit development test sources; a PhASAR-only image build has no STIG analyzer yet.	Installation alone does not run Kotlin tests. Select `integrationTest` or a source verification build.
Last complete local run, 2026-09-16 CEST	15:17:38–15:17:42; 64 executable passes, 1,612 individual passes, 2 skips, 21 disabled.	Report completed 15:18:38; 50/50 passed.	15:18:42–15:19:12; 1,231 cases, zero unexpected failures, 98 declared expected-failure matrix cells.

Execution reports are under ~/.cache/stig/stig-code-analyzer/build/phasar/ (phasar-unit-tests.xml and Testing/Temporary/LastTest.log), ~/.cache/stig/stig-code-analyzer/build/cmake/Testing/Temporary/LastTest.log, and stig-gradle-plugin/build/test-results/integrationTest/. The percentage report is stig-gradle-plugin/build/reports/taint-matrix-ranking.txt. Count PhASAR cases from the complete log; CTest truncates captured stdout in its XML. A later filtered Kotlin run can replace the XML reports without providing a full inventory.

Commands from the STIG source checkout

Run all three native-analysis test kinds and print the percentage table, renewing the native build and integration tasks explicitly:

set -o pipefail
./gradlew :stig-code-analyzer:stigBuildAnalyzer --rerun \
  :stig-gradle-plugin:integrationTest --rerun --console=plain --info \
  2>&1 | tee /tmp/stig-analyzer-tests.log

With a local ../phasar checkout this rebuilds incrementally and reruns its CTest gate. For a pinned/prebuilt installation the build script may reuse the dependency's successful installation marker even with Gradle --rerun. If that dependency's test build tree exists, its registered cases can be run explicitly with ctest --test-dir ~/.cache/stig/stig-code-analyzer/build/phasar --output-on-failure --no-tests=error --timeout 300. CTest alone does not rebuild changed sources. A published prebuilt installation need not retain that test tree.

Other source-checkout scopes:

Command	Scope
`./gradlew test --console=plain --info`	Selects JVM unit-test tasks across modules; does not select the Kotlin analyzer integration or TestKit functional suites. Native build prerequisites can still run their CTest gates.
`./gradlew :stig-gradle-plugin:check --console=plain --info`	Gradle-plugin unit, analyzer integration, functional and viewer checks, with native prerequisites.
`./gradlew build --console=plain --info`	All module builds and their check/test dependencies, including the three analysis test kinds.
`./gradlew releaseCoreVerification --console=plain --info`	Shared CI/local aggregate: all module builds plus notices and coverage.
`./gradlew preReleaseCheck --console=plain --info`	Release verification, adding static, documentation and IDEA checks to the core aggregate. Follow the ordered cleanup workflow for a checked submission.
`./gradlew :stig-gradle-plugin:showMatrixRanking --console=plain`	Prints the retained percentage report only; executes no tests.

Gradle can reuse up-to-date/cached task results. Use the task-local --rerun for a focused fresh run; repository-wide --rerun-tasks also reruns unrelated prerequisites and can be much more expensive.

There is a current integration-test freshness gap: the task declares the analyzer's path as a system property but does not declare the executable's contents as a Gradle input. A PhASAR-only change can rebuild the binary while leaving Kotlin tests up to date; the September 15 investigation observed this reuse. Until executable-content tracking is added, explicitly rerun integrationTest after native changes. The complete September 16 result above used the forced command and is not a cached pre-change result. Merely reaching integrationTest through an aggregate is not proof that it executed against the changed binary.

Commands from a consumer such as stig-pth

There is currently no consumer task that runs all three development suites. stigSelfTest exists, but its current implementation only prints the LLVM/toolchain banner; the fuller environment/matrix checks remain a scaffold. A consumer's test/build tasks verify that consumer's code and configured checks, not STIG's full development suites.

Consumer-root command	Actual scope
`./gradlew stigSelfTest --console=plain`	Toolchain information only; no correctness tests.
`./gradlew stigSetupAnalyzer --console=plain --info`	Build/resolve the analyzer. A local STIG composite delegates to its native build/CTest task; a published install follows the first-install rules above. No Kotlin matrix.
`./gradlew :stigCheckAnalysis --console=plain --info`	Analyze the consumer's configured activities and collect evidence/reports. This is not the synthetic regression suite and does not itself prove expected source/sink agreement.

With the sibling source checkout available, run its tests directly from the consumer directory:

set -o pipefail
../stig/gradlew -p ../stig :stig-code-analyzer:stigBuildAnalyzer --rerun \
  :stig-gradle-plugin:integrationTest --rerun --console=plain --info \
  2>&1 | tee /tmp/stig-analyzer-tests-from-consumer.log

For all STIG module tests/checks, use ../stig/gradlew -p ../stig build --console=plain --info. Without that source checkout, the published plugin does not ship these development suites; a consumer-facing packaged regression runner remains follow-up work. Run shared native-cache builds sequentially when several consumer/source checkouts use the same machine. See Local multi-repository development and Runtime monitoring for build selection and diagnostic controls.

23 October 2026