Test fixtures and generated sources
This note records the fixture policy for tests that exercise STIG model source, Gradle TestKit projects, report assets, and generated code.
Keep source fixtures as files
Do not embed substantial Kotlin, Gradle, JavaScript, JSON, SARIF, or other source artifacts as string literals in test classes. If a test needs source-like input, put that input in a real file under the appropriate source set:
src/test/...for files used only by tests in the same module.src/testFixtures/...for files shared by tests in multiple modules.src/testFixtures/resources/...for fixture source trees that tests copy into temporary projects or load into an IDE fixture.
Inline strings are acceptable for small, focused unit tests where the string itself is the assertion subject. They are not acceptable for reusable model fixtures, representative Kotlin safety models, Gradle project templates, viewer assets, or multi-file examples.
Use engineering meaning in model identifiers
Apply the reference naming guidance and terminology policy to Kotlin declaration identifiers and their references for every model entity kind, including test fixtures and examples. Names should describe the represented engineering role, responsibility, or behavior. For example, a Failure can be lossOfSignal or unintendedDisconnection, an interface point outputSignal, and its element signalController. A fixture's creation or editing workflow does not supply the modeled entity's engineering meaning.
The Kotlin declaration identifier and a human-facing name property are distinct. Keep each meaningful in its own role: lossOfSignal can have name = "Loss of signal". When intentionally renaming a fixture declaration, update its resolved references, expected canonical entity IDs, source-diff assertions, and rendered-control expectations together. Changing an identifier does not require changing its engineering name value.
Preserve deliberately bad names in naming-rule cases and exact spellings needed by focused symbol-resolution or identity regressions. Keep the intended violation or spelling-sensitive condition explicit so normal example cleanup does not remove the test input. This is authoring guidance; engineering meaning is assessed when writing and reviewing the model.
Shared rule fixtures
Rule-checking fixtures belong next to the rules in stig-core:
Downstream tests consume those fixtures:
stig-gradle-plugin/src/functionalTestcopies the physical source tree into a temporary TestKit project and checksstigCheckModelSARIF.stig-idea-plugin/src/testloads the same physical source files into IntelliJ's light test fixture and checks PSI/highlighting/quick-fix integration.stig-core/src/testvalidates that fixture metadata references real rules, real files, and real source-line markers.
The shared expectation catalog may be Kotlin code, but it contains metadata: paths, rule and entity IDs, input partitions/cardinalities, expected results, and quick-fix titles. It must not contain the fixture model source itself.
Automatic-rule case matrix
AutomaticRuleCases in the core test fixtures adds an adapter-neutral case contract. Each case names the rule, checked subject and source declaration, its input partitions, and hand-authored expected results. Results preserve severity, kind, related entity IDs and multiplicity. Expectations must not be calculated by executing the rule under test. A valid case, an invalid case, and a not-applicable/unknown-input case are different outcomes; an empty result from a rule that was never invoked is not a passing test.
Cases can also pin exact problem messages, remediations, and quick-fix titles. Both adapters check these against the same authored expectations, including fractional severity text and bracketed identities in relation lists. The severity-value cases cover all named classes and their Severity(Float) equivalents, tolerance boundaries, out-of-range fractions, NaN, and infinities; IDEA must preserve the same Severity.value Float inputs as the compiled model. Aliases and qualified references are resolved by symbol identity, including Kotlin and Java Float constants. Separate lookalike fixtures ensure that matching spellings alone cannot select a library constant.
The physical sources under model-rule-fixtures/automatic/ are compiled as core test fixtures and copied unchanged into both integration adapters. Scalar and ownership probes check the actual loaded nullable values and container relations as well as collection cardinalities. A case labelled "blank", "unclassified", or "inherited owner" must therefore really contain that input in both the compiled and PSI models.
Entry point | Contract checked |
|---|---|
| Compiled model, actual input cardinalities, invocation, and exact rule results for every case |
| Real |
| PSI model and input cardinalities, the same expected rule results, generated inspection descriptors, source anchors, and editor highlighting |
IDEA checks exact diagnostic counts and PSI ranges before the editor combines overlapping highlights. Findings anchor on property or class names, avoiding unrelated constructor documentation in the hover. Class-backed subjects still name their expected superclass constructor to validate the fixture declaration, independently of the name-based anchor. Batch SARIF parity compares complete findings with their multiplicities: several violations may share a rule and subject, or even the same message. A batch fixture runs the whole matrix in one TestKit project; do not start one Gradle build per individual rule case. Existing cross-module loading and editor-edit tests remain separate: a single-module rule matrix cannot prove direct-reference discovery, context-only ownership, or cache invalidation.
Declaration layout and naming cases set RuleCaseSubject.sourceDeclaration = true and identify a physical Kotlin declaration. Such a subject need not exist in the model registry; do not fabricate a MetamodelEntity.id for it. Core, batch, and IDEA feed the same declaration index from the physical fixture sources. Dependency sources can resolve inherited types without becoming checked subjects. Keep tests for classes with no model instances, exact identifier highlights, per-viewpoint module reporting, and source-only edits that invalidate checks without recompilation.
AutomaticRuleCaseCatalogTest enforces applicable valid/invalid pairs and the reviewed minimum input partitions for every executable DefaultRules registration. The coverage contracts must match that entire inventory; the retained shared-coverage-backlog.txt must remain empty. Adding an automatic rule without its shared cases fails this gate. Semantic reviews and Gradle task-local configuration checks are separate inventories, not automatic core rules.
All automatic core registrations have shared scenario coverage. This is not a claim of exhaustive JVM branch coverage. The matrix covers analysis bypass, outgoing/incoming relations, failure-to-interface consistency, ASIL/severity propagation, parent-requirement derivation/allocation consistency, and required-field/containment rules across CREATE. The presence cases distinguish omitted/null/empty/whitespace text, independent name/description presence, all ASIL values, both multiplicities, and required lists with zero/one/multiple entries. Containment cases cover nested and explicit ownership, absent and wrong-type containers, multiple declared children, and logical viewpoints independent of the declaration package. Reviewed required partitions are maintained separately from the case rows so deleting a scenario cannot silently remove its coverage obligation.
Hierarchy and allocation cases include missing/empty/multiple specifications, wrong subjects, nested Designs, duplicate references, all-invalid and mixed lists, and a covering requirement in first and last position. Interface membership cases distinguish whole-service ports, selected members, nested definitions, shared topics, unresolved members and foreign endpoints. Failure cases cover classification thresholds, multiplicity, self/duplicate/cyclic independent references, complete-name description matching and duplicate display names with distinct entity IDs. Requirement and Test cases include normative modals and legacy spellings, forbidden-phrase boundaries and overlaps, Given/When/Then ordering, and independent combinations of traceability fields.
An inventory rule that rejects every instance of a diagnostic placeholder, such as UnresolvedInterfaceMember, has its valid cases on resolved members and empty member inventories. These still require the real rule invocation; not-applicable cases on unknown classification or ownership do not substitute for applicable valid/invalid pairs.
Maintaining and deepening coverage
For each new or changed executable rule, review its implementation and the shared helpers it calls, then supply:
An applicable valid case and an applicable violation with exact findings. Not-applicable cases do not replace either one.
Both sides of each meaningful conditional, every enum alternative, nullable/absent inputs, and values below, equal to, and above classification or numeric thresholds. For compound predicates, vary each independent condition so another condition cannot mask the branch under test.
Zero, one, and multiple entries for every relevant collection, including nested collections. For universal checks, include all-valid, all-invalid, and mixed lists; for existential checks, include no sufficient entry and a sufficient entry in both first and last position. Verify the actual fixture cardinalities.
Dispatch and ownership coverage where relevant: no subjects, one/multiple containers, complete and incomplete required context, and findings restricted to the checked module. Loading failures must remain loading failures, not empty successful rule results.
The same cases through core, batch, and IDEA. Keep adapter-specific checks for source locations, quick fixes, configuration-cache reuse, and live edits in addition to the shared semantic assertions.
Registration coverage does not close every defensive or loading branch. AdditionalRuleBranchCases adds the source-representable gaps found by the implementation audit: omitted Design edges, incomplete ownership chains, unknown and invalid incoming severities, unknown allocation owners, non-finite propagation inputs and diagnostic formatting. Its required partitions extend the existing rule contracts, so removing a scenario does not remove its coverage obligation. InterfaceAnalysisCoverageRuleCases covers interface-tree paths, distinct branches and ports, selected members, shared definitions, missing/empty/disabled analyses and ownership boundaries.
RuleBranchBoundaryTest covers adapter boundaries separately: incomplete and context-only inventories, an element checked against context requirements, dispatch switches, custom interface/element implementations, missing source binding metadata, anonymous runtime declarations, cyclic containment and semantic-review evidence prerequisites. The IDEA materializer separately verifies that an unsupported anonymous object expression remains a loading diagnostic. Keep the multi-module loading and cache-invalidation tests alongside these cases. A core-only boundary test is not a shared adapter scenario.
Use :stig-core:koverXmlReport and :stig-core:koverHtmlReport to inspect uncovered implementation branches alongside the matrix. A high aggregate percentage does not prove per-rule coverage. Review compiler-generated or unreachable branches explicitly; if the Kotlin API forbids a malformed model, test the parser/loading diagnostic rather than fabricating a "valid" rule case that never invokes its predicate. Per-rule branch thresholds can become mandatory as these gaps are resolved; do not set a blanket threshold that is satisfied only by unrelated tests.
Review residual JVM branches against the Kotlin contract before adding artificial cases. For example, a safe call on Analysis.element after a non-null Analysis cannot take its null branch: the constructor requires an element. Other examples are nulls after a filtered non-null value or an ownership lookup that already established the same owner, empty fallback paths after a nonempty check, and numeric parsing of a digit-only regex capture. Unbound references fail when their canonical ID is read, before later quick-fix fallbacks can be reached. The viewpoint allow-list contains every enum value. Such paths do not represent an additional valid model scenario; retain the constructor, binding, dispatch and malformed-input tests that establish those preconditions. Do not claim literal 100% JVM branch coverage from registration or scenario counts.
Task-owned temporary directories
The shared JVM module convention sets java.io.tmpdir for each module's Gradle Test tasks to that task's build/tmp/<task>/jvm directory and recreates the directory before execution. Consequently, Files.createTempDirectory(...) in unit, functional, and integration tests is removed by the next task run or by clean instead of accumulating in the operating system's /tmp.
Do not rely on File.deleteOnExit() for a non-empty fixture tree. The JVM attempts to delete only the registered path; it does not recursively remove files that a TestKit build or analyzer created below it. A test that needs cleanup before its task finishes should delete its fixture recursively. The task-owned temporary root provides the repository-wide fallback.
The included build-logic build owns the convention plugin's TestKit tests. Its archive-relocation fixture injects the compiled plugin with withPluginClasspath() and runs offline. It checks timestamp-independent archive bytes and Java compilation cache reuse after moving the project; it does not configure STIG or a native analyzer.
Generated artifacts
Generated files are appropriate when generation is the product behavior being tested or delivered. Examples include:
lifecycle/scaffolding tasks such as
stigInitElement;report/viewer assets generated for publication;
generated inspection metadata/classes where the generator is itself part of the build contract.
Avoid generating .kt or .kts as a shortcut for test setup. Kotlin/Gradle source fixtures should normally be checked in as physical fixture files so they can be reviewed, searched, edited, and reused directly.