SUPPLEMENTAL FILE 2: IMPLEMENTATION DETAILS

Manuscript: A Structured Workflow for Transforming Cyber Threat Intelligence
            into Computable Detection Patterns
JoVE submission: 71144

This file documents the algorithmic and implementation details that support
Stages 4 and 5 of the protocol. The protocol main text presents the user-facing
procedure; this file presents the deterministic rules and validators that the
pipeline applies internally.

================================================================================
S2.1  STAGE 4 — Neo4j-ASSISTED IOC NORMALIZATION
================================================================================

S2.1.1  Graph schema
--------------------

The reference graph defines three node labels (:Path, :Registry, :CLI) with an
identical property schema (name, name_lower) and a single directed adjacency
relationship. The graph is populated from native Windows 8, 10, 11 and Windows
Server 2012, 2016, 2019, 2022. Full schema, property types, example chains, and
data-source coverage are listed in Table 2 of the manuscript. All matching uses
name_lower for case-insensitive lookup. A process-local cache keyed on
(label, component) eliminates redundant database round-trips.

S2.1.2  Cypher queries
----------------------

The normalization algorithm issues three parameterized queries, where <Label>
is one of Path, Registry, or CLI:

(i) Node-exists check on a component:

    MATCH (n:<Label>) WHERE n.name_lower = $c RETURN n LIMIT 1

(ii) Pairwise adjacency check between two candidate components:

    MATCH (a:<Label> {name_lower:$t1}), (b:<Label> {name_lower:$t2})
    MATCH (a)-[]->(b) RETURN count(*) AS cnt

(iii) Variable-length reachability (up to seven hops) from a component:

    MATCH (s:<Label> {name_lower:$c})-[*0..7]->(t) RETURN DISTINCT t.name

S2.1.3  Keep/discard decision rule (paths and command lines)
------------------------------------------------------------

For each decomposed component c_i in order:

  1. Pre-filter to discard if the token matches a drive identifier, a
     registry root (HKLM / HKCU / HKU / HKCR / HKCC / REGISTRY / REG), a
     file-extension tail, a SID pattern, or a GUID pattern.

  2. Otherwise, run query (iii) to obtain the reachability set R(c_i).
     If R(c_i) is empty, label discard.

  3. Label keep if any of the following holds:
       (a) c_i is the first valid component;
       (b) all preceding components are discard;
       (c) c_i ∈ R(c_j) for some previously kept c_j;
       (d) there exists c_{i+k} ∈ R(c_i).
     Otherwise label discard.

  4. CLI fallback: if no non-shell component is kept, force keep on
     cmd / cmd.exe and on /c or /k.

S2.1.4  Registry key normalization
----------------------------------

Registry keys use a separate longest-contiguous-substring procedure instead
of the per-component rule above. After stripping root hives, SIDs, and
GUIDs, extract the longest span of consecutive tokens for which:

  (a) every token satisfies query (i), and
  (b) every adjacent pair satisfies query (ii).

The resulting substring defines the canonical registry path.

================================================================================
S2.2  STAGE 5 — REGEX VALIDATION, SCORING, AND ITERATION
================================================================================

S2.2.1  Regex validation rules
------------------------------

Each candidate regex produced by the LLM is subjected to four deterministic
validators before being accepted. A candidate is accepted only when all four
succeed.

  1. Match test
     Compile with re.IGNORECASE and require a full match on the IOC. If
     matching fails, a token-by-token cumulative diagnostic records the
     last-successful prefix and the first failing token; both are passed
     to the optimization prompt.

  2. Static quality checks
     Reject if the regex fails to compile, places .* or .+ inside an
     alternation group, contains alternation without parentheses, lacks
     a leading (?i) flag, or matches a degenerate pattern such as (?i).*.
     Auto-repair inserts the missing (?i) prefix and collapses duplicate
     .*.* sequences.

  3. Forbidden-token check (boundary-aware)
     For path and registry IOCs, split the regex on path separators and
     reject if any discard component appears as a whole segment. For
     command-line IOCs, apply the same segment check for embedded paths
     and additionally reject if any discard token matches under word
     boundaries (?<![a-z0-9_])TOKEN(?![a-z0-9_]). This avoids false
     positives from substring collisions (for example, the ignore token
     "c:" must not match "sc").

  4. Over-generalization test
     Generate overgen_random_tests = 5 deterministic negative samples
     seeded by CRC32(ioc_type + ioc):
       - Path samples: random drive letter and 2–6 random path segments
         with a random extension.
       - Registry samples: random root hive and 2–6 random sub-keys.
       - Command-line samples: random fake command with an optional
         switch and 2–6 arguments.
     Reject the regex if it matches any of the five samples.

S2.2.2  Component-based scoring
-------------------------------

Structurally valid candidates are ranked by

    Score = n_cg − n_wc

The regex is first normalized by stripping (?i), leading/trailing .*,
optional groups (...)? , and character classes, then tokenized.

  - n_cg counts keep components with at least one token present in the
    normalized regex.
  - n_wc counts discard components present, plus regex tokens that map
    to no component at all.

Candidates with n_cg = 0 are rejected.

S2.2.3  Iteration control
-------------------------

The outer generation loop is capped at max_iterations = 10. Within each
outer iteration:

  - failed match tests enter a match-debug loop capped at
    debug_loop_cap = 5 optimization calls;
  - successful matches that still contain discard components enter a
    discard-validation loop capped at discard_validation_cap = 5.

If the discard loop breaks matching or exhausts its cap, the outer loop
restarts generation. When all outer iterations are exhausted without a
fully compliant regex, the pipeline falls back to the highest-scoring
regex that at least matched the IOC. This fallback is recorded in
telemetry as used_fallback = True.

================================================================================
END OF FILE
================================================================================