A Structured Workflow for Transforming Cyber Threat Intelligence into Computable Detection Patterns

Summary

Here, we present a protocol to convert indicators of compromise from cyber threat intelligence reports file paths, registry keys, and command-line indicators into validated regular expressions for security information and event management (SIEM) detection rules, using ensemble extraction with large language models (LLMs) and graph-assisted component labeling.

Abstract

Security Operations Centers (SOCs) routinely convert cyber threat intelligence (CTI) reports into operational detection content. A persistent bottleneck in this workflow is the translation of extracted indicators of compromise (IOCs) particularly file paths, registry keys, and command-line strings into deployable regular expressions (regexes) suitable for embedding in security information and event management (SIEM) correlation rules. Although prior work has improved automated indicator-of-compromise (IOC) extraction, transforming extracted strings into validated regex patterns remains largely manual, requires specialized expertise, and is prone to error. The goal of this protocol is to provide a standardized, reproducible procedure for IOC-to-regex translation. The workflow comprises five stages: (1) parsing heterogeneous CTI reports into a unified Markdown representation; (2) IOC extraction using multiple large language models (LLMs) with consensus voting; (3) rule-based normalization, categorization, and deduplication of extracted IOCs; (4) graph-assisted labeling of IOC components as keep (capture-group) or discard (non-capture-group); and (5) iterative regex generation with diagnostic validation against the original IOC strings. To assess utility, the workflow was applied to 3,156 CTI reports, and the resulting regexes were evaluated against more than 2,400 independently collected ground-truth strings from ten MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) Evaluation scenarios, yielding an average hit rate of 99.1 % and an average cross-IOC mismatch rate of 0.8 %. The protocol therefore documents a reproducible implementation for IOC-to-regex translation and explicitly delineates its current scope, operational assumptions, and known failure cases.

Introduction

Cybercrime continues to impose substantial operational and financial burdens on organizations across public and private sectors. In 2023, reported losses due to cybercrime in the United States exceeded $12.5 billion¹, highlighting the scale and persistence of malicious activity. Within this landscape, Security Operations Centers (SOCs) serve as the primary operational units responsible for detecting, analyzing, and responding to threats in real time.
Detection logic in many SOC workflows is implemented through rule-based mechanisms within Security Information and Event Management (SIEM) platforms, which are widely used because they are in....

Protocol

Use the following five-stage workflow to transform a CTI report into validated regex patterns with traceable intermediate outputs (see Figure 1 for an overview).

1. System setup

1. Install prerequisites.
   1. Install Python 3.8 or later, all Python dependencies listed in requirements.txt, and a Neo4j graph database.
      1. Confirm access to one or more application programming interfaces (APIs) for the chosen large language models and verify that the Neo4j service is running and reachable from the local machine.
   2. Confirm that the Materials Table i....

Results

This section presents representative outcomes produced by the IOC-to-regex protocol and summarizes the reference evaluation used to assess its operational applicability. The reference evaluation processed 3,156 CTI reports associated with MITRE ATT&CK techniques, analyzed more than 230,000 sentences, extracted more than 63,000 IOC candidates, and evaluated generated regexes against more than 2,400 independently collected ground-truth strings from ten MITRE ATT&CK Evaluation scenarios. These ground-truth strings a.......

Discussion

Translating unstructured CTI reports into executable detection logic remains a time-consuming and error-prone task in operational security workflows. While prior efforts have explored automation at the level of IOC extraction or high-level rule generation, practitioners still face substantial challenges in converting extracted IOC strings into regexes that are structurally correct, semantically precise, and suitable for downstream SIEM use. The protocol presented here addresses that gap through a staged workflow in which.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Computer (CPU)	—	≥ 4 cores recommended	No GPU required
LangChain	LangChain	≥ 0.1.x	LLM orchestration framework
LLM (IOC extraction, single-model)	OpenAI	gpt-5.1	Used for IOC extraction (Stage 2) when ensemble voting is disabled. temperature = 0.0; max_workers = 5. Accessed: 2025-12-15.
LLM (Regex generation)	OpenAI	gpt-5.1	Used for regex generation (Stage 5). temperature = 0.3 before downstream validation. Accessed: 2025-12-15.
LLM (Scalability characterization)	OpenAI	gpt-5.1	Used for the 6,000-IOC scalability run reported in Representative Results. Accessed: 2025-12-15.
Memory (RAM)	—	≥ 16 GB recommended	Required for document processing
Neo4j	Neo4j, Inc.	≥ 5.x	Graph database for IOC normalization
Neo4j Python Driver	Neo4j, Inc.	≥ 5.x	Python interface to Neo4j
Operating System	Microsoft / Apple / Linux	Windows, macOS, or Linux	Cross-platform support
PDF parsing — primary backend	Microsoft	MarkItDown ≥ 0.0.x	Stage 1 backend; converts PDF/DOCX/HTML/TXT inputs to Markdown. Parsed output chunked at 4,000 characters before LLM processing. Accessed: 2025-12-15. https://github.com/microsoft/markitdown
Pipeline configuration (Stage 2 — IOC extraction)	—	Reference defaults	Single-LLM mode: temperature = 0.0, max_workers = 5. Ensemble-voting mode defaults: repeats = 1 per configured model, min_votes = 2.
Pipeline configuration (Stage 5 — regex generation)	—	Reference defaults	Generation temperature = 0.3. Validation: overgen_random_tests = 5 deterministic negative samples per IOC. Iteration bounds: max_iterations = 10, debug_loop_cap = 5, discard_validation_cap = 5.
Python	Python Software Foundation	≥ 3.8	Required runtime environment
Regex Engine	Python Standard Library	re module	Used for regex validation and testing
Streamlit	Streamlit Inc.	≥ 1.25	Web-based user interface
Reference implementation source code	Authors / GitHub | GitHub repository		Source code for the Streamlit interface, LangChain pipeline, Neo4j-assisted normalization, regex generation, validation utilities, and example configuration files. Available at https://github.com/SOCautomatic/cti-ioc-regex-pipeline. Accessed: June 11, 2026.