A Large Language Model-Assisted Protocol for Metadata Classification and Governance in Electric Power Information Systems

Zusammenfassung

This protocol outlines a large language model-assisted workflow for classifying metadata in electric power information systems. By integrating structured prompting, rule-based post-processing, and mandatory human review, it provides a secure, auditable framework for data governance during smart grid digitalization.

Zusammenfassung

Electric power organizations must classify heterogeneous data assets before applying access, retention, and cybersecurity controls. This study presents a metadata-only protocol that combines a fixed large language model (LLM) prompt, structured JSON output, deterministic screening, and mandatory human adjudication across five governance classes. The protocol was evaluated on 240 metadata records from 10 business domains. Raw LLM predictions agreed with adjudicated human labels for 207 records (86.25%; macro-F1 = 0.827), outperforming an exploratory deterministic rule baseline (66.67%) and a majority-class baseline (41.25%). Eighty-two records (34.17%) were routed to manual review; after adjudication, all 82 matched the reference labels. Five unsafe downgrades involved Critical Operational records and were intercepted by the review pathway. The workflow is limited to metadata and does not process raw documents, full tables, telemetry, or logs. These findings support LLM use as a triage aid rather than an autonomous classifier for metadata in electric power data governance.

Einleitung

Digital transformation has fundamentally reshaped information management within electric power organizations. Modern enterprise systems integrate diverse data streams, ranging from routine administrative records to operational-support metadata, cybersecurity monitoring outputs, and customer-service data. This convergence is driven by the broader digitalization of smart grids, where interconnected infrastructure and artificial intelligence are deployed to optimize energy management and system monitoring¹. Consequently, effective data governance has become a critical prerequisite for secure information sharing, analytics, and operational reliability across distributed grid environments².

However, data assets in power systems are highly heterogeneous, demanding strict classification to prevent inappropriate exposure of sensitive information. Frameworks such as the NIST Cybersecurity Framework 2.0 emphasize that governance is a core operational function rather than a peripheral administrative task³. Currently, manual classification of these enterprise data assets is slow, inconsistent, and administratively burdensome. Data stewards must process vast volumes of metadata featuring uneven descriptions, legacy system names, and overlapping business domains.

LLMs have been applied to document classification in regulated sectors⁴, but electric power governance presents a different input and risk setting. Document classifiers generally receive content-rich prose, whereas the present governance unit is a short metadata record containing fields such as source system, update frequency, description, and access scope. An unclassified record is not technically unusable; rather, its missing governance label prevents an organization from determining whether its content may be disclosed to a model, which fields may be submitted, and who must approve the output. In parallel, operational smart-grid AI commonly assumes that input assets have already been curated, permissioned, and governed^5,6. Existing approaches, therefore, mismatch both the metadata input modality and the safeguards required for critical-infrastructure classification.

To address this gap, the goal of this method is to establish and evaluate an LLM-assisted protocol for metadata-level classification and governance in electric power information systems. The workflow uses language understanding to interpret structured records while preventing autonomous enforcement. Because under-classifying sensitive infrastructure metadata can create greater harm than conservative over-classification, the protocol emphasizes metadata-only input, fixed governance definitions, structured JSON output, contextual risk screening, and mandatory human confirmation^7,8,9.

By clearly defining what data may be submitted and how confidence scores trigger manual review, this method ensures institutional accountability. This protocol is appropriate for data stewards, cybersecurity officers, and enterprise architects who seek to reduce manual classification workloads and preserve auditability without compromising the secure governance of digital transformation in electric power environments¹⁰.

Protokoll

1. Study design and data-security scope

1. Design a metadata-level, large language model (LLM)-assisted protocol to classify and govern data within electric power information systems.
2. Target the workflow to process structured data asset records originating from diverse sources, including customer service, metering, outage management, enterprise resource planning, geographic information, and cybersecurity monitoring systems.
3. Define the core unit of analysis as a single data asset record, expressly avoiding the processing of individual customer accounts, specific control commands, or real-time grid events.
4. Process only metadata-level descriptions to prevent the exposure of regulated information¹¹. Ensure that raw files, full database tables, telemetry streams, and cybersecurity incident logs are entirely excluded from the environment.
5. Deploy the workflow strictly as a decision-support system. Use LLM outputs to assist classification, identify risk flags, and prioritize manual review, ensuring that the model does not automatically enforce access controls or retention schedules.

2. Study records and data asset register construction

1. Compile 240 metadata-level data asset records covering 21 controlled source-system labels and 10 business domains.
   NOTE: The 21 labels in the record-level workbook are AMI_MDMS, Audit Management, Compliance Repository, CRM, Customer Portal, Distribution Management System, EAM, ERP, Finance Ledger, GIS, HRIS, Maintenance Planning, Market Interface, Meter Data Management, Network Mapping, Outage Management System, SCADA Summary Register, Security Event Monitoring, Settlement System, SIEM, and Training Records.
2. Construct a metadata register assigning each record an asset ID, source system, business domain, asset name, asset description, data format, update frequency, and access scope.
3. Add screening indicators for customer data, operational control, cybersecurity, and regulatory relevance. Code each indicator as 1, 0, or unknown, and interpret it together with the description and access scope rather than as an automatic final class. Refer to Table 1 for field definitions.
4. Exclude records from the LLM workflow if they are missing essential fields such as the source system, asset description, or data owner role.
5. Return asset descriptions shorter than 10 words to the data steward for immediate revision. Shorten descriptions exceeding 80 words to a standardized 15 to 60 words while meticulously retaining the business purpose, sensitivity cues, and access context.
6. Strip all directly sensitive electric power information from the input texts to maintain an absolute security boundary¹². Remove names, accounts, meter readings, API keys, control-room logs, and equipment coordinates before prompt submission.

3. Governance classification schema application

1. Apply a five-class schema comprising Public, Internal, Confidential, Regulated, and Critical Operational.
   NOTE: Use these locked definitions. Public: approved for external disclosure and containing no restricted context. Internal: routine business metadata limited to ordinary employees but lacking contractual, personnel, financial, privacy, compliance, operational-control, or cyber-defense sensitivity. Confidential: commercially, financially, contractually, procurement-, vendor-, or personnel-sensitive metadata requiring role-based access. Regulated: metadata governed by privacy, customer, billing, audit, statutory-retention, or compliance obligations. Critical Operational: metadata whose disclosure or alteration could affect grid operations, outage response, dispatch, protection, control support, cyber-defense, infrastructure security, or operational situational awareness.
2. Separate regulated compliance data explicitly from operationally sensitive grid data, recognizing that these categories necessitate vastly different access controls and security-domain restrictions¹³.
3. When more than one definition is supported by the complete metadata context, select the highest-risk applicable class. Do not elevate a record solely because a broad domain indicator equals 1; confirm the risk signal in the description, source system, access scope, or rationale.
4. Lock these decision rules prior to model evaluation to prevent any post hoc adjustment of the governance logic.

4. Data preprocessing and standardization

1. Standardize all metadata records before constructing the LLM prompts. Identify duplicate asset IDs and resolve them immediately.
2. Map source-system names to a controlled vocabulary to standardize inputs across the dataset.
3. Code business domains and update frequencies utilizing predefined categorical labels.
4. Code the customer-data, operational-control, cybersecurity, and regulatory-relevance indicators strictly as 1 if the signal is explicit, and 0 if absent. Route any record with an unclear signal directly to manual review by marking it as unknown.
5. Utilize this rigorous standardization to minimize inconsistent model behavior stemming from uncontrolled enterprise terminology¹⁴.

5. Human reference classification

1. Recruit two independent reviewers with enterprise data-management or electric-power governance experience to classify all 240 records against the locked five-class schema.
2. Give both reviewers the same written definitions, highest-risk rule, contextual risk triggers, and decision examples before classification. Record reviewer role, relevant experience, and any relationship to dataset construction.
3. Require each reviewer to label records independently without seeing the LLM prediction. Preserve both initial labels and disagreement reasons. Resolve disagreements through a third adjudicator, with non-adjacent disagreements receiving an explicit operational-risk review.
4. Establish the final adjudicated label as the definitive reference standard for subsequent automated evaluation¹⁵. Lock the adjudicated reference label before model inference and conceal it from the LLM and the first-pass manual reviewer. Use the label only for evaluation and final discrepancy adjudication¹⁵.

6. LLM-assisted classification workflow execution

1. Execute the complete workflow spanning metadata validation, prompt construction, model classification, post-processing, and manual review. Refer to Figure 1 for the detailed workflow schematic.
2. Submit only the structured metadata fields through an enterprise-controlled API environment. Disable web browsing, external retrieval, and file upload, and exclude all raw documents, database contents, telemetry, and incident logs.
3. Require a single governance class, a confidence score, risk flags, a concise field-grounded rationale, and a recommended action in a valid JSON object.
4. Evaluate one enterprise-hosted LLM configuration with a temperature of 0.2, a maximum output length of 800 tokens, and one classification call per record¹⁶. Do not attribute the reported results to a specific LLM family or version. Record the provider, model name, and version, deployment identifier, and release date in all future runs.
   NOTE: The exact model identity, including whether the deployed model was GPT-4 or another model/version, was not retained in the study records and could not be reliably recovered retrospectively.

7. Prompt structure design

1. Use the complete fixed system prompt, user template, governance definitions, calibration examples, and required JSON schema provided in Supplementary File 1. Do not modify or abbreviate these components during replication.
2. Instruct the model to use only the supplied metadata, avoid inferring unobserved sensitive facts, cite the supplied fields in its rationale, and avoid downgrading when the complete context supports a higher-risk class.
3. Calibrate decision logic using fixed examples that are independent of the 240 evaluation records. Use these examples only to demonstrate the application of the predefined classification rules and risk-prioritization principles. Refer to Supplementary File 1 for the complete set of calibration examples.
4. Lock the prompt version before evaluation and store it for each record. Refer to Table 2 for parameters and review triggers.

8. Rule-based post-processing

1. Apply automated rule-based post-processing immediately following each API response.
2. Mark an output invalid if JSON parsing fails, a required key is missing, confidence is outside 0–1, more than one class is returned, or the predicted class is outside the locked schema. Route every invalid output to manual review.
3. Route a record to manual review when confidence is below 0.80, the rationale contradicts the prediction, the access scope conflicts with a public prediction, or the description and contextual indicators support a higher-risk class than the prediction.
4. Accept an output without manual review only when confidence is at least 0.80, JSON is valid, every required key is present, and no unresolved contextual conflict exists. Treat confidence as a triage signal rather than a calibrated probability¹⁷.

9. Governance action assignment

1. Link every confirmed governance class to a standardized set of predefined governance actions. Require strict publication approval for public records and role-based access logging for Confidential records.
2. Enforce privacy reviews and tokenization for regulated records, while mandating restricted operational-zone access, encryption, and dual-authority approval for critical operational data.
3. Apply these governance actions distinctly as decision-support controls without configuring the script to alter access privileges in the live source systems automatically.

10. Manual review and adjudication

1. Review every record triggered by low confidence, invalid output, rationale-prediction conflict, restricted-access conflict, or contextual sensitivity conflict.
2. Show the reviewer the raw metadata, predicted class, confidence, risk flags, and rationale, but conceal the adjudicated reference label until the reviewer records an independent decision and reason.
3. Record one of four outcomes: accept, upgrade, downgrade, or unresolved. After recording the independent decision, compare it with the locked reference; send any remaining discrepancy to an adjudicator who did not make the first-pass decision whenever staffing permits.

11. Quality-control checks

1. Check metadata completeness, JSON validity, class membership, reviewer independence, and consistency between each confirmed label and the written class definitions.
2. Exclude a record only when essential metadata remain insufficient, and no defensible reference label can be established. Retain difficult boundary cases and model errors in the evaluation set.

12. Stability assessment

1. Select a random 10% subset of the dataset to rigorously assess output stability¹⁸.
2. Re-run the LLM classification on this subset utilizing the exact same prompt version, model parameters, and schema constraints.
3. Calculate the repeat-run consistency by measuring the proportion of records that receive the exact same predicted class across both the initial and subsequent runs.

13. Evaluation metrics and statistical analysis

1. Compare each raw LLM prediction with the locked adjudicated reference label to calculate overall accuracy.
2. Calculate class-specific precision, recall, and F1 scores and the macro-averaged F1 score.
3. Add two exploratory post hoc benchmarks: (i) a deterministic rule baseline that assigns Critical Operational when operational_control or cybersecurity equals 1, otherwise Regulated when customer_data or regulatory_relevance equals 1, otherwise Public for a public access scope, Confidential for a restricted access scope, and Internal otherwise; and (ii) a majority-class baseline that predicts Confidential for every record. Report these as descriptive comparators rather than preregistered model benchmarks.
4. Calculate manual-review rate, invalid-output rate, high-risk downgrading count and rate, and the confusion matrix.
5. Estimate Wilson 95% confidence intervals for accuracy, manual-review rate, invalid-output rate, and high-risk downgrading rate¹⁹. Refer to Table 3 for formulas and observed values.

14. Audit trail and reproducibility control

1. Create an audit record for every call containing asset ID, timestamp, complete prompt version, provider/model/deployment identifier, input metadata, raw JSON, parsed fields, post-processing triggers, reviewer identity/role, independent review decision, and final adjudication.
2. Enforce the metadata-only boundary consistently, ensuring that no internal IP addresses, protection-setting values, or full customer profiles enter the model context.
3. Retain the audit record, prompt, class definitions, source data, analysis code, and figure source data under version control to support reproducibility and governance accountability²⁰.

Ergebnisse

Dataset composition and overall classification performance

The evaluation included 240 metadata records across 10 business domains; Metering (34 records), Customer Service (32), and Distribution Operations (30) were the largest groups (Table 4). The source_system field contained 21 controlled labels. The adjudicated reference distribution was Public 10, Internal 64, Confidential 99, Regulated 33, and Critical Operational 34. Raw LLM predictions were Public 8, Internal 70, Confidential 99, Regulated 31, and Critical Operational 32 (Figure 2). Thus, similar marginal totals did not imply record-level agreement.

Raw LLM predictions matched 207 of 240 adjudicated labels, giving 86.25% accuracy (Wilson 95% CI: 81.32%-90.04%) and a macro-F1 score of 0.827. Confidential had the highest class-specific F1 score (0.919); Critical operational had an F1 score of 0.758 (Table 5). In an exploratory post hoc comparison, a deterministic metadata-rule baseline achieved 66.67% accuracy and macro-F1 0.693, while a majority-class baseline that assigned every record to Confidential achieved 41.25% accuracy and macro-F1 0.117. These unoptimized baselines show added discriminatory value over simple rules but do not establish superiority to trained machine-learning classifiers.

Misclassification patterns and high-risk downgrading analysis

The confusion matrix contained 33 disagreements (Table 6). Five were high-risk downgrades: two critical operational records were predicted as Internal, and three as Confidential. Four additional critical operational records were predicted as regulated; these retained a high-risk label and were not counted as unsafe downgrades under the prespecified definition. Seven regulated records were conservatively predicted as critical operational. The high-risk downgrading rate was therefore 7.46% (5/67; Wilson 95% CI: 3.23%-16.31%) among records whose reference class was Regulated or Critical Operational.

Confidence thresholds and manual-review outcomes

The distribution of LLM confidence scores effectively triaged the records according to the protocol's manual-review logic (Figure 3 and Table 7). A total of 158 records achieved confidence scores of 0.80 or higher without triggering any mandatory override flags. Among these provisionally accepted records, 155 matched the reference labels. The three remaining incorrect high-confidence cases involved subtle boundary distinctions between Internal, Confidential, and Critical Operational classes. Conversely, records with confidence scores from 0.60 to 0.79 (64 records) and below 0.60 (18 records) were routed to manual review. Correctly classified records exhibited higher mean confidence scores (0.829) compared with incorrectly classified records (0.667), indicating that confidence scores provided useful but imperfect discrimination between correct and incorrect predictions. Overall, the protocol routed 82 records to manual review, corresponding to a review rate of 34.17% (Wilson 95% CI: 28.46%–40.37%). The rule-based post-processing and confidence thresholds successfully captured 30 of the 33 total disagreement cases, significantly mitigating residual classification error. Following manual review, 16 records were upgraded to address LLM risk underestimation, and 14 were downgraded to correct risk overestimation. The final governance register incorporated both the directly accepted and the manually adjudicated classifications, reducing the immediate human workload while preserving critical oversight. Confidence separated easily from difficult cases but was not perfectly calibrated (Figure 3 and Table 7). After review, 52 predictions were accepted, 16 were upgraded, and 14 were downgraded. All 82 reviewed records ultimately matched the adjudicated reference, but this 100% agreement reflects adjudication against the locked reference rather than an independent external validation.

Representative-case analysis showed that all five unsafe downgrades contained explicit operational-control, cybersecurity, operational-security-zone, control-room, or dual-authority cues, yet the raw model assigned Internal or Confidential (Table 8). Because the public workbook does not retain verbatim model rationales, Table 8 separates retained fields and risk flags from an author interpretation and does not attribute that interpretation to the model.

The business domain did not map one-to-one to the governance class. Every domain contained multiple reference classes, confirming that domain identity provides context but cannot determine the label by itself (Table 9).

Workflow stability and governance action validation

A random 10% subset (24 records) was classified twice under the same prompt and parameter settings. Twenty-two records received the same class, giving 91.7% repeat-run consistency. The two inconsistent cases involved overlapping operational and regulated cues, and neither produced a high-risk downgrade. The public register preserves a prompt version, confidence, risk flags, recommended action, manual action, and final class for all records. It does not preserve the verbatim rationale, raw JSON, reviewer-level annotations, or exact model deployment identifier; these omissions limit independent auditability. The current Zenodo record (https://zenodo.org/records/21393865) contains the de-identified record-level workbook used for the reported analyses.

Figure 1: Metadata-only LLM-assisted classification and governance workflow. A single non-duplicated pathway proceeds from register preparation and metadata validation to structured LLM inference, deterministic output screening, provisional acceptance or manual review, final governance assignment, and audit. Reference labels are locked before inference and used only for evaluation and final adjudication. Abbreviation: LLM = large language model. Please click here to view a larger version of this figure.

Figure 2: Adjudicated reference and raw LLM-predicted governance-class distributions. Bars show record counts across 240 metadata records. Reference/predicted counts were Public 10/8, Internal 64/70, Confidential 99/99, Regulated 33/31, and Critical Operational 34/32. Source data are provided in the repository-deposit package. Abbreviation: LLM = large language model. Please click here to view a larger version of this figure.

Figure 3: Confidence category, raw prediction correctness, and manual-review routing. Stacked bars show correct/incorrect counts of 155/3 at confidence ≥0.80, 41/23 at 0.60–0.79, and 11/7 at <0.60. All 82 records below 0.80 were routed to review; the 158 records at or above 0.80 were not routed by the confidence rule. Source data are provided in the repository-deposit package. Abbreviation: LLM = large language model. Please click here to view a larger version of this figure.

Table 1: Metadata fields and governance coding rules for electric power information-system data asset records. This table lists the required metadata fields, their data types, descriptive examples, and the underlying governance purpose for constructing the structured data asset register. Please click here to download this Table.

Table 2: Prompt architecture, model parameters, classification thresholds, and manual-review triggers. The table summarizes the fixed components of the LLM prompt, operational model parameters such as temperature and output length, and the mandatory rule-based conditions that trigger manual review. Please click here to download this Table.

Table 3: Evaluation metrics, formulas, scopes, and observed results. The table reports all the observed values, including accuracy, class-specific precision/recall/F1, macro-F1, review and invalid-output rates, repeat consistency, high-risk downgrading, and Wilson intervals. Please click here to download this Table.

Table 4: Composition of 240 metadata records by 10 business domains. Counts and percentages were recalculated directly from the record-level workbook. Please click here to download this Table.

Table 5: Raw LLM and exploratory baseline performance. The table reports class-specific raw LLM metrics and overall accuracy/macro-F1 for the raw LLM, deterministic metadata-rule baseline, and majority-class baseline. Abbreviation: LLM = large language model. Please click here to download this Table.

Table 6: Confusion matrix of adjudicated reference and raw LLM-predicted classes. All cells and marginal totals were recalculated directly from the 240 record-level observations. Abbreviation: LLM = large language model. Please click here to download this Table.

Table 7: Confidence categories, raw LLM prediction correctness, and manual-review routing outcomes. Counts were recalculated from individual confidence scores and prediction_correct indicators. Please click here to download this Table.

Table 8: Representative successful classifications and high-risk downgrade cases. The table reports retained metadata, reference and predicted labels, confidence, risk flags, manual outcome, and an explicitly labeled author interpretation; verbatim LLM rationales were unavailable. Please click here to download this Table.

Table 9: Cross-tabulation of 10 business domains and five adjudicated governance classes. The distribution demonstrates that all domains span multiple classes and that domain membership is not a deterministic class mapping. Please click here to download this Table.

Supplementary File 1: Complete LLM Prompt, User Template, Governance Definitions, Calibration Examples, and JSON Output Schema. Please click here to download this file.

Diskussion

The most critical protocol step is the separation of model inference from final governance authority. Raw LLM accuracy was 86.25%, but five Critical Operational records were downgraded to Internal or Confidential. Deterministic screening routed all five to review, illustrating why structured output, contextual conflict checks, and independent human adjudication are required in critical-infrastructure governance^21,22. The representative cases in Table 8 also show that operational sensitivity can be embedded in routine-sounding descriptions, so a broad domain label or confidence score must never substitute for field-level reasoning.

Troubleshooting centers on invalid JSON, missing keys, contradictory rationales, ambiguous descriptions, and confidence below 0.80. Any of these conditions triggers a review rather than an automatic correction. The workflow captured 30 of 33 raw errors while routing 34.17% of records, demonstrating a risk-coverage trade-off: lower automation coverage concentrates human effort on uncertain records¹⁷. Confidence remains a triage signal, not a calibrated probability; institutions should monitor error and review rates over time and revise thresholds only with a separate validation set^23,24.

This study has several technical limitations. It processes metadata only and cannot classify raw documents, complete tables, telemetry, or unmasked logs. The controlled, single-organization-style dataset contains repeated description patterns and may understate real catalog noise. Only one LLM configuration was evaluated, and the study records did not retain sufficient deployment information to reliably recover the exact model identity or version, including whether the model was GPT-4. No trained random-forest or support-vector-machine benchmark was available; the added rule-based and majority-class baselines were post hoc. Confidence was self-reported and not calibrated. Individual reviewer labels, inter-rater agreement, reviewer identities, verbatim model rationales, and reviewer-overlap provenance were not preserved, so grader dependence and confirmation bias cannot be excluded. Finally, the five-class schema and contextual triggers require external validation before transfer to another utility or jurisdiction.

The protocol's contribution is therefore a comparative, human-referenced evaluation and an auditable workflow rather than a claim that an LLM can replace expert classification. The raw LLM exceeded simple deterministic and majority baselines, but its main operational value was rapid first-pass triage combined with a defined rejection pathway. This framing distinguishes the method from smart-grid AI studies that assume already governed inputs and from document classifiers that receive substantially richer text^25,26,27.

In conclusion, the results support a narrow use of LLMs as metadata-classification assistants. The workflow reduced the number of records requiring immediate review to 82 of 240 while retaining human authority over every flagged case. The findings do not support autonomous governance enforcement or generalization beyond the evaluated model, schema, and controlled dataset. Future work should preregister multiple-model and trained-classifier benchmarks, calibrate uncertainty on external utility datasets, preserve reviewer-level provenance, and monitor drift longitudinally²⁸.

Materialien

Liste der in diesem Artikel verwendeten Materialien

Name	Unternehmen	Katalognummer	Kommentare
Enterprise large-language-model API	Microsoft Azure	NA	Generate one structured JSON governance classification for each metadata record.
Spreadsheet software	Microsoft Corporation	Microsoft Excel 365	Maintain the data-asset register and the separately submitted editable tables. Other spreadsheet and document-processing software with equivalent functionality may be used.
Secure enterprise workstation	Dell Technologies	Institution-managed workstation; exact hardware identifier not retained	Run metadata preprocessing, model calls, human review, statistical analysis, figure generation, and audit logging. Other equivalent workstations may be used.