MAS4SysML: A Multi-Agent Framework for SysML v2 Model Generation from Natural Language

Summary

This protocol presents MAS4SysML, a multi-agent approach that automatically generates SysML v2 code through coordinated task division, requiring few repair iterations and significantly reducing manual modeling time while improving system modeling efficiency.

Abstract

Automatically generating accurate SysML models from natural-language requirements can substantially accelerate the adoption of Model-Based Systems Engineering (MBSE) in complex system development. However, using large language models (LLMs) to generate model code often fails to meet the strict syntactic constraints of formal modeling languages, and consistently ensuring semantic alignment between generated models and requirements remains challenging. To address these challenges, this paper presents MAS4SysML, a multi-agent collaborative framework for SysML v2 code generation that improves syntactic correctness and semantic consistency under a constrained repair budget. The framework decomposes a modeling task into hierarchical subtasks, formalizes them as structured task cards, and generates model code in a bottom-up manner. During generation, an official validation environment is used for syntax diagnostics; after completion, the framework verifies semantic consistency between the code and the task cards. If syntax or semantic validation fails, the framework iteratively repairs and revalidates the code within a predefined repair budget, guided by diagnostic feedback, until the validation criteria are satisfied or the budget is exhausted. To evaluate the proposed method, we construct a SysML v2 dataset spanning five core task types—requirements, use cases, structure, parametrics, and state machines—and conduct comparative experiments. Results show that MAS4SysML reduces the average syntax error rate to 2.63, increases semantic similarity to 0.91, and outperforms existing code-generation methods overall.

Introduction

MBSE has become a key methodology for requirements analysis, system architecture design, and verification planning in the development of complex equipment in domains such as aviation and aerospace¹. Using unified modeling languages such as SysML as the modeling backbone, information—including requirements, structure, behavior, and constraints—can be organized into a coherent model framework, improving process structure and the efficiency of cross-disciplinary collaboration². However, as system scale continues to grow, the number of models that must be developed increases accordingly, leading to a sustained rise in the workload of manual SysML modeling. Moreover, modelers must work under strict syntactic and methodological constraints, which require substantial expertise and strong abstraction capabilities. These factors have become a major bottleneck in the engineering adoption of MBSE³.

In recent years, LLMs have demonstrated strong capabilities in natural language understanding, structured information representation, and code generation, enabling new opportunities for automating MBSE modeling from natural language to model code⁴. Prior studies have explored the direct generation of SysML model code using LLMs⁵. Nevertheless, significant challenges remain. Syntactically, SysML's type system, scoping mechanisms, and reference semantics are governed by strict formal constraints and are often more intricate than those of general-purpose programming languages⁶. Semantically, LLMs may fail to fully capture behavioral logic, structural relationships, and cross-layer constraints, resulting in missing relations, logical inconsistencies, or incomplete models⁷. These limitations hinder the reliability, controllability, and interpretability of direct-generation approaches.

To address these challenges, this paper presents MAS4SysML, a SysML v2 code-generation framework comprising four agent roles. Rather than directly targeting the one-shot generation of a complete cross-view system model, this study focuses on generating and iteratively repairing multiple representative SysML v2 modeling tasks. Driven by task decomposition and structured task cards, and integrating code generation, tool-level syntax diagnostics, and semantic consistency verification, MAS4SysML establishes a closed-loop workflow of generation, validation, and repair. This design improves the syntactic correctness, semantic consistency, and structural completeness of the generated code under a constrained repair budget.

The main contributions are summarized as follows: (1) MAS4SysML framework. We propose MAS4SysML, a multi-agent, LLM-driven framework that enables end-to-end generation from natural-language requirements to executable SysML v2 model code, providing a practical pathway to reduce modeling cost and improve modeling efficiency. (2) Task parsing and dual validation. We introduce a task-structure-tree parsing mechanism and a dual validation scheme (syntax and semantics). In task parsing, modeling objectives are hierarchically decomposed and formalized into structured task cards. For validation, syntax diagnostics leverage an official validation environment⁸ to check generated code and return repair-oriented diagnostics, while semantic validation uses key fields in the task cards as references to assess overall consistency between the generated model and modeling objectives. (3) Experimental evaluation. We conduct comparative experiments using syntax error rate and semantic consistency score as the primary metrics. The results show that MAS4SysML reduces the average syntax error rate to 2.63 and increases semantic similarity to 0.91, outperforming baseline methods in generation accuracy and automation.

Protocol

The code generation process of the MAS4SysML framework is summarized in Supplementary File 1. It should be noted that this study does not aim to achieve the one-shot generation of a complete system model from natural language with strict cross-view consistency, including requirements, structure, parametrics, and behavior. Instead, the protocol focuses on generating several representative types of SysML v2 view code.

Phase I: Task analysis
The workflow begins with task parsing. The system provides the natural-language modeling intent to the Task Structure Generation Agent, which outputs a task-card set. To ensure that subsequent generations are executable and reproducible, each task card must include, at a minimum, (i) a task identifier, (ii) dependency relationships, and (iii) key modeling information for validation, such as the modeling objective, constraints/boundary conditions, parameter slots, and instantiation values, and the expected outputs. This stage outputs task_card_set, which serves as the unified basis for subsequent model code generation.

Phase II: Iterative code generation
In iterative generation, the system initializes the code context prev_code to an empty state and generates code for each task card sequentially according to an order determined by the dependency fields. For each task card, the Code Generation Agent takes the current task card and the contextual code as input to produce candidate_code and then immediately invokes the Syntax Validation Module for checking. The module validates the code using the official SysML v2 validation environment and returns diagnostic results. If validation succeeds, the candidate_code is used to update prev_code and supports subsequent generation. If validation fails, the Code Repair Agent is triggered and performs minimal, targeted edits guided by the returned diagnostics, after which the repaired code is resubmitted for revalidation. This repair revalidation loop is bounded by the maximum repair budget K_max. If validation succeeds within the budget, the passing version updates prev_code; otherwise, after K_max attempts, the system logs the failure and continues with subsequent task card generation using the last repaired version as prev_code to avoid blocking the workflow while maintaining contextual continuity.

Phase III: Semantic validation
After the code has been generated for all task cards, the workflow proceeds to semantic validation. The Semantic Validation Agent assesses consistency between the final code and the modeling intent using key fields in task_card_set as references and outputs the semantic validation results. If validation succeeds, prev_code is accepted as the final SysML v2 model code. Otherwise, the system generates a Semantic Deviation Report that identifies unmet task card fields and the required revision scope. The Code Repair Agent then revises the code accordingly and outputs the revised model code as the final result.

Model architecture and methodology
Model architecture
The MAS4SysML framework, illustrated in Figure 1, comprises four collaborative agents: the task structure generation agent, code generation agent, code repair agent, and semantic validation agent. The corresponding prompt templates are presented in Figure 2.

The task structure generation agent performs semantic analysis of the input modeling intent and generates executable, structured task cards. It first applies a hierarchical task decomposition mechanism (see hierarchical task decomposition mechanism) to decompose the overall modeling objective into task nodes with well-defined semantic boundaries and then constructs a structured task card for each node. Subsequently, the task cards are ordered according to their modeling dependency fields to ensure that the execution sequence aligns with the eventual code structure, thereby laying the foundation for bottom-up code generation driven by the global modeling objective.

The code generation agent progressively generates SysML v2-compliant model code according to the modeling dependencies. Building on the code artifacts produced by parent tasks, the agent performs the corresponding code generation operations based on the requirements specified in each task card, thereby enabling a stepwise construction process—from local components to the complete model.

The code repair agent corrects errors in the generated code based on the outcomes of the syntax validation module (see syntax validation module) and the semantic validation results. For syntactic repair, it leverages the error type, position, and contextual information returned by the syntax validator to synthesize targeted repair strategies and generate corrected code. For semantic repair, it adjusts structural and logical relations according to the semantic-validation results, ensuring semantic consistency and structural completeness in the final model.

The semantic validation agent assesses the semantic consistency between the fully generated code and the task cards using a dedicated semantic validation mechanism (see semantic validation mechanism). Through quantitative assessment, it ensures that the generated code accurately reflects the original modeling intent, thereby achieving precise alignment between the model code and the specified modeling requirements.

Hierarchical task decomposition mechanism
As a formal modeling language for complex systems, SysML v2 features tightly coupled syntax, deeply nested hierarchical structures, and cross-level semantic constraints. For example, a system structure block may contain multiple subparts, attributes, and ports while simultaneously expressing performance or behavioral requirements through cross-layer constraints. These structures and constraints create top-down structural dependencies and bottom-up semantic feedback relationships. With a flat, one-shot generation approach, accurately mapping such hierarchical dependencies becomes challenging, often resulting in missing relations, semantic inconsistencies, or the loss of constraint information.

To address this challenge, we develop a task-tree-based modeling intent parsing method that hierarchically decomposes natural-language modeling requirements. As illustrated in Figure 3, complex modeling goals are decomposed into structured and traceable task nodes, enabling the system to interpret modeling semantics in a top-down manner and identify dependency relationships. Specifically, when the task structure generation agent receives the user input, it first leverages the semantic parsing capabilities of LLMs to identify core modeling objectives, key entities, and their dependencies. It then recursively decomposes the top-level goal into semantically independent subtasks and further refines them into atomic tasks that can be directly mapped to SysML v2 modeling operations, ultimately forming a complete task structure tree. After the task tree is constructed, the agent generates a structured task card for each task node based on a predefined template. The format of the task card is defined as follows:

TC = {id,O,N,K,P,V,C,D} (1)

Where id as the unique identifier of the task node, O as the task objective, N as the natural-language description of the task, K denotes the core SysML v2 semantic elements that may be involved in the task, mainly including requirement def/requirement, part def/part, port def/port, item def, attribute def/attribute, and state/transition. Relationships among these elements are primarily expressed through connect (structural connections), input/output items on ports (information/material flows), and trigger/guard conditions of state machine transitions (e.g., commands, health status, and threshold constraints), C as the semantic rules or boundary conditions, P as the parameterizable slots within the task such as attribute names, data types or composite types, V as the instantiated values for each slot and D as the modeling dependencies between tasks where depend_on specifies required outputs from other tasks before generating the current task code, provides denotes outputs produced after the task is completed and consumes represents external inputs required by the task.

Syntax validation module
A syntax validation module is constructed based on the SysML v2 Pilot Implementation. By invoking its parser and validator interfaces, the module parses and verifies the syntactic correctness of the generated SysML v2 model code. The module's validation criteria are primarily derived from the SysML v2 language specification, as well as the grammar rules, scope-resolution rules, and related constraint-checking mechanisms implemented in the Pilot tool. Specifically, the validation examines whether element declarations are well-formed, whether block structures are complete, whether type annotations are valid, whether names and references can be successfully resolved, and whether modeling constructs such as ports, connections, states, and transitions comply with the language's requirements.

After the code generation agent produces the code fragment for the current task, the output is forwarded to the syntax validation module, where the validation script analyzes the code and returns the results in the form of structured diagnostic information. The validation results are reported as follows:

e₁ = (type_i,pos_i,msg_i) (2)

Where e_i denotes the list of detected issues for the current modeling task, each entry containing the error type type_i, error location pos_i, and diagnostic message msg_i .

For example, if the generated code contains a syntax error such as "an attribute is not typed by an attribute definition," the validation module returns the following diagnostic message:

'type' : 'error'
'message' : 'ERROR: An attribute must be typed by attribute definition.' (3)
'position' : 'line 7 column: 3'

When the validation result e_i ≠ 0, the collected error information e_i is forwarded to the code repair agent for further correction. Therefore, the code repair process is not an unconstrained modification procedure, but a targeted revision guided by the explicit diagnostic information returned by the parser and validator.

Semantic validation mechanism
The semantic validation mechanism uses key task card fields that have explicit and traceable correspondences to the model code as semantic anchors. It assesses semantic consistency at the model level, thereby providing explicit, actionable criteria for subsequent model repair. Specifically, for each task card TC_i, the following fields are used as key semantic references: (i) the modeling objective O_i, (ii) semantic constraints and boundary conditions C_i, (iii) instantiated parameter-slot values V_i, and (iv) expected outputs after task completion D_i['provide']. These fields impose complementary semantic constraints on the generated model from multiple perspectives: realization of modeling intent, satisfaction of constraints, consistency of parameter instantiation, and completeness of model outputs—enabling a principled decision on whether the model code satisfies the modeling requirements without requiring extra assumptions.

Based on these key fields, we define a multi-field semantic consistency decision function:

(4)

where I(·) denotes an indicator function that equals 1 if all sub-decision functions inside the parentheses hold, and 0 otherwise. This binary decision explicitly distinguishes between the states of satisfying the modeling requirements and requiring further repair, providing a deterministic trigger condition for the subsequent semantic repair process. The overall decision is jointly determined by the following four sub-decision functions:

(1) Modeling objective consistency:

Φ₀ (TC_i,c_f) = I(consist(c_f,0_i)) (5)

where Consist(c_f,0_i) indicates whether the model code c_f is semantically consistent with the modeling objective 0_i specified in the task card.

(2) Semantic constraint satisfaction:

Φ_c (TC_i,c_f) = I(Satisfy(c_f,C_i)) (6)

where Satisfy(c_f,C_i) indicates whether the model code c_f satisfies the semantic constraints and boundary conditions C_i specified in the task card.

(3) Parameter consistency:

Φ_c (TC_i,c_f) = I(Instant(c_f,V_i)) (7)

where Instant(c_f,V_i) indicates whether the instantiated parameter values V_i in the task card are reflected consistently in the model code.

(4) Output consistency:

 (8)

where Artifacts(c_j) indicates whether the outputs expected by the task card are present in the final model code, serving as a measure of the completeness of the generated result.

These consistency judgments are implemented by the Semantic Validation Agent by leveraging the semantic understanding capability of the LLM; the agent's internal reasoning process does not alter the formal definition or usage of the consistency function.

Through this multi-field semantic consistency checking, the generated model can be validated field by field to ensure that each modeling objective, constraint condition, parameter configuration, and expected output is adequately satisfied. This process not only provides an explicit trigger for subsequent semantic repair but also supplies traceable semantic evidence throughout the generation pipeline, thereby improving the reliability and consistency of the generated model.

Experimental data and evaluation
Experimental data
SysML v2 model code is not ordinary software code; its generated artifacts exhibit distinctive characteristics of formal modeling. Different views typically involve distinct categories of core modeling elements, such as requirements, parts, ports, attributes, states, and transitions, which differ substantially in their declaration styles, organizational forms, and compositional structures. In addition, the model code must satisfy multiple constraints, including type referencing, hierarchical nesting, connection constraints, and semantic reuse among elements.

To comprehensively evaluate the performance of the proposed method under varying modeling complexities, a code dataset covering five representative model view types—requirements, use cases, structure, parametrics, and state machines—is constructed. These model views correspond to requirement specification, functional interaction, structural composition, parametric constraint representation, and behavioral logic description in system modeling, respectively. Evaluating the framework separately on different model view types enables a finer-grained analysis of its applicability under diverse code structure characteristics and modeling constraint conditions.

Each model view type contains 15 manually created model instances, resulting in a dataset of N = 75 SysML v2 models. The dataset spans multiple engineering domains, including aerospace, automotive, medical, and smart home systems, and all models successfully passed the official SysML v2 validation environment, ensuring strict syntactic compliance.

Subsequently, we generated a corresponding natural-language modeling intent description for each model. To improve construction efficiency, we used the prompt template shown in Supplementary File 2 and employed GPT-4o to produce initial descriptions. GPT-4o was selected for its strong semantic understanding and information extraction capabilities, enabling it to accurately capture core model elements without hallucination and generate human-like modeling intent descriptions⁹. To ensure accuracy and eliminate ambiguity, all generated descriptions were manually reviewed and refined by researchers with a systems engineering background. Representative examples for different model types are shown in Table 1.

Evaluation metrics
We employ the following three key metrics to evaluate the quality of the generated SysML v2 model code:

Average syntactic error rate (SER)
This metric quantifies the proportion of syntactic errors detected when the generated model code is validated against the official SysML v2 syntax rules. It is computed as:

   (9)

where E_i denotes the number of syntactic errors identified in the i-th generated model. This metric reflects the extent to which the generated model code adheres to the formal SysML v2 syntax specification.

Semantic consistency score (SCS)
This metric evaluates how accurately and comprehensively the generated model code captures the semantic intent expressed in the natural-language modeling specifications. Specifically, we extract semantic units from the modeling intent—such as system entities, participating components, core functions or behavioral scenarios, and key conditions or constraints—and compare them with the semantic units present in the generated model code. The semantic consistency is computed as:

    (10)

where U represents the set of semantic units extracted from the modeling intent, and represents the set of semantic units identified in the generated code.  indicates the number of units correctly captured by the generated model code. A higher SCS value indicates stronger semantic coverage and alignment.

Human quality evaluation
Traditional automated metrics such as BLEU and CodeBLEU primarily assess surface level similarity or code executability, but they do not capture whether the model truly understands or correctly expresses the intended modeling semantics. These metrics are limited in evaluating semantic consistency, completeness of key elements, and alignment with the modeling intent¹⁰. In contrast, human evaluation can more accurately identify issues such as missing semantic elements, logical inconsistencies, structural redundancy, or unsupported hallucinations, thereby providing a more reliable assessment¹¹. Motivated by these limitations, we design a human evaluation framework for generated SysML v2 models consisting of three criteria: (1) Correctness: the generated model must accurately reflect the modeling intent, maintain structural and logical consistency with the task objectives, and contain no semantic ambiguity, missing elements, or erroneous extensions. (2) Readability: the model code should be clear and easy to understand, with consistent naming, coherent structure, and a well-organized hierarchy that supports inspection and subsequent maintenance. (3) Integrity: the model should exhibit complete structural logic, consistent cross-element references, and no undefined types or broken dependency chains, ensuring its usability for downstream analysis and integration. We invited researchers with SysML modeling experience to score each generated model on a three-point scale, where 1 denotes the lowest quality and 3 the highest. During evaluation, assessors were allowed to compare the generated model code with the ground-truth model to ensure a more accurate and comprehensive assessment.

Base line
We selected multiple evaluation baselines for comparative testing against the proposed method, including:
CodeCoT¹²: Combines chain-of-thought reasoning with a self-check mechanism, enabling the model to explicitly reason during generation and self-correct syntactic errors, thereby improving code quality and semantic consistency.

Self-planning¹³: Introduces a two-stage code generation pipeline in which the model first plans the solution steps and then generates code according to the plan, effectively enhancing logical coherence and interpretability for complex tasks.

Self-edit¹⁴: Adopts an iterative generate and edit paradigm that executes the generated code and automatically corrects errors based on runtime feedback, continuously refining the output.
CodeChain¹⁵: Uses modular generation and iterative revision by decomposing complex tasks into independent functional modules and improving structural soundness and overall quality through multiple optimization rounds.

Self-debugging¹⁶: Endows the model with autonomous debugging and explanation capabilities. Through a closed-loop process of generation, execution, and debugging, it substantially improves correctness on complex programming tasks without human intervention.

MapCoder¹⁷: Constructs a multi-stage collaborative framework consisting of four agents—retrieval, planning, coding, and debugging—closely simulating the human programming workflow and enabling closed-loop generation from task understanding to result verification.

Self-Collaboration¹⁸: Organizes the system as a virtual programming team with roles such as analyst, programmer, and tester, improving overall performance on complex code generation through role-based collaboration and iterative feedback.

Experimental setup
To ensure fairness and comparability across experiments, we first evaluated several mainstream LLMs using a direct code generation approach to establish baseline performance. Based on these initial results, the best-performing LLM was selected as the unified backbone model for all subsequent experiments. We subsequently compared the proposed MAS4SysML framework with multiple representative code generation methods. All LLM interactions were conducted using a fixed temperature setting (T = 0.2) to minimize randomness during generation. For each modeling task, the maximum number of repair iterations in MAS4SysML was set to K_max = 3. All baseline methods were executed under the same experimental configuration as MAS4SysML to ensure result consistency and experimental fairness. The Python script of the MAS4SysML method is provided as Supplementary File 3.

Results

Baseline model evaluation
We first selected several mainstream LLMs and conducted preliminary performance tests using direct model-to-code generation, including CodeX(175B)¹⁹, CodeGen-Mono(16.1B)²⁰, PaLM Coder(62B)²¹, Alphacode(1.1B)²², Incoder(6.7B)²³, and code-davinci-002(175B)²⁴. As shown in Table 2, code-davinci-002(175B)

Discussion

We propose MAS4SysML, a multi-agent collaborative framework for semi-automated SysML v2 model code generation. The framework consists of four functionally complementary agents. During generation, it (i) hierarchically decomposes natural-language modeling requirements using a task-tree–based structure and formalizes them into structured task cards, and (ii) generates SysML v2 model code in a bottom-up manner guided by the constraints and dependency relations specified in these cards. Throughout generation, a syntax ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
LangChain	LangChain (open-source project)	v1.0.8; https://github.com/langchain-ai/langchain	Framework for LLM interaction and agent orchestration
LangGraph	LangChain (open-source project)	v1.0.3; https://github.com/langchain-ai/langgraph	Multi-agent workflow execution framework
Python	Python Software Foundation	3.10.x; https://www.python.org/downloads/release/python-3100/	Main programming language for MAS4SysML implementation
SysML v2 Pilot Implementation	Object Management Group (OMG)	(provide release/tag version); https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation	Used for syntax validation and model parsing