Baseline model evaluation
We first selected several mainstream LLMs and conducted preliminary performance tests using direct model-to-code generation, including CodeX(175B)19, CodeGen-Mono(16.1B)20, PaLM Coder(62B)21, Alphacode(1.1B)22, Incoder(6.7B)23, and code-davinci-002(175B)24. As shown in Table 2, code-davinci-002(175B)24 exhibited the best performance on both the SER and SCS metrics. Therefore, code-davinci-002(175B) was selected as the base LLM for evaluating various code-generation strategies in this study. The results of these strategy evaluations are summarized in the lower half of Table 2.
Among single-agent methods, CodeCoT leverages chain-of-thought reasoning and self-checking to correct certain local errors, but its improvements remain largely constrained to local adjustments: its SER remains as high as 4.98, and its SCS reaches only 0.72. In contrast, Self-Planning, Self-Edit, and CodeChain introduce task planning and iterative generation, which improve structural coherence, reducing SER to 3.82, 3.61, and 3.23 and increasing SCS to 0.77, 0.75, and 0.78, respectively. Notably, Self-Debugging benefits from the self-explanation capability of LLMs and achieves superior semantic performance with an SCS of 0.83, but its syntactic accuracy remains inadequate, indicating that self-debugging alone is insufficient to ensure both syntactic correctness and semantic consistency. For multi-agent collaborative methods, MapCoder enhances contextual sharing through a multi stage process comprising retrieval, planning, coding, and debugging, yielding SER and SCS scores of 3.42 and 0.88. Self-Collaboration simulates the workflow of a virtual team, forming a feedback loop across analysis, generation, and validation, and further improving semantic alignment, with SER and SCS of 3.38 and 0.87.
In contrast, the proposed MAS4SysML achieves the best performance across both metrics, reducing SER to 2.63 and increasing SCS to 0.91. These substantial performance gains stem from the integration of explicit task-tree decomposition, progressive layer-by-layer code generation, and a unified syntax–semantic dual-validation mechanism, which collectively ensure optimal structural consistency and semantic alignment in the generated models.
Performance analysis of different LLMs
To systematically evaluate LLM performance in SysML v2 code generation, we selected several mainstream conversational models for baseline testing, including Fastchat(3B)25, ChatGLM(6B)26, MPT(7B)27, Vicuna(7B/13B)28, HuggingChat(30B)29, Dromedary(65B)30, and the high-performing GPT-4o9. We evaluated each model under two settings: (1) direct generation of SysML v2 model code and (2) serving as the base model within MAS4SysML, to distinguish the inherent capabilities of each LLM from the gains introduced by our collaborative framework.
As shown in Table 3, overall code generation quality improves substantially as model size increases. Models below 13B typically yield SER values around 30 and SCS values below 0.6, indicating limited capability in handling SysML v2 syntax and semantic alignment, and even with MAS4SysML, the improvements remain marginal. When model size increases to 30B and 65B, SER decreases to 20.21 and 12.96, while SCS increases to 0.78 and 0.82, suggesting that these models achieve more stable syntactic correctness and semantic consistency, and that MAS4SysML provides greater benefits at this scale. Among all tested models, GPT-4o achieves the best performance, with SER reduced to 3.78 and SCS reaching 0.85, demonstrating its strong capability for generating syntactically compliant and semantically aligned SysML v2 models. Overall, model size emerges as the primary factor influencing SysML v2 code generation quality. When model size is sufficiently large, MAS4SysML can significantly enhance syntactic correctness and semantic alignment; however, when model size is small, the potential gains offered by MAS4SysML are substantially constrained.
We additionally evaluated several LLMs with strong code-generation capabilities, including text-davinci-002(175B)31, code-davinci-002(175B)24, code-cushman-001(13B)32, and Code LLaMA (7B/13B/34B)33. Among them, code-davinci-00231, code-cushman-00132, and the Code LLaMA series33 were trained on large-scale programming corpora, providing stronger syntactic control and structural code modeling capabilities. The results are presented in Table 4. The results indicate that for smaller models, MAS4SysML yields only limited improvements due to the inherent capability ceilings of the underlying LLMs. When model size increases to 34B, the benefits of the multi-agent framework become more apparent, with SER decreasing to 9.83 and SCS improving to 0.75. At the same 175B scale, code-davinci-002(175B)24 significantly outperforms the general-purpose model text-davinci-002(175B)31, demonstrating that code oriented training substantially enhances code generation capabilities. However, the absence of a large and high-quality SysML v2 training corpus limits domain-specific fine-tuning, leaving current SysML generation performance heavily dependent on the LLMs' pretraining capability.
Overall, LLM performance on SysML v2 code generation improves substantially with larger model sizes and more specialized training paradigms. When model size exceeds 30B parameters, both syntactic correctness and semantic consistency exhibit notable gains, and the collaborative benefits of MAS4SysML become increasingly pronounced. Furthermore, specialized code training plays a crucial role in improving generation quality. Future work leveraging large-scale SysML v2 corpora for domain-adapted training may further enhance the reliability and quality of SysML model generation.
Cost analysis
In general, multi-agent frameworks incur higher runtime overhead than single-agent setups. Accordingly, for cost analysis, we compare MAS4SysML with one single-agent code-generation baseline (Self-Debugging) and two multi-agent baselines (MapCoder and Self-Collaboration). Table 5 summarizes the evaluation results for these frameworks. We monitored runtime overhead using the Python library psutil34, recording execution time, CPU utilization, memory usage, disk I/O, and network I/O to assess the cost efficiency of MAS4SysML. All experiments were performed on a dedicated server to minimize interference from other services or processes.
The results indicate that, although MAS4SysML requires more runtime than the single-agent baseline, it achieves the best efficiency among the multi-agent methods. Specifically, MAS4SysML completes execution in 174.6 s, compared with 194.7 s for MapCoder and 231.5 s for Self-Collaboration. Moreover, MAS4SysML and MapCoder exhibit similarly low CPU utilization (1.2%), both lower than Self-Collaboration (2.3%), suggesting that MAS4SysML maintains low CPU demand while achieving competitive throughput.
In terms of memory consumption, MAS4SysML uses 0.17 GB, whereas Self-Collaboration reaches 0.37 GB, indicating higher memory pressure for Self-Collaboration. For disk write I/O, Self-Collaboration produces markedly higher write volume (112.5 MB) than the other frameworks, while MAS4SysML writes only 14.7 MB, reflecting more conservative storage activity.
Network overhead shows a similar pattern. Self-Collaboration yields the largest transmission volume (34.3 MB), likely due to repeated inter-agent exchanges, whereas MAS4SysML transfers 12.9 MB, which is lower than the other multi-agent baselines. This result suggests that MAS4SysML's collaboration protocol is comparatively communication-efficient.
Overall, MAS4SysML exhibits lower overhead than the other multi-agent code-generation frameworks considered in this study. We attribute this advantage primarily to its task-structure decomposition strategy: by organizing complex modeling requests into hierarchically structured subtasks, MAS4SysML can generate code incrementally under explicit constraints, reducing unnecessary regeneration and redundant reasoning, and thereby lowering overall computational and resource costs.
Ablation evaluation
In this section, we systematically analyze the contribution of each key agent in the MAS4SysML framework through a series of ablation studies. To this end, we compare the full MAS4SysML configuration with the following five variants: (1) w/o Task Planning: this variant removes the task-structure generation agent. The model directly generates code from the natural-language modeling intent and is followed only by syntax checking, error repair, and semantic validation. (2) w/o Syntax Validation: this variant omits the syntax-validation module and performs semantic validation directly after code generation. (3) w/o Code Repair: this variant excludes the code-repair agent. Errors detected by the syntax validator are not automatically corrected but are instead passed back to the code-generation agent, which attempts self-correction in the next iteration. (4) w/o Semantic Validation: this variant performs syntax-level verification only, without semantic-consistency checking, and is used to assess structural completeness and intent alignment when semantic constraints are absent. (5) Direct: this baseline includes no planning, validation, or repair. The model generates the full code in a single pass and serves as a reference for quantifying the performance improvements brought by the complete MAS4SysML framework.
As shown in Figure 4, all key components of the MAS4SysML framework contribute substantially to overall performance. When the task structure generation agent is removed, SER rises to 3.84 and SCS falls to 0.82, indicating that without the task tree, the model's ability to interpret hierarchical structures and cross-element dependencies is substantially weakened, increasing its susceptibility to structural and syntactic errors in complex models. The syntax validation module exerts the most pronounced impact: removing it causes SER to surge to 6.78, with errors primarily involving missing structural elements, omitted keywords, and syntactic conflicts. Eliminating the code repair agent also raises SER to 5.12, demonstrating that deterministic verification and repair are essential for preventing error accumulation and maintaining generation stability. In contrast, removing the semantic validation agent has a smaller effect on SER, but SCS declines markedly to 0.75. Further inspection shows that although the generated code passes syntactic validation under this condition, it frequently exhibits semantic deviations such as missing constraints, incomplete role relationships, or unwarranted extensions. This highlights the critical role of semantic validation in maintaining logical coherence and ensuring faithful alignment with the modeling intent.
Overall, the components operate in a complementary manner: the task structure generation agent provides hierarchical task organization and semantic dependency parsing; the syntax validation module and code repair agent ensure structural completeness and compilability; and the semantic validation agent enforces global semantic alignment and consistency. Together, these components enable MAS4SysML to achieve stable, high-quality automated SysML v2 code generation for complex modeling tasks.
Impact of maximum repair iterations
To evaluate the impact of the maximum number of repair iterations Kmax on final code quality, we conducted a controlled comparison by varying its value. The results, shown in Figure 5, reveal a clear downward trend in SER as Kmax increases. The most substantial improvement occurs when Kmax increases from 0 to 1: SER drops from 6.31 to 3.62, indicating that even a single repair iteration can correct a large proportion of syntax errors. Meanwhile, SCS also increases markedly, suggesting that the repair mechanism enhances not only structural correctness but also semantic alignment between the generated code and the modeling intent. As Kmax continues to increase, performance improvements persist but gradually diminish. With Kmax=2, SER decreases by an additional 0.84 compared with the previous setting, although the SCS gain becomes modest. At Kmax=3, performance stabilizes: SER decreases by only 0.16, and SCS drops slightly by 0.02, while the computational cost for generation and validation increases noticeably. Overall, the results indicate that multiple repair iterations can effectively improve code-generation quality, but Kmax=2 provides the best balance between performance gains and computational cost. Therefore, setting the maximum number of repair iterations to 2 represents the optimal choice under the trade-off between quality and efficiency.
We further examined the small number of cases that still failed to fully satisfy the validation criteria under Kmax=3. These cases were primarily associated with (i) incomplete task-card specifications resulting from implicit or underspecified modeling intent, (ii) structurally complex initial generations involving nested declarations, references, or connection logic, and (iii) syntactically valid yet semantically incomplete outputs that could not be fully corrected through explicit diagnostic feedback alone. This finding suggests that increasing the number of repair iterations is effective for resolving explicit syntax-related issues; however, its effectiveness diminishes when residual errors stem from upstream intent interpretation or deeper semantic incompleteness.
Human evaluation
We adopt the human assessment metrics described earlier to subjectively score the generated SysML v2 model code. Twenty modeling tasks were randomly selected from the dataset, and five code versions were prepared for each task: True, Self-Debugging16, MapCoder17, Self-Collaboration18, and MAS4SysML. During evaluation, each developer reviewed the five code versions for the same task simultaneously and assigned comparative scores. A total of ten developers with 2–5 years of SysML modeling experience participated in the study. The evaluation was conducted using anonymous questionnaires, and the presentation order of the five methods was randomized to mitigate potential bias.
The human evaluation results are shown in Figure 6. In terms of correctness, MAS4SysML exhibits a score distribution closest to the True baseline, with most scores concentrated in the 2.5–3 range, indicating that its generated code is highly reliable in terms of structural consistency, semantic accuracy, and constraint logic. In contrast, Self-Debugging and MapCoder show more dispersed distributions with a greater number of scores in the 1–2 range, reflecting issues such as logical omissions and inconsistent constraints. Regarding readability, MAS4SysML again performs best, producing code with more standardized naming, clearer hierarchical organization, and more coherent logical structure. Self-Collaboration benefits from its multi-role cooperation mechanism, yielding improved structural clarity and ranking second in readability. For integrity, MAS4SysML leverages hierarchical task decomposition and semantic validation to substantially reduce missing fields, incomplete constraints, and undefined elements, with scores concentrated between 2 and 3. Although Self-Debugging can address some local errors, it lacks sufficient coverage of modeling intent and cross element dependencies, resulting in the lowest integrity scores. Self-Collaboration and MapCoder, despite their collaborative advantages, frequently produce incomplete attributes, interfaces, or binding relations, leading to scores predominantly in the 1–2 range.
Overall, the human evaluation demonstrates that MAS4SysML achieves the highest performance across correctness, readability, and integrity, with scores approaching those of the True baseline. This confirms that MAS4SysML can achieve modeling quality comparable to human created SysML V2 models in practical modeling tasks.
Case study
We conduct qualitative evaluations through representative case studies to assess the effectiveness of MAS4SysML across diverse modeling scenarios. The first case considers structural-model generation for a smart home system and demonstrates how multiple agents collaborate throughout the modeling workflow. The second case addresses requirements, structural, and behavioral modeling for a launch-vehicle propulsion subsystem, encompassing requirements modeling, structural modeling, and state-machine modeling, to evaluate the method's ability to capture diverse modeling objectives in complex equipment systems.
In the smart home structural modeling task, the agents within MAS4SysML sequentially execute task parsing, code generation, syntax validation, code repair, and semantic-consistency verification (as illustrated in Figure 7). (1) The task structure generation agent constructs a task tree based on requirement semantics, decomposing the overall modeling objective into structured task cards that define explicit modeling goals and semantic constraints. (2) The code generation agent produces model code in a layer-by-layer manner according to the task cards, progressively extending previously generated content to maintain structural continuity and contextual coherence. (3) The syntax validation module uses the official SysML v2 validation environment to perform formal checks on the generated code. During one of these checks, it detected a syntax error on line 6, indicating that "HomeState" was undefined. (4) The code repair agent then modifies the code precisely based on the validation feedback, correcting the erroneous reference from "HomeState" to the correct type "HomeStatus." (5) Finally, the semantic validation agent compares the generated code with the task cards and confirms that the model satisfies both structural and semantic requirements, thereby completing the end-to-end generation process.
In the second case study, MAS4SysML is applied to the modeling task of a launch-vehicle propulsion subsystem. The evaluation considers three core task types: requirements modeling, structural modeling, and behavioral modeling. Together, these subcases capture key design information from the perspectives of requirements, architecture, and control logic, and are used to assess the proposed framework's ability to automate modeling across heterogeneous task types.
Requirements modeling. First, we construct a requirements view of the propulsion subsystem to establish a requirements baseline for subsequent structural and behavioral modeling. In this task, MAS4SysML is applied to specify the subsystem's thrust-provision capability during main-stage operation, critical safety constraints, and control responsiveness. The task specification is as follows: Construct the requirements view of the propulsion subsystem and define four key requirements: (i) during main-stage operation, the propulsion subsystem shall provide at least the target thrust and support thrust build-up and stable operation; (ii) the combustion-chamber pressure shall remain within the specified range, and protection or shutdown shall be triggered if it exceeds this range; (iii) the propulsion subsystem shall implement interlocks for ignition, propellant supply, and shutdown, and support abort under abnormal conditions; and (iv) the propulsion subsystem shall respond to flight control commands to execute ignition, throttling, and shutdown, supporting closed loop control execution. The generated model code is visualized using Graphviz, as shown in Figure 8.
Structural modeling. Second, the structural modeling task identifies key internal components and their interaction relationships. MAS4SysML is used to model the propellant tank, rocket engine, and engine controller, together with the associated material transfer and information-interaction links. The task specification is as follows: Construct the structural diagram of the launch-vehicle propulsion subsystem. The subsystem comprises three components: a propellant tank, a rocket engine, and an engine controller. The tank supplies propellant to the engine, and the engine converts propellant into thrust under controller-issued commands. The propulsion subsystem receives external control commands; these commands are first routed to the controller, which generates executable control commands and transmits them to the engine. Meanwhile, the controller outputs engine status to external interfaces as telemetry information. The resulting model is shown in Figure 9.
Behavioral modeling. Finally, we construct a state-machine model for the engine control logic to describe state evolution and safety handling strategies under different commands and health conditions. MAS4SysML is applied to formalize the state sequence, transition guards, and abnormal handling paths. The task specification is as follows: Build a state machine model for the engine control logic of the launch vehicle propulsion subsystem. The engine starts in the Off state, and the external command interface issues three commands: Start, Shutdown, and Abort. Upon receiving Start under nominal health status, the engine transitions through Pre cooling, Ignition, and Thrust ramp up, and then enters Steady state operation. Entry into steady state is permitted only if, during thrust ramp up, the combustion-chamber pressure is detected within the specified lower and upper bounds. If Shutdown is issued during steady state, the engine enters the shutdown process and returns to Off only after thrust drops to zero. At any stage, if an Abort command is received or health status degrades, the engine shall immediately transition to Abort to execute protective actions; when conditions permit, it may return to Off via Shutdown. The resulting model is shown in Figure 10.
Data Availability
All raw data generated and analyzed during this study have been made publicly available. The dataset is deposited on Zenodo with DOI: 10.5281/zenodo.18439316.

Figure 1: MAS4SysML framework. The framework utilizes multi-agent collaboration to perform task decomposition, code generation, syntax validation, error repair, and semantic consistency checking, enabling the automated transformation from modeling intent to SysML v2 code. Please click here to view a larger version of this figure.

Figure 2: Prompt templates for agents in the MAS4SysML framework. This figure presents the prompt templates used by the four core agent types within the framework, specifying the input structure, generation objectives, and operational rules for each agent. Please click here to view a larger version of this figure.

Figure 3: Illustration of the hierarchical task decomposition mechanism. This figure shows how the task structure generation agent iteratively decomposes the overall modeling objective into structured subtasks through a task tree. Each node corresponds to a task card that provides clear semantic constraints and operational guidance for subsequent automatic code generation. Please click here to view a larger version of this figure.

Figure 4: Ablation result. This figure shows how the model's performance changes when different modules are removed, including variations in SER and SCS. Please click here to view a larger version of this figure.

Figure 5: Impact of maximum repair attempts. This figure illustrates the trends in SER and SCS under different maximum repair attempt settings Kmax. Please click here to view a larger version of this figure.

Figure 6: Human evaluation results. This figure presents the score distributions given by human reviewers for different code generation methods across three evaluation criteria: readability, correctness, and integrity. Please click here to view a larger version of this figure.

Figure 7: Structural model generation process for a smart home system. This figure illustrates the complete execution process of MAS4SysML in the structural modeling task for a smart home system, including task structure generation, code generation, syntax validation, code repair, and semantic validation. Please click here to view a larger version of this figure.

Figure 8: Requirements modeling view of the launch vehicle propulsion subsystem generated by MAS4SysML. This figure presents the propulsion subsystem requirements modeling view automatically generated by MAS4SysML. It includes key requirements such as thrust requirements, combustion-chamber pressure constraints, interlock/abort safety requirements, and command-response requirements, and represents them in a structured manner using a unified Requirement definition. Please click here to view a larger version of this figure.

Figure 9: Structural model of the launch vehicle propulsion subsystem generated by MAS4SysML.This figure shows the structural decomposition and interaction relationships of the propulsion subsystem, including three core components:PropellantTanks, RocketEngine, and EngineController. Please click here to view a larger version of this figure.

Figure 10: State machine model for engine control logic generated by MAS4SysML.This figure illustrates the state machine model of the engine control logic, consisting of states Off, Chilldown, Ignition, RampUp, Steady, Shutdown, and Abort, together with state transitions triggered by Start/Shutdown/Abort commands and the health status. Entry into the steady-state is constrained by combustion-chamber pressure bounds, and completion of shutdown is constrained by the thrust dropping to zero. Please click here to view a larger version of this figure.
| Model type | Description |
| Requirements Model | Construct a requirements diagram for a home security system focusing on intrusion detection and alarm response. The system is intended to identify unauthorized entry in real time and promptly notify the user and security service when an intrusion occurs. In addition, the system must include mechanisms to reduce false alarms caused by pets or environmental disturbances. |
| Use Case Model | Construct a use case diagram for a satellite telemetry and telecontrol system. The system subject is an on-orbit satellite, which exchanges status information and control commands with the ground station and the mission control center. The satellite must perform three primary tasks: provide operational status and health parameters to the ground station, receive and execute telecommands from the mission control center, and maintain or adjust its attitude and orbit according to mission requirements. |
| Structure Model | Construct a system structure diagram for an unmanned aerial vehicle (UAV). The system is composed of an avionics subsystem, a propulsion subsystem, a mission payload subsystem, a communication subsystem, and a power subsystem. These subsystems are responsible for flight control, thrust generation, mission execution, ground communication, and energy supply, respectively. The model should also define the data link and control interfaces between the UAV and the ground control station. |
| Parametric Model | Construct a parametric diagram that describes the endurance performance constraints of a small UAV during mission execution. The model takes the UAV’s battery capacity, baseline power consumption, flight speed, aerodynamic drag coefficient, and payload weight as input parameters, and uses a power consumption model to represent the influence of these factors on actual power usage. Based on this, the endurance time is determined by the proportional relationship between battery capacity and actual power consumption. |
| State Machine Model | Construct a state machine diagram representing the launch process of a launch vehicle from pre-launch preparation to orbital insertion. The system subject is the launch vehicle executing the mission, and the mission process is composed of sequential phases including pre-launch checks, propellant fueling, countdown, ignition, liftoff, ascent, stage separation, and orbital insertion. Additionally, a safety mechanism is included that triggers an abort sequence if any anomaly occurs during any stage of the process. |
Table 1: Examples of modeling intents for different model types. This table presents modeling intent examples for various model types,including requirements, use case, structural, parametric, and state machine models. These examples were automatically generated by GPT-4o based on manually constructed model code and subsequently validated by human review.
| Approach | SER | ΔSER | SCS | ΔSCS |
| Direct Generation (Baseline) | | | | |
| CodeX (175B) | 10.16 | 1.49 | 0.71 | −0.02 |
| PaLM Coder (62B) | 15.34 | 6.67 | 0.62 | −0.11 |
| CodeGen-Mono (16.1B) | 23.63 | 14.96 | 0.61 | −0.12 |
| Incoder (6.7B) | 29.51 | 20.84 | 0.57 | −0.16 |
| AlphaCode (1.1B) | 31 | 22.33 | 0.54 | −0.19 |
| code-davinci-002 (175B) | 8.67 | 0 | 0.73 | 0 |
| Methods Based on code-davinci-002 | | | | |
| CodeCoT | 4.98 | −3.69 | 0.72 | −0.01 |
| Self-Planning | 3.82 | −4.85 | 0.77 | 0.04 |
| Self-Edit | 3.61 | −5.06 | 0.75 | 0.02 |
| CodeChain | 3.23 | −5.44 | 0.78 | 0.05 |
| Self-Debugging | 3.65 | −5.02 | 0.83 | 0.1 |
| MapCoder | 3.42 | −5.25 | 0.88 | 0.15 |
| Self-Collaboration | 3.38 | −5.29 | 0.87 | 0.14 |
| MAS4SysML | 2.63 | −6.04 | 0.91 | 0.18 |
Table 2: Baseline model evaluation. This table presents the performance comparison of various code generation models on SysML v2 automatic modeling tasks. The evaluation includes SER and SCS metrics, as well as their changes relative to the baseline model (ΔSER and ΔSCS), providing a comprehensive assessment of each model's syntactic correctness and semantic alignment.
| Model | SER | SCS | SER (Direct) | SCS (Direct) |
| Fastchat (3B) | 32.83 | 0.54 | 34.42 | 0.53 |
| ChatGLM (6B) | 32.51 | 0.51 | 32.24 | 0.51 |
| MPT (7B) | 31.75 | 0.57 | 33.98 | 0.56 |
| Vicuna (7B) | 28.23 | 0.53 | 30.76 | 0.54 |
| Vicuna (13B) | 23.93 | 0.66 | 28.32 | 0.57 |
| HuggingChat (30B) | 20.21 | 0.78 | 25.77 | 0.62 |
| Dromedary (65B) | 12.96 | 0.82 | 16.85 | 0.63 |
| GPT-4o | 3.78 | 0.85 | 7.23 | 0.81 |
Table 3: Performance analysis of different mainstream conversational LLMs. This table compares the performance of several mainstream conversational LLMs on SysML v2 automatic modeling tasks.
| Model | SER | SCS | SER (Direct) | SCS (Direct) |
| code-davinci-002(175B) | 2.63 | 0.91 | 8.67 | 0.71 |
| text-davinci-002(175B) | 3.12 | 0.9 | 9.21 | 0.74 |
| code-cushman-001(13B) | 15.98 | 0.71 | 18.64 | 0.6 |
| Code LLaMA(34B) | 9.83 | 0.75 | 13.21 | 0.67 |
| Code LLaMA(13B) | 18.64 | 0.67 | 19.86 | 0.61 |
| Code LLaMA(7B) | 25.41 | 0.64 | 27.93 | 0.58 |
Table 4: Performance analysis of code generation LLMs. This table compares the performance of several representative code generation LLMs on SysML v2 automatic modeling tasks.
| Method | Run Time (s) | CPU Usage (%) | Memory Usage (GB) | Disk Write (MB) | Net Traffic (MB) |
| Self-Debugging | 168.4 | 0.8 | 0.16 | 21.9 | 7.4 |
| MapCoder | 194.7 | 1.2 | 0.19 | 46.3 | 28.1 |
| Self-Collaboration | 231.5 | 2.3 | 0.37 | 112.5 | 34.3 |
| MAS4SysML | 174.6 | 1.2 | 0.17 | 14.7 | 12.9 |
Table 5: Runtime overhead comparison of different methods in the prototype system experiments. This table summarizes the resource consumption of each method across multiple overhead metrics.
Supplementary File 1: Automatic model code generation process based on the MAS4SysML. This table presents the overall automatic code generation workflow of MAS4SysML.Please click here to download this file.
Supplementary File 2: Prompt template for modeling intent generation. This table presents the prompt template used to generate natural-language modeling intent from SysML v2 model code. The template specifies rules for describing modeling goals, identifying core system elements, and summarizing functional or behavioral logic, while explicitly prohibiting line-by-line code enumeration or the direct use of SysML specific terminology, ensuring the generated intent descriptions are concise and semantically accurate.Please click here to download this file.
Supplementary File 3: Python script of the MAS4SysML method. This file provides the full implementation details of the MAS4SysML method, covering the core modules, including task structure generation, code generation, syntax validation, code repair, and semantic verification, and the key functions that support the complete workflow of automated SysML v2 model code generation.Please click here to download this file.