Automated User Interface Prototyping Framework Integrating Cognitive and Visual Design Principles to Improve Usability and Layout Clarity

Summary

This study presents an automated user interface prototyping framework integrating cognitive design principles, perceptual color modeling, and deep learning. The system improves accessibility, layout clarity, color harmony, and cross-screen consistency, resulting in coherent, user-centered interface designs.

Abstract

Effective user interface (UI) design requires a harmonious balance between visual appeal, cognitive usability, and layout consistency. However, current automatic UI generation approaches primarily focus on visual appearance or component detection and lack a unified framework that integrates cognitive principles, color intelligence, and structural reasoning. Furthermore, existing methods suffer from limited layout generalization, poor interpretability, static color selection, and inconsistent behavior across screens. In this context, this work proposes an automated UI prototyping framework that integrates the Faster region-based convolutional neural network (Faster R-CNN)-based component detection and CIECAM02 uniform color space (CAM02-UCS)-driven perceptual color modeling, enriched with cognitive and visual design principles. The Faster R-CNN is used to identify UI components and infer hierarchical structures from large-scale interface datasets. An enhanced color generation module analyzes brand or reference images to ensure perceptually uniform, harmonious, and usability-compliant color themes using CAM02-UCS. These outputs are further optimized through cognitive design rules, including Gestalt grouping, Fitts’ and Hick’s laws, attention-based spacing, and visual hierarchy modeling, to automatically generate refined, task-oriented UI layouts. Experiments conducted on RICO, ENRICO, and Guo’s UI Color Datasets show that the proposed system achieves 92.4% component detection accuracy, improves layout clarity and reading order accuracy by 18.7%, and produces color palettes rated 24.5% more harmonious by designers compared to baseline methods. User evaluations also indicate a 31% reduction in perceived cognitive load and a 28% increase in design consistency across screens. These findings demonstrate that combining deep learning–based structural understanding with perceptually grounded color modeling and cognitive design principles produces UI prototypes that are highly efficient, aesthetically coherent, and user-friendly. This framework establishes a novel, end-to-end approach to intelligent and human-centered automated UI prototyping.

Introduction

Graphical user interfaces (GUIs) serve as a fundamental medium of communication between humans and computer systems, significantly influencing usability, accessibility, and overall user experience (UX)^1,2. Modern software systems rely on intuitive and visually appealing interfaces to attract and retain users. Traditionally, UI design involves creating graphical layouts that are later translated into front-end code by engineers. However, differences in platform-specific protocols across operating systems require repeated adaptation, increasing development complexity and effort. Automated UI code generation has therefore emerged as an important research direction, reducing manual effort and improving development efficiency.

Early approaches to automated UI generation combine traditional image processing techniques with deep learning models for region detection and classification^3,4. Currently, dominant strategies for GUI code generation apply computer vision techniques to analyze UI images and convert them into structured front-end representations^5,6,7. While image processing methods rely on handcrafted features, deep learning approaches extract hierarchical representations from large-scale datasets. As a result, researchers have explored hybrid approaches that combine deep learning with domain-specific image processing techniques to improve robustness and accuracy.

Color is another critical factor in UI design, influencing user perception, usability, and aesthetic appeal¹. Maintaining consistency between image content and UI color schemes becomes challenging when visual inputs vary in hue and context^8,9,10. Consequently, significant research has focused on automated color palette generation and perceptual color modeling. Existing methods for color generation include techniques based on the CIELAB color space¹¹. Other approaches use clustering algorithms, such as k-means, to extract dominant colors from images¹². As a result, recent work has increasingly adopted data-driven, machine learning–based approaches for color modeling.

In parallel, deep learning approaches have significantly improved UI component detection and layout understanding. Neural networks have enabled more accurate structural parsing of mobile interfaces¹³. However, these methods primarily focus on detection accuracy and often overlook higher-level aspects such as cognitive usability, layout hierarchy, and interaction efficiency. Generative models for UI layout synthesis have also been proposed^14,15, but they face challenges in maintaining cross-screen consistency and providing interpretable design decisions¹⁶. Other approaches focus on visual similarity or rendering quality but lack a unified framework that combines component semantics, color harmony, and usability constraints¹⁷. Text recognition is also important for understanding UI content. Techniques such as convolutional recurrent neural networks (CRNNs) enable accurate recognition of complex text patterns in interface screens^18,19,20.

Several systems have been proposed to generate UI code from sketches or images. Sketch2Code uses object detection to convert sketches into code representations^21,22, but it is sensitive to image quality. REDRAW provides an automated pipeline for data collection and model training, combining convolutional and recurrent neural networks to generate structured UI representations^23,24. Subsequent work has introduced improved evaluation metrics to assess generated UI quality²⁵. More recent approaches include diffusion-based and transformer-based models for layout generation, such as diffusion layout transformers without autoencoder methods, GUI layout generation using transformer-based GUI arrangement graphs, and large language model–guided UI layout generation^26,27,28. A comparative overview of these approaches is provided in Table 1.

References and Key Method(s)	Objective	Advantages	Disadvantages
Automated Image-Based User Interface Color Theme Generation: prominent-color extraction + CAM02-UCS perceptual color modeling¹	Automatically generate UI color themes from reference images while optimizing harmony and usability.	Strong perceptual grounding (CAM02-UCS); explicitly balances harmony and usability (contrast and diversity); includes web-based implementation and designer evaluation.	Focused only on color/theme (not layout or component structure); rule/heuristic-based rather than end-to-end learned UI synthesis.
Unifying Layout Generation with a Decoupled Diffusion Model (LDGM)²⁵	Unified, flexible layout generation for graphic scenes and UIs.	State-of-the-art diversity and controllability for layout generation; supports conditional generation and multiple attribute types.	Diffusion-based outputs may exhibit alignment or fine-grained layout issues unless constrained; higher model complexity and inference cost.
Diffusion Layout Transformers without Autoencoder Methods: transformer + diffusion for layout generation (no autoencoder)²⁶	Improve fidelity and alignment for layout generation benchmarks (FID, alignment, and overlap metrics).	Better capture of semantic correlations between neighboring objects; strong performance on FID and alignment metrics.	Primarily focuses on layout geometry (positions, sizes, categories) rather than UI semantics.
GUI Layout Generation with Transformer-based Models from GUI Arrangement Graphs (GUI-AGs)²⁷	Create GUI layouts from positional/graph constraints to assist designers.	Graph representations capture relational constraints; transformer enables flexible, constraint-conditioned generation.	May require explicit constraint graphs from designers; limited emphasis on color and usability metrics.
UI Layout Generation with LLMs Guided by UI Grammar: Large Language Models (LLMs) + hierarchical UI grammar²⁸	Explore LLMs for layout generation and improve explainability/control via grammar representations.	High-level controllability and explainability; can leverage in-context learning from LLMs (GPT-type).	Early-stage work with limited empirical validation; grammar design and robustness across UI s

Table 1: Comparison of existing UI color modeling and layout generation methods. The table summarizes representative approaches for user interface design, including color theme generation, diffusion-based layout generation, transformer-based methods, and large language model–guided layout generation. For each method, the underlying technique, objective, advantages, and limitations are presented, highlighting differences in perceptual modeling, layout generation capability, controllability, and usability considerations.

Despite these advances, a key limitation remains: no existing system jointly integrates component detection, perceptual color modeling, cognitive design principles, and multiscreen layout consistency within a single end-to-end framework¹. Current approaches typically optimize only one or two aspects—such as detection, layout, or color—without considering their interdependencies. This fragmentation highlights a critical research gap in developing unified, human-centered automated UI prototyping systems. In particular, cognitive design principles such as Gestalt laws, Fitts’ law, and Hick’s law are often treated conceptually rather than being formally integrated into computational models.

To address these limitations, this study proposes an end-to-end automated UI prototyping framework that integrates component detection, perceptual color modeling, and cognitive design principles within a unified system. The framework is designed to improve layout quality, reduce cognitive load, enhance color harmony, and increase overall usability. These improvements are validated through experiments on the RICO, ENRICO, and Guo’s UI Color Datasets, demonstrating the effectiveness of combining structural, perceptual, and cognitive aspects within a single automated UI prototyping pipeline. It is hypothesized that integrating deep learning–based component detection, perceptual color modeling, and cognitive design principles within a unified framework will significantly improve UI prototype quality compared to existing approaches. Specifically, H1 proposes that the framework enhances layout quality, including alignment, grouping, and reading order. H2 posits that the framework reduces user cognitive load. H3 suggests that perceptual color modeling improves color coordination and visual harmony. H4 states that the overall usability and aesthetic quality of UI prototypes are improved.

Protocol

This study uses publicly available datasets and does not involve human or animal subjects.

The proposed automated UI prototyping framework integrates deep learning–based structural understanding, perceptually uniform color modeling, and cognitive design principles to generate coherent and user-centered interface prototypes. The methodology begins with a Faster R-CNN architecture trained on the RICO and ENRICO datasets to detect UI components and extract their hierarchical relationships, enabling accurate interpretation of diverse screen layouts. The detected components are then processed by a color intelligence module, which extracts brand- or image-based color cues, models perceptual similarity using CAM02-UCS, and generates harmonious, contrast-aware, and accessibility-compliant color palettes. Subsequently, cognitive design principles—including Gestalt grouping laws, Fitts’ law, Hick’s law, attention-based spacing, and visual hierarchy constraints—are applied through a cognitive optimization engine to refine spatial organization and component alignment. Finally, a layout reasoning layer integrates structural, perceptual, and cognitive constraints to generate multiscreen consistent UI prototypes that balance usability, aesthetics, and functional clarity. This integrated pipeline ensures perceptual coherence, reduces cognitive load, and adheres to human-centered design principles. The complete workflow of the proposed method is illustrated in Figure 1.

Figure 1. Overall architecture of the proposed automated user interface prototyping framework. The diagram illustrates the complete pipeline, starting from an input UI image. Component detection is performed using Faster R-CNN to identify interface elements. The detected components are then processed using CAM02-UCS-based perceptual modeling for hierarchy and layout extraction. Cognitive optimization is subsequently applied using Gestalt principles, Fitts’ law, Hick’s law, and attention-based adjustments. Arrows indicate the flow of data between modules, resulting in the final UI prototype output. Please click here to view a larger version of this figure.

Overview of the framework

Figure 1 illustrates the end-to-end workflow of the proposed automated UI prototyping framework, which transforms a raw UI screenshot into a cognitively refined prototype. The process begins with the input UI image, which is passed to the component detection module based on Faster R-CNN. This module identifies interface elements such as buttons, icons, text fields, and containers, and outputs their corresponding bounding boxes and class labels. The detected components are then processed in the hierarchy and layout extraction stage. Based on spatial relationships and structural patterns, the system constructs a hierarchical layout tree that reflects how users naturally perceive screen organization. This representation captures visual grouping and structural dependencies among UI elements. The extracted layout is subsequently refined through cognitive optimization by applying human-centered design principles. These include Gestalt grouping for visual clustering, Fitts’ law for optimizing touch-target size and accessibility, Hick’s law for reducing decision complexity, and attention-based spacing for improving visual hierarchy. As a result, the layout becomes more intuitive, readable, and efficient for user interaction. Finally, the optimized layout is combined with perceptual color modeling to generate a coherent UI prototype. The resulting interface is structurally accurate, visually harmonious, and aligned with human usability expectations.

The framework was implemented using PyTorch 2.0 on Ubuntu 22.04. Experiments were conducted on a system equipped with an NVIDIA RTX 4090 GPU (24 GB VRAM). The Faster R-CNN model employed a ResNet-50 backbone pre-trained on the ImageNet dataset. Training was performed using the stochastic gradient descent (SGD) optimizer with a learning rate of 0.001, batch size of 16, and up to 60 epochs. Early stopping was applied to prevent overfitting using a validation set comprising 20% of the data. Although training was performed for up to 60 epochs, convergence was typically achieved earlier through early stopping based on validation loss. Momentum and weight decay were set to 0.9 and 0.0005, respectively. Data augmentation included normalization, random horizontal flipping, and random cropping and resizing.

Dataset preparation

To support the proposed automated UI prototyping framework, three complementary datasets were utilized: ENRICO²⁹, RICO³⁰, and Guo’s UI Color Dataset¹. The RICO dataset contains 66,261 Android UI screens collected from 9,700 applications, providing large-scale diversity for component detection and hierarchical layout extraction. ENRICO includes 1,464 high-quality UI screenshots manually categorized into 20 design patterns, enabling improved generalization for structural reasoning and layout consistency modeling. Guo’s UI Color Dataset consists of 128 curated UI images with annotated color themes, which are used for perceptual color modeling and palette evaluation. However, due to its relatively smaller size, this dataset may introduce some variability in layout characteristics. For this study, a combined dataset of 5,128 screens was prepared for training and evaluation. Specifically, approximately 4,000 images from RICO were used to train the Faster R-CNN model for component detection, 1,000 images from ENRICO were used for layout pattern learning and structural consistency modeling, and 128 images from Guo’s dataset were used for color evaluation tasks. All images were normalized to a resolution of 480 × 800 pixels, converted to a uniform sRGB color space, and re-annotated with standardized bounding boxes and hierarchical metadata to ensure compatibility across datasets. An overview of the datasets used in this study is provided in Table 2. The RICO dataset supports large-scale detection of UI components and structural analysis, while ENRICO enhances the model’s ability to recognize layout patterns and enforce cross-screen consistency. Guo’s UI Color Dataset, although smaller in size, plays a critical role in evaluating perceptual color harmony and contrast modeling. Together, these datasets provide a comprehensive and well-balanced foundation for training, validation, and evaluation of the proposed automated UI prototyping framework.

Dataset	Total Images Available	Images Used in Study	Purpose in Proposed Framework
RICO Dataset30	66,261	4,000	UI component detection, structural layout extraction
ENRICO Dataset29	1,464	1,000	Design pattern learning, layout consistency modeling
Guo’s UI Color Dataset1	128	128	Color theme extraction, perceptual color evaluation
Combined Total	67,853	5,128	Comprehensive dataset for detection, layout, color modeling

Table 2: Summary of datasets used in the proposed framework. The table presents the datasets utilized for training and evaluation, including the RICO, ENRICO, and Guo’s UI Color Datasets. It reports the total number of images available, the subset used in this study, and the specific role of each dataset in component detection, layout modeling, and perceptual color analysis within the proposed framework.

A stratified sampling strategy was employed to preserve diversity in UI components, layout structures, and color distributions across RICO, ENRICO, and Guo datasets. The dataset was split into training (70%), validation (15%), and testing (15%) subsets using stratified sampling. Images were normalized using mean (0.485, 0.456, and 0.406) and standard deviation (0.229, 0.224, and 0.225). Data augmentation included random horizontal flipping (probability = 0.5) and random cropping (scale range: 0.8–1.0), followed by resizing to 224 × 224 pixels applied during training.

Component annotation and preprocessing

The RICO, ENRICO, and Guo’s UI Color Dataset collectively support the three core functional components of the proposed automated UI prototyping framework: component detection, layout modeling, and perceptual color optimization. The RICO dataset contains a large-scale collection of more than 66,000 mobile UI screens and is primarily used for training the component detection module. Its diversity enables the Faster R-CNN model to learn robust representations of common UI elements such as buttons, images, and text fields across a wide range of applications. This ensures accurate detection and localization of UI components under varying design conditions. The ENRICO dataset provides high-quality, pattern-labeled UI screens for learning structural relationships and layout patterns. It enables the system to understand inter-component relationships, hierarchical organization, and common design templates. This dataset plays a critical role in modeling layout structures and enforcing multiscreen consistency, allowing the framework to generate layouts that align with established design conventions. In contrast, Guo’s UI Color Dataset is specifically used for perceptual and cognitive color modeling. Unlike RICO and ENRICO, which focus on structural aspects, this dataset contains curated UI images with annotated color themes. These annotations are used to train color extraction and harmony evaluation modules. By mapping color relationships into perceptual color spaces such as CAM02-UCS, the framework learns to generate color palettes that balance contrast, accessibility, and aesthetic coherence. Together, these datasets form an integrated pipeline: RICO supports component detection, ENRICO enables layout pattern understanding and structural consistency, and Guo’s dataset facilitates perceptual color optimization. This integration ensures that the generated UI prototypes are structurally accurate, visually harmonious, and cognitively optimized.

Normalization was performed using a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225]. Data augmentation techniques, including random cropping, horizontal flipping, and rotation, were applied to improve model generalization. In addition, pixel values were scaled to the range [0, 1], and all images were resized to a resolution of 480 × 800 pixels to ensure consistency across datasets. Images resized to 224 × 224 pixels are used for augmentation during training, while the original resolution of 480 × 800 pixels is preserved for layout analysis. Bounding boxes were adjusted to filter out irrelevant annotations using predefined thresholds. To achieve uniformity across datasets, all UI elements were manually annotated using the LabelImg annotation tool. A predefined schema was established prior to annotation to classify UI elements into categories such as button, text, image, icon, and container. For bounding box representation, a uniform coordinate system was applied, and hierarchical information was constructed based on spatial relationships between elements and their parent–child associations within the layout tree. In addition, guidelines were established to ensure consistent annotation of elements, proper handling of overlapping components, and enforcement of minimum size thresholds across the RICO, ENRICO, and Guo datasets.

Detailed mathematical formulations for component detection training and layout pattern extraction are provided in Supplementary File 1.

The Faster R-CNN model was trained using SGD with a momentum of 0.9 and a weight decay of 0.0005. The initial learning rate was set to 0.001 and adjusted using a step decay policy based on validation performance. Training was performed for up to 60 epochs with a batch size of 16. Early stopping was applied to prevent overfitting, using a validation set comprising 20% of the training data.

The mapping function f(.) takes detected UI elements as input and produces a hierarchical layout representation as output. It computes adjacency relationships between UI elements based on spatial distance and alignment criteria, and constructs an adjacency matrix to represent these relationships. A clustering algorithm, such as DBSCAN, is then applied to group related components. Finally, clustered elements are organized into hierarchical structures based on parent–child relationships, forming a structured layout representation.

Detailed formulations for color harmony modeling and the unified optimization objective are provided in Supplementary File 1.

This objective function mathematically formalizes the integration of structural detection, layout reasoning, and perceptual color modeling, ensuring that all components of the framework contribute jointly to generating coherent and user-centered UI prototypes. Optimization is performed in a modular manner for each component of the framework. SGD is used for training the component detection module, while the Adam optimizer is used during joint optimization of the unified objective. The combined objective is used to guide overall system performance. For joint optimization stages, the objective function is minimized using the Adam optimizer with a learning rate of 0.001 and a batch size of 16. Training is performed for up to 60 epochs, with early stopping applied when the validation loss change is less than 10⁻⁴ for five consecutive epochs.

Component detection using Faster R-CNN

The proposed framework employs Faster R-CNN for automatic detection of UI components, including buttons, icons, text fields, images, and containers. Faster R-CNN is well suited for this task because it combines high object detection accuracy with efficient region proposal generation, which is essential for handling complex UI layouts containing densely packed elements. The model utilizes a ResNet-50 backbone pre-trained on the ImageNet dataset to extract convolutional feature maps from each UI screen. During training, the early layers were frozen to retain general visual features, while the higher layers (conv4_x and conv5_x) were fine-tuned for UI-specific component detection. These feature maps capture visual structures, edges, textures, and component boundaries. During detection, the region proposal network (RPN) identifies candidate regions (anchors) that are likely to contain UI components. Anchors are defined using multiple scales (128, 256, and 512) and aspect ratios (1:1, 1:2, and 2:1) to accommodate UI elements of varying sizes. During training, anchors with an intersection-over-union (IoU) greater than 0.7 with ground truth boxes are labeled as positive, while those with IoU less than 0.3 are labeled as negative; anchors with intermediate IoU values are ignored. Each anchor is assigned a probability indicating the presence of a component. Non-maximum suppression (NMS) is then applied to remove redundant and overlapping proposals, ensuring efficient localization of meaningful UI elements even in cluttered interfaces. Once candidate regions are generated, each proposal is classified into predefined component categories, and its bounding-box coordinates are refined. A region of interest (ROI) Align operation extracts fixed-size feature representations for each proposal, which are then processed by fully connected layers for classification and regression. The classification branch outputs class probabilities, while the regression branch predicts precise bounding-box coordinates. The entire Faster R-CNN model is trained end-to-end by optimizing a joint loss function that combines the RPN loss with the final detection loss. This enables the system to not only accurately recognize UI components but also precisely localize them across diverse interface structures. A detailed description of Faster R-CNN component detection, including the RPN stage, ROI classification, bounding-box refinement, and final optimization objective, is provided in Supplementary File 1.

Algorithm S1 provides the detailed component detection and structural extraction workflow (Supplementary File 1). It describes the complete process of automatic UI component detection and structural extraction from a raw screen image. The input image is first preprocessed and passed through a CNN backbone to extract deep visual features. These features are used by the RPN to generate candidate bounding boxes, which are then refined through classification and regression. NMS removes redundant detections to ensure accurate localization. Following detection, components are grouped based on spatial proximity using density-based spatial clustering of applications with noise (DBSCAN). This clustering step enables the construction of a hierarchical UI tree that captures parent–child relationships and logical groupings among components. This hierarchical representation forms the foundation for subsequent layout analysis and UI understanding. Figure 2 illustrates the Faster R-CNN component detection process on a mobile UI screen. The input interface (left) contains UI elements such as buttons and text blocks, which are automatically enclosed within bounding boxes. These detected regions are then classified into component categories (e.g., Button and Text), as indicated by labeled connections. The Faster R-CNN detection module (right) refines bounding-box coordinates, assigns semantic labels, and outputs structured component-level information. This step serves as a critical foundation for downstream processes, including layout extraction, cognitive optimization, and UI prototype generation, by providing accurate, semantically meaningful representations of UI components.

Figure 2. Component detection of user interface elements using Faster R-CNN. The figure illustrates the detection of UI components from an input interface screenshot. Regions of interest are identified using bounding boxes, and elements such as buttons and text fields are classified using Faster R-CNN. Arrows indicate the mapping of detected components to their corresponding semantic labels. Please click here to view a larger version of this figure.

Layout pattern extraction and hierarchical modeling

Following component detection using Faster R-CNN, each UI element is represented by a bounding box. However, these bounding boxes alone do not convey how elements are structurally organized within the interface, for example, which elements belong to headers, navigation bars, or grouped content regions. To address this limitation, the detected components are transformed into a logical UI tree, where each component is treated as a node and functionally or visually related components are grouped under common parent nodes. To identify meaningful layout groups, clustering techniques such as DBSCAN or hierarchical agglomerative clustering are applied. These methods analyze spatial proximity and alignment patterns among components to detect relationships between UI elements. In a manner analogous to human design practices, the system learns to recognize common structural patterns such as headers, footers, grids, and content blocks. Once grouping is established, additional structural properties—including alignment, grid organization, and reading order—are analyzed to construct a comprehensive layout representation. For example, components aligned in equal-width columns may indicate a card-based layout, whereas vertically stacked elements may indicate a form- or feed-based interface. By integrating clustering, spatial reasoning, and alignment rules, the framework constructs a hierarchical UI tree that captures both visual grouping and functional relationships. This hierarchical representation serves as the foundation for subsequent layout refinement and multiscreen consistency modeling, enabling the system to generate structured, coherent, and user-centered interface designs.

Detailed formulations for spatial distance and alignment similarity are provided in Supplementary File 1.

Clustering for layout grouping (DBSCAN)

To identify layout groups, DBSCAN is applied. DBSCAN uses two parameters: (1) ε = maximum distance between neighboring components and (2) = minimum number of components required to form a cluster. For this analysis, the DBSCAN parameters were empirically selected based on the normalized UI resolution (480 × 800 pixels). The neighborhood distance parameter was set to ε = 50 pixels to capture spatial proximity between adjacent UI components. The minimum number of points required to form a cluster was set to MinPts = 3 to avoid the formation of insignificant or spurious groups of UI components.

The detailed logical UI tree construction formulation is provided in Supplementary File 1.

Perceptual color modeling with CAM02-UCS

Perceptual color modeling ensures that colors used in the generated UI are harmonious, legible, and cognitively efficient. Dominant colors are extracted from input sources such as UI images using clustering techniques. Specifically, K-means clustering with and K-means++ initialization is applied for 300 iterations to obtain representative color palettes. However, the RGB color space does not accurately reflect human visual perception, meaning that numerically similar colors may appear perceptually different. To address this limitation, all extracted colors are transformed into CAM02-UCS, which is perceptually uniform. In this space, equal distances correspond to equal perceived color differences, making it suitable for modeling color harmony and contrast. Once mapped into CAM02-UCS, perceptual color differences are computed using Euclidean distance, enabling the system to quantify contrast, harmony, and variation between colors. Colors that are too similar are separated to improve legibility, while excessively contrasting colors are moderated to avoid visual discomfort. The generated palette also adheres to accessibility standards such as WCAG AA and AAA, ensuring sufficient contrast between foreground and background elements. Furthermore, color harmony is enforced by constraining palette relationships to complementary, analogous, or triadic schemes, depending on the intended UI theme. This ensures that the resulting color palette is not only visually appealing but also functionally accessible and cognitively optimized. To further refine the palette, an optimization process is applied under constraints of perceptual similarity, contrast, harmony, and brand consistency. A primary color is selected (e.g., from brand identity), and secondary colors are adjusted in terms of luminance and chroma to preserve visual hierarchy. A rule-based evaluation mechanism assesses candidate palettes based on perceptual distances and accessibility metrics, minimizing cognitive strain while maintaining aesthetic balance. As a result, the framework produces UI color schemes that exhibit professional-level coherence, readability, and perceptual consistency.

Detailed mathematical expressions for CAM02-UCS-based perceptual color modeling are provided in Supplementary File 1.

Algorithm S2 (Supplementary File 1) describes the CAM02-UCS-based perceptual color extraction and optimization workflow under harmony and accessibility constraints. Initially, dominant colors are extracted from the input image and transformed into CAM02-UCS to ensure that color differences correspond to human perception. Perceptual distances between colors are then computed and constrained within predefined bounds to maintain both clarity and harmony. Subsequently, accessibility is enforced by evaluating luminance contrast ratios according to WCAG guidelines. The final palette is obtained by optimizing a combined objective function that balances perceptual spacing, contrast requirements, and color harmony constraints. This ensures that the generated UI color schemes are not only visually appealing but also readable, accessible, and perceptually consistent.

Cognitive design optimization

Cognitive design optimization ensures that the automatically generated UI prototypes are not only visually consistent but also easy to understand and interact with. This module integrates key cognitive design principles, including Gestalt principles, Fitts’ law, and Hick’s law, to guide the arrangement, grouping, and spacing of UI components. Detailed mathematical formulations for cognitive design optimization are provided in Supplementary File 1.

Integrated cognitive optimization objective

The mathematical expression for the integrated cognitive optimization objective is provided in Supplementary File 1. Coefficients representing the importance of visual grouping, interaction efficiency, and decision complexity are denoted by alpha, beta, and gamma, respectively. In this study, the values of alpha, beta, and gamma were empirically determined using the validation dataset. For the present experiment, the coefficients were set as follows: α = 0.5, β = 0.3, and γ = 0.2. This configuration provides an effective balance between visual grouping, interaction efficiency, and decision simplicity.

Multiscreen consistency and UI generation

Multiscreen consistency modeling ensures that different screens within an application share a coherent visual identity, structural layout, and interaction pattern. As users navigate across screens, they develop expectations regarding element placement, typography, spacing, and visual hierarchy. To meet these expectations, the system analyzes cross-screen patterns using templates derived from the RICO and ENRICO datasets. These templates capture common UI structures such as home–detail layouts, list–detail patterns, multistep forms, and dashboard flows. By comparing component placement and grouping behavior, the system constructs a cross-screen structural profile that identifies which elements should remain consistent and which may vary. Once structural patterns are identified, the framework enforces consistency in typography scales, spacing ratios, grid layouts, and navigation anchors. For example, header bars maintain consistent height, primary buttons retain uniform size and alignment, and content blocks follow predictable visual ordering. This is achieved through a layout regularization process that evaluates each screen against global structural constraints. If a screen deviates from learned templates—such as inconsistent padding or misaligned titles—the system automatically adjusts the layout to restore coherence. These corrections help ensure a seamless UX and reduce cognitive switching costs across screens. The framework further analyzes the navigation graph between screens to maintain consistency in interaction flow. Common navigation patterns, including forward/backward transitions, tabbed navigation, and multistep workflows, are identified and enforced. Each transition is validated to ensure alignment with the user’s mental model, thereby improving usability and reducing confusion. As a result, the multiscreen consistency model minimizes visual noise, enhances familiarity, and promotes a unified interaction experience.

Detailed mathematical formulations for multiscreen consistency and UI generation are provided in Supplementary File 1.

Finally, the rendering engine generates visual outputs in formats such as SVG wireframes, HTML/CSS prototypes, or pixel-based UI mockups. The resulting UI screens exhibit correct reading order, harmonious color relationships, precise grid alignment, and consistent visual hierarchy across multiple screens.

Algorithm S3 provides the cognitive–visual layout optimization and multiscreen consistency workflow (Supplementary File 1). Algorithm S3 describes the integrated cognitive and structural optimization process used to generate user-friendly UI layouts. It combines perceptual grouping (Gestalt principles), interaction efficiency (Fitts’ law), and decision simplicity (Hick’s law) within a unified optimization framework. Initially, spatial relationships between components are evaluated to establish perceptual groupings. Interaction efficiency is then optimized by adjusting component size and placement, while decision complexity is controlled by limiting the number of choices presented to users. In addition, the algorithm enforces multiscreen consistency by aligning each screen with learned layout templates, ensuring uniformity in typography, spacing, and structural organization. A multiobjective optimization function then refines the layout by balancing alignment, spacing, and cognitive cost, resulting in an interface that is both visually coherent and cognitively efficient. Overall, this algorithm ensures that generated multiscreen layouts maintain high levels of visual consistency, usability, and perceptual clarity.

Results

The proposed automated UI prototyping framework was assessed using an integrated dataset of 5,128 UI screens from three sources: 4,000 RICO images, 1,000 ENRICO screens, and 128 color-annotated UI samples from Guo’s UI Color Dataset. This blended dataset provides a well-balanced benchmark for evaluating component detection accuracy, layout structural clarity, multiscreen consistency, and perceptual color harmony.

Experimental results demonstrate that the Faster R-CNN–based detection model achieves component-detection accuracy of 92.4% and generalizes effectively across diverse mobile UI styles. Furthermore, incorporating ENRICO pattern annotations enhances layout reasoning, resulting in a 18.7% increase in layout clarity and improved preservation of reading order. In color evaluation experiments, the CAM02-UCS-driven color modeling module generates palettes that are 24.5% more harmonious, according to designer assessments, and achieves higher perceptual uniformity compared to traditional RGB- and LAB-based palette generation methods. Moreover, multiscreen consistency modeling achieves a 28% improvement in cross-screen alignment and typography consistency. User studies involving 30 participants demonstrate a 31% reduction in perceived cognitive load when interacting with prototypes produced by the proposed system. Together, these results indicate that combining component detection, perceptual color modeling, and cognitive optimization produces UI prototypes that are not only structurally accurate but also visually coherent and cognitively efficient. Table 3 highlights the simulation environment and technical settings used to assess the proposed UI prototyping framework. The computationally intensive training procedures of Faster R-CNN and layout consistency modules were supported by a high-performance NVIDIA RTX 4090 GPU. All experiments were conducted in an Ubuntu 22.04 environment using PyTorch 2.0 for deep learning implementation.

Parameter	Configuration
Hardware	NVIDIA RTX 4090 GPU (24 GB), Intel i9 Processor, 64 GB RAM
Operating System	Ubuntu 22.04 LTS
Deep Learning Framework	PyTorch 2.0
Faster R-CNN Backbone	ResNet-50 + FPN
Image Resolution	480 × 800 (normalized across all datasets)
Training Batch Size	8
Optimizer	AdamW (LR = 1e−4, Weight Decay = 0.01)
Epochs	40
Color Modeling Space	CAM02-UCS
Clustering for Palette Extraction	K-means (k = 5)
Layout Clustering	DBSCAN (ε = 20, min_samples = 3)

Table 3: Implementation parameters and experimental configuration of the proposed framework. The table summarizes the hardware and software setup used for model training and evaluation, including computational resources, operating system, deep learning framework, model architecture, image resolution, training parameters, optimization strategy, color modeling space, and clustering methods used for palette extraction and layout grouping.

The Faster R-CNN architecture employs a ResNet-50 backbone with an FPN to detect a wide range of fine-grained UI components. All UI images are uniformly resized to 480 × 800 pixels prior to processing. Training is performed over 60 epochs using the SGD optimizer, with a batch size of 16 to ensure stable convergence. The color palette generator uses CAM02-UCS with K-means clustering, while DBSCAN is used for structural layout clustering. These technical settings ensure that the generated UI results are reproducible, stable, and high-fidelity. To provide a comprehensive assessment of the proposed automated UI prototyping framework, four categories of evaluation metrics are employed:

i) Component detection performance, analyzed using precision, recall, F1-score, intersection-over-union (IoU), and mean average precision (mAP), which quantify the accuracy of the Faster R-CNN model in identifying UI components across the RICO and ENRICO datasets;

ii) Layout clarity and structural consistency, measured through alignment error (AE), grouping accuracy, reading-order correctness, and cross-screen consistency scores derived from design-pattern constraints;

iii) Perceptual color quality, evaluated using CAM02-UCS perceptual distance (ΔE_UCS), WCAG 2.1 contrast ratios, diversity index, and color harmony scores inspired by Guo’s UI color evaluation framework;

iv) User-centered cognitive efficiency metrics, including cognitive load measured by the NASA task load index (NASA-TLX), aesthetic coherence ratings, and task completion efficiency.

These metrics enable evaluation of the framework not only in terms of detection accuracy but also in terms of perceptual harmony, usability quality, and cognitive experience.

Component detection metrics

Precision describes the model's exactness in predicting UI components without generating false positives. High precision implies that most predicted bounding boxes correspond to valid UI elements. Recall measures the model’s ability to identify all relevant UI components on a screen. High recall indicates that the model successfully detects most ground-truth components. The F1-score, defined as the harmonic average of precision and recall, provides a balanced evaluation and is particularly useful when UI components are unevenly distributed across datasets.

The IoU metric measures how well the predicted bounding box overlaps with the ground truth bounding box. In object detection tasks, IoU thresholds of 0.5 and 0.75 are commonly used to represent moderate and strict evaluation conditions. mAP summarizes detection performance across multiple IoU thresholds. The mAP@0.5:0.95 metric provides a more robust evaluation by averaging precision across thresholds from 0.5 to 0.95 in steps of 0.05 and is therefore widely accepted as a standard benchmark for object detection.

The detection results across the three datasets indicate that the proposed component detection module performs consistently well. Quantitative results are presented in Table 4. The RICO dataset achieves the highest performance, with a precision of 0.91, a recall of 0.88, and an F1-score of 0.89, owing to its large, diverse set of UI component annotations. ENRICO shows slightly lower scores due to its more simplified structure and fewer annotated component types. The Guo dataset records the lowest F1-score (0.81), likely due to greater visual variation and fewer standardized layout patterns, which make detection more challenging. Overall, these results confirm that the model maintains robust component detection performance across different UI design styles and dataset characteristics.

Dataset	Precision	Recall	F1-score
RICO	0.91	0.88	0.89
ENRICO	0.87	0.84	0.85
Guo	0.83	0.8	0.81

Table 4: Component detection performance across datasets. The table presents precision, recall, and F1-score for UI component detection across the RICO, ENRICO, and Guo datasets, demonstrating the effectiveness of the detection model.

Figure 3 shows the model’s performance across the RICO, ENRICO, and Guo datasets at IoU thresholds of 0.5 and 0.75. As expected, mAP values are higher at IoU 0.5 due to the more relaxed overlap requirement. RICO consistently reports the highest mAP values (0.89 at IoU 0.5 and 0.78 at IoU 0.75), reflecting the richness and coherence of its annotations. ENRICO shows slightly lower values, while Guo reports the lowest scores due to greater variation in layout style and component density. The declining trend with stricter IoU thresholds illustrates that dataset complexity directly influences detection robustness.

Figure 3. Mean average precision (mAP) at intersection over union (IoU) thresholds of 0.5 and 0.75 across datasets. The line plot compares component detection performance across the RICO, ENRICO, and Guo datasets. The x-axis represents the datasets, and the y-axis shows mAP values. Two curves correspond to IoU thresholds of 0.5 and 0.75, illustrating performance variation under different detection stringency levels. Please click here to view a larger version of this figure.

Figure 4 presents mAP@IoU(0.5:0.95) for each dataset, providing a broader evaluation of detection performance across ten IoU thresholds. The results reveal a clear descending trend from RICO (0.61) to ENRICO (0.57) and Guo (0.52), confirming that the proposed detection model generalizes best on larger and more structured datasets. This stricter averaged metric highlights subtle performance degradations that may not be evident under single-threshold evaluation. These findings indicate that, although the model performs strongly across all datasets, increased layout diversity and component variability introduce additional challenges under rigorous COCO-style evaluation. Detection results are presented in Figures 3 and 4, which provide quantitative comparisons of mAP across different IoU levels.

Figure 4. Mean average precision (mAP) averaged over intersection over union (IoU) thresholds from 0.5 to 0.95 across datasets. The line plot shows component detection performance across the RICO, ENRICO, and Guo datasets using the mAP metric averaged over IoU thresholds ranging from 0.5 to 0.95. The x-axis represents the datasets, and the y-axis indicates the corresponding mAP values reported on 0–1 scale, reflecting model performance under varying detection thresholds. Please click here to view a larger version of this figure.

Layout quality metrics

Layout quality is evaluated using AE, grouping accuracy (GA), and reading-order accuracy (ROA), which collectively assess structural correctness, perceptual organization, and cognitive readability of the generated UI layouts.

Alignment error.

AE (pixel-based deviation) indicates how well UI elements are aligned with ideal alignment lines, such as left, right, top, or baseline guides. A lower AE indicates that elements are consistently arranged with minimal visual distortion, improving readability and overall design clarity. This metric is particularly important for evaluating cognitive layout refinement, as misalignment disrupts visual flow and increases perceived complexity. Figure 5 illustrates AE across the RICO, ENRICO, and Guo datasets, measured as the average pixel deviation from ideal alignment lines. The results show that RICO achieves the lowest AE (4.2 px), indicating that the UI components in this dataset exhibit more consistent structural patterns, which makes alignment easier for the model. ENRICO follows with a moderate alignment deviation of 6.1 px, reflecting a combination of well-structured and varied screen layouts. The Guo dataset records the highest AE (7.4 px), due to its higher diversity in component placement and non-standard layout structures, which challenge alignment-based refinement. Overall, these results demonstrate that the model maintains strong alignment accuracy across datasets, despite increasing layout complexity.

Figure 5. Alignment error across datasets. The figure shows alignment error across datasets; lower values indicate better alignment. Please click here to view a larger version of this figure.

Grouping accuracy (GA).

GA quantifies how effectively the model groups related UI components, for example, based on Gestalt principles such as proximity, similarity, and continuity. Higher grouping accuracy indicates that the model preserves the intended structure of UI elements, enabling users to interpret the interface more naturally. This metric is particularly useful for evaluating whether the layout refinement module successfully organizes content into meaningful visual groups. Figure 6 presents the distribution of GA achieved by the proposed framework across the three datasets. The ENRICO dataset exhibits the highest median GA and the smallest interquartile range, indicating stable and consistent grouping performance due to its well-defined, pattern-labelled layouts. The RICO dataset shows moderate grouping accuracy with higher variability, likely due to its greater diversity and layout complexity. The Guo UI Color dataset records lower grouping accuracy, as its samples are visually heterogeneous and less focused on structural layout. Overall, the results indicate that the cognitive layout refinement module performs reliably across datasets, achieving the best grouping performance on structurally consistent data.

Figure 6. Distribution of grouping accuracy across datasets. The box plot shows grouping accuracy based on Gestalt principles across the RICO, ENRICO, and Guo’s UI Color Datasets. The boxes represent the interquartile range, the central line indicates the median, and the whiskers denote the range of values. The x-axis represents the datasets, and the y-axis shows grouping accuracy scores. The sample size n=5128 user interface screens (RICO: 4000, ENRICO: 1000, and Guo: 128). Please click here to view a larger version of this figure.

Reading order accuracy (ROA).

Reading order accuracy (ROA) measures how accurately the model predicts the natural reading flow of a UI screen, typically following a top-to-bottom and left-to-right pattern. Maintaining correct reading order is essential for usability, navigation clarity, and consistency with established design conventions. Higher ROA indicates that the system preserves expected cognitive scanning patterns. Figure 7 shows ROA across different evaluation stages for the RICO, ENRICO, and Guo datasets. ENRICO consistently achieves the highest ROA due to its regular and pattern-oriented layout structure, which enables more accurate prediction of human-like reading sequences. RICO demonstrates moderate performance with slight variability, reflecting its greater diversity and real-world complexity. The Guo dataset exhibits the lowest ROA, as many of its screens are visually rich but lack clearly defined reading hierarchies. Overall, these results indicate that the proposed cognitive layout refinement module performs most reliably on structured datasets while maintaining stable reading-order predictions across diverse UI designs.

Figure 7. Reading order accuracy across evaluation stages for multiple datasets. The line plot shows reading order accuracy across evaluation steps for the RICO, ENRICO, and Guo’s UI Color Datasets. The x-axis represents evaluation stage, and the y-axis indicates reading order accuracy. Each curve corresponds to a dataset, illustrating changes in the preservation of visual flow across successive evaluation stages. The evaluation stage is a representation of the suggested framework’s progressive refining stages. Please click here to view a larger version of this figure.

Layout consistency score (LCS)

The layout consistency score (LCS) measures how consistently the generated UI layouts are maintained across multiple screens within the same application. This includes consistency in spacing, alignment rules, typography regions, and grouping patterns. A higher score indicates improved cross-screen coherence, resulting in a more unified and intuitive UX. Figure 8 presents the LCS measured across multiple evaluation stages for the RICO, ENRICO, and Guo’s UI Color Datasets. The ENRICO dataset consistently achieves the highest LCS values due to its pattern-labeled and structurally uniform UI screens, which facilitate more stable learning of cross-screen consistency. In contrast, the RICO dataset demonstrates moderate but steadily improving consistency, attributable to the model’s ability to generalize across highly diverse UI layouts. As expected, the Guo dataset records the lowest LCS values, since it is primarily focused on color characteristics rather than structural layout, making multiscreen consistency more challenging. Overall, these results indicate that the proposed cross-screen consistency module performs most effectively on pattern-oriented datasets while still providing measurable improvements across all datasets.

Figure 8. Layout consistency score across evaluation stages for multiple datasets. The line plot shows layout consistency scores across evaluation stage for the RICO, ENRICO, and Guo’s UI Color Datasets. The x-axis represents evaluation stage and the y-axis indicates layout consistency scores. Each curve corresponds to a dataset, illustrating the improvement in structural coherence of generated interfaces over successive evaluation stages. Please click here to view a larger version of this figure.

Color quality metrics

The generated UI color themes are evaluated using five perceptually grounded metrics derived from CAM02-UCS: ΔE (perceptual color difference), which quantifies perceptual uniformity between palette colors; contrast ratio (based on WCAG 2.1), which assesses readability between foreground and background elements; color harmony index (CHI), which measures aesthetic compatibility between hues; color diversity score (CDS), which reflects variation within the perceptual color space; and color prominence score (CPS), which evaluates the balance and dominance of colors within the palette. Together, these metrics provide a comprehensive assessment of both aesthetic quality and usability-oriented color performance. The results demonstrate that the proposed method achieves a lower ΔE value of 4.12, indicating improved perceptual uniformity across colors. A contrast ratio of 6.21 confirms compliance with accessibility standards, ensuring enhanced readability. Table 5 presents a quantitative comparison between the proposed color modeling module and baseline methods. The CHI increases from 0.61 to 0.83, indicating improved aesthetic cohesion in color transitions. Additionally, the CDS increases by over 50%, demonstrating that the generated palettes maintain richer variation without introducing visual clutter. Higher CPS values indicate a balanced distribution of dominant colors, preventing oversaturation by any single hue. Overall, these results establish the effectiveness of the CAM02-UCS-based color modeling module in generating harmonious, readable, and perceptually optimized UI color schemes.

Metric	Definition	Proposed Method	Baseline1	Improvement (%)
ΔE (CAM02-UCS)	Perceptual color difference	4.12	7.85	47.5% lower
Contrast Ratio (WCAG)	Readability of FG–BG pairs	6.21	4.38	41.80%
CHI (Color Harmony Index)	Hue & chroma harmony	0.83	0.61	36.10%
CDS (Color Diversity Score)	Variance in J′a′b′ space	18.70	12.40	50.80%
CPS (Color Prominence Score)	Stability of dominant colors	0.72	0.55	30.90%

Table 5: Quantitative comparison of color quality metrics between the proposed and baseline methods. The table presents evaluation metrics for color quality, including perceptual color difference (ΔE in CAM02-UCS), contrast ratio (WCAG), color harmony index (CHI), color diversity score (CDS), and color prominence score (CPS). Results of the proposed method are compared with baseline values, along with percentage improvements.

Participant demographics and study design

A total of 30 participants were involved in the evaluation process. The participants were aged between 20 and 35 years (mean age: 26.4 ± 3.2 years). The cohort included individuals from diverse backgrounds, including software engineers, students, and UI/UX designers. Based on self-reported computer proficiency, 40% of participants were classified as intermediate users, 35% as advanced users, and 25% as beginners. Approximately half of the participants had prior experience in UI/UX design or related fields, while the remaining participants did not. This diversity ensured a balanced evaluation across different levels of technical expertise and design familiarity.

User study metrics

User-centered evaluation was conducted using multiple metrics, including NASA-TLX, system usability scale (SUS), aesthetic harmony score (AHS), search efficiency time (SET), and cross-screen consistency rating (CSCR), to assess cognitive load, usability, visual appeal, efficiency, and consistency.

The NASA-TLX metric quantifies mental workload by measuring factors such as mental demand, effort, and frustration. Lower scores indicate reduced cognitive load and improved ease of interaction. The proposed framework consistently resulted in lower NASA-TLX scores across all datasets, indicating reduced mental effort. This improvement can be attributed to the integration of cognitive design principles, including Gestalt grouping, optimized spacing, and enhanced visual hierarchy, which collectively facilitate more intuitive navigation. The SUS metric provides a standardized usability score ranging from 0 to 100, where higher values indicate improved usability and clarity. The proposed framework achieved higher SUS scores compared to baseline methods, reflecting improvements in both structural layout and color clarity. Enhanced alignment, spacing, and navigation flow contributed to interfaces that users perceived as more intuitive and functionally coherent. AHS, measured on a 7-point Likert scale, evaluates perceived visual appeal and color harmony. Interfaces generated using the CAM02-UCS-based color modeling module achieved higher AHS values, indicating smoother color transitions and more visually balanced palettes. Participants consistently preferred these interfaces due to improved contrast, hue coherence, and reduced visual clutter, highlighting the importance of perceptual color modeling in UI design. SET measures the time required for users to locate target UI elements during visual search tasks. Lower SET values indicate improved visual organization and hierarchy. The proposed framework significantly reduced SET across all datasets, demonstrating that integrating Gestalt grouping, attention-guided spacing, and refined layout structures enables faster, more efficient interaction. The CSCR metric evaluates the consistency of UI design across multiple screens. Higher scores indicate better continuity in layout, color, and structural organization. The proposed framework achieved higher CSCR values, reflecting the effectiveness of the multiscreen consistency module in maintaining stable color schemes, spacing, and hierarchical structures across interfaces.

Figure 9 presents the comparative user study results for all metrics (NASA-TLX, SUS, AHS, SET, and CSCR) across the RICO, ENRICO, and Guo datasets. Overall, consistent improvements are observed across all evaluation metrics. Notably, the Guo dataset demonstrates superior performance across user-centered metrics, including lower cognitive load (NASA-TLX), higher usability (SUS), improved aesthetic harmony (AHS), reduced search efficiency time (SET), and improved cross-screen consistency (CSCR). However, these results require careful interpretation. The improved UX metrics observed for the Guo dataset are primarily attributed to its strong color consistency and relatively simpler visual composition, rather than superior structural layout quality. While users perceive these interfaces as more visually harmonious and less cognitively demanding, earlier results demonstrate that the Guo dataset performs poorly in terms of alignment, grouping, and reading order. This highlights an important distinction between perceptual and structural factors in UI design. While perceptual attributes, such as color harmony, significantly enhance the UX, structural accuracy and layout consistency remain critical for functional usability. Therefore, optimal UI design requires a balance between perceptual harmony and structural correctness.

Figure 9. Comparison of user study metrics across datasets. The grouped bar chart compares user study metrics across the RICO, ENRICO, and Guo’s UI Color Datasets. The x-axis represents evaluation metrics, including NASA Task Load Index (NASA-TLX), system usability scale (SUS), aesthetic harmony score (AHS), structural efficiency time (SET), and cross-screen consistency rate (CSCR), while the y-axis indicates the corresponding scores. Bars represent performance for each dataset, illustrating differences in usability, cognitive load, aesthetic quality, and interface consistency. NASA-TLX (0–100, lower is better), SUS (0–100, higher is better), AHS (1–7 Likert), SET (s, lower is better), and CSCR (0–1, higher is better). Please click here to view a larger version of this figure.

Statistical validation of hypotheses

To further validate the proposed hypotheses (H1–H4), statistical significance testing was performed on key evaluation metrics, including layout quality, cognitive load, color harmony, and usability measures (Table 6). Results are reported as mean ± standard deviation, and statistical significance was evaluated using paired t-tests with a 95% confidence interval (p < 0.05). The results indicate that the proposed framework achieves statistically significant improvements over baseline approaches across multiple metrics, including alignment accuracy, grouping accuracy, reading order, cognitive load (NASA-TLX), and color harmony measures. Notably, reductions in cognitive load and improvements in usability metrics demonstrate highly significant differences (p < 0.01). These findings confirm that integrating cognitive design principles and perceptual color modeling leads to measurable, statistically significant improvements in UI quality, thereby supporting hypotheses H1–H4.

Metric	Baseline (Mean ± SD)	Proposed (Mean ± SD)	Improvement (%)	p-value	Significance
Alignment Error (↓)	7.8 ± 1.2	4.2 ± 0.9	46.10%	0.003	Significant
Grouping Accuracy (↑)	0.68 ± 0.05	0.82 ± 0.04	20.50%	0.001	Significant
Reading Order Accuracy (↑)	0.72 ± 0.06	0.87 ± 0.03	20.80%	0.002	Significant
NASA-TLX (↓)	68.5 ± 5.4	47.2 ± 4.8	31.10%	0.0005	Highly Significant
SUS Score (↑)	71.3 ± 6.2	88.9 ± 4.5	24.70%	0.0008	Highly Significant
Color Harmony Index (↑)	0.61 ± 0.07	0.83 ± 0.05	36.00%	0.001	Significant
Contrast Ratio (↑)	4.1 ± 0.6	6.2 ± 0.5	51.20%	0.002	Significant

Table 6: Statistical comparison of performance metrics between baseline and proposed methods. The table presents mean and standard deviation (mean ± SD) for key performance metrics, including alignment error, grouping accuracy, reading order accuracy, NASA Task Load Index (NASA-TLX), system usability scale (SUS), color harmony index, and contrast ratio. Percentage improvements, p-values, and statistical significance levels are reported. (↑) indicates higher values are better, and (↓) indicates lower values are better. Statistical significance was evaluated at p < 0.05.

Dataset-specific observations

The relatively lower performance observed in structural metrics on the Guo dataset can be attributed to its inherent characteristics. The Guo dataset is smaller than RICO and ENRICO and primarily focuses on perceptual color features rather than structured layout annotations. As a result, it exhibits higher variability in component placement, weaker hierarchical organization, and less consistent layout structures across screens. These properties make structural learning and layout optimization more challenging, explaining the lower performance in alignment, grouping, and reading-order metrics, despite strong performance in perceptual and user-centered evaluations.

Ablation study

The ablation study evaluates the contribution of each module within the proposed framework by systematically removing individual components and analyzing their impact on layout quality, color modeling, and detection performance. Layout quality is assessed using AE, GA, ROA, and LCS. AE reflects the positional deviation of UI elements, where higher values indicate weaker spatial structure. GA evaluates how effectively components are organized according to Gestalt principles. ROA measures the preservation of logical visual flow, while LCS quantifies cross-screen structural consistency in terms of alignment, spacing, and layout organization. Color quality degradation is evaluated using ΔE (perceptual color difference), CHI, and CDS, which collectively assess perceptual uniformity, aesthetic coherence, and palette richness. Detection performance is evaluated using mAP, precision, and recall, which quantify the impact of replacing the Faster R-CNN module with a simpler detection model. Together, these metrics provide a comprehensive evaluation of how each module contributes to overall system performance.

Table 7 summarizes the contribution of each module and compares the proposed framework with existing UI generation approaches. The full model achieves the best performance across all layout quality, color modeling, and detection metrics, demonstrating that cognitive design, perceptual color modeling, and deep visual understanding have a synergistic effect. When the color modeling module is removed, ΔE increases significantly, while CHI and CDS decrease substantially, indicating a loss of perceptual uniformity and color harmony. This confirms the importance of CAM02-UCS-based modeling in maintaining aesthetic and perceptual quality. Removing the cognitive layout module results in a significant increase in AE and notable decreases in GA and ROA, demonstrating that cognitive design principles are essential for producing structurally coherent and readable layouts. The removal of the multiscreen consistency module leads to a reduction in LCS, confirming its role in maintaining consistent layout patterns across multiple screens. Replacing the Faster R-CNN detector with a simpler CNN results in a sharp decline in mAP, precision, and recall, highlighting the critical importance of robust component detection in the overall pipeline. In comparison with baseline methods, including pix2code, REDRAW, and LDGM, the proposed framework consistently outperforms all approaches across layout accuracy, visual coherence, and perceptual color quality.

Method / Module	AE ↓	GA ↑	ROA ↑	LCS ↑	ΔE ↓	CHI ↑	CDS ↑	mAP ↑
Color Module Removed	0.14	0.86	0.92	0.84	7.85	0.61	12.4	0.9
Cognitive Layout Module Removed	0.18	0.72	0.73	0.69	4.21	0.79	17.9	0.89
Multi-Screen Consistency Removed	0.17	0.81	0.88	0.62	4.18	0.81	17.3	0.9
Faster R-CNN Replaced with Simple CNN	0.16	0.85	0.91	0.85	4.15	0.82	18.1	0.74
Pix2Code	0.25	0.61	0.65	0.51	10.2	0.48	9.6	0.65
REDRAW	0.21	0.66	0.72	0.56	8.7	0.52	11.4	0.72
LDGM (Layout Diffusion)	0.19	0.74	0.79	0.63	6.9	0.59	13.8	0.8
Proposed Full Model	0.12	0.89	0.94	0.87	4.12	0.83	18.7	0.91

Table 7: Ablation study and comparative performance analysis of the proposed framework and baseline methods. The table evaluates the impact of individual components by comparing the full proposed model with variants in which specific modules are removed (color module, cognitive layout module, multiscreen consistency, and Faster R-CNN replaced with a simple CNN), as well as baseline methods (pix2code, REDRAW, and LDGM). Performance is assessed using alignment error (AE), grouping accuracy (GA), reading order accuracy (ROA), layout consistency score (LCS), perceptual color difference (ΔE), color harmony index (CHI), color diversity score (CDS), and mean average precision (mAP). (↑) indicates higher values are better, and (↓) indicates lower values are better.

DATA AVAILABILITY:

The datasets used in this study are available at the following links: RICO dataset: https://interactionmining.org/rico; ENRICO dataset: https://github.com/luileito/enrico; Guo’s UI Color Dataset: https://github.com/guozhongke/UI-Color-Dataset.

Supplementary File 1. Mathematical formulations and extended implementation details. This file provides the detailed mathematical formulations and algorithmic workflows supporting the proposed automated UI prototyping framework. It includes component detection training, layout pattern extraction, color harmony modeling, the unified UI prototyping objective, Faster R-CNN detection details, spatial distance and alignment similarity calculations, logical UI tree construction, CAM02-UCS-based perceptual color modeling, cognitive design optimization, multiscreen consistency modeling, and Algorithms S1–S3.Please click here to download this file.

Discussion

The findings demonstrate statistically significant improvements in usability, structural organization, and perceptual quality (p < 0.05), confirming the effectiveness of incorporating cognitive design principles into automated UI prototyping. Unlike traditional UI generation systems that rely primarily on visual detection or rule-based heuristics, the proposed framework integrates Gestalt grouping, hierarchy modeling, spacing constraints, and perceptual readability throughout the entire UI reconstruction process. Improvements observed in NASA-TLX, SUS, and SET further confirm a statistically significant reduction in cognitive workload (NASA-TLX, p.< 0.01). These results indicate that automated UI generation should extend beyond visual reconstruction accuracy and incorporate human-centered design knowledge to achieve practical usability. Additionally, all formulated hypotheses (H1–H4) are empirically supported by the experimental results.

Beyond layout improvements, the integration of CAM02-UCS-based perceptual color modeling results in statistically significant gains in color harmony, accessibility, and perceptual stability (p < 0.05), as reflected by improvements in ΔE, CHI, CDS, and contrast ratio. Compared to prior work such as automated image-based UI color theme generation, which primarily focuses on color optimization, the proposed framework integrates both color and layout refinement within a unified pipeline. Furthermore, the inclusion of a multiscreen consistency module addresses a key limitation of previous approaches that focus only on single-screen UI generation. A comparison with existing methods, including pix2code³¹, REDRAW^23,24, and LDGM³², shows that these approaches primarily emphasize structural reconstruction or generative modeling without incorporating cognitive principles, perceptual color harmonization, or multiscreen reasoning. The proposed framework demonstrates statistically significant improvements over these methods (p < 0.05) across layout accuracy metrics (AE, GA, and ROA), perceptual metrics (CHI and ΔE), and usability measures (SUS and CSCR). These findings highlight the advantage of combining cognitive design principles, perceptual color modeling, and deep learning within a unified system.

An important observation from the results is the distinction between structural and perceptual factors influencing UX. Structural optimization enhances functional correctness and layout clarity, while perceptual optimization—particularly color harmony—significantly influences user perception and cognitive load. The findings indicate that optimal UI design requires a balance between structural accuracy and perceptual coherence. Despite the demonstrated improvements, certain limitations remain. For example, performance on smaller and more perceptually oriented datasets, such as the Guo UI Color dataset, is lower in structural metrics due to variability in layout organization and limited structural annotations. These characteristics introduce challenges in learning consistent layout patterns and hierarchical relationships. Future work will focus on improving robustness across such datasets.

From a theoretical perspective, this research demonstrates that UI generation can be significantly enhanced by embedding cognitive and perceptual principles directly into deep learning workflows. The study challenges the conventional view that UI generation is solely a computer vision or code-generation problem. Instead, it highlights the importance of incorporating perceptual uniformity (via CAM02-UCS), cognitive load reduction (via Gestalt principles and hierarchy modeling), and multiscreen design coherence as core components of automated UI systems. This work contributes to a shift toward human-centered computational models, where UI generation accounts not only for structural correctness but also for psychological and perceptual factors. As such, it establishes a new theoretical direction for automated design intelligence.

From a practical standpoint, the proposed framework provides a deployable solution for UI designers, product teams, and prototyping tools. By reducing cognitive load and improving search efficiency, the system generates UI layouts that require less manual refinement to meet professional design standards. The use of perceptually optimized color themes ensures compliance with accessibility guidelines such as WCAG 2.1, enabling immediate usability in real-world applications. Additionally, the multiscreen consistency module facilitates faster development of multipage applications by maintaining consistent layout and design patterns without requiring manual adjustments. Overall, the framework has the potential to significantly reduce UI development time while improving both the quality and usability of generated designs.

The proposed framework introduces a human-centered approach to automated UI prototyping by integrating cognitive design principles, perceptual color modeling (CAM02-UCS), and multiscreen layout consistency into a unified computational pipeline. Unlike prior approaches focused primarily on structural reconstruction or visual detection, the framework explicitly models Gestalt grouping, hierarchy, contrast optimization, and perceptual uniformity. Experimental results demonstrate significant improvements in usability (SUS and NASA-TLX), visual harmony (AHS, CHI, and ΔE), and structural performance (GA, ROA, LCS, and mAP), confirming that cognitive and perceptual modeling enhances UI quality and UX. Future work will explore the integration of transformer-based vision–language models to improve semantic understanding and structural reasoning. Additionally, incorporating adaptive personalization, device-responsive layouts, and human-in-the-loop refinement may further enhance the practical applicability of the system. Overall, this work provides a foundation for next-generation AI-driven UI design systems that are not only visually accurate but also cognitively efficient, perceptually harmonious, and practically deployable.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
CPU (Processor)	AMD Ryzen 9 7950X (16-core, 32-thread, 4.5–5.7 GHz)	Advanced Micro Devices (AMD), USA	Ryzen 9 7950X
CUDA Toolkit	Version 11.8	NVIDIA Corporation, USA	CUDA Toolkit 11.8
cuDNN	Version 8.9	NVIDIA Corporation, USA	cuDNN v8.9
Faster R-CNN	Object Detection Model (with Region Proposal Network)	Meta AI Research / Detectron2	Detectron2 framework
Matplotlib	Version 3.7	Matplotlib Development Team	v3.7.0
NVIDIA RTX 4090 GPU	24 GB GDDR6X Graphics Card	NVIDIA Corporation, Santa Clara, CA, USA	RTX 4090
NumPy	Version 1.24	NumPy Developers	v1.24.0
OpenCV	Version 4.8	OpenCV Foundation	v4.8.0
PyTorch	Version 2.0	PyTorch Foundation (Meta AI)	v2.0.0
ResNet-50 with FPN	Backbone CNN with Feature Pyramid Network	Microsoft Research / Detectron2	ResNet-50-FPN
Scikit-learn	Version 1.3	Scikit-learn Developers	v1.3.0
SciPy	Version 1.10	SciPy Community	v1.10.0
System Memory (RAM)	64 GB DDR5	Corsair / Kingston (or equivalent)	DDR5 64GB Kit
Ubuntu Operating System	Version 22.04 LTS	Canonical Ltd., UK	Ubuntu 22.04 LTS