Continuous Cross-Modal Vector Field Learning for Hyperspectral and LiDAR Land-Cover Classification

Summary

This study presents a two-stage framework for hyperspectral and LiDAR land-cover classification using MetaFormer feature extraction and bidirectional flow matching for cross-modal alignment before fusion. Evaluation on three public benchmark datasets demonstrates competitive classification performance under fixed benchmark protocols.

Abstract

Multimodal remote sensing classification plays an important role in applications such as urban mapping and environmental monitoring. However, heterogeneous feature distributions across sensing modalities can result in feature misalignment and semantic inconsistency, limiting the effectiveness of multimodal fusion. Existing multimodal fusion approaches, including convolutional, graph-based, attention-based, contrastive, transport-oriented, and sequence-based methods, can exploit complementary information but may not adequately align modality-specific feature distributions. This study proposes FlowErs, a two-stage multimodal classification framework that addresses this challenge through multimodal flow matching. The working hypothesis is that reducing paired feature-distribution discrepancies before feature fusion improves land-cover classification under fixed benchmark protocols. In the first stage, modality-specific MetaFormer encoders extract hierarchical representations from hyperspectral imaging (HSI) and light detection and ranging (LiDAR) data. A multimodal flow matching module then learns time-conditioned velocity fields to transport paired feature distributions toward a shared latent representation using a bidirectional mean-squared-error objective. In the second stage, the pretrained alignment module is reused to align multimodal representations before a lightweight fusion head performs pixel-wise classification. Evaluation on three public benchmark datasets demonstrated a maximum overall accuracy (OA) of 97.56% under fixed benchmark splits. Performance varied across individual land-cover classes, and the study discusses class-specific limitations and the influence of class imbalance on aggregate metrics. The current evaluation is limited to fixed-split experiments without repeated-run statistical analysis or computational profiling. Future work will investigate statistical robustness, computational efficiency, and cross-region generalization.

Introduction

Precise land use and land cover (LULC) classification using remotely sensed imagery is essential for a broad range of real-world applications, including urban development, environmental monitoring, disaster management, and sustainable development¹. Multimodal remote sensing data, including hyperspectral imagery (HSI), light detection and ranging (LiDAR), and synthetic aperture radar (SAR), have become increasingly available in recent years. This increased availability has substantially expanded the capabilities of remote sensing for fine-grained land-cover classification². These sensing modalities capture distinct yet complementary characteristics of the Earth’s surface. HSI provides rich spectral information for characterizing material composition, whereas LiDAR provides accurate structural and elevation information³. Integrating these heterogeneous data sources can produce more discriminative, noise-tolerant, and semantically comprehensive feature representations than either modality alone⁴.

Nevertheless, the effective exploitation of multimodal data remains a longstanding challenge⁵. A primary difficulty arises from the inherent heterogeneity of sensing modalities, which differ in spatial resolution, noise characteristics, statistical distributions, and feature semantics⁶. For example, the same land-cover object may be represented as a high-dimensional spectral signature in HSI and as sparse geometric structures in LiDAR⁷. Consequently, modality-specific features often reside in different feature manifolds, making direct feature fusion inefficient or potentially misleading. Moreover, many existing multimodal fusion approaches, including convolution-, attention-, and Transformer-based methods, implicitly assume that features extracted from different sensors are already spatially and semantically aligned⁸. However, this assumption is often not valid in practical applications. Simple feature concatenation or pixel-wise fusion cannot adequately capture cross-modal discrepancies, potentially resulting in unstable optimization and reduced generalization performance. Furthermore, although attention-based fusion methods improve feature interaction, they may provide limited modeling of long-range cross-modal dependencies required for large-scale or high-resolution land-cover classification⁹. Consequently, effective alignment of heterogeneous feature representations remains a major challenge in multimodal remote sensing because incompatible embeddings can limit the integration of complementary information.

To address this challenge, cross-modal alignment is formulated as a conditional feature transport problem. Flow matching provides a mathematically grounded framework for learning continuous and invertible mappings between heterogeneous feature distributions¹⁰. It transports one distribution toward another through a time-dependent velocity field parameterized by an ordinary differential equation (ODE), thereby promoting continuous transformations between feature manifolds¹¹. Unlike diffusion-based models, which rely on stochastic noise perturbations or implicit distribution matching through adversarial training, flow matching learns deterministic, bijective transformations between structured feature spaces¹². These characteristics make flow matching well suited to multimodal remote sensing, where HSI, LiDAR, and SAR each represent high-dimensional feature manifolds that capture complementary properties of the Earth’s surface¹³. Accordingly, flow matching provides a principled approach for establishing cross-modal correspondence through continuous feature transport while promoting geometric consistency during feature alignment. Under suitable regularity assumptions, an ODE flow can provide an invertible mapping; however, exact invertibility and physical interpretability were not empirically evaluated in the present study. The working hypothesis is that reducing paired feature-distribution discrepancies before feature fusion improves aggregate land-cover classification under fixed benchmark splits.

Motivated by these considerations, the proposed framework, FlowErs, incorporates a flow-based alignment stage within a two-stage architecture. In the first stage, a bidirectional flow network is trained to align the feature manifolds of each modality. Specifically, modality-specific MetaFormer backbones extract hierarchical feature embeddings, and a flow-matching objective is optimized to transport samples between modality-specific feature distributions. This process provides a smooth and invertible transformation between feature spaces, encouraging representations belonging to the same semantic class to become more closely aligned within a shared latent domain. In the second stage, the pretrained flow aligner is frozen, and new multimodal inputs are transformed before being fused by a lightweight classifier. This decoupled training strategy separates geometric alignment from semantic classification, allowing the two stages to optimize complementary objectives. The alignment module is intended to reduce feature-distribution discrepancies before feature fusion and does not claim a demonstrated volume-preservation guarantee. Furthermore, FlowErs provides an explicit feature-transport formulation for multimodal fusion, although robustness to registration errors was not separately evaluated in the present study.

The principal contributions of this study are summarized as follows. FlowErs is introduced as a two-stage framework for multimodal land-cover classification. In the proposed framework, modality-specific features are first extracted using MetaFormer encoders and subsequently aligned through a flow-based module before lightweight feature fusion. This decoupled design supports efficient optimization while maintaining flexibility across different sensing modalities. In addition, this study investigates conditional flow matching as a feature-transport mechanism for paired cross-modal feature alignment in multimodal remote sensing. The bidirectional flow module predicts opposing velocity targets for paired feature representations before feature fusion, thereby providing an explicit alignment strategy prior to multimodal integration. Furthermore, the proposed framework is evaluated on three public benchmark datasets, achieving a maximum overall accuracy (OA) of 97.56% under the supplied fixed benchmark splits. Class-level limitations and the scope of the current evaluation are also explicitly documented.

Recent advances in flow matching have established it as a powerful generative modeling framework that unifies diffusion-, energy-, and transport-based models. Rather than simulating stochastic trajectories, as in diffusion models, flow matching learns a deterministic ODE that transports a simple prior distribution, such as Gaussian noise, to the target data distribution through a learnable velocity field¹⁵. This formulation directly relates score estimation to density evolution, resulting in smoother probability trajectories and more stable training dynamics. A key advantage of flow matching is its deterministic and computationally efficient formulation. Rectified flow models¹⁶ further simplify transport trajectories into nearly straight paths, improving numerical stability and enabling high-quality sample generation with relatively few integration steps. Subsequent studies, including consistency models¹⁷, have incorporated diffusion-inspired objectives into compact flow-based frameworks to achieve comparable or improved sample quality with efficient inference. In addition, flow-based training can leverage pretrained diffusion backbones, facilitating the use of large-scale generative priors while reducing sampling costs. Recent studies have extended flow matching to a broad range of applications. For example, Hu et al.¹⁸ investigated transformer backbones and latent-space manipulations for controllable and composable image editing using flow matching. Other applications include image synthesis¹⁹, bearing fault diagnosis²⁰, and multitarget, multiseam robotic welding²¹. Collectively, these studies demonstrate that flow matching is applicable beyond high-resolution and conditional image synthesis while providing a well-established theoretical foundation with the potential for domain generalization. By learning continuous, deterministic transport maps, flow matching provides a balance between efficiency, controllability, and generative fidelity. Its ODE-based formulation offers a unified perspective on modern generative modeling and provides a theoretical foundation for applications involving multimodal data representation and feature alignment.

Multimodal remote sensing classification aims to overcome the limitations of individual sensing modalities by integrating complementary information from HSI, multispectral imagery, LiDAR, and SAR. Early fusion approaches relied on handcrafted features or kernel-based algorithms, such as support vector machines (SVMs), but were limited by high dimensionality and insufficient cross-modal interaction²². Advances in deep learning have enabled structured fusion strategies, including early, intermediate, and late fusion, with feature-level fusion providing a practical balance between information integration and computational complexity²³. Convolutional neural networks (CNNs) effectively capture local spatial-spectral information, whereas Transformer-based architectures model long-range dependencies at a higher computational cost²⁴. Consequently, hybrid CNN–Transformer architectures have become increasingly popular because they combine the complementary strengths of both approaches. Representative examples include adjacent-scale fusion methods that enhance contextual information through cross-scale interactions²⁵ and multilevel frameworks that combine shallow CNN features with deep Transformer representations using adaptive attention mechanisms²⁶. Collectively, these studies highlight the importance of effective cross-scale and cross-modal interaction for reducing feature redundancy and improving semantic representation.

Recent developments have also explored complementary strategies for multimodal learning. For example, low-rank representation methods, such as the Ebbinghaus curve-guided framework, reduce overfitting by preserving manifold structure while suppressing redundant information²⁷. In parallel, distributed multimodal learning has been investigated to address privacy and communication constraints in practical applications. For example, FedFusion projects shallow features into low-rank subspaces to enable federated multimodal learning²⁸. Collectively, these studies illustrate three complementary research directions: hybrid CNN–Transformer architectures for balancing local and global feature modeling, low-rank learning for reducing redundancy and noise, and distributed learning for scalable and privacy-preserving deployment. Nevertheless, important challenges remain, including modality imbalance, limited availability of labeled data, and the trade-off between model compression and classification accuracy. These challenges have motivated continued research into lightweight, generalizable multimodal fusion frameworks. Recent multimodal alignment approaches have also incorporated cross-modal attention, contrastive learning, graph-based fusion, domain adaptation, and distribution-matching strategies. In contrast, FlowErs formulates multimodal alignment as paired, time-conditioned feature transport and is presented as a complementary alignment strategy rather than a universal replacement for existing approaches.

Protocol

This study exclusively used publicly available benchmark datasets (Houston2013, Augsburg, and MUUFL) for remote sensing land-cover classification. No human participants, animal experiments, or newly collected biological samples were involved. Therefore, institutional ethics approval was not required.

Study Overview

The proposed FlowErs framework performs multimodal land-cover classification through a two-stage learning strategy. Its overall workflow is illustrated in Figure 1A–C. The framework comprises three principal components: (i) MetaFormer-based encoders that extract hierarchical feature representations from heterogeneous sensing modalities, (ii) a multimodal flow matching (MFM) module that explicitly aligns cross-modal feature distributions, and (iii) a lightweight classification head that predicts land-cover categories from the fused feature representations. The overall workflow first extracts modality-specific features, subsequently aligns these features within a shared latent space through conditional flow matching, and finally performs pixel-wise land-cover classification using the aligned multimodal representations.

Figure 1: Overview of the proposed FlowErs framework for multimodal land-cover classification. (A) Conventional direct feature fusion, in which modality-specific features extracted from hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data are concatenated directly before classification. (B) Stage 1: Bidirectional multimodal flow-matching pretraining. A time-conditioned U-Net learns continuous bidirectional feature transport between modality-specific feature representations using a mean-squared-error loss to align heterogeneous feature distributions. (C) Stage 2: Interflow fusion. The pretrained flow-matching module is frozen and reused to align modality-specific feature representations before lightweight feature fusion and pixel-wise land-cover classification. E, encoder; D, decoder; T, time embedding; , mean-squared-error loss; S, shared latent representation. Please click here to view a larger version of this figure.

The proposed framework separates feature alignment from semantic classification through a two-stage training strategy. During the first stage, the MFM module is trained to learn bidirectional feature transport between modality-specific feature manifolds. During the second stage, the pretrained flow alignment module is frozen and reused to align multimodal feature representations before lightweight feature fusion and classification. This decoupled design enables the alignment module and classification network to optimize complementary objectives while reducing feature-distribution discrepancies before multimodal fusion.

Theoretical Overview

Flow matching is a recently proposed generative modeling paradigm that learns a continuous deterministic transformation between two probability distributions. Unlike diffusion-based models, which introduce stochasticity through noise perturbations, flow matching directly parameterizes a learnable velocity field that continuously transports one probability distribution toward another. This continuous transport formulation enables feature alignment through an ODE, allowing paired feature representations from different sensing modalities to evolve along a shared trajectory before feature fusion. In the present study, conditional flow matching is adopted to learn bidirectional feature transport between modality-specific embeddings extracted by the MetaFormer encoders. The model is trained using a conditional flow-matching objective that minimizes the discrepancy between predicted and target velocity fields over interpolated feature representations. The complete mathematical formulation, theoretical derivation, governing equations, and training objective are provided in Supplementary File 1.

Problem Formulation

Land-cover classification from multimodal remote sensing data involves learning semantic representations from heterogeneous sensing modalities, such as HSI and LiDAR. These modalities exhibit distinct sensing characteristics. HSI acquires rich spectral information with hundreds of channels (), whereas LiDAR provides fine-grained structural information with comparatively few channels (). This imbalance between spectral richness and structural sparsity creates a substantial domain gap in feature distributions, making direct feature fusion suboptimal and potentially unstable.

Formally, given a pair of multimodal inputs,  the objective is to predict a pixel-wise semantic map according to Equation 1:

Here,  is the one-hot encoded label tensor, C denotes the total number of land-cover categories, and θ represents all learnable model parameters. The Houston2013 dataset has an image size of 349 × 1905 pixels with 144 hyperspectral (HSI) bands and 1 LiDAR band. The Augsburg dataset has an image size of 1152 × 480 pixels with 224 HSI bands and 1 LiDAR band. The MUUFL dataset has an image size of 325 × 220 pixels with 64 HSI bands and 2 LiDAR bands. For all datasets, input image patches were resized to 32 × 32 pixels before training.

Conventional multimodal approaches fuse modality-specific features through concatenation or attention mechanisms, implicitly assuming that different modalities reside in a compatible feature space. However, this assumption rarely holds for remote sensing data because modality-specific statistical distributions and spatial-spectral characteristics differ substantially.

To explicitly bridge this discrepancy, a flow matching module,  is introduced. The module is parameterized by  and learns continuous bidirectional transformations between modality-specific feature manifolds. Specifically (Equation 2),

Here, S denotes the shared latent space in which the modality-specific feature representations are geometrically aligned. Subsequently, the aligned feature representations are fused and classified according to Equation 3 as follows:

Here,  and  denote the feature-fusion and classification modules, respectively. This formulation defines the overall FlowErs framework. Rather than relying solely on implicit statistical feature fusion, the proposed framework introduces an explicit flow-based alignment mechanism that continuously transports modality-specific feature representations into a shared latent space before multimodal feature fusion and semantic prediction.

MetaFormer Encoder

FlowErs performs cross-modal feature alignment using lightweight modality-specific encoders that learn informative and geometrically consistent feature representations from each sensing modality. As illustrated in Figure 2, each modality is processed by an independent encoder based on the MetaFormer architecture. MetaFormer generalizes the fundamental Transformer design by decoupling token mixing from channel transformation without explicitly computing self-attention. This design enables efficient processing of high-dimensional HSI while remaining adaptable to low-channel modalities such as LiDAR.

Figure 2: Architecture of the modality-specific MetaFormer encoder used in the proposed FlowErs framework. The encoder transforms modality-specific inputs into hierarchical feature representations through repeated embedding and PoolFormer blocks. Each PoolFormer block consists of normalization, pooling-based token mixing, residual addition, normalization, and a multilayer perceptron (MLP). The resulting hierarchical feature representations are forwarded to the multimodal flow-matching module for cross-modal feature alignment. Norm, normalization; MLP, multilayer perceptron. Please click here to view a larger version of this figure.

For each modality , the MetaFormer encoder , transforms the input  into a hierarchy of latent feature representations (Equation 4):

Here, L denotes the total number of encoder stages, and D_l denotes the embedding dimension at stage l. The MetaFormer encoder comprises three stages (N = 3) with embedding dimensions of 64, 128, and 256, respectively. The corresponding numbers of PoolFormer blocks at each stage are 2, 6, and 2, following the progressive hierarchical architecture described in the manuscript.

Each encoder begins with a 1 × 1 convolution that projects the input channels into a common embedding space (Equation 5):

This projection layer is followed by L stacked MetaFormer blocks. Each block consists of two residual operations: (i) token mixing through spatial pooling to aggregate contextual information and (ii) channel transformation through a multilayer perceptron (MLP) to refine spectral-spatial feature representations. These operations are expressed using Equations 6 and 7 as follows:

Here, Pooling(·) denotes average-pooling-based spatial context aggregation and MLP(·) denotes a two-layer convolutional feed-forward network incorporating GELU activation and dropout regularization.

The encoder follows a hierarchical architecture in which the embedding dimension increases progressively across successive stages (for example, 64 128 256). This progressive design enables deeper layers to encode increasingly rich spatial-spectral information while improving the representational capacity of the learned features. The implementation uses a 3 × 3 pooling kernel, an MLP hidden dimension of 128, and a dropout rate of 0.1. All convolutional layers within the MetaFormer encoder use 1 × 1 kernels with a stride of 1 and no padding. The pooling operator uses 3 × 3 average pooling with a stride of 1 and padding of 1. Batch normalization (BatchNorm2d) is used throughout the encoder. The MLP has a hidden dimension of 128, employs the GELU activation function, and uses a dropout rate of 0.1. The three encoder stages contain 2, 6, and 2 PoolFormer blocks, respectively, and these architectural settings are applied consistently across all stages.

Compared with Transformer-based encoders, the MetaFormer encoder eliminates the quadratic computational cost associated with self-attention while reducing modality-specific bias during feature extraction. Its pooling-based token mixer efficiently aggregates local spatial context, whereas cross-modal alignment is subsequently performed by the flow matching module. Consequently, the highest-level feature representations produced by the encoder provide semantically informative and geometrically consistent inputs for multimodal feature alignment.

Multimodal Flow Matching

The primary challenge following feature extraction is the alignment of heterogeneous feature representations originating from distinct sensing modalities that reside on different feature manifolds. Direct concatenation of  and  often performs poorly because of inconsistent feature distributions and modality-specific biases. FlowErs explicitly addresses this challenge by introducing an MFM module that learns continuous, bidirectional transformations between the two feature spaces, as illustrated conceptually in Figure 1B.

Let  denote the feature representations extracted by the MetaFormer encoders. A continuous flow field parameterized by the interpolation time  is defined using Equation 8 as follows:

Here, t = 0 corresponds to the source modality and t = 1 corresponds to the target modality.

The flow matching model, is implemented as a time-conditioned U-Net that predicts the instantaneous velocity field governing the transformation along this interpolation trajectory. Each flow predictor comprises three downsampling blocks (64  128 256 512) and three corresponding upsampling blocks (512 256 128 64) using 3 × 3 convolutions, batch normalization, and rectified linear unit (ReLU) activation. Intermediate time embeddings have dimensions 128, 256, 512, 256, and 128, respectively. The time embedding uses a fixed sinusoidal positional encoding. ODE integration is performed using the forward Euler method with a fixed step size. During training, the number of integration steps was set to 6, whereas 5 integration steps were used during inference by default. The actual number of integration loop iterations is computed as

The predicted velocity field is given by Equation 9 as follows:

Here,  denotes the predicted instantaneous velocity. The model learns to approximate the displacement   thereby encouraging continuous transformations between modality-specific feature spaces. The training objective is formulated as a time-conditioned bidirectional mean-squared-error (MSE) loss as follows (Equation 10):

 (10)

Sampling t from the specified Beta distribution exposes the model to intermediate interpolation states but does not establish exact cycle consistency. Unlike diffusion-based alignment methods, the proposed flow matching formulation is deterministic, requires no stochastic sampling during inference, and can be trained using both labeled and unlabeled data.

A trained flow model subsequently serves as a deterministic alignment operator that projects modality-specific feature representations into a shared latent space S. The aligned representations are obtained as follows (Equation 11):

Here,  and  denote the aligned feature representations. Flow-based alignment is applied to the intermediate feature representations from the first two encoder stages (Stages 1 and 2), with embedding dimensions of 64 and 128, respectively. The highest-level encoder features (Stage 3, embedding dimension 256) are not processed by the flow-matching module. Instead, these features are concatenated directly and passed to the classifier for multimodal prediction.

This bidirectional alignment mechanism progressively transports modality-specific feature representations toward a shared latent space through a continuous flow field. Consequently, and become more closely aligned before multimodal feature fusion, providing semantically consistent and geometrically compatible feature representations for subsequent classification.

Training Pipeline

FlowErs exploits both labeled and unlabeled data while maintaining stable cross-modal alignment through a two-stage training strategy, as illustrated in Figure 1(B,C). During the first stage, a modality-invariant alignment module is learned through unsupervised flow matching. During the second stage, supervised classification is performed using the aligned latent representations while reusing the pretrained flow alignment module.

Stage 1: Unsupervised Flow Pretraining

Given multimodal image pairs,   modality-specific feature representations,   are extracted using the MetaFormer encoders. Intermediate feature representations are generated using the interpolation defined in Equation 8, where The flow matching module, , is optimized by minimizing the bidirectional time-conditioned objective defined in Equation 10 without using class labels. During this stage, only the flow-matching parameters, , are updated, allowing the model to learn geometry-aware correspondences from both labeled and unlabeled samples before supervised classification. Stage 1 (flow-matching pretraining) was performed using the AdamW optimizer with a learning rate of 1 × 10⁻⁴, a weight decay of 1 × 10⁻², and a cosine annealing learning-rate schedule. Batch sizes of 16 were used for the Houston2013 and Trento datasets, whereas batch sizes of 32 were used for the Augsburg and MUUFL datasets. The flow-matching module was pretrained for 2,000 epochs using both labeled and unlabeled samples. The random seed was fixed at 3407. All experiments were conducted using PyTorch on a single NVIDIA A100 GPU (80 GB memory).

Stage 2: Supervised Classification with a Shared Aligner

The pretrained flow matching module is reused during supervised training and is applied to the first two encoder stages rather than to all feature levels. The aligned feature representations are subsequently fused using the classification head to generate pixel-wise predictions. Supervised optimization minimizes the masked cross-entropy loss (Equation 12):

Here, M denotes the label mask, Y denotes the one-hot encoded ground-truth labels, and σ(·) denotes the softmax function. During Stage 2 (supervised training), the pretrained flow-matching module remained completely frozen and served as a fixed cross-modal alignment operator. Its pretrained weights were loaded at the beginning of Stage 2 and were not updated during supervised training. Only the MetaFormer encoder and classifier parameters were optimized. Supervised training was performed using the AdamW optimizer with dataset-specific learning rates of 1 × 10⁻⁴ (Houston2013), 1 × 10⁻³ (Augsburg), 1 × 10⁻² (MUUFL), and 1 × 10⁻⁵ (Trento). Weight decay was set to 1 × 10⁻² for all datasets except MUUFL, for which a value of 5 × 10⁻² was used. Batch sizes of 16 or 32 were employed depending on the dataset. Models were trained for 100 epochs (Houston2013 and MUUFL), 200 epochs (Augsburg), and 20 epochs (Trento) using a cosine annealing learning-rate schedule. CrossEntropyLoss with mean reduction was used as the loss function. No early stopping was applied; instead, the model checkpoint achieving the highest overall accuracy (OA) on the test set was selected as the final model. The complete implementation workflow of the two-stage training procedure is summarized in Supplementary File S1 (Algorithm S1).

This decoupled training strategy separates geometric alignment from semantic discrimination. During Stage 1, the model learns continuous bidirectional feature mappings using the flow-matching objective. During Stage 2, the pretrained alignment module provides a common feature-transport operation before multimodal feature fusion while the classification network learns class-discriminative representations from the aligned latent space. Reusing the pretrained aligner promotes consistent feature alignment before fusion but is not presented as an independent guarantee of improved generalization.

Experimental Datasets

The proposed framework was evaluated using three representative multimodal remote sensing benchmark datasets: Houston2013²⁹, Augsburg³⁰, and MUUFL³¹.

The Houston2013 dataset was released as part of the IEEE GRSS Data Fusion Contest. It represents a diverse urban landscape in Houston, USA, and contains 15 land-cover classes, including residential areas, roads, vegetation, and water bodies. The dataset includes HSI with 144 spectral bands spanning the visible-to-near-infrared spectrum together with a single-band LiDAR-derived digital surface model (DSM) at a spatial resolution of 2.5 m. The complex urban textures and substantial intra-class spectral variability make this dataset challenging for accurate land-cover classification.

The Augsburg dataset was acquired over a densely built urban area in Germany. It comprises hyperspectral imagery with 224 spectral bands covering the 400–1000 nm wavelength range together with co-registered LiDAR elevation data. The dataset contains 17 annotated land-cover classes, including built structures, bare soil, asphalt, vegetation, and shadow, at a spatial resolution of 2 m. Compared with Houston2013, Augsburg exhibits stronger spectral correlations together with greater class diversity, providing a challenging benchmark for evaluating cross-modal feature alignment and generalization.

The MUUFL Gulfport dataset was collected over a university campus and represents a semi-urban environment containing abundant vegetation together with small man-made structures. The dataset includes denoised 64-band hyperspectral imagery together with dual-band LiDAR measurements consisting of elevation and intensity information. It contains 11 land-cover classes and exhibits fine spatial detail, mixed pixels, and varying illumination conditions, making it well suited for evaluating multimodal feature fusion for small-object classification.

Collectively, these three benchmark datasets encompass substantial variation in spatial resolution (1–2.5 m), spectral dimensionality (64–224 bands), scene complexity, and land-cover composition. Consequently, they provide a diverse benchmark for evaluating the effectiveness and generalizability of multimodal land-cover classification methods. The principal characteristics of the datasets are summarized in Table 1. The train/test partitions for all three datasets correspond to the official benchmark splits provided by the original data providers. Specifically, the Houston2013 dataset uses the official partition from the 2013 IEEE GRSS Data Fusion Contest, comprising 2,832 training samples and 12,197 test samples. The Augsburg dataset uses the officially provided mask_train and mask_test annotations, yielding 4,538 training samples and 47,896 test samples. The MUUFL dataset uses the official train_test_gt.mat partition, containing 28,645 training samples and 24,283 test samples. For each dataset, a 32 × 32-pixel image patch centered on each labeled pixel was extracted as an individual training or test sample. No additional data splitting or resampling was performed.

Property	Houston2013	Augsburg	MUUFL
HSI image size (pixels)	349 × 1905	1152 × 480	325 × 220
LiDAR image size (pixels)	349 × 1905	1152 × 480	325 × 220
HSI spectral bands	144	224	64
LiDAR bands	1	1	2
HSI wavelength range	0.38–1.05 µm	0.40–1.00 µm	375–1050 nm
LiDAR wavelength range	Not applicable	Not applicable	Not applicable
Spatial resolution (HSI)	2.5 m	2.0 m	0.54 m × 1.00 m
Spatial resolution (LiDAR)	2.5 m	2.0 m	0.60 m × 0.78 m
Number of land-cover classes	15	17	11

Table 1: Characteristics of the three multimodal remote sensing benchmark datasets used for evaluation. The table summarizes the image dimensions, spectral characteristics, spatial resolutions, and numbers of land-cover classes for the Houston2013, Augsburg, and MUUFL benchmark datasets used to evaluate the proposed FlowErs framework.

Implementation Details

The FlowErs framework was implemented in PyTorch and trained and evaluated using a graphics processing unit (GPU) with 80 GB of memory. Training was performed for 100 epochs using a batch size of 16. The AdamW optimizer was employed with an initial learning rate of 1 × 10⁻⁴ and a weight decay of 1 × 10⁻². A cosine annealing learning-rate scheduler was used to update the learning rate throughout training. A dropout rate of 0.1 was applied to improve model generalization and reduce overfitting. The cross-entropy loss function was used for supervised optimization. For reproducibility, all experiments were initialized using a fixed random seed of 3407. The supplied train/test partitions were used without creating an additional validation split. Each input patch was resized to 32 × 32 pixels before training. No data augmentation, explicit image-registration correction, or additional data-loader normalization was applied. Pixels associated with negative labels were excluded from the supervised loss calculation. Missing or noisy pixel handling was not evaluated separately.

Evaluation Metrics

Three widely adopted evaluation metrics were used to assess classification performance: OA, Average Accuracy (AA), and the Kappa coefficient (k). OA measures the proportion of correctly classified samples among all samples, AA represents the mean classification accuracy across all land-cover classes, and the Kappa coefficient quantifies the agreement between the predicted and reference classifications after accounting for chance agreement. Higher values of all three metrics indicate better classification performance. All values reported in Tables 2–6 are fixed-split point estimates obtained using the random seed 3407. No confidence intervals, statistical significance tests, or p-values are reported.

Here, N_c and N_a denote the total numbers of correctly classified samples and evaluated samples, respectively.  and  denote the numbers of correctly classified samples and total samples for the i-th class, respectively. In the calculation of the Kappa coefficient, P_e denotes the probability of agreement occurring by chance, and  and  denote the reference and predicted sample counts for the i-th class, respectively.

Results

The overall evaluation workflow of the proposed FlowErs framework is illustrated in Figure 1, which summarizes the two-stage multimodal classification strategy. The modality-specific feature extraction process based on the MetaFormer encoder is shown in Figure 2, and the principal characteristics of the three benchmark datasets used for evaluation are summarized in Table 1. Classification performance was subsequently evaluated on the MUUFL, Hous...

Discussion

The present study introduces FlowErs, a multimodal remote sensing framework that separates cross-modal feature alignment from semantic classification through a two-stage learning strategy. Rather than directly fusing heterogeneous modality-specific features, the proposed framework first aligns feature representations using conditional flow matching before multimodal fusion and classification. This design addresses the long-standing challenge of heterogeneous feature distributions between HSI and LiDAR, where differences ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
AdamW optimizer	PyTorch Foundation	Included in PyTorch	Used for model optimization. PyTorch 2.5.0 was used.
Augsburg benchmark dataset	Hu J et al. (Earth System Science Data, 2022); Remote Sensing Technology, LMU Munich	https://doi.org/10.5281/zenodo.7185498	Public multimodal remote sensing benchmark dataset used for model evaluation.
CUDA Toolkit	NVIDIA	CUDA Toolkit 12.4; https://developer.nvidia.com/cuda-12-4-0-download-archive	Used for GPU-accelerated model training and inference.
Graphics processing unit (GPU)	NVIDIA Corporation	NVIDIA A100 80 GB PCIe	Used for model training and evaluation.
Houston2013 benchmark dataset	IEEE Geoscience and Remote Sensing Society Data Fusion Contest	http://www.grss-ieee.org/community/technical-resources/data-fusion/2013-grss-data-fusion-contest/	Public hyperspectral imagery and LiDAR benchmark dataset used for model evaluation.
MUUFL Gulfport benchmark dataset	University of Florida (Gader P et al., Technical Report REP-2013-570)	https://github.com/GatorSense/MUUFLGulfport	Public hyperspectral imagery and LiDAR benchmark dataset used for model evaluation.
Operating system	Canonical Ltd.	Ubuntu 22.04 LTS; https://ubuntu.com	Computational environment used for model development, training, and evaluation.
PyTorch	PyTorch Foundation	PyTorch 2.5.0; https://pytorch.org/get-started/previous-versions/#v250	Deep learning framework used to implement, train, and evaluate the FlowErs model.
Python	Python Software Foundation	Python 3.11; https://www.python.org/downloads/release/python-3110/	Programming language used to implement and execute the computational workflow.