Research Article

RareCode: An Unsupervised Deep Learning Framework for Anomaly Detection in Colorectal Cancer Histopathological Images

9 views

⸱

DOI:

10.3791/72303

⸱

September 22nd, 2026

 ,  ,  ,  ,  ,  ,  ,  , 
1School of Computer Science and Engineering, Anhui University of Science and Technology, 2School of Medicine, Anhui University of Science and Technology, 3School of Electrical and Information Engineering, Anhui University of Science and Technology, 4The First Hospital, Anhui University of Science and Technology, 5School of Safety Science and Engineering, Anhui University of Science and Technology, 6Department of Pathology, Shenzhen People's Hospital, The Second Clinical Medical College, Jinan University, 7The First Affiliated Hospital, Southern University of Science and Technology, 8Anhui Provincial Academician Workstation of Anhui University of Science and Technology for Autoimmune Disease Diagnostic Technology

In This Article

Summary

This study presents RareCode, an unsupervised deep learning framework that detects anomalies in colorectal cancer histopathological images by analyzing Codebook Activation Rarity (CAR), with the potential to assist pre-screening without labeled training data.

Abstract

Unsupervised anomaly detection in histopathological images has been widely explored using reconstruction-based approaches that measure reconstruction errors, yet these methods often fail to capture subtle, semantic-level pathological variations due to the inherent heterogeneity of tissue textures. To address this limitation, this study presents RareCode, a vector quantization-based framework that extends the detection paradigm beyond pixel-level reconstruction to incorporate semantic-level codebook activation analysis. The core innovation is Codebook Activation Rarity (CAR) scoring, which profiles the activation frequency of each codebook entry during training on exclusively normal samples and flags infrequent activations as anomaly indicators at inference time, thereby complementing reconstruction errors with semantic-level discrimination. Building upon this single-scale CAR mechanism, a Multi-scale Hierarchical Codebook (MHC) module is further introduced, employing codebooks of varying sizes with learnable fusion weights to capture pathological patterns spanning from coarse tissue-level structures to fine cellular-level details. Leveraging the multi-scale design, hierarchical anomaly heatmaps are generated at each codebook granularity, providing pathologists with interpretable, multi-dimensional visual localization cues indicating both where and at what structural level anomalies occur. Five-fold cross-validation on a clinically annotated colorectal cancer histopathological image dataset obtained from Shenzhen People's Hospital demonstrates that RareCode achieves an area under the curve (AUC) of 96.82%, outperforming baseline methods. Per-class analysis showed stronger performance for cancer detection (AUC = 99.44%, specificity = 94.21%) than for inflammation detection (AUC = 94.30%, specificity = 60.44%). These findings suggest that CAR analysis may offer a promising unsupervised approach for histopathological anomaly detection, potentially aiding clinical pre-screening in colorectal cancer diagnosis.

Introduction

Reliable histopathological screening for colorectal cancer remains constrained by expert annotation and manual review, particularly when inflammatory lesions and malignancies show overlapping morphology1,2. These constraints motivate unsupervised computational approaches that learn from normal tissue without requiring exhaustive abnormal annotations3,4.

The clinical urgency of automated histopathological screening is underscored by the growing imbalance between diagnostic demand and available pathology expertise, with the diagnostic workload per US pathologist rising by over 40% in a single decade as the pathologist workforce continued to shrink5. In colorectal cancer (CRC), organized screening programs have been associated with mortality reductions of 29-68%6, yet the capacity for manual pathological review remains limited by workforce availability. An automated pre-screening system that reliably identifies suspicious cases for priority review could help alleviate this burden and improve diagnostic turnaround times. However, deploying such systems requires both high sensitivity to avoid missing true positives and adequate specificity to prevent excessive false alarms7.

Unsupervised anomaly detection (UAD) has become increasingly relevant in medical image analysis8 because it can model normal tissue distributions without requiring abnormal training labels3,9,10. Reconstruction-based models, including autoencoders, variational autoencoders, generative adversarial networks, and memory-augmented variants, identify anomalies through reconstruction errors, but high-capacity decoders may still reconstruct abnormal regions with high fidelity11,12,13,14,15,16,17. Feature-based methods such as PatchCore and PaDiM use pretrained representations, although features learned from natural images may not fully capture microscopic atypia in histopathology18,19,20,21. Pathology-specific foundation models and diffusion-based anomaly detectors provide powerful alternatives, but their data requirements or computational costs may limit their direct use in high-throughput screening22,23. Recent automated and evolutionary methods, such as EvoAAE24 and MoARNN-AM25, further illustrate the value of adaptive model optimization for anomaly detection, although their application contexts differ from histopathological image analysis. In contrast, vector quantization (VQ)-based methods impose discrete codebook constraints that can reduce identity mapping; however, existing VQ-based approaches still rely primarily on spatial reconstruction errors and underuse the semantic information contained in codebook activation patterns26,27.

Although the aforementioned methods demonstrate promising performance in general domains, their application to complex histopathological scenarios faces specific challenges. Histopathological images are characterized by complex tissue heterogeneity across multiple structural levels28. In practice, image analysis typically operates on local image patches extracted from tissue sections, and pathological diagnosis is inherently cross-scale: pathologists assess glandular and tissue morphology at low magnification while examining nuclear pleomorphism and mitotic figures at high magnification29. We identify three key limitations of current approaches: First, pathological anomalies and normal tissues exhibit high similarity in low-level visual features such as staining styles and local textures, causing reconstruction-based methods to fail in detecting subtle anomalies, as both normal and anomalous samples may produce similar reconstruction quality at the semantic level. Second, most existing methods employ single-scale feature extraction, making it challenging to simultaneously capture anomalous features at both tissue and cellular levels30,31. Third, although mainstream methods can generate pixel-level heatmaps, they often fail to intuitively capture the hierarchical attributes of anomalies, limiting the usefulness of such models as clinical diagnostic aids.

To address the above challenges, we propose RareCode, a UAD framework that explores codebook activation patterns and hierarchical feature representation at multiple levels of abstraction. The framework introduces three main components: (1) Codebook Activation Rarity (CAR) Scoring Mechanism, which computes rarity scores based on entry activation frequencies trained exclusively on normal samples, distinguishing normal from anomalous samples at the semantic level and providing complementary discriminative signals to traditional spatial reconstruction errors. (2) Multi-scale Hierarchical Codebook (MHC) Fusion Architecture, which captures pathological features at multiple levels of granularity—from coarse structural patterns to fine-grained cellular details—by employing codebooks with varying capacities, with adaptive fusion achieved through learnable weights. (3) Hierarchical Interpretable Localization Module, which leverages the multi-scale architecture to generate anomaly heatmaps at different granularities, providing pathologists with multi-scale semantically interpretable diagnostic references.

The primary hypothesis of this study was that codebook activation-frequency profiles derived from codebooks trained exclusively on normal colorectal histopathological images would encode anomaly-relevant semantic information beyond pixel-level reconstruction error, and that multi-scale integration of these complementary signals would improve the discrimination of images containing cancerous or inflammatory tissue from normal images relative to representative UAD methods, as assessed primarily by AUC. To test this hypothesis, we evaluated RareCode using 5-fold cross-validation on a clinically annotated CRC dataset and performed ablation and localization analyses to examine the contributions and interpretability of its core components.

Protocol

This study was approved by the Clinical Research Ethics Committee of Shenzhen People's Hospital (Approval No. LL-KY-2025300-01). The requirement for informed consent was waived by the ethics committee because this retrospective study used existing histopathological images and clinical materials without direct patient contact or intervention. All clinical data and histopathological images were de-identified before analysis, and no personally identifiable information was used in this study. All data used in this research were handled in accordance with the ethical standards of the institutional and national research committees.

Framework overview
The overall architecture of the proposed UAD framework, RareCode, is illustrated in Figure 1. The framework employs a parallel dual-branch architecture. For each 512 x 512 input image, non-overlapping 32 x 32 and 64 x 64 patches are extracted as fine- and coarse-scale inputs, respectively. The 64 x 64 patches are resized to 32 x 32 before being passed to the network so that both branches share the same encoder-decoder input size while preserving different receptive fields. Encoders in each branch map the extracted features into a latent space, where the MHC module imposes discretization constraints. Subsequently, decoders reconstruct the image from quantized features to compute spatial-domain reconstruction errors. The CAR mechanism then measures the degree of semantic-level anomaly by profiling codebook activation frequencies. Finally, the model fuses reconstruction scores and CAR scores through a weighted combination to obtain image-level anomaly scores. The RareCode framework is trained end-to-end using only normal samples, requiring no anomaly annotations. The dual-branch design was used to capture both local cellular features and broader glandular structures. The MHC module was used to model these features with codebooks of different capacities.

Encoder-decoder architecture
RareCode constructs structurally independent encoders and decoders for the fine- and coarse-scale branches. Each branch uses the same encoder-decoder design but does not share parameters. The encoder consists of four convolutional blocks, each including a 3 x 3 convolution, batch normalization, ReLU activation, and dropout. The channel width increases from 64 to 128, then to 256, and the final feature map is flattened and projected into a 64-dimensional latent embedding. The decoder mirrors the encoder with a fully connected projection layer followed by four transposed convolution layers, reconstructing each patch to the original input size of the network. Reconstruction error is calculated as the mean squared error between the input and reconstructed patches. By combining this encoder-decoder structure with discrete codebook constraints, RareCode learns compact representations of normal tissue and measures deviations during inference.

Multi-scale hierarchical codebook
To capture pathological features at multiple levels of abstraction—from broad tissue patterns to localized cellular variations—the MHC module employs K discrete codebooks figure-protocol-1 with varying capacities n1, n2, ..., nK. In the final FourScales configuration, each branch contains four codebooks with 64, 128, 256, and 512 entries, respectively, and each codebook entry has a 64-dimensional embedding. For each encoded patch feature, the nearest codebook entry is selected according to Euclidean distance. The quantized outputs from different codebooks are then fused using learnable weights normalized across codebooks. Smaller codebooks, due to stronger compression constraints, are expected to encode coarse-grained prototypical patterns that abstract away local variations, whereas larger codebooks preserve finer-grained features that capture more specific structural characteristics. For each codebook figure-protocol-2, where ek(i) figure-protocol-3 Rd denotes the i-th embedding vector in the k-th codebook, continuous features are mapped to the discrete codebook space via nearest neighbor lookup (Eq. 1 and 2):

figure-protocol-4

figure-protocol-5

To achieve adaptive fusion of multi-scale features, we introduce learnable weights . After softmax normalization, weighted summation is performed over the quantized outputs from each codebook (Eq. 3):

figure-protocol-6

where σ(·) denotes the softmax function, ensuring that weights satisfy Σk σ(wk) = 1. This design enables the model to automatically adjust the contribution ratios of codebooks at different granularities based on the semantic content of the input features.

To optimize codebook learning, we adopt the standard VQ loss function (Eq. 4):

figure-protocol-7

where sg[·] denotes the stop-gradient operation, and β = 0.25 is the commitment loss weight coefficient. The first term encourages codebook vectors to move toward encoder outputs, while the second term encourages encoder outputs to remain consistent with their corresponding codebook vectors. The complete mechanism of multi-scale VQ is illustrated in Figure 2.

Codebook activation rarity scoring
CAR measures semantic-level anomalies based on codebook activation statistics estimated from normal training samples. After model training, all normal training images were passed through the encoder and MHC module without data augmentation. For each branch and each codebook, the assignment index of every patch feature was recorded, and the number of assignments to each codebook entry was counted. The counts were then normalized to obtain the activation-frequency distribution for that codebook (Eq. 5):

figure-protocol-8

During inference, each test image was processed through the same patch extraction, encoding, and nearest-neighbor codebook assignment procedure. For each assigned codebook entry, the rarity score is calculated as the negative logarithm of its activation frequency in the normal training set (Eq. 6):

figure-protocol-9

where ε is a small constant used for numerical stability. A low-frequency codebook entry, therefore, receives a higher rarity score, suggesting that the corresponding patch feature is less consistent with the learned normal-tissue distribution. Patch-level rarity scores were averaged within each image and across the fine- and coarse-scale branches to obtain an image-level rarity response.

To complement activation-frequency rarity, we also compute a percentile-based quantization distance score (Eq. 7):

figure-protocol-10

This score is derived from the Euclidean distance between each encoded feature vector and its nearest codebook entry. The distance distribution is estimated from normal training samples, and test-sample distances are converted into percentile scores. Larger percentile scores indicate that the encoded feature is more difficult to represent using the learned normal codebook prototypes.

The final CAR score combines activation rarity and quantization distance across all codebooks using the learnable codebook weights (Eq. 8):

figure-protocol-11

The same normalized codebook weights used for multi-scale feature fusion are applied to score-level fusion, maintaining consistency between representation learning and anomaly scoring. The resulting CAR score is calculated at the image level and is then combined with the reconstruction score to obtain the final anomaly score. The overall CAR scoring mechanism is illustrated in Figure 3.

Final anomaly score computation
We fuse spatial-domain reconstruction errors with semantic-level CAR scores. Before fusion, both scores are separately normalized using percentile-based min-max normalization to reduce the influence of extreme outliers. The final image-level anomaly score is then defined as (Eq. 9):

figure-protocol-12

where Srecon denotes the normalized mean squared reconstruction error, reflecting sample reconstruction quality in pixel space; SCAR is the CAR score described above, capturing degrees of semantic-level anomaly. The hyperparameter α figure-protocol-13 [0,1] controls the relative weighting between the two components, with optimal values determined on validation sets.

Training objective
The RareCode framework is trained end-to-end using only normal samples. The total training loss comprises the following components (Eq. 10):

figure-protocol-14

Each term is defined as follows: Reconstruction Loss: figure-protocol-15, measuring the mean squared error between input patches p and reconstructed patches D(zq) after quantization. This loss is applied separately to the 32 × 32 patch branch (figure-protocol-16) and 64 × 64 patch branch (figure-protocol-17) to ensure effective learning of features at both spatial scales. 

Dataset description
To evaluate RareCode, experiments were conducted on a clinically annotated CRC histopathological image dataset comprising images of colorectal tissue sections stained with hematoxylin and eosin (H&E), obtained from Shenzhen People's Hospital. All images in this dataset originate from real clinical cases. Images were digitized at 40x magnification with an original resolution of 1024 × 1024 pixels. To ensure annotation reliability and quality, all sample category labels were jointly reviewed and confirmed by two or more senior professional pathologists following a double-blind protocol.

Based on histopathological characteristics, the dataset was categorized into three classes: normal tissue (1,226 images), cancer (1,215 images), and inflammation (1,214 images). In the binary UAD setting, cancer and inflammation samples were treated as anomalous, while normal samples served as the reference distribution. This formulation reflects the intended triage task of separating cases requiring further pathological review from those that are morphologically normal.

All H&E-stained images were resized to 512 × 512 pixels before being used as network input. Stratified 5-fold cross-validation was used for evaluation. In each fold, the normal samples were first divided into training, validation, and held-out test subsets. RareCode was trained only on the training normal samples. The validation subset, which also contained only normal samples, was used for model selection, hyperparameter tuning, and selection of the fusion weight α. The held-out test set, consisting of unseen normal samples and anomalous samples including cancer and inflammation, was used only for final performance evaluation and was not used for model training, hyperparameter selection, or α selection. The representative categories, dataset structure, and validation protocol are shown in Figure 4.

Implementation details
The proposed RareCode framework and comparison baselines were implemented using PyTorch. All experiments were conducted on a workstation equipped with a single NVIDIA GeForce RTX 4060 Ti GPU (16 GB). For network architecture configuration, to capture features at different spatial extents, the dual-branch architecture processes non-overlapping patches at two scales: 32 × 32 pixels (smaller receptive field, capturing local texture patterns) and 64 × 64 pixels (larger receptive field, capturing broader spatial context), respectively. The continuous feature embedding dimension produced by both branch encoders is uniformly set to 64. In the MHC module, we configure four codebooks with different capacities for each branch, with discrete prototype quantities of 64, 128, 256, and 512, respectively.

During optimization, the network was trained end-to-end for 50 epochs with a batch size of 8. We employ the AdamW optimizer for weight updates, with an initial learning rate of 5 × and weight decay of to prevent overfitting. Additionally, to ensure smooth training convergence, a cosine annealing learning rate schedule was used, enabling finer optimization in later training stages. Patch sizes and codebook sizes were selected to balance multi-scale representation and computational cost. The final FourScales configuration was supported by ablation analysis, and α was selected on the validation set before test-set evaluation.

For the decoder-complexity ablation, a complex-decoder variant was implemented based on the FourScales configuration. This variant used the same data split, codebook sizes, training epochs, optimizer, learning rate, validation-based α selection, and evaluation metrics as the FourScales model. The basic decoder was replaced with an EMCAD-style decoder containing multi-scale depthwise convolution blocks and Convolutional Block Attention Modules (CBAM). After each of the first three transposed-convolution upsampling stages, parallel 3 × 3, 5 × 5, and 7 × 7 depthwise convolutions were applied, followed by 1 × 1 pointwise fusion, batch normalization, ReLU activation, a residual connection, and CBAM attention. CBAM included channel attention based on average- and max-pooling descriptors and spatial attention using a 7 × 7 convolution. The final stage used transposed convolution and sigmoid activation to reconstruct 32 × 32 patches.

Results

Comparison with baseline methods
To evaluate the RareCode framework, multiple representative UAD methods were selected as baselines, spanning reconstruction-based methods (CAE32, VAE12, SAE33, MAE34, PatchSAE35), memory-augmented reconstruction (MemAE36), knowledge distillation (STFPM37), synthetic anomaly generation (DRAEM38), and pretrained feature extraction (PatchCore18). Baseline selection focused on methods that can be trained or applied under comparable computational budgets and data assumptions (access only to unlabeled normal training samples), thereby ensuring that performance differences reflect methodological contributions rather than disparities in pretraining data scale. To ensure fair comparison, all methods were evaluated using the same 5-fold data splits, preprocessing pipeline, evaluation metrics, and computational environment. Models were trained under the same UAD protocol using only normal training samples, with hyperparameters and thresholds selected on the validation set and final performance assessed only on the held-out test set. Table 1 summarizes 5-fold cross-validation results, and Figure 5 provides multi-metric radar chart comparisons.

As shown in Table 1 and Figure 5, RareCode achieves favorable detection performance and statistical stability on the CRC dataset. In terms of AUC, RareCode (96.82 ± 0.23%) outperforms multiple reconstruction-based baseline models, including MemAE (94.97 ± 0.81%). Statistical analysis confirms that RareCode outperforms all reconstruction-based baselines (paired t-tests, p < 0.05), with the improvement over MemAE reaching statistical significance (p < 0.01). RareCode exhibits low performance variance (standard deviation of 0.23%), indicating consistent cross-fold stability. The limited effectiveness of DRAEM, based on synthetic anomalies, likely reflects the fact that simple texture overlay strategies cannot adequately simulate complex pathological atypia.

Regarding specificity, RareCode achieves 58.24 ± 5.99%, outperforming all comparison methods in reducing false positive rates. This represents approximately a 25-percentage-point improvement over the reconstruction-only baseline (NoCAR, 33.76%), demonstrating the CAR mechanism's contribution to reducing false positives. The lower specificity of STFPM and PatchCore, which rely natural-image-pretrained features, may partially reflect the domain gap between natural and histopathological images. Notably, STFPM and DRAEM exhibit high specificity variance (±33.53% and ±7.09%), with per-fold specificity ranging from near-zero to moderate levels, raising concerns about their deployment reliability.

Per-class detection performance
Per-class analysis was conducted by separately comparing cancer and inflammation samples against normal samples (Table 2). RareCode achieved higher performance for cancer detection (AUC = 99.44 ± 0.12%, recall = 98.13 ± 0.29%, specificity = 94.21 ± 2.22%) than for inflammation detection (AUC = 94.30 ± 0.44%, recall = 96.55 ± 0.78%, specificity = 60.44 ± 4.34%). These findings suggest that RareCode may show stronger discrimination for malignant abnormalities than for inflammatory changes, whereas the inflammation-normal boundary appears to contribute substantially to the reduced overall specificity.

Ablation study
To investigate the contributions of key components within the RareCode framework, ablation experiments were conducted to assess the impact of the CAR mechanism and the MHC module on detection performance. Results are detailed in Table 3. As shown in Figure 6A, adding CAR to the reconstruction-only baseline improved AUC from 92.81% to 95.92%, and multi-scale codebook fusion further increased AUC to 96.82%. Figure 6B shows that CAR also improved specificity, increasing it from 33.76% in the NoCAR baseline to 58.24% in the full FourScales model. These results support the complementary value of codebook activation rarity and multi-scale fusion.

Additionally, a decoder-complexity ablation was conducted using the FourScales configuration. As shown in Table 3, the complex-decoder variant with multi-scale depthwise convolution blocks and Convolutional Block Attention Modules (CBAM) showed lower AUC than the basic FourScales model (95.17 ± 0.44% vs. 96.82 ± 0.23%). This result suggests that, within the present VQ-AE-based RareCode setting, increasing decoder capacity did not improve anomaly detection performance and may reduce the effectiveness of the codebook-constrained representation.

Hierarchical spatial response analysis
To examine the spatial responses generated by RareCode, patch-level reconstruction error and codebook rarity maps were aggregated at the image level and compared between normal and anomalous samples. Energy ratio, Cohen's d, and AUC were used to quantify image-level response separation. Detailed results are presented in Table 4 and Figure 7 and Figure 8.

Results show that codebook rarity scores at all scales exhibit response elevation on anomalous samples (energy ratios > 1). Smaller-capacity codebooks (Codebook 64) achieve stronger localization-level discrimination (78.79% AUC), consistent with the expectation that higher compression rates encourage learning of more abstract prototypical patterns that are more sensitive to deviations from normal tissue archetypes. Reconstruction error alone provides substantial anomaly capture capability (93.31% AUC), while CAR scores offer complementary signals from the semantic frequency dimension.

Qualitative examination of Figure 7 showed that elevated responses in cancer samples overlapped with irregular and crowded glands, nuclear enlargement, hyperchromasia, pseudostratification, and loss of epithelial polarity. In inflammatory samples, stronger responses were observed around distorted crypts and dense inflammatory-cell infiltrates. Lower-capacity codebooks tended to capture broader architectural deviations, whereas higher-capacity codebooks produced more localized cellular-level responses. These findings provide qualitative morphological interpretation but not pixel-level validation.

Fusion parameter analysis
The fusion weight α balances the contributions of spatial reconstruction error and the CAR score to the final anomaly score. Optimal α values for each fold were determined via a grid search (step size 0.05) on the validation sets, with the results presented in Table 5 and  Figure 9. Optimal α values primarily concentrate in the 0.85-0.95 range, with a mean of 0.90 ± 0.05. Under optimal configurations, the model achieves an average AUC of 96.82 ± 0.23% on test sets. Higher optimal weights (approximately 0.90) suggest that CAR scores contribute more to final detection than reconstruction error, while reconstruction error provides auxiliary local constraints. Compared with the α = 0 setting in the sensitivity analysis (AUC = 93.29%), the validation-selected fusion setting improved AUC by approximately 3.53 percentage points.

To summarize the present work presents the RareCode framework for unsupervised anomaly detection in histopathological images, aiming to mitigate the identity-mapping problem encountered by traditional generative models. The CAR mechanism reduces over-reliance on pixel-level reconstruction errors, while the MHC module provides hierarchical feature representations at multiple granularities, enabling complementary anomaly discrimination at different levels of abstraction. Experiments on the CRC dataset demonstrate that RareCode achieves 96.82% AUC, with per-class analysis showing effective cancer detection (AUC = 99.44%) and stronger discrimination for cancer than inflammation, suggesting that inflammation-normal overlap remains a major challenge. Ablation studies confirm that integrating discrete semantic features from VQ with multi-scale tissue morphological representations improves detection performance. RareCode's high recall (98.40%) and moderate specificity (58.24%) suggest its potential as a pre-screening tool for prioritizing suspicious histopathological images; however, its actual effect on pathologist workload will require prospective workflow evaluation.

Data Availability:
The CRC histopathological image dataset used in this study was obtained from Shenzhen People's Hospital under institutional ethics approval (Approval No. LL-KY-2025300-01). Due to patient privacy and institutional data sharing policies, the dataset is not publicly available. Access may be granted upon reasonable request to the corresponding author, subject to institutional approval and data use agreements. The source code for the RareCode framework is publicly available at https://github.com/XL-alg/RareCode.

figure-results-1
Figure 1: Overall architecture and data flow of the RareCode framework. A 512 × 512 H&E-stained histopathological image is divided into non-overlapping 32 × 32 and 64 × 64 patches as fine- and coarse-scale inputs. The two branches encode patches into latent embeddings, apply Multi-scale Hierarchical Codebook (MHC) discretization, and reconstruct patches to compute reconstruction-error scores. In parallel, the Codebook Activation Rarity (CAR) mechanism estimates semantic rarity from codebook activation-frequency profiles learned from normal training samples. Normalized reconstruction and CAR scores are then fused to generate the final image-level anomaly score, while patch-level responses are projected back to the image plane for hierarchical localization. Please click here to view a larger version of this figure.

figure-results-2
Figure 2: Multi-scale vector quantization mechanism. (A) Distance computation between encoder output features and codebook embedding vectors. (B) Nearest neighbor selection and weighted fusion across codebooks with capacities of 64, 128, 256, and 512. (C) Reconstruction from quantized features via transposed convolutional decoder. Please click here to view a larger version of this figure.

figure-results-3
Figure 3: CAR scoring mechanism. This figure illustrates four stages: Stage 1: activation frequency statistics are computed solely on normal training samples. Stage 2: test samples obtain activation indices and distances through encoder and vector quantization. Stage 3: rarity scores and distance scores are computed based on training-set frequency profiles. Stage 4: scores from each codebook scale are fused through learnable weights to produce the final CAR score. Please click here to view a larger version of this figure.

figure-results-4
Figure 4: CRC dataset composition and experimental protocol. (A–C) Representative H&E-stained histopathological images of normal colorectal tissue, colorectal cancer, and colorectal inflammation. (D) 5-fold cross-validation protocol for UAD, in which normal samples were used for training and validation, while held-out normal and anomalous samples were used for testing. Images were digitized at 40x magnification and processed into 32 x 32 and 64 x 64 patches. Green, light green, and orange indicate training, validation, and test sets, respectively. Please click here to view a larger version of this figure.

figure-results-5
Figure 5: Multi-metric comparison radar chart of anomaly detection methods. Radar chart comparing RareCode against baseline methods across AUC, average precision (AP), F1-score, precision, recall, and precision at 90% recall (P@R90). All values represent mean performance across 5-fold cross-validation. Please click here to view a larger version of this figure.

figure-results-6
Figure 6: Incremental contribution analysis of ablation study components. (A) AUC performance comparison showing the progressive improvement from reconstruction-only baseline (NoCAR) through single-codebook to multi-scale configurations. (B) Specificity performance comparison demonstrating the CAR mechanism's effect on false positive reduction. All error bars represent standard deviation (SD) across 5-fold cross-validation. Please click here to view a larger version of this figure.

figure-results-7
Figure 7: Multi-scale codebook hierarchical localization heatmaps. Representative examples are shown for (A) normal, (B) cancer, and (C) inflammation samples. Each row includes the original image, overlay, reconstruction-error map, rarity-score maps from codebooks of different capacities (64, 128, 256, and 512), and the final fused localization map. Smaller-capacity codebooks highlight coarse-grained patterns through stronger compression abstraction, whereas larger-capacity codebooks preserve finer structural details. High-response regions qualitatively correspond to cancer- or inflammation-related histopathological features, but were not validated with pixel-level expert annotations. Please click here to view a larger version of this figure.

figure-results-8
Figure 8: Distribution histograms of different score types. Histograms compare score distributions between normal and anomalous samples for (A) reconstruction error, (B) combined CAR score, (C) Codebook 64, (D) Codebook 128, (E) Codebook 256, and (F) Codebook 512. Green and red histograms indicate normal and anomalous samples, respectively. Please click here to view a larger version of this figure.

figure-results-9
Figure 9: Alpha parameter sensitivity analysis. (A) Validation-set AUC used for α selection. (B) Test-set AUC under different α values. The validation-selected α values ranged from 0.85 to 0.95, with a mean of 0.90 ± 0.05, consistent with Table 5. AUC variation remained below 0.5% when α varied within the [0.70, 1.00] range, indicating stable performance across a wide parameter range. Please click here to view a larger version of this figure.

MethodAUC (%)AP (%)F1 (%)Precision (%)Recall (%)Specificity (%)P@R90 (%)
CAE90.37 ± 0.9498.62 ± 0.1895.34 ± 0.1591.64 ± 0.4099.35 ± 0.2228.14 ± 3.8895.66 ± 0.32
SAE90.58 ± 0.9498.66 ± 0.1895.34 ± 0.1591.67 ± 0.4099.32 ± 0.2428.46 ± 3.9795.71 ± 0.30
VAE93.60 ± 0.8299.13 ± 0.1295.86 ± 0.1992.81 ± 0.4999.12 ± 0.2939.07 ± 4.7096.94 ± 0.43
MAE91.65 ± 2.7598.76 ± 0.5095.61 ± 0.6092.87 ± 1.5698.55 ± 0.5539.74 ± 14.3296.55 ± 0.97
MemAE94.97 ± 0.8199.35 ± 0.1195.78 ± 0.3392.82 ± 1.0798.95 ± 0.6039.15 ± 9.9997.57 ± 0.33
PatchSAE94.86 ± 0.3699.34 ± 0.0595.39 ± 0.1392.32 ± 0.2798.67 ± 0.1234.91 ± 2.5798.13 ± 0.24
STFPM90.23 ± 3.0598.70 ± 0.4494.32 ± 0.2191.90 ± 3.5197.14 ± 3.3829.82 ± 33.5395.93 ± 1.96
DRAEM76.34 ± 2.8295.34 ± 0.9094.19 ± 0.0789.39 ± 0.6599.56 ± 0.656.19 ± 7.0992.69 ± 0.57
PatchCore74.64 ± 1.8294.76 ± 0.5594.73 ± 0.0590.52 ± 0.1599.36 ± 0.1217.49 ± 1.5492.54 ± 0.15
RareCode96.82 ± 0.2399.59 ± 0.0396.63 ± 0.1594.93 ± 0.6798.40 ± 0.4858.24 ± 5.9998.80 ± 0.10

Table 1: Comparison results with baseline methods (5-fold cross-validation). Detection performance of RareCode and nine baseline UAD methods on the CRC dataset. Metrics include AUC, average precision (AP), recall, specificity, F1-score, precision at 90% recall (P@R90), and precision. All values represent mean ± SD across 5-fold cross-validation. Bold values indicate the best performance for each metric.

CategoryAUC (%)AP (%)F1 (%)Recall (%)Specificity (%)
Cancer99.44 ± 0.1299.86 ± 0.0398.34 ± 0.1798.13 ± 0.2994.21 ± 2.22
Inflammation94.30 ± 0.4498.48 ± 0.1293.44 ± 0.2996.55 ± 0.7860.44 ± 4.34

Table 2: Per-class detection performance across anomaly subtypes. Separate evaluation of RareCode's detection performance for cancer and inflammation samples against normal samples. Per-class metrics, including AUC, AP, F1-score, recall, and specificity, were computed using class-specific optimal thresholds.

VariantCodebook ConfigCARAUC (%)AP (%)F1 (%)Specificity (%)P@R90 (%)
NoCAR  figure-results-10  figure-results-1192.81 ± 0.1299.05 ± 0.0295.30 ± 0.0833.76 ± 3.8296.72 ± 0.12
SingleCodebook[256]  figure-results-1295.92 ± 0.4099.47 ± 0.0596.25 ± 0.1455.37 ± 6.5198.31 ± 0.43
TwoScales[128, 512]  figure-results-1396.57 ± 0.2799.56 ± 0.0496.45 ± 0.1257.75 ± 4.1498.75 ± 0.12
ThreeScales[128, 256, 512]  figure-results-1496.36 ± 0.3799.53 ± 0.0596.52 ± 0.1757.99 ± 4.6598.60 ± 0.23
FourScales[64, 128, 256, 512]  figure-results-1596.82 ± 0.2399.59 ± 0.0396.63 ± 0.1558.24 ± 5.9998.80 ± 0.10
FourScales + complex decoder[64, 128, 256, 512]  figure-results-1695.17 ± 0.4499.39 ± 0.0695.32 ± 0.2053.83 ± 21.4598.40 ± 0.37

Table 3: Ablation study results. Performance comparison of RareCode configurations with progressive addition of components: reconstruction-only baseline (NoCAR), single-codebook CAR (SingleCodebook), and multi-scale configurations (TwoScales, ThreeScales, FourScales). An additional decoder-complexity ablation using the FourScales configuration is also included. Metrics include AUC, average precision (AP), F1-score, specificity, and precision at 90% recall (P@R90).

Score TypeEnergy RatioCohen's dAUC (%)
Reconstruction Error2.6232.00693.31
Combined CAR1.0781.00274.85
Codebook 641.1091.20778.79
Codebook 1281.0851.06776.19
Codebook 2561.0650.91672.80
Codebook 5121.0640.83370.38

Table 4: Image-level analysis of localization-derived responses. Image-averaged reconstruction error and codebook rarity responses were compared between normal and anomalous samples using the energy ratio, Cohen's d, and AUC.

FoldOptimal αVal AUC (%)Test AUC (%)
10.9596.2296.91
20.996.3896.39
30.8596.7196.82
40.8596.2296.93
50.9596.7297.05
Mean0.90 ± 0.0596.45 ± 0.2396.82 ± 0.23

Table 5: Optimal alpha values across folds. Optimal fusion weight alpha determined via grid search (step size 0.05) on validation sets for each cross-validation fold, with mean and SD statistics.

Discussion

The results suggest that codebook activation patterns may provide complementary information to spatial reconstruction errors. The improvement from NoCAR to the full FourScales model supports the potential contribution of CAR scoring, while the selected fusion weights indicate that semantic rarity may play an important role in the final anomaly score. One possible explanation is that CRC-related abnormalities can span different morphological scales, from nuclear atypia to glandular architectural disruption, and therefore may benefit from feature representations at multiple granularities14. The current model captures different spatial extents using 32 × 32 and 64 × 64 patches at the same magnification level; future integration with whole-slide image (WSI) processing frameworks may further support hierarchical slide-level analysis39.

Clinically, RareCode may be best considered as a pre-screening triage tool rather than an autonomous diagnostic system. Its high recall (98.40%) suggests a relatively low risk of missing anomalous cases40, whereas its moderate specificity (58.24%) likely reflects the difficulty of distinguishing inflammatory changes from normal tissue in an unsupervised setting. The higher cancer-specific specificity (94.21%) further suggests that false alarms may be less frequent for the most clinically critical subtype. These performance characteristics support the potential use of RareCode as a pre-screening tool for prioritizing cases that require expert assessment and streamlining pathology review5,6.

Per-class findings may further support RareCode’s potential role in triage. The relatively stronger performance for cancer than for inflammation may reflect the more pronounced architectural and cytological deviations often seen in malignant tissue, including loss of polarity, nuclear pleomorphism, and stromal desmoplasia14,39. In contrast, inflammatory changes may overlap more closely with benign tissue variation, which could partly explain the lower inflammation-specific specificity. Therefore, the overall specificity should be interpreted with caution, with cancer detection serving as a clinically important use case rather than a definitive autonomous diagnosis.

RareCode’s potential failure modes should be interpreted in the context of its one-class learning objective. Because the model learns patterns of normal tissue rather than disease-specific categories, non-neoplastic changes such as regenerative epithelium, fibrosis, necrosis, or marked inflammation may also receive high anomaly scores. Conversely, subtle dysplasia or well-differentiated carcinoma with near-normal glandular architecture may produce weaker responses. Technical factors, including staining variation, tissue folds, sectioning artifacts, defocus, and image compression, may also influence reconstruction errors and codebook activation frequencies. These considerations suggest that clinical translation would require image-quality control, stain harmonization, scanner-aware calibration, and multicenter validation39,41.

Scope of baseline comparisons. The experimental evaluation in this study focuses on methods that operate under comparable data and computational assumptions—specifically, methods that can be trained end-to-end using only small-scale unlabeled normal samples without requiring external pretraining corpora. This scope encompasses the major paradigms in unsupervised anomaly detection: reconstruction-based (CAE, VAE, SAE, MAE, PatchSAE), memory-augmented (MemAE), knowledge distillation (STFPM), synthetic augmentation (DRAEM), and pretrained feature matching (PatchCore). We acknowledge that two categories of methods were not included as direct baselines. First, pathology-specific foundation models (UNI, CONCH, CTransPath) leverage pretraining on millions of curated pathology images, and their performance advantages are primarily attributable to the scale and diversity of pretraining data rather than the anomaly detection mechanism; direct comparison would thus conflate the contributions of pretraining data with the contributions of the detection methodology. The suboptimal performance of STFPM and PatchCore—both utilizing ImageNet-pretrained backbones—empirically illustrates the impact of domain gap when general-purpose pretrained features are applied to histopathological scenarios and suggests that pathology-specific pretraining may bridge this gap. Second, diffusion-based anomaly detection methods, while promising, impose substantially higher computational overhead during inference (typically requiring hundreds of iterative denoising steps), limiting their applicability in high-throughput clinical screening workflows where processing efficiency is a practical requirement. Future work will investigate whether incorporating pathology-specific pretrained encoders into the RareCode architecture—replacing the current end-to-end-trained encoder—can further improve detection performance while preserving the CAR mechanism's interpretability advantages.

Several limitations should be acknowledged. First, validation was limited to a single-center CRC dataset, and further multicenter evaluation across institutions, scanners, and staining protocols is needed. Second, although cancer-specific specificity was encouraging, overall specificity remained affected by the inflammation-normal boundary. Third, pixel-level expert annotations were unavailable; therefore, Dice and IoU could not be calculated, and localization evaluation was limited to qualitative visualization and image-level analysis of aggregated spatial responses. Fourth, formal reader studies are still needed to determine whether hierarchical heatmaps can improve diagnostic accuracy or review efficiency. Finally, RareCode currently performs patch- and image-level analysis, and WSI-level deployment would require integration with additional processing frameworks39,41,42.

Future work should evaluate RareCode on public benchmarks and multicenter cohorts after integration with WSI processing frameworks. Prospective reader studies may help assess its potential impact on diagnostic accuracy, review time, and interobserver agreement42. Additional directions include improving computational efficiency, extending the model toward multiclass anomaly subtyping, and integrating multiple-instance learning for slide-level diagnosis28,39.

Disclosures

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. During the preparation of this work, we used ChatGPT to improve the language and readability of the manuscript and to assist with generating data visualization code. After using this tool, we reviewed and edited the content as needed and take full responsibility for the published article.

Acknowledgements

This research was supported by the Yong Dai Academician's Workstation (Diagnostic Technology for Autoimmune Diseases) (Wan Ke Science and Technology [2023] No. 317), the Graduate Innovation Fund Project of Hefei National Research Center for Health Science and Technology Joint Research Center for Occupational Medicine and Health Open Fund (No.OMH-2023-04), the Clinical and Translational Research Project of Anhui Province (No.202427610020132).

Materials

List of materials used in this article
NameCompanyCatalog NumberComments
GeForce RTX 4060 Ti GPUNVIDIAN/A16 GB VRAM; single GPU used for all experiments (RRID:SCR_022858)
MatplotlibMatplotlib Development TeamVersion 3.8.4Data visualization and figure generation (RRID:SCR_008624)
NumPyNumPy DevelopersVersion 1.26.4Numerical computing and array operations (RRID:SCR_008633)
pandaspandas Development TeamVersion 2.2.3Data organization and tabular result processing (RRID:SCR_018214)
PillowPython Imaging Library / Pillow ContributorsVersion 10.3.0Image loading and preprocessing
PythonPython Software FoundationVersion 3.12.3Programming language used for model development and data analysis (RRID:SCR_008394)
PyTorchMeta Platforms, Inc.Version 2.7.0+cu118Deep learning framework used for model implementation and training (RRID:SCR_018536)
scikit-learnscikit-learn DevelopersVersion 1.4.2Cross-validation, train-test splitting, and evaluation metrics (RRID:SCR_002577)
SciPySciPy DevelopersVersion 1.13.1Statistical analysis (RRID:SCR_008058)
torchvisionPyTorch ProjectVersion 0.22.0+cu118Image preprocessing and data transformation
tqdmtqdm DevelopersVersion 4.66.4Training and evaluation progress monitoring

References

  1. Fan J, Sun Q, Di Y, et al. DIPathMamba: A domain-incremental weakly supervised state space model for pathology image segmentation. Med Image Anal. 2025;103:103563.
  2. Nan T, Zheng S, Qiao S, et al. Deep learning quantifies pathologists' visual patterns for whole slide image diagnosis. Nat Commun. 2025;16(1):5493.
  3. Lagogiannis I, Meissen F, Kaissis G, et al. Unsupervised pathology detection: a deep dive into the state of the art. IEEE Trans Med Imaging. 2024;43(1):241-252.
  4. Tian Y, Liu F, Pang G, et al. Self-supervised pseudo multiclass pre-training for unsupervised anomaly detection and segmentation in medical images. Med Image Anal. 2023;90:102930.
  5. Metter DM, Colgan TJ, Leung ST, et al. Trends in the US and Canadian pathologist workforces from 2007 to 2017. JAMA Netw Open. 2019;2(5):e194337.
  6. Shaukat A, Levin TR. Current and future colorectal cancer screening strategies. Nat Rev Gastroenterol Hepatol. 2022;19(8):521-531.
  7. Ancker JS, Edwards A, Nosal S, et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17(1):36.
  8. Litjens G, Kooi T, Ehteshami Bejnordi B, et al. A survey on deep learning in medical image analysis. Med Image Anal. 2017;42:60-88.
  9. Stepec D, Skocaj D. Unsupervised detection of cancerous regions in histology imagery using image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2021:3785-3792.
  10. Li Y, Lao Q, Kang Q, et al. Self-supervised anomaly detection, staging and segmentation for retinal images. Med Image Anal. 2023;87:102805.
  11. Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science. 2006;313(5786):504-507.
  12. Kingma DP, Welling M. Auto-encoding variational Bayes. In: Proceedings of the 2nd International Conference on Learning Representations (ICLR). 2014.
  13. Goodfellow IJ, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. In: Advances in Neural Information Processing Systems. 2014;27:2672-2680.
  14. Lu S, Zhang W, Zhao H, et al. Anomaly detection for medical images using heterogeneous auto-encoder. IEEE Trans Image Process. 2024;33:2770-2782.
  15. Yang Z, Zhang T, Soltani Bozchalooi I, et al. Memory-augmented generative adversarial networks for anomaly detection. IEEE Trans Neural Netw Learn Syst. 2022;33(6):2324-2334.
  16. Huyan N, Quan D, Zhang X, et al. Unsupervised outlier detection using memory and contrastive learning. IEEE Trans Image Process. 2022;31:6440-6454.
  17. Jézéquel L, Beaudet J, Histace A, et al. Unified anomaly detection via multi-scale contrasted memory. IEEE Trans Image Process. 2026;35:2802-2815.
  18. Roth K, Pemula L, Zepeda J, et al. Towards total recall in industrial anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022:14298-14308.
  19. Defard T, Setkov A, Loesch A, et al. PaDiM: a patch distribution modeling framework for anomaly detection and localization. In: International Conference on Pattern Recognition. Cham: Springer; 2021:475-489.
  20. Zingman I, Stierstorfer B, Lempp C, et al. Learning image representations for anomaly detection: application to discovery of histological alterations in drug development. Med Image Anal. 2024;92:103067.
  21. Jiang Y, Cao Y, Shen W. Prototypical learning guided context-aware segmentation network for few-shot anomaly detection. IEEE Trans Neural Netw Learn Syst. 2025;36(7):12016-12026.
  22. Wang X, Zhao J, Marostica E, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature. 2024;634(8035):970-978.
  23. Frotscher A, Kapoor J, Wolfers T, et al. Unsupervised anomaly detection in medical imaging using aggregated normative diffusion. Med Image Anal. 2026;109:103895.
  24. Zeng GQ, Yang YW, Lu KD, et al. Evolutionary adversarial autoencoder for unsupervised anomaly detection of industrial Internet of Things. IEEE Trans Reliab. 2025;74(3):3454-3468.
  25. Lu KD, Zhang BX, Xu Y, et al. MoARNN-AM: multi-objective automated recurrent neural network with attention mechanism for cyber-attack detection of UAV. IEEE Trans Consum Electron. 2026;72(1):1738-1749.
  26. van den Oord A, Vinyals O, Kavukcuoglu K. Neural discrete representation learning. In: Advances in Neural Information Processing Systems. 2017;30:6306-6315.
  27. Ghafourian A, Shui H, Upadhyay D, et al. Targeted collapse regularized autoencoder for anomaly detection: black hole at the center. IEEE Trans Neural Netw Learn Syst. 2025;36(6):10348-10358.
  28. Gao Z, Mao A, Dong Y, et al. SMMILe enables accurate spatial quantification in digital pathology using multiple-instance learning. Nat Cancer. 2025;6(12):2025-2041.
  29. Wang X, Liu H, Zhang Y, et al. Joint modeling histology and molecular markers for cancer classification. Med Image Anal. 2025;102:103505.
  30. Jin H, Shen J, Cui L, et al. Dynamic graph based weakly supervised deep hashing for whole slide image classification and retrieval. Med Image Anal. 2025;101:103468.
  31. Schmitz R, Madesta F, Nielsen M, et al. Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture. Med Image Anal. 2021;70:101996.
  32. Masci J, Meier U, Ciresan D, et al. Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks. Berlin: Springer; 2011:52-59.
  33. Ng A. Sparse autoencoder. CS294A Lecture Notes. 2011;72:1-19.
  34. He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022:16000-16009.
  35. Lim H, Choi J, Choo J, et al. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In: Proceedings of the 13th International Conference on Learning Representations (ICLR). 2025.
  36. Gong D, Liu L, Le V, et al. Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019:1705-1714.
  37. Wang G, Han S, Ding E, et al. Student-teacher feature pyramid matching for anomaly detection. In: Proceedings of the 32nd British Machine Vision Conference (BMVC). 2021.
  38. Zavrtanik V, Kristan M, Skocaj D. DRAEM: a discriminatively trained reconstruction embedding for surface anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021:8330-8339.
  39. Niazi MKK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. Lancet Oncol. 2019;20(5):e253-e261.
  40. Raab SS, Grzybicki DM, Janosky JE, et al. Clinical impact and frequency of anatomic pathology errors in cancer diagnoses. Cancer. 2005;104(10):2205-2213.
  41. Schömig-Markiefka B, Pryalukhin A, Hulla W, et al. Quality control stress test for deep learning-based diagnostic model in digital pathology. Mod Pathol. 2021;34(12):2098-2108.
  42. Steiner DF, MacDonald R, Liu Y, et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol. 2018;42(12):1636-1646.

Reprints and Permissions

Tags

Unsupervised LearningCodebook ActivationVector QuantizationMulti-Scale CodebookCancer DetectionSemantic Analysis
Contact Us
Recommend to Library

Privacy

Terms of Use

Policies

Copyright © 2026 MyJoVE Corporation. All rights reserved.