Reproducible Multimodal Artificial Intelligence Workflow for Historical Digital Archive Discovery

Summary

Here, we present a protocol to improve discovery in historical digital archives by integrating post-correction optical character recognition, visual document classification, metadata enrichment, and retrieval evaluation into an auditable workflow.

Abstract

Historical digital archives are increasingly searchable, but discovery remains limited when optical character recognition errors, visually heterogeneous document types, and sparse metadata are handled separately. This study aimed to develop a reproducible protocol to evaluate whether a conservative multimodal artificial intelligence workflow can improve archival retrieval without replacing archivist review. A corpus of 1,600 digitized archival records from eight document classes was assembled, and a 420-query benchmark was used to compare five retrieval conditions: baseline indexing, optical character recognition correction alone, visual classification alone, metadata enrichment alone, and full multimodal integration. Record-level outcomes included character error rate (CER), word error rate (WER), named-entity recall, document-type classification accuracy, prediction confidence, metadata completeness, and subject-heading match. Query-level outcomes included P@10, R@10, nDCG@10, time to first relevant result, and successful-search rate. Optical character recognition correction reduced WER most strongly for handwritten letters, ledgers, and registry books; visual fine-tuning improved classification accuracy most for maps, posters, newspapers, and registry books; and metadata enrichment increased completeness across all historical periods. The full multimodal condition achieved the highest retrieval performance (P@10 = 0.624, R@10 = 0.492, nDCG@10 = 0.634) and reduced the mean time to first relevant result from 156.079 s to 58.485 s. These results support a modular, auditable workflow in which text, image, and descriptive signals are combined under explicit thresholds and human review.

Introduction

Digital archives have expanded rapidly and now support research in history, cultural heritage, public administration, and the digital humanities. However, access remains uneven because archival records often combine degraded optical character recognition output, visually diverse page layouts, inconsistent cataloging practices, and historically variable names, places, and institutions. These limitations affect not only transcription quality but also retrieval, filtering, and downstream reuse of digitized collections^{1,2,3,4,5,6,7}.

Existing document artificial intelligence and archival-retrieval studies have improved individual components such as optical character recognition post-correction, document-image classification, metadata normalization, topic modeling, word embeddings, neural retrieval, and data-governance practice^{8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25}. Yet many archival systems still evaluate these components separately, making it difficult to determine whether a combined workflow improves discovery under realistic search conditions. The methodological gap addressed here is the lack of a reproducible, auditable protocol that connects text, image, and metadata modules to retrieval outcomes in a single archival workflow.

The central objective was to test whether optical character recognition correction, visual document classification, and rule-constrained metadata enrichment produce complementary improvements when evaluated against the same retrieval benchmark.

We hypothesized that the full multimodal condition would outperform each single-component condition because archival discovery depends on the interaction of readable text, visually interpretable document structure, and searchable descriptive metadata. The workflow was designed conservatively: automated outputs could enrich retrieval indices, but low-confidence predictions were flagged for review rather than used as authoritative archival description.

Protocol

This study used digitized archival records and simulated retrieval queries as units of analysis. Archival repositories may restrict access to sensitive or culturally restricted records. Follow repository access rules, institutional data-security procedures, and de-identification requirements before processing or sharing any records. No hazardous chemical, biological, sharps, radioactive, or other regulated laboratory waste was generated by this digital workflow.

Study design and corpus construction

Figure 1 presents the complete study workflow, including corpus construction, reference labeling, image preprocessing, OCR generation and post-correction, visual classification, metadata enrichment, index construction, retrieval evaluation, statistical analysis, and reproducibility packaging.

Corpus construction was conducted from January 15 to April 30, 2025, followed by system design and debugging from May 1 to October 31, 2025. The corpus contained 1,600 digitized archival records selected from eight depositories representing state, city, religious, museum, university, and library systems. The sampling frame was stratified by repository and document class to preserve archival heterogeneity across handwritten letters, ledgers, maps, newspapers, photographs with captions, posters, registry books, and typed correspondence.

For each candidate record, the selection log recorded the repository or collection identifier, catalog identifier, document class, approximate historical period, dominant language context, available image format, image resolution, page count, and baseline descriptive fields. Inclusion required all of the following: a stable catalog identifier; at least one TIFF, JPEG, or PNG page image; source image resolution of at least 150 dpi; readable textual, tabular, or document-structural content; and assignment to one of the eight predefined document classes. Exclusion criteria were exact duplicates, blank reverse pages, images cropped to the point that more than 30% of the text-bearing area was missing, and records judged unreadable after manual inspection.

The retrieval benchmark contained 420 short natural-language queries generated between August 5 and August 20, 2025. Query generation followed a reproducible template: candidate information needs were sampled from people, institutions, places, date spans, occupations, events, and topical themes; each query was normalized to 2–9 words; and target relevant records were identified before system comparison. Each query was evaluated under five retrieval conditions, producing 2,100 matched query-condition observations.

Reference labeling and quality control

Reference labeling was performed from May 1 to July 15, 2025, by three annotators: two archival staff members experienced in descriptive cataloging and one digital-humanities researcher experienced in historical document assessment. The annotation guide defined document classes, language-context categories, historical-period bins, metadata-completeness fields, named-entity categories, and rules for selecting OCR evaluation regions.

For OCR evaluation, annotators selected representative text-bearing regions that included headings, names, dates, institutional terms, marginal notes, tabular entries, and, when present, noisy layout areas. When a page contained multiple text regions, the selected region had to be relevant to retrieval and had to include enough characters for CER and WER calculation. Disagreements were first reconciled through discussion between the two primary annotators; unresolved cases were adjudicated by the digital humanities researcher.

Quality control used a random 12% subset of records. Inter-annotator agreement for the primary document type was Cohen's kappa = 0.86, supporting substantial agreement. The adjudication log retained the original and final labels, the disagreement category, and the reason for resolution so that future users can audit class definitions and edge cases.

Image preprocessing

All images were processed at the record level using Python 3.11.8 with OpenCV 4.9.0, Pillow 10.3.0, and NumPy 1.26.4. Each input page was converted to 8-bit grayscale unless color carried semantic information, such as maps, posters, stamps, or color-coded annotations. Skew was estimated using projection profiles and Hough-line detection; deskewing was applied only when the absolute skew angle exceeded 1.5 degrees and was capped at 8.0 degrees to avoid distortion.

Contrast enhancement used contrast-limited adaptive histogram equalization with clipLimit = 2.0 and tileGridSize = 8 × 8. A median filter with a 3 × 10⁻²³ kernel was applied only when the estimated background-noise ratio exceeded 0.35. Images captured at 150–299 dpi were resampled to 300 dpi for optical character recognition; images already at 300 dpi or higher were not upscaled. No super-resolution, generative restoration, or content hallucination was used.

Bleed-through, edge darkening, uneven paper texture, marginal stamps, handwriting, ruled lines, and table boundaries were retained unless they prevented OCR region selection. This rule preserved archival evidence while standardizing image inputs sufficiently for reproducible OCR and classification.

OCR base generation and post-correction

Baseline OCR was generated with Tesseract OCR 5.3.0. The primary language pack was selected from the catalog language and visible script; multilingual decoding was enabled when the catalog language and page script did not match. OCR outputs were grouped at the record level and stored with page identifiers, OCR confidence, bounding-box coordinates when available, and raw text before correction.

Post-OCR correction used a three-stage conservative procedure. First, character-level normalization corrected frequent historical recognition confusions, including ligatures, long-s/short-s substitutions, fragmented punctuation, and digit-letter substitutions. Second, token-level correction used edit distance and frequency-ranked candidates from an archive-specific lexicon of personal names, place names, institutions, administrative terms, and historical spelling variants. Third, rule-based sequence checks validated named entities and dates against evidence from context and metadata.

Candidate replacements were accepted only when posterior correction confidence exceeded 0.80; named-entity replacements required confidence of at least 0.85. Candidates below the threshold were retained as OCR text but flagged for review. CER was calculated as the Levenshtein edit distance divided by the number of characters in the reference transcription. WER used the same formula at the token level. Named-entity recall was calculated as correctly recognized reference entities divided by all reference entities in the selected evaluation region.

Visual document classification

The visual classification module assigned each record to one of eight archival document classes: handwritten letter, ledger, map, newspaper, photograph with caption, poster, registry book, and typed correspondence. The predicted document type was used as a retrieval and filtering signal, not as an authoritative replacement for archival cataloging.

The model was implemented in PyTorch 2.2.2 and torchvision 0.17.2 using an ImageNet-pretrained EfficientNet-B3 backbone. Record-level thumbnails were resized to 384 x 384 pixels. To reduce leakage, records were split by repository and document class into training, validation, and test sets at a 70:15:15 ratio, corresponding to approximately 1,120, 240, and 240 records.

Training used AdamW with initial learning rate = 1 × 10^-4, weight decay = 1 × 10⁻², batch size = 16, label smoothing = 0.05, and early stopping when validation loss failed to improve for five consecutive epochs. Horizontal flipping was disabled because mirrored archival layouts are not realistic. Augmentation was limited to rotation from -3 degrees to +3 degrees and brightness variation within ±10%.

Epoch-level model diagnostics are summarized in 20 training rows, each containing training loss, validation loss, training accuracy, validation accuracy, macro-F1, weighted-F1, ROC-AUC, and PR-AUC.

Metadata enrichment workflow

Metadata enrichment was designed to improve discoverability while preserving archival conservatism. Existing catalog values were retained unless they contained an explicit contradiction, such as an impossible date or a document-type conflict confirmed by both OCR and visual evidence. Candidate enrichment fields included normalized title, document type, approximate date, institution mentions, geographic mentions, personal-name mentions, subject headings, and keywords.

Evidence priority followed a fixed order: (1) existing catalog metadata, (2) corrected OCR output, and (3) visual document-type prediction. Controlled vocabulary matching was used for subject headings, and nearest-neighbor semantic expansion with sentence-transformers 2.7.0 was used only when cosine similarity was at least 0.72. Date normalization required confidence of at least 0.85 or concordance between OCR and metadata. Every automatically added field retained its source, confidence, and rule identifier.

Metadata completeness was defined as the number of populated core descriptive fields divided by the number of applicable core fields for that record. Subject-heading match was defined as the presence of at least one controlled-vocabulary subject label supported by record-level content evidence and judged relevant to the query category.

Retrieval system construction

Five retrieval indices were constructed to isolate module contributions: baseline indexing; optical character recognition correction only; visual classification only; metadata enrichment only; and full multimodal integration. All indices were built in the search-engine software listed in the Table of Materials (version 8.11.1) at the record level. BM25 parameters were fixed at k1 = 1.2 and b = 0.75 before evaluation.

The baseline index used original metadata and baseline OCR text. The OCR condition replaced baseline OCR with corrected OCR. The visual condition added document-type priors and class-aware ranking boosts. The metadata condition added enriched descriptive fields. The full multimodal condition combined corrected OCR, visual priors, and enriched metadata. Field weights were fixed before testing: normalized title = 3.0, subject headings = 2.5, person and institution mentions = 2.2, place mentions = 2.0, document type = 1.8, corrected OCR body text = 1.0, and baseline OCR body text = 0.8, where applicable.

Exact phrase matching was enabled for quoted queries. Query expansion was permitted only in the metadata-enrichment and full-multimodal conditions and was limited to accepted controlled-vocabulary synonyms and subject-heading variants. Search logs were generated from August 25 to August 28, 2025, in the containerized Linux environment listed in the Table of Materials, with fixed seeds and frozen index configurations.

Query design and outcome measures

The 420 queries represented common archival search behaviors rather than keyword-only benchmark prompts. Topics included personal names, institutions, places, dates and date ranges, occupations, events, and thematic subjects. Each query was stored with its normalized wording, target record set, query-topic category, and difficulty score.

Difficulty was scored before retrieval using target ambiguity, lexical specificity, and expected frequency of expression. Composite scores below 0.40 were classified as low difficulty, scores from 0.40–0.69 as moderate difficulty, and scores of 0.70 or higher as high difficulty. Relevance judgments were completed before system comparison and then checked against the finalized collection.

Retrieval outcomes included P@10, R@10, nDCG@10, time to first relevant result, and successful search rate. P@10 was the proportion of the top 10 records judged relevant. R@10 was the proportion of all known relevant records retrieved in the top 10. nDCG@10 used graded relevance where available and binary relevance otherwise. Time to first relevant result was measured from query submission until the first relevant record appeared in the ranked output. A successful search was defined as at least one relevant result among the top 10 records.

Statistics collection

All analyses were performed using the software versions listed in the Table of Materials. Continuous outcomes were summarized as mean ± SD when approximately symmetric and as median with interquartile range otherwise. Categorical outcomes were summarized as counts and percentages.

Normality of paired differences was assessed by the Shapiro-Wilk test and distribution plots. OCR and metadata pre/post outcomes were compared with paired t tests when paired differences were approximately normal and Wilcoxon signed-rank tests otherwise. Classification performance before and after fine-tuning was compared using McNemar's test on paired correct/incorrect labels. Matched multi-arm retrieval outcomes were compared with the Friedman test followed by Holm-adjusted pairwise Wilcoxon signed-rank tests.

All statistical tests were two-tailed, and P < 0.05 was considered statistically significant. Confidence intervals were estimated using 2,000 bootstrap resamples with the random seed fixed at 2025. Effects were reported as mean differences, matched odds ratios, or rank-based effect sizes according to outcome type.

Reproducibility, scope, and resource availability

All pipeline parameters were fixed before retrieval testing. No parameter was optimized on the held-out retrieval benchmark. Configuration files, controlled vocabularies, lexicon tables, query templates, field mappings, annotation guidelines, statistical scripts, and figure-generation scripts were maintained in the version-control system listed in the Table of Materials with random seed 2025.

To support independent validation, the source-data workbook includes a de-identified 1,600-record table with 17 variables used for optical character recognition, metadata, and visual-classification analyses. Derived Table 1, Table 2, Table 3, Table 4, and Table 5, figure-source summaries, benchmark-query definitions, relevance-judgment summaries, retrieval-log surrogates, training diagnostics, lexicon categories, vocabulary-mapping rules, and annotation-guideline fields are provided as separate uploadable tables or materials files where applicable. Materials that cannot be publicly shared because of repository restrictions are represented by de-identified metadata fields and reproducible sampling fields.

This protocol evaluates retrieval-oriented discovery rather than the truth of historical claims. It does not replace archivist appraisal, provenance assessment, privacy review, or repository-specific access rules.

Results

OCR correction improved transcription, especially in handwritten and tabular records

Optical character recognition correction improved transcription quality across all eight archival document classes (n = 1,600 records). Mean WER decreased from 0.529 ± 0.172 to 0.384 ± 0.123, with a mean paired change of −0.144 (95% CI, −0.149 to −0.139; Wilcoxon signed-rank P < 0.001). Mean CER decreased from 0.377 ± 0.126 to 0.258 ± 0.086, with a mean paired change of −0.119 (95% CI, −0.123 to −0.116; Wilcoxon signed-rank P < 0.001). Table 2 shows that baseline WER was highest for handwritten letters, ledgers, and registry books, indicating that the most visually complex records also had the greatest initial recognition burden.

After correction, WER decreased across all document classes. The largest mean paired WER reductions occurred in handwritten letters (n = 262; −0.200; 95% CI, −0.213 to −0.188), ledgers (n = 263; −0.195; 95% CI, −0.206 to −0.183), and registry books (n = 194; −0.173; 95% CI, −0.187 to −0.160). CER followed the same pattern, supporting the interpretation that the post-correction module primarily benefited difficult handwritten, tabular, and mixed-layout records.

Named-entity recall also improved across all document types, as shown in the summary table. The largest increases occurred in handwritten letters (0.302–0.440), ledgers (0.336–0.471), and registry books (0.369–0.504). Typed correspondence reached the highest post-correction named-entity recall (0.646), but its absolute gain was smaller because baseline recognition was already stronger for this class. Figure 2 summarizes the record-level baseline CER-WER relationship, the association between handwriting intensity and correction benefit, and the post-correction shift in record-level discoverability.

Visual document classification improved overall, but performance gains were uneven across classes

Visual document classification improved overall across the 1,600-record test set. Top-1 accuracy increased from 0.717 to 0.796 (absolute change, 0.079; McNemar chi-squared = 27.33; P < 0.001), and mean prediction confidence increased from 0.736 ± 0.131 to 0.800 ± 0.108 (mean paired change, 0.064; 95% CI, 0.057–0.071; paired t test P < 0.001). As shown in Table 3, seven of eight classes improved after archive-specific fine-tuning, with the largest gains for maps, posters, newspapers, and registry books.

Typed correspondence did not improve materially after fine-tuning (0.796–0.791), suggesting that its baseline visual features were already stable and that the training procedure added little additional class separation for this category.

Figure 3 summarizes visual document classification performance before and after archive-specific fine-tuning; the source data workbook lists 20 epochs of train/validation loss and accuracy, along with macro-F1, weighted-F1, ROC-AUC, and PR-AUC diagnostic summaries.

Metadata enrichment improved descriptive completeness across all historical periods (n = 1,600 records). Mean completeness increased from 0.657 ± 0.125 to 0.907 ± 0.070, with a mean paired change of 0.250 (95% CI, 0.243–0.257; Wilcoxon signed-rank P < 0.001). As shown in Table 4, baseline completeness was lowest for 1850–1879 and increased across later periods before enrichment. Figure 4 summarizes the improvement in metadata completeness, newly realized metadata fields by document type, and subject-heading match gains across historical periods.

After enrichment, completeness increased in every historical period: 1850–1879 (n = 281; mean change, 0.207; 95% CI, 0.191–0.223), 1880–1909 (n = 474; mean change, 0.236; 95% CI, 0.223–0.248), 1910–1939 (n = 549; mean change, 0.277; 95% CI, 0.265–0.288), and 1940–1969 (n = 296; mean change, 0.263; 95% CI, 0.247–0.279). Mean counts of populated fields also increased consistently across periods.

Subject-heading match improved in parallel. Baseline match rates ranged from 64.3% to 69.4%, whereas post-enrichment match rates ranged from 82.1% to 85.4%. The largest absolute gain occurred in the earliest period (Delta = 0.178), supporting the value of conservative enrichment for records with sparse initial description.

Retrieval performance improved in all enhanced systems, with the strongest gains under full multimodal integration

Retrieval effectiveness improved under each enhancement condition relative to baseline, but the magnitude of improvement varied by system arm (n = 420 matched benchmark queries). Overall retrieval results are listed in Table 5. Baseline performance was low: P@10 = 0.394, R@10 = 0.238, nDCG@10 = 0.392, mean time to first relevant result = 156.079 s, and successful-search rate = 5.0% (21/420; 95% CI, 3.3%–7.5%). Figure 5 summarizes retrieval performance across the five experimental system arms, including overall retrieval metrics, topic-level success rates, and query-difficulty strata.

All three single-component systems outperformed the baseline in aggregate. Optical character recognition correction increased P@10 to 0.484, R@10 to 0.346, and nDCG@10 to 0.475, while reducing time to first relevant result to 124.261 s and increasing successful-search rate to 30.2% (127/420; 95% CI, 26.0%–34.8%). Visual classification produced P@10 = 0.490, R@10 = 0.302, nDCG@10 = 0.478, mean time to first relevant result = 131.985 s, and successful-search rate = 22.9% (96/420; 95% CI, 19.1%–27.1%). Metadata enrichment achieved the strongest single-component retrieval performance, with P@10 = 0.507, R@10 = 0.347, nDCG@10 = 0.513, mean time to first relevant result = 117.474 s, and a successful search rate of 38.3% (161/420; 95% CI, 33.8%–43.1%).

The full multimodal system performed best across all retrieval endpoints. P@10 increased from 0.394 at baseline to 0.624, R@10 from 0.238 to 0.492, and nDCG@10 from 0.392 to 0.634. Mean time to first relevant result decreased from 156.079 s to 58.485 s, and the successful-search rate increased from 5.0% to 95.5% (401/420; 95% CI, 93.0%–97.1%). These values correspond to absolute gains of 0.230 for P@10, 0.254 for R@10, and 0.242 for nDCG@10.

Topic-specific retrieval also improved under full multimodal integration. Successful-search rates increased across people, institutions, places, occupations, events, and subject themes, with the occupation category showing the largest practical gain, as its baseline success rate was 0.0% and rose to 90.0% under the full multimodal condition. Place-name queries had the strongest baseline performance but still benefited from multimodal integration.

Multimodal gains varied across document, period, and language strata

Subgroup analyses showed that multimodal gains were broadly distributed across document types, historical periods, and language contexts, although the magnitude of improvement varied. This heterogeneity indicates that the workflow should be evaluated within each target collection rather than assumed to produce uniform gains. Figure 6 shows the heterogeneity of multimodal retrieval gains across target document type, historical period, and language context.

At the document-type level, nDCG@10 improved across all target classes. Photographs with captions, handwritten letters, maps, ledgers, and registry books showed strong gains, while newspapers and typed correspondence improved more modestly. These patterns suggest that records with weak initial metadata or complex visual structure benefit most from combined text, image, and metadata signals.

At the historical-period level, R@10 increased from 0.175 to 0.429 for 1850–1879 and reached 0.506 for 1880–1909, 0.501 for 1910–1939, and 0.519 for 1940–1969 under full multimodal integration. Gains were similar in absolute scale across periods, suggesting that enrichment and corrected recognition helped both early sparse records and later records with richer baseline metadata.

Language-context results showed a similar pattern. P@10 under the full multimodal condition reached 0.607 for mixed Latin-script records, 0.628 for English, 0.618 for French, 0.661 for German, and 0.639 for Portuguese and Spanish. The largest precision gain occurred for mixed Latin-script records, which had the weakest baseline precision.

Component-level patterns: complementary rather than redundant contributions

Together, the results show that the protocol components are complementary rather than redundant. OCR correction improved text recognizability, visual classification added structure-aware document-type signals, and metadata enrichment expanded the searchable descriptive layer. The full multimodal condition produced the strongest and most consistent retrieval gains, supporting the study objective of testing an auditable, retrieval-oriented workflow for heterogeneous digital archives.

These findings support a protocol model in which multimodal artificial intelligence augments archival search while preserving the need for repository governance, data provenance, and expert validation.

DATA AVAILABILITY:

The de-identified 1,600-record dataset used for the main analyses is available on Zenodo as data.xlsx (https://doi.org/10.5281/zenodo.20774456).

Figure 1: Reproducible workflow for multimodal archival discovery. (A) Corpus construction from repository-level archival records with inclusion/exclusion screening. (B) Reference labeling and quality control, including document type, OCR reference regions, language context, historical period, and metadata completeness. (C) Image preprocessing, OCR generation and post-correction, visual document classification, and conservative metadata enrichment. (D) Index construction, five-arm retrieval evaluation, statistical analysis, and resource packaging. Abbreviations: OCR, optical character recognition; CER, character error rate; WER, word error rate. Please click here to view a larger version of this figure.

Figure 2: Record-level optical character recognition error structure and discoverability after post-correction. (A) Baseline CER versus baseline WER across 1,600 archival records, colored by document type for representative classes. (B) Association between scaled handwriting intensity and change in WER after correction; negative values indicate lower WER after correction. The fitted line and gray band show the linear trend and 95% confidence interval. (C) Record-level discoverability score before and after correction by document type, illustrating post-correction shifts in searchable evidence. Abbreviations: CER, character error rate; WER, word error rate. Please click here to view a larger version of this figure.

Figure 3: Visual document classification before and after archive-specific fine-tuning. (A) Top-1 accuracy by document type at baseline and after fine-tuning. (B) Confidence distributions for correct and incorrect predictions under baseline and post-tuning conditions. (C) Class-level performance matrix summarizing top-1 accuracy and mean posterior confidence before and after tuning. Model diagnostic data include 20 epochs of loss/accuracy curves, macro-F1, weighted-F1, ROC-AUC, and PR-AUC. Please click here to view a larger version of this figure.

Figure 4: Metadata completeness and subject-heading match after conservative enrichment. (A) Record-level metadata completeness before and after enrichment. (B) Newly realized metadata fields by document type. (C) Gain in subject-heading match across historical periods. Completeness was calculated as the number of populated applicable core fields divided by the total number of applicable core fields. Please click here to view a larger version of this figure.

Figure 5: Retrieval performance across the five experimental system arms. (A) Overall P@10, R@10, nDCG@10, time to first relevant result, and successful-search rate for baseline, optical character recognition correction, visual classification, metadata enrichment, and full multimodal integration. (B) Bubble plot of successful-search rate by query topic and system arm; color indicates system arm, and bubble size indicates percentage success. (C) P@10, R@10, and nDCG@10 across easy, moderate, and hard query-difficulty strata. (D) Alternative topic-level success-rate view showing the distribution of successful searches across query topics and system arms. Abbreviations: P@10, precision at 10; R@10, recall at 10; nDCG@10, normalized discounted cumulative gain at 10. Please click here to view a larger version of this figure.

Figure 6: Heterogeneity of multimodal retrieval gains across archival subgroups. (A) nDCG@10 by target document type and retrieval condition. (B) R@10 by historical period and retrieval condition. (C) P@10 by target language context and retrieval condition. Conditions are Baseline, OCR correction, visual classification, metadata enrichment, Full multimodal integration, OCR+Text, Text+Visual+Metadata, and Visual+Metadata. The figure highlights that multimodal gains are widespread but not uniform across document, period, and language strata. Please click here to view a larger version of this figure.

Document type	n records	Median pages	Handwritten records	Mean baseline WER	Mean baseline metadata completeness	Mean baseline discoverability
			(%)
Handwritten letter	262	2	95	0.729	0.648	68.2
Ledger	263	7	63.5	0.679	0.674	72.2
Map	102	1	2.9	0.457	0.552	74.1
Newspaper	261	8	0.8	0.495	0.66	75.5
Photograph with caption	179	1	1.1	0.374	0.601	73.6
Poster	87	1	0	0.413	0.571	74.8
Registry book	194	10	18	0.616	0.703	71.3
Typed correspondence	252	3	4	0.315	0.689	76.4

Table 1: Characteristics of the archival collection by document type. The table reports the number of records, median page count, percentage of records containing handwriting, mean baseline WER, mean baseline metadata completeness, and mean baseline discoverability score for each of the eight document classes.

Document Type	Sample Size	CER	CER	WER	WER	Named-entity Recall	Named-entity Recall	ΔWER
	(n)	(Baseline)	(Post)	(Baseline)	(Post)	(Baseline)	(Post)
Handwritten letter	262	0.531	0.363	0.729	0.531	0.302	0.44	−0.198
Ledger	263	0.49	0.326	0.679	0.485	0.336	0.471	−0.194
Map	102	0.322	0.228	0.457	0.347	0.38	0.503	−0.11
Newspaper	261	0.354	0.256	0.495	0.381	0.436	0.558	−0.114
Photograph with caption	179	0.257	0.181	0.374	0.286	0.494	0.616	−0.088
Poster	87	0.287	0.199	0.413	0.316	0.448	0.57	−0.097
Registry book	194	0.442	0.293	0.616	0.439	0.369	0.504	−0.177
Typed correspondence	252	0.219	0.155	0.315	0.232	0.524	0.646	−.083

Table 2: OCR performance before and after post-correction by document type. CER and WER were calculated against reference transcriptions; named-entity recall was calculated against manually labeled person, place, institution, and date entities. Delta WER is post-correction WER minus baseline WER.

Document type	n	Accuracy, baseline	Accuracy, post	Confidence, baseline	Confidence, post	ΔAccuracy
Handwritten letter	262	0.615	0.645	0.616	0.655	0.03
Ledger	263	0.707	0.749	0.725	0.767	0.042
Map	102	0.765	0.902	0.801	0.899	0.137
Newspaper	261	0.716	0.843	0.728	0.83	0.127
Photograph with caption	179	0.815	0.869	0.824	0.89	0.054
Poster	87	0.771	0.903	0.792	0.889	0.132
Registry book	194	0.687	0.793	0.701	0.787	0.106
Typed correspondence	252	0.796	0.791	0.804	0.82	-0.005

Table 3: Visual document classification performance before and after archive-specific fine-tuning. The table reports sample size, top-1 accuracy, mean posterior confidence, and the change in accuracy by document type. The diagnostic data include 20 epoch-level rows with macro-F1, weighted-F1, ROC-AUC, PR-AUC, training loss, validation loss, training accuracy, and validation accuracy.

Completeness	Completeness	Populated Fields	Populated Fields	Subject-heading Match	Subject-heading Match	Δ Completeness	Δ Subject-heading Match
(Baseline)	(Post)	(Baseline)	(Post)	(Baseline)	(Post)
0.534	0.868	4.274	8.584	0.643	0.821	0.334	0.178
0.607	0.898	4.838	8.709	0.672	0.841	0.291	0.169
0.68	0.927	5.426	9.095	0.686	0.849	0.247	0.163
0.686	0.922	5.48	8.953	0.694	0.854	0.236	0.16

Table 4: Metadata enrichment by historical period. Completeness is the proportion of applicable core descriptive fields populated for a record; populated fields are reported as mean counts; subject-heading match indicates whether record-level evidence supported at least one controlled-vocabulary subject label.

System arm	n queries	P@10	R@10	nDCG@10	Time to first relevant result (s)	Successful search rate
						(%)
Baseline	420	0.394	0.238	0.392	156.079	5
OCR correction	420	0.484	0.346	0.475	124.261	30.2
Visual classification	420	0.49	0.302	0.478	131.985	22.9
Metadata enrichment	420	0.507	0.347	0.513	117.474	38.3
Full multimodal	420	0.624	0.492	0.634	58.485	95.5

Table 5: Retrieval performance for the five experimental system arms. P@10, R@10, nDCG@10, time to first relevant result, and successful search rate were calculated across the 420-query benchmark under matched-query conditions.

Discussion

This protocol demonstrates that discovery in historical digital archives improved most when OCR correction, visual document understanding, and conservative metadata enrichment were evaluated as connected retrieval components rather than as isolated technical upgrades. The largest OCR gains occurred in handwritten, tabular, and registry materials, supporting the value of post-correction for document classes where baseline recognition is weakest. At the same time, residual error after correction shows that OCR cleanup cannot be treated as definitive transcription and must remain auditable.

The visual classification results also require cautious interpretation. Fine-tuning improved visually distinctive classes such as maps, posters, newspapers, and registry books, but gains were smaller for handwritten letters and ledgers, where layouts overlap, and class boundaries are less visually separable. The relatively small corpus size means that classification performance should be interpreted as protocol evidence rather than proof that a production multimodal model can be trained from 1,600 records alone. Future validation should compare EfficientNet-B3 with document transformers, Vision Transformers, Swin Transformers, ConvNeXt, and multimodal document-foundation models^{8,11,12,13,14,15,16,17,18}.

Metadata enrichment contributed significantly to retrieval by expanding sparse descriptive records into searchable fields while retaining source and confidence information. This finding is consistent with the archival need for richer access points, but it also raises governance risks. Automated enrichment can amplify OCR errors, normalize historically ambiguous names too aggressively, or introduce bias if controlled vocabularies are incomplete. For that reason, the workflow uses confidence thresholds, source logging, and review flags rather than fully automated replacement of catalog records.

The retrieval evaluation provides evidence of complementary module effects, but it has limitations. The present comparison used baseline and internal module-ablation conditions; it did not benchmark against dense retrieval, late-interaction rankers such as ColBERT, hybrid sparse-dense retrieval, neural re-ranking, or retrieval-augmented generation systems^22,23,24. These approaches should be added in future work to determine whether the observed multimodal gains remain competitive against current retrieval architectures.

Implementation also requires attention to data leakage, computational constraints, and scalability. Enrichment used optical character recognition outputs, visual predictions, and existing metadata, so train/validation/test splitting and query evaluation must be kept separate from enrichment decisions. Future deployments should report hardware, runtime, memory use, index size, and cost per 1,000 records. Imaging-restoration approaches from adjacent computer-vision work, including multi-scale and attention-based occlusion-removal models, may inspire future preprocessing experiments, but they should be tested carefully because archival workflows must not hallucinate or alter evidentiary content^26,27.

Overall, the workflow is best understood as a reproducible access protocol rather than a replacement for archival description. Multimodal artificial intelligence can improve discovery when its outputs are constrained, logged, and reviewed, but archivists remain essential for provenance assessment, access-governance decisions, and interpretation of historically complex records^21,25.

CONCLUSIONS:
This study tested whether optical character recognition correction, visual document classification, and conservative metadata enrichment can serve as complementary retrieval components in historical digital archives. The protocol improved transcription, classification, metadata completeness, and aggregate retrieval performance while retaining explicit thresholds, source logging, and human-review safeguards. The workflow should therefore be used as an auditable access-support protocol, not as a replacement for archivist judgment or repository governance.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Python	Python Software Foundation	v3.11.8; https://www.python.org/	Workflow scripting, data processing, optical character recognition post-correction, and statistical support
R	R Foundation for Statistical Computing	v4.3.3; https://www.r-project.org/	Statistical analysis and final figure generation
Tesseract OCR	Tesseract OCR Project	v5.3.0; https://github.com/tesseract-ocr/tesseract	Baseline optical character recognition generation
Tesseract language packs	Tesseract OCR Project	Study-specific language packs; https://github.com/tesseract-ocr/tessdata	Primary and mixed-language optical character recognition decoding
OpenCV	OpenCV Team	v4.9.0; https://opencv.org/	Deskewing, noise estimation, and image preprocessing
Pillow	Python Imaging Library contributors	v10.3.0; https://python-pillow.org/	Image input/output and format normalization
NumPy	NumPy developers	v1.26.4; https://numpy.org/	Array computation for image and metric processing
pandas	pandas development team	v2.2.2; https://pandas.pydata.org/	Tabular data handling and summary exports
SciPy	SciPy developers	v1.13.1; https://scipy.org/	Statistical tests and numerical utilities
statsmodels	statsmodels developers	v0.14.2; https://www.statsmodels.org/	Statistical modeling and paired-comparison support
scikit-learn	scikit-learn developers	v1.4.2; https://scikit-learn.org/	Classification metrics, ROC/PR diagnostics, and validation utilities
rapidfuzz	Max Bachmann and contributors	v3.9.0; https://github.com/rapidfuzz/RapidFuzz	Edit-distance candidate ranking for optical character recognition correction
spaCy	Explosion AI	v3.7.4; https://spacy.io/	Named-entity processing with custom lexicon tables
PyTorch	PyTorch Foundation	v2.2.2; https://pytorch.org/	Visual document classifier training
torchvision	PyTorch Foundation	v0.17.2; https://pytorch.org/vision/	EfficientNet-B3 implementation and image transforms
EfficientNet-B3 backbone	torchvision / ImageNet pretrained weights	EfficientNet-B3; https://pytorch.org/vision/stable/models/efficientnet.html	Visual document classification backbone
sentence-transformers	UKP Lab / Hugging Face ecosystem	v2.7.0; https://www.sbert.net/	Semantic expansion for controlled-vocabulary candidates
Search-engine software	Elastic	Elasticsearch v8.11.1; https://www.elastic.co/elasticsearch	BM25 indexing, retrieval, and ranking
Container engine	Docker Inc.	Docker v26.1.1; https://www.docker.com/	Containerized retrieval environment
Linux operating system	Canonical	Ubuntu 22.04 LTS; https://ubuntu.com/	Fixed operating environment for retrieval runs
Version-control system	Git project	Git v2.44.0; https://git-scm.com/	Version control for configuration, vocabularies, scripts, and figure code
ggplot2	tidyverse	v3.5.1; https://ggplot2.tidyverse.org/	R-based figure rendering
Matplotlib	Matplotlib developers	v3.8.4; https://matplotlib.org/	Supplementary visualization and quality-assurance plots
Annotation guideline	Authors	5 annotation sections; 8 document-class labels; OCR regions; entity labels; relevance judgments; adjudication rules	Definitions for document classes, OCR regions, entities, relevance judgments, and adjudication
Archive-specific OCR lexicon	Authors	6 lexicon categories: person variants, place names, institution names, date patterns, administrative terms, abbreviations	Names, places, institutions, administrative terms, and spelling variants
Controlled vocabulary mapping	Authors / archival repositories	5 mapping rules linking OCR entities, visual class, title keywords, date coverage, and query topic to metadata fields	Subject-heading matching and semantic expansion
Benchmark query set	Authors	420 benchmark queries; 6 query topics; 3 difficulty levels; 8 target document classes; 4 historical periods; 6 language contexts	Matched retrieval benchmark across five system arms
Relevance judgments	Authors / annotators	2,100 query-system rows; 420 queries x 5 system arms; relevant totals, relevant-in-top-10 counts, graded relevance, and success flags	Ground-truth surrogate for P@10, R@10, nDCG@10, and successful-search rate
Training diagnostics	Authors	20 epochs; training loss; validation loss; training accuracy; validation accuracy; macro-F1; weighted-F1; ROC-AUC; PR-AUC	Epoch-level model diagnostic summaries
Computational hardware	Authors	8 CPU cores; 32 GB RAM; NVIDIA GPU-class training environment	Reproducibility resource detail for JoVE submission
Dataset file	Authors	data.xlsx; 1,600 records; 17 variables; DOI listed in manuscript Data Availability	Record-level source data for optical character recognition, metadata completeness, discoverability, and visual classification analyses