Here, we present a protocol to improve discovery in historical digital archives by integrating post-correction optical character recognition, visual document classification, metadata enrichment, and retrieval evaluation into an auditable workflow.
A subscription to JoVE is required to view this content. Sign in or start your free trial.
Research Article
Here, we present a protocol to improve discovery in historical digital archives by integrating post-correction optical character recognition, visual document classification, metadata enrichment, and retrieval evaluation into an auditable workflow.
Historical digital archives are increasingly searchable, but discovery remains limited when optical character recognition errors, visually heterogeneous document types, and sparse metadata are handled separately. This study aimed to develop a reproducible protocol to evaluate whether a conservative multimodal artificial intelligence workflow can improve archival retrieval without replacing archivist review. A corpus of 1,600 digitized archival records from eight document classes was assembled, and a 420-query benchmark was used to compare five retrieval conditions: baseline indexing, optical character recognition correction alone, visual classification alone, metadata enrichment alone, and full multimodal integration. Record-level outcomes included character error rate (CER), word error rate (WER), named-entity recall, document-type classification accuracy, prediction confidence, metadata completeness, and subject-heading match. Query-level outcomes included P@10, R@10, nDCG@10, time to first relevant result, and successful-search rate. Optical character recognition correction reduced WER most strongly for handwritten letters, ledgers, and registry books; visual fine-tuning improved classification accuracy most for maps, posters, newspapers, and registry books; and metadata enrichment increased completeness across all historical periods. The full multimodal condition achieved the highest retrieval performance (P@10 = 0.624, R@10 = 0.492, nDCG@10 = 0.634) and reduced the mean time to first relevant result from 156.079 s to 58.485 s. These results support a modular, auditable workflow in which text, image, and descriptive signals are combined under explicit thresholds and human review.
Digital archives have expanded rapidly and now support research in history, cultural heritage, public administration, and the digital humanities. However, access remains uneven because archival records often combine degraded optical character recognition output, visually diverse page layouts, inconsistent cataloging practices, and historically variable names, places, and institutions. These limitations affect not only transcription quality but also retrieval, filtering, and downstream reuse of digitized collections1,2,3,4,5,6,7.
Existing document artificial intelligence and archival-retrieval studies have improved individual components such as optical character recognition post-correction, document-image classification, metadata normalization, topic modeling, word embeddings, neural retrieval, and data-governance practice8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25. Yet many archival systems still evaluate these components separately, making it difficult to determine whether a combined workflow improves discovery under realistic search conditions. The methodological gap addressed here is the lack of a reproducible, auditable protocol that connects text, image, and metadata modules to retrieval outcomes in a single archival workflow.
The central objective was to test whether optical character recognition correction, visual document classification, and rule-constrained metadata enrichment produce complementary improvements when evaluated against the same retrieval benchmark.
We hypothesized that the full multimodal condition would outperform each single-component condition because archival discovery depends on the interaction of readable text, visually interpretable document structure, and searchable descriptive metadata. The workflow was designed conservatively: automated outputs could enrich retrieval indices, but low-confidence predictions were flagged for review rather than used as authoritative archival description.
Access restricted. Please log in or start a trial to view this content.
This study used digitized archival records and simulated retrieval queries as units of analysis. Archival repositories may restrict access to sensitive or culturally restricted records. Follow repository access rules, institutional data-security procedures, and de-identification requirements before processing or sharing any records. No hazardous chemical, biological, sharps, radioactive, or other regulated laboratory waste was generated by this digital workflow.
Study design and corpus construction
Figure 1 presents the complete study workflow, including corpus construction, reference labeling, image preprocessing, OCR generation and post-correction, visual classification, metadata enrichment, index construction, retrieval evaluation, statistical analysis, and reproducibility packaging.
Corpus construction was conducted from January 15 to April 30, 2025, followed by system design and debugging from May 1 to October 31, 2025. The corpus contained 1,600 digitized archival records selected from eight depositories representing state, city, religious, museum, university, and library systems. The sampling frame was stratified by repository and document class to preserve archival heterogeneity across handwritten letters, ledgers, maps, newspapers, photographs with captions, posters, registry books, and typed correspondence.
For each candidate record, the selection log recorded the repository or collection identifier, catalog identifier, document class, approximate historical period, dominant language context, available image format, image resolution, page count, and baseline descriptive fields. Inclusion required all of the following: a stable catalog identifier; at least one TIFF, JPEG, or PNG page image; source image resolution of at least 150 dpi; readable textual, tabular, or document-structural content; and assignment to one of the eight predefined document classes. Exclusion criteria were exact duplicates, blank reverse pages, images cropped to the point that more than 30% of the text-bearing area was missing, and records judged unreadable after manual inspection.
The retrieval benchmark contained 420 short natural-language queries generated between August 5 and August 20, 2025. Query generation followed a reproducible template: candidate information needs were sampled from people, institutions, places, date spans, occupations, events, and topical themes; each query was normalized to 2–9 words; and target relevant records were identified before system comparison. Each query was evaluated under five retrieval conditions, producing 2,100 matched query-condition observations.
Reference labeling and quality control
Reference labeling was performed from May 1 to July 15, 2025, by three annotators: two archival staff members experienced in descriptive cataloging and one digital-humanities researcher experienced in historical document assessment. The annotation guide defined document classes, language-context categories, historical-period bins, metadata-completeness fields, named-entity categories, and rules for selecting OCR evaluation regions.
For OCR evaluation, annotators selected representative text-bearing regions that included headings, names, dates, institutional terms, marginal notes, tabular entries, and, when present, noisy layout areas. When a page contained multiple text regions, the selected region had to be relevant to retrieval and had to include enough characters for CER and WER calculation. Disagreements were first reconciled through discussion between the two primary annotators; unresolved cases were adjudicated by the digital humanities researcher.
Quality control used a random 12% subset of records. Inter-annotator agreement for the primary document type was Cohen's kappa = 0.86, supporting substantial agreement. The adjudication log retained the original and final labels, the disagreement category, and the reason for resolution so that future users can audit class definitions and edge cases.
Image preprocessing
All images were processed at the record level using Python 3.11.8 with OpenCV 4.9.0, Pillow 10.3.0, and NumPy 1.26.4. Each input page was converted to 8-bit grayscale unless color carried semantic information, such as maps, posters, stamps, or color-coded annotations. Skew was estimated using projection profiles and Hough-line detection; deskewing was applied only when the absolute skew angle exceeded 1.5 degrees and was capped at 8.0 degrees to avoid distortion.
Contrast enhancement used contrast-limited adaptive histogram equalization with clipLimit = 2.0 and tileGridSize = 8 × 8. A median filter with a 3 × 10−23 kernel was applied only when the estimated background-noise ratio exceeded 0.35. Images captured at 150–299 dpi were resampled to 300 dpi for optical character recognition; images already at 300 dpi or higher were not upscaled. No super-resolution, generative restoration, or content hallucination was used.
Bleed-through, edge darkening, uneven paper texture, marginal stamps, handwriting, ruled lines, and table boundaries were retained unless they prevented OCR region selection. This rule preserved archival evidence while standardizing image inputs sufficiently for reproducible OCR and classification.
OCR base generation and post-correction
Baseline OCR was generated with Tesseract OCR 5.3.0. The primary language pack was selected from the catalog language and visible script; multilingual decoding was enabled when the catalog language and page script did not match. OCR outputs were grouped at the record level and stored with page identifiers, OCR confidence, bounding-box coordinates when available, and raw text before correction.
Post-OCR correction used a three-stage conservative procedure. First, character-level normalization corrected frequent historical recognition confusions, including ligatures, long-s/short-s substitutions, fragmented punctuation, and digit-letter substitutions. Second, token-level correction used edit distance and frequency-ranked candidates from an archive-specific lexicon of personal names, place names, institutions, administrative terms, and historical spelling variants. Third, rule-based sequence checks validated named entities and dates against evidence from context and metadata.
Candidate replacements were accepted only when posterior correction confidence exceeded 0.80; named-entity replacements required confidence of at least 0.85. Candidates below the threshold were retained as OCR text but flagged for review. CER was calculated as the Levenshtein edit distance divided by the number of characters in the reference transcription. WER used the same formula at the token level. Named-entity recall was calculated as correctly recognized reference entities divided by all reference entities in the selected evaluation region.
Visual document classification
The visual classification module assigned each record to one of eight archival document classes: handwritten letter, ledger, map, newspaper, photograph with caption, poster, registry book, and typed correspondence. The predicted document type was used as a retrieval and filtering signal, not as an authoritative replacement for archival cataloging.
The model was implemented in PyTorch 2.2.2 and torchvision 0.17.2 using an ImageNet-pretrained EfficientNet-B3 backbone. Record-level thumbnails were resized to 384 x 384 pixels. To reduce leakage, records were split by repository and document class into training, validation, and test sets at a 70:15:15 ratio, corresponding to approximately 1,120, 240, and 240 records.
Training used AdamW with initial learning rate = 1 × 10-4, weight decay = 1 × 10−2, batch size = 16, label smoothing = 0.05, and early stopping when validation loss failed to improve for five consecutive epochs. Horizontal flipping was disabled because mirrored archival layouts are not realistic. Augmentation was limited to rotation from -3 degrees to +3 degrees and brightness variation within ±10%.
Epoch-level model diagnostics are summarized in 20 training rows, each containing training loss, validation loss, training accuracy, validation accuracy, macro-F1, weighted-F1, ROC-AUC, and PR-AUC.
Metadata enrichment workflow
Metadata enrichment was designed to improve discoverability while preserving archival conservatism. Existing catalog values were retained unless they contained an explicit contradiction, such as an impossible date or a document-type conflict confirmed by both OCR and visual evidence. Candidate enrichment fields included normalized title, document type, approximate date, institution mentions, geographic mentions, personal-name mentions, subject headings, and keywords.
Evidence priority followed a fixed order: (1) existing catalog metadata, (2) corrected OCR output, and (3) visual document-type prediction. Controlled vocabulary matching was used for subject headings, and nearest-neighbor semantic expansion with sentence-transformers 2.7.0 was used only when cosine similarity was at least 0.72. Date normalization required confidence of at least 0.85 or concordance between OCR and metadata. Every automatically added field retained its source, confidence, and rule identifier.
Metadata completeness was defined as the number of populated core descriptive fields divided by the number of applicable core fields for that record. Subject-heading match was defined as the presence of at least one controlled-vocabulary subject label supported by record-level content evidence and judged relevant to the query category.
Retrieval system construction
Five retrieval indices were constructed to isolate module contributions: baseline indexing; optical character recognition correction only; visual classification only; metadata enrichment only; and full multimodal integration. All indices were built in the search-engine software listed in the Table of Materials (version 8.11.1) at the record level. BM25 parameters were fixed at k1 = 1.2 and b = 0.75 before evaluation.
The baseline index used original metadata and baseline OCR text. The OCR condition replaced baseline OCR with corrected OCR. The visual condition added document-type priors and class-aware ranking boosts. The metadata condition added enriched descriptive fields. The full multimodal condition combined corrected OCR, visual priors, and enriched metadata. Field weights were fixed before testing: normalized title = 3.0, subject headings = 2.5, person and institution mentions = 2.2, place mentions = 2.0, document type = 1.8, corrected OCR body text = 1.0, and baseline OCR body text = 0.8, where applicable.
Exact phrase matching was enabled for quoted queries. Query expansion was permitted only in the metadata-enrichment and full-multimodal conditions and was limited to accepted controlled-vocabulary synonyms and subject-heading variants. Search logs were generated from August 25 to August 28, 2025, in the containerized Linux environment listed in the Table of Materials, with fixed seeds and frozen index configurations.
Query design and outcome measures
The 420 queries represented common archival search behaviors rather than keyword-only benchmark prompts. Topics included personal names, institutions, places, dates and date ranges, occupations, events, and thematic subjects. Each query was stored with its normalized wording, target record set, query-topic category, and difficulty score.
Difficulty was scored before retrieval using target ambiguity, lexical specificity, and expected frequency of expression. Composite scores below 0.40 were classified as low difficulty, scores from 0.40–0.69 as moderate difficulty, and scores of 0.70 or higher as high difficulty. Relevance judgments were completed before system comparison and then checked against the finalized collection.
Retrieval outcomes included P@10, R@10, nDCG@10, time to first relevant result, and successful search rate. P@10 was the proportion of the top 10 records judged relevant. R@10 was the proportion of all known relevant records retrieved in the top 10. nDCG@10 used graded relevance where available and binary relevance otherwise. Time to first relevant result was measured from query submission until the first relevant record appeared in the ranked output. A successful search was defined as at least one relevant result among the top 10 records.
Statistics collection
All analyses were performed using the software versions listed in the Table of Materials. Continuous outcomes were summarized as mean ± SD when approximately symmetric and as median with interquartile range otherwise. Categorical outcomes were summarized as counts and percentages.
Normality of paired differences was assessed by the Shapiro-Wilk test and distribution plots. OCR and metadata pre/post outcomes were compared with paired t tests when paired differences were approximately normal and Wilcoxon signed-rank tests otherwise. Classification performance before and after fine-tuning was compared using McNemar's test on paired correct/incorrect labels. Matched multi-arm retrieval outcomes were compared with the Friedman test followed by Holm-adjusted pairwise Wilcoxon signed-rank tests.
All statistical tests were two-tailed, and P < 0.05 was considered statistically significant. Confidence intervals were estimated using 2,000 bootstrap resamples with the random seed fixed at 2025. Effects were reported as mean differences, matched odds ratios, or rank-based effect sizes according to outcome type.
Reproducibility, scope, and resource availability
All pipeline parameters were fixed before retrieval testing. No parameter was optimized on the held-out retrieval benchmark. Configuration files, controlled vocabularies, lexicon tables, query templates, field mappings, annotation guidelines, statistical scripts, and figure-generation scripts were maintained in the version-control system listed in the Table of Materials with random seed 2025.
To support independent validation, the source-data workbook includes a de-identified 1,600-record table with 17 variables used for optical character recognition, metadata, and visual-classification analyses. Derived Table 1, Table 2, Table 3, Table 4, and Table 5, figure-source summaries, benchmark-query definitions, relevance-judgment summaries, retrieval-log surrogates, training diagnostics, lexicon categories, vocabulary-mapping rules, and annotation-guideline fields are provided as separate uploadable tables or materials files where applicable. Materials that cannot be publicly shared because of repository restrictions are represented by de-identified metadata fields and reproducible sampling fields.
This protocol evaluates retrieval-oriented discovery rather than the truth of historical claims. It does not replace archivist appraisal, provenance assessment, privacy review, or repository-specific access rules.
Access restricted. Please log in or start a trial to view this content.
OCR correction improved transcription, especially in handwritten and tabular records
Optical character recognition correction improved transcription quality across all eight archival document classes (n = 1,600 records). Mean WER decreased from 0.529 ± 0.172 to 0.384 ± 0.123, with a mean paired change of −0.144 (95% CI, −0.149 to −0.139; Wilcoxon signed-rank P < 0.001). Mean CER decreased from 0.377 ± 0.126 to 0.258 ± 0.086, with a mean paired change of −0.119 (95%...
Access restricted. Please log in or start a trial to view this content.
This protocol demonstrates that discovery in historical digital archives improved most when OCR correction, visual document understanding, and conservative metadata enrichment were evaluated as connected retrieval components rather than as isolated technical upgrades. The largest OCR gains occurred in handwritten, tabular, and registry materials, supporting the value of post-correction for document classes where baseline recognition is weakest. At the same time, residual error after correction shows that OCR cleanup cann...
Access restricted. Please log in or start a trial to view this content.
The authors have nothing to disclose.
The authors sincerely thank the two experienced archival staff members and the professional digital humanities researcher for their valuable assistance with text annotation and quality control. This research was financially supported by the Jiangsu Provincial Archives Science and Technology Project Plan (Grant No. 2024-9) for the construction of an intelligent speech-based oral archives system, and the Jiangsu Provincial Science and Technology Development Plan for Traditional Chinese Medicine (Grant No. MS2025025) for studying the inheritance and development of TCM schools from an archival perspective.
Access restricted. Please log in or start a trial to view this content.
| Name | Company | Catalog Number | Comments |
|---|---|---|---|
| Python | Python Software Foundation | v3.11.8; https://www.python.org/ | Workflow scripting, data processing, optical character recognition post-correction, and statistical support |
| R | R Foundation for Statistical Computing | v4.3.3; https://www.r-project.org/ | Statistical analysis and final figure generation |
| Tesseract OCR | Tesseract OCR Project | v5.3.0; https://github.com/tesseract-ocr/tesseract | Baseline optical character recognition generation |
| Tesseract language packs | Tesseract OCR Project | Study-specific language packs; https://github.com/tesseract-ocr/tessdata | Primary and mixed-language optical character recognition decoding |
| OpenCV | OpenCV Team | v4.9.0; https://opencv.org/ | Deskewing, noise estimation, and image preprocessing |
| Pillow | Python Imaging Library contributors | v10.3.0; https://python-pillow.org/ | Image input/output and format normalization |
| NumPy | NumPy developers | v1.26.4; https://numpy.org/ | Array computation for image and metric processing |
| pandas | pandas development team | v2.2.2; https://pandas.pydata.org/ | Tabular data handling and summary exports |
| SciPy | SciPy developers | v1.13.1; https://scipy.org/ | Statistical tests and numerical utilities |
| statsmodels | statsmodels developers | v0.14.2; https://www.statsmodels.org/ | Statistical modeling and paired-comparison support |
| scikit-learn | scikit-learn developers | v1.4.2; https://scikit-learn.org/ | Classification metrics, ROC/PR diagnostics, and validation utilities |
| rapidfuzz | Max Bachmann and contributors | v3.9.0; https://github.com/rapidfuzz/RapidFuzz | Edit-distance candidate ranking for optical character recognition correction |
| spaCy | Explosion AI | v3.7.4; https://spacy.io/ | Named-entity processing with custom lexicon tables |
| PyTorch | PyTorch Foundation | v2.2.2; https://pytorch.org/ | Visual document classifier training |
| torchvision | PyTorch Foundation | v0.17.2; https://pytorch.org/vision/ | EfficientNet-B3 implementation and image transforms |
| EfficientNet-B3 backbone | torchvision / ImageNet pretrained weights | EfficientNet-B3; https://pytorch.org/vision/stable/models/efficientnet.html | Visual document classification backbone |
| sentence-transformers | UKP Lab / Hugging Face ecosystem | v2.7.0; https://www.sbert.net/ | Semantic expansion for controlled-vocabulary candidates |
| Search-engine software | Elastic | Elasticsearch v8.11.1; https://www.elastic.co/elasticsearch | BM25 indexing, retrieval, and ranking |
| Container engine | Docker Inc. | Docker v26.1.1; https://www.docker.com/ | Containerized retrieval environment |
| Linux operating system | Canonical | Ubuntu 22.04 LTS; https://ubuntu.com/ | Fixed operating environment for retrieval runs |
| Version-control system | Git project | Git v2.44.0; https://git-scm.com/ | Version control for configuration, vocabularies, scripts, and figure code |
| ggplot2 | tidyverse | v3.5.1; https://ggplot2.tidyverse.org/ | R-based figure rendering |
| Matplotlib | Matplotlib developers | v3.8.4; https://matplotlib.org/ | Supplementary visualization and quality-assurance plots |
| Annotation guideline | Authors | 5 annotation sections; 8 document-class labels; OCR regions; entity labels; relevance judgments; adjudication rules | Definitions for document classes, OCR regions, entities, relevance judgments, and adjudication |
| Archive-specific OCR lexicon | Authors | 6 lexicon categories: person variants, place names, institution names, date patterns, administrative terms, abbreviations | Names, places, institutions, administrative terms, and spelling variants |
| Controlled vocabulary mapping | Authors / archival repositories | 5 mapping rules linking OCR entities, visual class, title keywords, date coverage, and query topic to metadata fields | Subject-heading matching and semantic expansion |
| Benchmark query set | Authors | 420 benchmark queries; 6 query topics; 3 difficulty levels; 8 target document classes; 4 historical periods; 6 language contexts | Matched retrieval benchmark across five system arms |
| Relevance judgments | Authors / annotators | 2,100 query-system rows; 420 queries x 5 system arms; relevant totals, relevant-in-top-10 counts, graded relevance, and success flags | Ground-truth surrogate for P@10, R@10, nDCG@10, and successful-search rate |
| Training diagnostics | Authors | 20 epochs; training loss; validation loss; training accuracy; validation accuracy; macro-F1; weighted-F1; ROC-AUC; PR-AUC | Epoch-level model diagnostic summaries |
| Computational hardware | Authors | 8 CPU cores; 32 GB RAM; NVIDIA GPU-class training environment | Reproducibility resource detail for JoVE submission |
| Dataset file | Authors | data.xlsx; 1,600 records; 17 variables; DOI listed in manuscript Data Availability | Record-level source data for optical character recognition, metadata completeness, discoverability, and visual classification analyses |
Access restricted. Please log in or start a trial to view this content.