Multimodal Knowledge Graphs Based on Rule-Based Linguistic Analysis and Computer Vision

Summary

VISHAM-KG is a multimodal framework that constructs knowledge graphs from Hindi visual documents by aligning textual and visual entities. It combines rule-based linguistic analysis with computer vision techniques to produce subject-relation-object triplets in low-resource Indic language settings.

Abstract

Visual-Semantic Hindi-Aligned Multimodal Knowledge Graph (VISHAM-KG) is a framework designed to construct consistent multimodal knowledge graphs (KGs) from Hindi visual documents by systematically aligning visual-text entities. The aim of this study is to integrate rule-based linguistic analysis with computer vision-based object detection that supports the structured semantic representation and grounded reasoning in low-resource Indic languages. The proposed algorithm begins with the preparation of Natural Language Processing (NLP) Hindi visual documents, followed by optical character recognition (OCR) for Devanagari script extraction and linguistic preprocessing, which includes various processes such as tokenization, lemmatization, part-of-speech tagging, and dependency parsing. In parallel, visual entities are extracted from images using object detection and filtered using confidence thresholds. Textual and visual entities are embedded into a shared semantic space using the multilingual transformer model XLM-R, along with CLIP-ViT, and aligned using cosine similarity-based thresholds. These aligned entities are combined with rule-based dependency relations to generate multimodal triplets. The protocol produces a structured multimodal knowledge graph encoded as subject-relation-object triplets with explicit visual grounding based on the Indian knowledge base. This resulting output will support cross-modal querying, entity alignment, and knowledge graph reasoning for Hindi visual documents and provide a replicable framework for multimodal knowledge construction in low-resource linguistic settings.

Introduction

Knowledge graphs (KGs) are structured semantic graphical representations in which entities are modelled as nodes and relationships as edges. It enables efficient knowledge retrieval and contextual reasoning across various applications such as question answering, recommendation systems, and information extraction¹. Over the past decade, KG construction methodologies have been developed substantially. However, most existing approaches are designed for resource-rich languages, which rely predominantly on large-scale textual corpora². As a result, low-resource languages remain underrepresented, constraining the applicability of KG-based technologies in culturally and linguistically diverse settings³. In parallel, a growing proportion of real-world documents-particularly in educational, cultural, and heritage domains have rich visual information that is insufficiently captured by text-centric graph construction methods⁴.

Multimodal knowledge graphs (MMKGs) extend conventional KGs by integrating non-textual modalities such as images, audio, or video to enable grounded semantic representation⁵. Prior MMKG frameworks, including IMGpedia, Richpedia, and ImageGraph, demonstrate the value of associating visual information with textual entities for improved semantic querying and reasoning^6,7,8. Despite these advances, existing methods are largely English-centric, depend on curated metadata or static datasets, and provide limited procedural guidance for constructing MMKGs directly from unstructured visual documents. Moreover, these frameworks do not explicitly address challenges inherent to low-resource languages, such as script-specific Optical Character Recognition (OCR) errors, morphological variability, and sparse annotated data^9,10.

The objective of this framework is to execute a step-by-step methodology for constructing a multimodal knowledge graph from Hindi visual documents by systematically aligning textual and visual entities. The proposed framework, Visual-Semantic Hindi-Aligned Multimodal Knowledge Graph (VISHAM-KG), integrates rule-based linguistic analysis with computer vision, which is based on object extraction, to enable dynamic graph construction of visual documents. Unlike existing MMKG approaches, VISHAM-KG directly extracts entities and relations from raw Hindi text and images, applies dependency-based grammatical rules for relation identification, and performs cross-modal entity alignment using embedding-based similarity thresholds rather than relying on external^11,12.

VISHAM-KG is intended for illustrated documents in which textual and visual content are related semantically, such as children's stories¹³, educational material, newspaper¹¹ and culturally grounded narratives. Some limitations, such as dependency on optical character recognition quality, object detection coverage, and domain-specific vocabulary availability, have been encountered while executing the mentioned framework. By explicitly documenting each procedural step, VISHAM-KG provides a replicable protocol for multimodal knowledge graph construction in low-resource linguistic contexts while supporting grounded semantic reasoning and cross-modal analysis.

VISHAM-KG differs from existing MMKG approaches by directly extracting entities and relations from unstructured Hindi text and images; employing rule-based dependency parsing for relation extraction; and aligning textual and visual entities through embedding-based similarity thresholds rather than metadata matching^8,10(Figure 1).

Figure 1: End-to-end framework. The figure illustrates end-to-end framework for multimodal knowledge Graph VISHAM-KG. Please click here to view a larger version of this figure.

This protocol is applicable to illustrated documents with aligned text-image content, such as educational material and cultural narratives. In this framework, YOLOv8 is chosen for its efficiency and robustness in object detection on visual documents. XLM-R is selected for its strong cross-lingual representations, which are well-suited for low-resource Hindi text processing, and CLIP-ViT is employed for its proven capability in learning shared visual-text embedding spaces, which enable effective cross-modal alignment. But it is limited by OCR accuracy, object detection coverage, and domain-specific vocabulary constraints.

Related work

A traditional knowledge graph G=(E,R,F) consists of entities E, relationships R, and factual triplets F, where each triplet is of the form (h,r,t)⁸. Extending this, a Multi-Modal Knowledge Graph (MMKG) incorporates E entities associated with non-textual modalities such as images, audio, and video¹⁴.

Two main strategies are used in MMKGs to represent visual data:
As attributes attached to textual entities
As visual entities connected through a specific annotated relation

One notable study is IMGpedia, which enhances Wikimedia image data by incorporating visual descriptors and similarity measures. This model addresses the limitations of traditional datasets that primarily include metadata, enabling visual-semantic querying and similarity assessment by linking images with DBpedia Commons⁹.

Similarly, another MMKG Richpedia tackles the challenge of incomplete knowledge graphs in scholarly research. It aggregates 2,883,162 visual entities from Wikipedia and 30,638 textual entities from Wikidata. Richpedia supports aspect-level querying and employs methods for extracting semantic relationships from unstructured content, including image elements, associated text, and hyperlinks¹⁵.

ImageGraph extends this study by constructing a relational knowledge graph based on the FB15K dataset, enriched with 829,931 web-crawled images and captions. It includes 14,870 entities and 1,330 relation types, allowing for visual-contextual querying and more accurate responses by supporting concept-based query parameters¹⁶.

VisualSem is another comprehensive multilingual knowledge graph that integrates visual and textual information. It comprises 89,896 entities, over 1.3 million glosses, and 938,100 images. Designed for applications such as data augmentation and grounding, VisualSem enhances semantic interpretation across languages and can be seamlessly incorporated into various processing pipelines¹.

Several MMKG models are also developed to support tasks such as link prediction, triplet classification, and entity matching. These models address limitations of single-modal graphs, particularly their inability to capture the complexity of cross-modal information^16,17,18.

The critical comparison between language-based MMKG models along with VISHAM-KG is provided in Table 1. It is specifically focused on their strength and limitations in the context of low-resource languages like Hindi, Tamil, or Sanskrit. These methods often assume access to high-quality textual corpora, reliable linguistic annotations, and large-scale pretrained models. These factors constrain their applicability to low-resource languages. In particular, OCR-dependent pipelines are frequently optimized for Latin scripts and exhibit reduced accuracy for Indic scripts, which leads to noisy or incomplete text extraction. Furthermore, linguistic pre-processing, part-of-speech tagging, and named entity recognition are commonly trained on high-resource languages. They show drastically degraded performance when applied to morphologically rich, syntactically flexible languages such as Hindi.

MMKG Model	Strengths	Limitations in Low-Resource Settings
IMGpedia	Integrates images with DBpedia	Focuses only on English content
	Supports visual similarity queries	No support for nonLatin scripts
		Limited cultural context for regional- visuals
Richpedia	Combines visual and textual entities from Wikipedia and Wikidata	Inadequate representation of Indic or folk knowledge
	Aspectlevel querying available	Assumes highquality alignment, which is lacking in regional datasets
ImageGraph	Relational KG with images and captions	Entity and relation extraction tuned for English corpora
	Supports extended tripletbased querying	Fails in environments with sparse captions or missing metadata
VisualSem	Multilingual support	Poor representation of Asian low-resource languages
	Useful in neural semantic pipelines	No support for Devanagari or culturally grounded visual semantics
VISHAM-KG	Relational KG with images in Indic Langauges	Language dependent
	Semantic pipelines for morphologically rich syntax	Depends on different POS tag of different langauges.

Table 1: Critical comparison of MMKGs with limitations in low-resource languages.

Existing MMKG models rely on static knowledge graphs, not adapting to dynamic real-world contexts where new entity types and associations develop due to their single-dataset training. This makes it crucial to develop models with dynamic capabilities¹⁶. The following limitations are found in this context: incorrect use of textual data in visual activities like object identification, extraction, and annotation; developing scalable methods for constructing multi-modal knowledge graphs from heterogeneous sources; and incorporating contextual information into multi-modal knowledge graphs for improved understanding and interpretation.

In these conditions, VISHAM-KG differs from earlier approaches by employing advanced visual extraction techniques to define nodes and relationships directly from visual documents. It combines standard text processing steps such as tokenization, stop-word removal, and part-of-speech tagging with semantic graph techniques to structure the extracted knowledge. By fusing computer vision and ontology, the system offers several advantages¹⁹: enhanced adaptability, allowing the knowledge base to evolve with application-specific needs; improved semantic representation that supports interoperability across systems; and better semantic inference and retrieval, enabling contextual-level knowledge base enhancement.

Protocol

No ethical approval is required for this protocol as it exclusively uses publicly available, non-human, non-sensitive visual and textual data. Table 2 provides all tools and techniques along with their dependencies. All source code, configuration files, and scripts required to reproduce the multimodal knowledge graph construction pipeline are available in a public GitHub repository(preeti017phdit22-wq/VISHAM_KG.). The repository includes installation instructions, and dependency specifications to facilitate reproducibility.

Module	Model / Tool	Version	Framework	Purpose
OCR	EasyOCR	v1.7.1	PyTorch	Hindi text extraction
POS + Dependency Parsing	Stanza (hi)	v1.6.1	PyTorch	Linguistic parsing
NER	BiLSTM-CRF	Custom-trained	PyTorch	Hindi entity recognition
Object Detection	YOLOv8	v8.0.208	Ultralytics	Visual entity extraction
Text Embeddings	XLM-R Base	2023-05	HuggingFace	Multilingual text encoding
Visual Embeddings	CLIP-ViT-B/32	2022-09	OpenAI	Image encoding
Graph Storage	Neo4j	v5.13	Neo4j	KG construction
Similarity	Cosine Similarity	–	NumPy	Cross-modal alignment

Table 2: Tools and techniques used at each step in construction of VISHAM-KG.

1. Knowledge graph construction

1. Data preparation
   1. Collect 10 children's story documents from the mentioned sources^11,13. Confirm the availability of image documents for each document.
   2. Store each document as a structured unit containing image files (PNG or JPG) and corresponding Hindi text.
   3. Assign a unique document identifier linking each image to its associated text.
2. Text extraction and preprocessing
   1. Extract text from scanned images using EasyOCR (configured for Devanagari script) to extract Hindi text from document images (Figure 2).
   2. Normalize extracted text by removing OCR artifacts and removing extraneous symbols.
   3. Perform sentence segmentation and tokenization. Tokenize the text into words. Remove stop words using a predefined Hindi stop-word list.
   4. Perform part-of-speech tagging and dependency parsing using a Hindi-compatible NLP using Stanza (Hi).
   5. Identify named entities using a BiLSTM-CRF model.
   6. Extract subject-relation-object triplets using dependency-rule templates. Generate a dependency tree with labeled grammatical relations for constructing meaningful triplets (Figure 3).

Figure 2: Extraction of a simple subject-verb-object triplet from Hindi text using verb-only relations. The flowchart describes extraction of a simple subject-verb-object triplet from Hindi text using verb-only relations. Please click here to view a larger version of this figure.

Figure 3: Extended Verb Preposition relationship. The figure illustrates extended Verb Preposition relationship to identify the triplet formation. Please click here to view a larger version of this figure.

3. Visual entity extraction
   1. Load each image and apply object detection using the YOLOv8 object detection model (Figure 4).
   2. Extract bounding boxes, class labels, and confidence scores of identified objects in the image (Figure 5).
   3. Filter detected objects by retaining detections with confidence scores ≥ 0.50. Record filtered objects as visual entities (OPTIONAL). Save visual entities with bounding box coordinates and generate a list of these entities.

Figure 4: Object Detection. The figure illustrates object detection Using YOLOv8. Please click here to view a larger version of this figure.

Figure 5: Visual feature extraction and object detection and identification. The figure illustrates visual feature extraction using convolutional layers and YOLOv8, followed by region refinement and alignment based on similarity scores. Please click here to view a larger version of this figure.

4. Entity embedding and alignment
   1. Generate contextual embeddings for textual entities using XLM-R embeddings. Generate visual embeddings for detected objects using CLIP-ViT embeddings (Figure 6).
   2. Project text and visual embeddings into a shared latent space and normalize them to unit length.
   3. Compute cosine similarity between each textual and visual embedding pair. Align entities when similarity ≥ predefined threshold τ (default τ = 0.65). Produce a list of aligned text-image entity pairs.

Figure 6: Visual object detection and POS tags fusion. The figure illustrates visual object detection and POS tags fusion for knowledge graph triplet extraction. YOLO and CIFAR-100 identify objects demonstrating multi-modal alignment. Please click here to view a larger version of this figure.

5. Triplet extraction
   1. Extract textual triplets using dependency rules that map subject-verb-object structures.
   2. Derive visual relations using spatial proximity and co-occurrence rules.
   3. Generate multimodal triplets by linking aligned textual and visual entities using relation labels. Validate triplets for syntactic and semantic consistency.
6. Knowledge graph construction
   1. Convert aligned entities into RDF-compatible triplets. Merge textual and visual triplets into a unified graph.
   2. Insert entities as nodes and relations as edges. Encode multimodal links using explicit predicates. Store the resulting graph in Neo4j (OPTIONAL). A finalized multimodal knowledge graph with aligned text-image triplets is now generated.
      NOTE: A systematic approach for constructing a multimodal knowledge graph from Hindi visual documents is shown in Figure 7.

Figure 7: Pipeline for multimodal knowledge graph construction. The flowchart represents pipelines for VISHAM-KG. Please click here to view a larger version of this figure.

3. Use the pseudocode below for knowledge graph construction.
   Input:
   D : Set of Hindi text-image document
   τ : Similarity threshold for alignment
   Preprocess each document pair (T,I)∈D
   If T is scanned, extract text T' using OCRPerform tokenization, lemmatization, and stop word removal
   Apply POS tagging and dependency parsing using Stanza
   Detect objects in I using YOLOv8
   Extract bounding boxes, labels L_i, and confidence scores > 0.5
   Generate Embedding
   Identify named entities E_T from T' using BiLSTM-CRF
   Extract visual entities E_V from L_i
   Compute textual embeddings E_T with XLM-R.
   Compute visual embeddings E_V with CLIP-ViT
   Entity Alignment with Triplet Extraction
   For each pair (e_t,e_v) in E_T x E_V:
   Compute cosine similarity S = cos(E_V,E_T)
   Set Threshold τ=0.6
   If s≥τ, add triplet (e_t, has_image,e_v) to set F.
   Extract (h,r,t) triplets from T' using dependency rules.
   Derive visual relations from spatial or caption-based co-occurrence.
   Project E_t and E_v into a shared latent space.
   Score triplets and retain those above the confidence threshold.
   Add validated triplets and entities to graph G.
   Output: final KG in Neo4j.

2. Evaluation procedure

NOTE: Hindi kids' stories are chosen for evaluation of the VISHAM-KG framework because they provide controlled, visually grounded narratives with clear entities and relations, enabling reliable validation of multimodal alignment, graph construction, and inference before domain-scale deployment. All hyperparameter settings are given in Table 3.

Module	Hyperparameter	Dimensions
OCR	Confidence threshold	0.5
Entity extraction	Embedding dimension	300
Object detection	Confidence threshold	0.5
	Input image size	640 × 640
Text embedding	Language model	XLM-R
	Embedding dimension	768
Image embedding	Vision model	CLIP-ViT-B/32
	Embedding dimension	768
Alignment	Similarity metric	Cosine similarity
Text-Image Alignment	Cosine Similarity threshold (τ)	0.6
Link prediction	Embedding dimension	100
	Training epochs	50
	Negative sampling	Uniform
Evaluation	Train–test split	80 / 20

Table 3: Hyperparameter setting for the framework.

Component	Count
Document Images	10
Textual Entities	186
Visual Entities	97
Text-derived Relations	105
Visual-derived Relations	41
Textual and Visual Triplets	312

Table 4: Knowledge Graph and triplets Statistics.

1. Dataset composition and partitioning
   1. The evaluation dataset consists of 10 kids' stories, each accompanied by illustrative images. Execute the entity extraction process mentioned in steps 1.2-1.4. The results are represented in Table 4.
   2. Construct two graph variants: one Text-only Knowledge Graph (T-KG) using only textual triplets and another Multimodal Knowledge Graph (MM-KG) using fused textual and visual triplets.
   3. To ensure a controlled evaluation, for both graphs, use identical data splits.
   4. Randomly partition extracted triplets at 80:20, that is 80% for graph construction (training set) and 20% held out for evaluation (test set). Apply this split consistently to both textual KG and MMKG to ensure fair comparison.
2. Baseline and evaluation metrics
   1. The textual KG serves as the baseline. The proposed framework, VISHAM KG, represents the proposed method. For both graphs, use an identical ontology with entity identifiers and evaluation queries. The only difference between the two graphs is the inclusion of visual entities in VISHAM-KG.
3. Evaluation metrics and link prediction
   1. Use the standard link prediction metrics²⁰: Mean Reciprocal Rank (MRR), Hits@1, Hits@3, Hits@10. Hit@K, defined as the proportion of cases where the correct entity appears in the top N ranks.
   2. For each test triplet (head, relation, tail), mask either the head or tail entity. Rank all candidate entities based on cosine similarity in the shared embedding space (Table 5).

Textual Entity	Visual Entity	Cosine Similarity
शेर		0.78
लोमड़ी		0.82

Table 5: Cosine similarity scores between text and image embeddings.

3. Generate predictions independently for text-only embeddings and multimodal embeddings (VISHAM-KG).
4. Calculate results using Mean Reciprocal Rank (MRR), as the average of reciprocal ranks of the correct entity across all queries²¹. Using Table 6, express all results in decimal format for consistency across experiments²².

Model	MRR	Hits@1	Hits@3	Hits@10
TransE	0.42	0.21	0.48	0.72
ComplEx	0.47	0.26	0.52	0.74
RotatE	0.51	0.31	0.58	0.74
VISHAM-KG(Textual)	0.49	0.36	0.62	0.76

Table 6: Link prediction performance on text-only triplets.

5. Use the metrics to validate the predictive power of the multimodal knowledge graph in recovering missing links, as shown in Table 7.

Model	MRR	Hits@1	Hits@3	Hits@10
IKRL	0.46	0.34	0.63	0.72
VisualBERT	0.52	0.35	0.61	0.72
ViLBERT	0.54	0.38	0.64	0.75
VISHAM-KG	0.57	0.41	0.66	0.79

Table 7: Performance on cross-modal triplet prediction tasks.

4. Use the pseudocode below for evaluation.
   For each Knowledge Graph variant G∈{G_T,G_MM}:
   Triplet Partitioning
   Extract all triplets T_all from G.
   Randomly partition T_all into training set (80%) and test set T_test(20%).
   Construct graph G_train using triplets in T_train.
   Similarity Score and Embedding
   For each test triplet (h,r,t)∈T_test:
   Mask head or tail entity to form query (h,r,?) or (?,r,t).
   Generate candidate entity set C from entities in G_train.
   Compute embedding similarity score S=cos(e_query,e_c) for each e_c ∈ C.
   Rank all candidate entities based on descending similarity score.
   Metric Computation
   Compute rank of the correct entity for each query.
   Calculate Mean Reciprocal Rank (MRR) over all test queries.
   Calculate Hits@1, Hits@3, and Hits@10.
   Compare evaluation scores between Text-only KG G_T and Multimodal KG G_MM.
   Output: Provide qualitative and quantitative results directly attributable for multimodal integration
5. Cross-modal similarity
   1. Compute similarity scores to evaluate alignment between textual and visual embeddings. Normalize both textual embeddings and visual embeddings to unit length to ensure consistency in scale. Use cosine similarity as the primary metric²².
   2. For each pair (e_t, e_v) of textual entity embedding and visual entity embedding, calculate the similarity score²³.
      Score(e_t,e_v) = λ · sim_text(e_t,e_v) + (1-λ) · sim_visual (e_t,e_v) .
      where:
      λ∈ [0,1] is the modality weighting parameter,
      sim_text is the cosine similarity between textual embeddings,
      sim_visual is the cosine similarity between visual embeddings.

Results

The proposed VISHAM-KG is evaluated through similarity score computation and link prediction tasks commonly used in the knowledge representation benchmark dataset.

Experimental setup

Evaluate the constructed multimodal knowledge graph using two established tasks: (i) cross-modal similarity assessment and (ii) knowledge graph link prediction. Perform all evaluations exclusively on the finalized graph output generated at the endpoint of the protocol. ...

Discussion

The performance of the VISHAM-KG framework is primarily based on three critical components: OCR for Devanagari text (step 1.2), confidence-based visual object detection using Clip-ViT (step 1.3) and embedding-based cross-modal alignment (step 1.4). OCR accuracy directly influences the downstream linguistic parsing and entity extraction. The errors introduced at this stage propagate to relation identification and reduce alignment precision. This effect is mitigated through Hindi-specific normalization, lemmatization, and ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
BiLSTM-CRF and Indic NER Model	Custom-trained	PyTorch	Named entity recognition
CLIP-ViT-B/32	2022-09	OpenAI	Visual embedding generation
CPU	Intel i9	Intel	General computation
EasyOCR	v1.7.1	Jaided AI	Hindi text extraction from images
GPU	NVIDIA RTX 3090	NVIDIA	Model inference acceleration
Hindi Kids Stories	10 stories	Curated dataset	Evaluation corpus
Neo4j	v5.13	Neo4j Inc.	Knowledge graph storage
NumPy	v1.24	NumPy Community	Numerical computations
Pandas	v2.0	Pandas Community	Data handling
Python	v3.10	Python Software Foundation	Pipeline implementation
PyTorch	v2.0	Meta AI	Deep learning framework
Stanza (Hindi Model)	v1.6.1	Stanford NLP	POS tagging and dependency parsing
XLM-R (Base)	2023-05	HuggingFace	Text embedding generation
YOLOv8	v8.0.208	Ultralytics	Visual object detection