Closed Domain Semantic Question Answering System as a Use Case of Transformer-Based BERT Models

Summary

This study designs an adaptive learning system for extracting knowledge from text to answer questions, comparing Google-BERT, DistilBERT, RoBERTa, and TF-IDF model, using cosine similarity, on latency and semantic generalization. Google-BERT performs best, though the system remains sensitive to spelling errors, emphasizing the need for strong preprocessing for practical application.

Abstract

To extract knowledge from text files and answer users' questions by finding the right information in context, an adaptive learning system, such as a Question Answering System (QAS), is designed for this purpose. This focus encourages further study on direct-answer systems and the use of large-scale tests for Question Answering (QA) tasks. To facilitate this, a Semantic Closed-domain QA (SCD-QA) dataset, which encompasses both factoid and non-factoid questions, is employed in conjunction with pre-trained transformer models. In this study, the ability to make inferences is measured and compared between three pre-trained transformer models, like Google-BERT, DistilBERT, and RoBERTa, on the SQuAD dataset, and a classic keyword-based TF-IDF model with cosine similarity. Results show Google-BERT performs best, with a mean Exact Match (EM) score of 90.0 and a mean latency of 1.27 s. The system also performs well on questions with synonyms, showing a strong understanding of meaning. But it performs inadequately on questions with spelling errors, indicating it is sensitive to misspellings and requires better early processing in use.

Introduction

Natural Language Processing (NLP) is a subdomain of Artificial Intelligence (AI) where QAS can be constructed, allowing computer systems to automatically generate answers to questions¹. NLP primarily focuses on the comprehension and interpretation of written language in a manner that emulates the cognitive processes of the human brain². Engaging in these tasks establishes NLP as a pivotal technique in constructing a system capable of generating human-like responses.

QAS, which responds to the user's questions, is one example of how NLP is used³. QAS is a study area that incorporates research from various domains with common issues, such as Information Retrieval (IR), Information Extraction (IE), and NLP. At present, contemporary search engines primarily perform document retrieval tasks⁴. In other words, when provided with specific keywords, these search engines exclusively yield a list of relevant documents ranked based on relevance and containing the specified keywords. They do not provide a precise answer. The purpose of QAS is to assist individuals in locating precise answers to particular queries within limited subject areas⁵. The grouping of QAS can be done based on the following categories outlined in Figure 1⁶. Both the factoid and non-factoid QAS operate using natural language (NL) and are designed to respond to questions posed in NL. The system utilizes NLP techniques to provide answers based on the available corpus⁷.

Figure 1: Categorization of QA systems. The figure illustrates a mind map of the primarycomponents of a QA system. At the center is the QA System, which handles various types ofquestions, including factoids, non-factoids, hybrids, visuals, conversational, and multiple-choicequestions. Please click here to view a larger version of this figure.

This study presents a Semantic Closed-Domain Question Answering (SCD-QA) system for responding to users in NL about Ph.D. admission policies and procedures. The system uses the Ph.D. rules and regulations PDF⁸ as its core context and implements the transformer-based Bidirectional Encoder Representations from Transformers (BERT)⁹ model as the main framework for generating responses. The goal is to design a semantically conversational FAQ-based QAS that delivers accurate answers to common questions, making it easier for newly admitted students to quickly find the essential information they need.

To achieve this goal, three widely well-known transformer-based BERT models were employed: Google-BERT⁹, DistilBERT^10,11, and RoBERTa^12,13, all trained on the Stanford Question Answering Dataset (SQuAD)¹⁴. Their performance was assessed using two critical parameters: semantic matching, measured by Exact Match (EM)¹⁴, and model Latency¹⁵, analyzed by mean reaction times. Additionally, a Vector Database (VD)^16,17 was used to store semantic embeddings, facilitating the retrieval of the most semantically relevant context for a user's question based on semantic similarity scores.

Literature review

Transformer-based models have overhauled the field of NLP, yielding significant improvements in QAS. The introduction of BERT⁹ has firmly emphasized transformer architectures as an essential component in developing robust and precise QAS. In this section, a comprehensive review of recent improvements in transformer-based models and semantic embeddings, with a focus on their relevance to the Semantic Conversational Dialogue (SeCD) QA, is presented. A comparative analysis is performed to justify the integration of these models in this work, emphasizing their viability to drive meaningful insights in the field.

Transformer-based architectures like BERT, DistilBERT, and RoBERTa use self-attention methodology to capture complex contextual relationships in text, which makes them ideal for tasks that need significant semantic understanding, such as QAS. These models, along with specialized sentence transformers, generate semantic embeddings, help to enable swift and precise context retrieval through vector-based similarity searches in SeCD QA. Recent research focuses on improving these models and embedding techniques to boost throughput, scalability, and efficiency, especially for task-specific uses like SCD-QA.

Sengupta et al.¹⁸introduced a creative approach to boost Multiple-choice QAS (MCQAS) for science-based questions by fine-tuning BERT models. In their framework, they first assessed distributed semantic similarity between the question-answer pairs and then fed these similarity scores, along with original features, into a classification layer. This combined approach achieved an F1 score of 90.2% on the SciQ dataset, highlighting BERT's efficacy in task-specific QASs that require nuanced understanding and precise classification.

Cichecki et al.¹⁹ compared ChatGPT and fine-tuned BERT models for smart design support systems and found that fine-tuned BERT models excel general-purpose Large Language Models (LLMs) like ChatGPT on domain-specific tasks, achieving an F1 score of 88.5% on a custom-designed dataset. The study shows the importance of domain-specific fine-tuning for improving model performance in specialized applications.

Guo et al.²⁰ examined the efficiency of task-specific fine-tuning strategies for QASs under limited annotation budgets. Their work revealed that a double fine-tuning approach-initially on a general QA dataset such as SQuAD, followed by fine-tuning on a task-specific dataset-achieves a significant 15% advancement in F1 score on datasets like COVID-QA. This finding highlights the critical role of sequential fine-tuning in enhancing the performance of SeCD QASs.

Pudasaini and Shakya²¹ examined the application of fine-tuned BERT models for QA in biomedical research papers, reporting EM and F1 scores of 83.89% and 89.67%, respectively, on a custom biomedical QA dataset sourced from PubMed. By leveraging transfer learning with PubMedBERT, their technique illustrated a notable ability to process complex biomedical queries, emphasizing the potential of task-specific fine-tuning in this field.

Baek et al.²² proposed the Knowledge-Augmented Language Model Prompting framework for zero-shot Knowledge Graph (KG) QA. This approach uses semantic similarities to retrieve pertinent facts from a KG and incorporates them into the input question. As a result, it achieved an F1 score of approximately 85% on KG-QA datasets. The work illustrates the potential of combining KGs and transformer models, highlighting the benefits of knowledge augmentation in enhancing QA performance.

Hu et al.²³ described a QAS designed for the household registration domain, leveraging a KG alongside BERT-based models (RoBERTa-BiLSTM-MultiHeadAttention) for intent classification and semantic analysis. By simplifying questions into single-event entities and intention relationships, this achieved high accuracy in querying structured data. Furthermore, the methodology's scalability to other domains emphasizes its potential for broad applicability in KG-based QAS.

Luo et al.²⁴ introduced a BERT-based schema for Knowledge Base (KB) QA, integrating a Multi-Granularity Pruning Model (MGPM) with relation-aware attention. Their approach achieves significant accuracies of 94.4% on SimpleQuestions, 68.6% on WebQuestionsSP, and 63.7% on WebQuestions. These results show BERT's effectiveness in leveraging semantic similarity for structured data queries. Overall, this study highlights the potential of transformer-based models for KB-QA, particularly in scenarios where exact entity and relation identification are important.

Wu et al.²⁵ designed a retrieval-augmented generation framework for QA in construction management, with a specific focus on safety and compliance queries. By integrating BERT-based semantic embeddings with a generative transformer model and a task-specific KG, their approach achieved an F1 score of 88.7% on a construction-related QA dataset. This study demonstrates the potential of combining semantic understanding with KG-based retrieval to improve accuracy on a task-specific QA dataset for construction management.

Peng et al.²⁶ systematically evaluated LLMs, including BERT and GPT-based models, for medical QA. By combining BERT's contextual understanding and GPT's generative abilities, they used retrieval-augmented generation and domain-specific fine-tuning to achieve an F1 score of 86.2% on a dataset from PubMed and clinical notes. This study highlights the potential of ensemble models to improve inference accuracy in healthcare, where both context and precise generation are critical.

Gardazi et al.²⁷reviewed BERT's use in NLP tasks such as QA, sentiment analysis, and Named Entity Recognition (NER). Their analysis showed that fine-tuned BERT models achieve F1 scores of 85%-92% on task-specific QA datasets such as SQuAD. This shows that BERT reliably produces effective contextual embeddings and adds KGs, making it well-suited for SeCD QAS.

Transformer-based models have become the decisive choice for SeCD QASs due to their unique ability to capture context-sensitive processing, scale efficiently, and adapt to domain-specific tasks. Table 1 illustrates this decisive advantage by comparing transformer-based models with alternative methods examined in this study^{18,20,22,23,24,25}.

Approach	Key Features	Advantages	Limitations	Performance Metrics (SQuAD)	Use Case Suitability
Transformer-Based Models (BERT, RoBERTa, DistilBERT)	Bidirectional context modeling, self-attention, fine-tuning on domain-specific data 16, 17, 19, 24, 25	High accuracy, robust semantic understanding, scalable with vector databases 18, 20, 22, 24, 25	Higher computational cost, requires pre-training 17, 18	EM: 80–90%, F1: 85–93% 16, 17, 19, 24, 25	Ideal for Closed Domain (CD) -QA, FAQ systems, and semantic search 16, 17, 19, 20, 22, 24, 25
Traditional Rule-Based Systems	Hand-crafted rules, keyword matching 16	Low computational cost, simple implementation 16	Limited scalability, poor handling of complex queries 16, 18	EM: 50–60%, F1: 55–65% 16	Basic QAS with structured data 16
Recurrent Neural Networks (RNNs)	Sequential processing, LSTM/GRU architectures 19	Moderate performance for sequential tasks 19	Struggles with long-range dependencies, slower inference 19, 9	EM: 65–75%, F1: 70–80% 19	General NLP tasks, less effective for CD-QA 19, 9
Keyword-Based Retrieval Systems	TF-IDF, BM25 algorithms 18, 22	Fast retrieval, low resource usage 18, 22	Limited to surface-level matching, poor semantic understanding 18, 22	EM: 40–50%, F1: 45–55% 18, 22	Document retrieval, not suitable for precise QAS 18, 22
Convolutional Neural Networks (CNNs)	Local feature extraction, convolutional layers 25	Efficient for short-text classification, fast inference 25	Limited contextual understanding, less effective for complex QAS 25	EM: 60–70%, F1: 65–75% 25	Text classification, not ideal for CD-QA 25
Graph Neural Networks (GNNs)	Graph-based modeling, relational reasoning 20	Effective for multi-hop reasoning, captures document relationships 20	High computational complexity, requires structured data 20	EM: 70–80%, F1: 75–85% 20	Multi-hop QAS, less suited for single-context CD-QA 20
Knowledge Graph-Based Systems	Structured knowledge representation, entity linking 21	Strong for structured data queries, leverages external knowledge 21	Requires pre-built knowledge graphs, limited to known entities 21	EM: 65–75%, F1: 70–80% 21	Domain-specific QAS with structured data 21
Attention-Augmented Models	Enhanced attention mechanisms, context-focused processing 24	Improved focus on relevant text segments, high accuracy 24	Higher computational cost, complex implementation 24	EM: 85–90%, F1: 88–92% 24	Complex CD-QA, non-factoid questions 24
Bag-of-Words Models	Word frequency-based representation 22	Simple, computationally lightweight 22	Poor semantic understanding, ineffective for complex queries 22,25	EM: 35–45%, F1: 40–50% 22	Basic text retrieval, not suitable for QAS 22, 25
Hybrid Neural Models (CNN+RNN)	Combines local and sequential processing 25	Balances local and contextual understanding 25	Moderate complexity, struggles with long contexts 25	EM: 68–78%, F1: 72–82% 25	General NLP tasks, moderate fit for CD-QA 25
Statistical Language Models	Probabilistic word associations 18	Fast for simple queries, low resource usage 18	Limited contextual understanding, poor scalability 18	EM: 45–55%, F1: 50–60% 18	Basic QAS, not suited for complex CD-QA 18

Table 1: Comparison of transformer-based models with other approaches for QAS. The table provides a side-by-side comparison of various QA system approaches, including Transformer-Based Models, Rule-Based Systems, RNNs, and others. It summarizes their main features, strengths, weaknesses, performance on SQuAD, and the tasks for which they are best suited, such as CD-QA, semantic search, and text classification.

The main objective of this research is to enhance students' access to institutional information by developing efficient, scalable, and precise SCD-QA systems in NLP. Specifically, this confirmatory study applies and validates pre-trained transformer-based models within the domain of Ph.D. admission policies. The aim is to demonstrate their efficacy and practical feasibility by incorporating semantic matching, model latency assessment, and VD-driven semantic retrieval. This approach offers an in-depth analysis for delivering accurate, domain-specific answers in a focused institutional context. Figure 2 illustrates the system's holistic architectural framework.

Figure 2: Overall schema of the SCD-QA system. The figure illustrates a flowchart of a QA system that utilizes large language models (LLMs). It begins with a context file and a user's question, which are handled by three models: Google-BERT, DistilBERT, and RoBERTa, and one keyword-based model: TF-IDF. The system evaluates each model's performance using a checklist, then selects the best one based on the results. Please click here to view a larger version of this figure.

Protocol

Comprehensive Theoretical Basis

QAS Component: The QAS framework consists of three segments, as shown in Figure 3: i) Question processing module (QPM), ii) Document processing module (DPM), and iii) Answer extraction and formulation module (AEFM). The system receives questions that fall into two main categories: Factoid and Non-Factoid. Factoid questions typically use interrogative words like what, where, when, or who, while non-Factoid questions use words such as how and why.

DPM: From the provided list, the user may select a specific passage. Next, each token in the passage is tagged using a Part-of-Speech (POS) Tagger. To extract verbs, identify all tokens tagged as verbs. Combine these verbs with a list of unconventional verbs and apply logic for regular verbs. Create a data structure (array) containing the extracted verbs, their tenses, and their -ing forms.

QPM: The system receives input in the form of a question from the user. The text is tokenized using the StringTokenizer class, and the resulting tokens are stored in a separate data structure. This data structure is then returned for further utilization within the programme.

AEFM: The first step is identifying the verb in the given question. The verb that has been recently identified is now matched with the tokens that were generated during the document processing stage. The chosen case for a specific type of factual question (such as what or when) is utilized to extract and construct the answer in a more precise manner.

Before choosing the sort of questioning, the user is first prompted to select the passage of their choice. The QPM is responsible for processing the user's question and forwarding it to the AEFM. The AEFM utilizes the extractions obtained from the DPM and the processed documents that contain the tagged format of the original input document. The module will pass the required algorithms to the formulation module to obtain the desired answer.

Figure 3: Components of QAS. The figure illustrates the four-step process of a QA system: Question, Question Processing, Answer Processing, and Answer. In Question Processing, the system classifies the question. In Answer Processing, it finds and reviews documents and passages to produce the answer. Please click here to view a larger version of this figure.

BERT model: To discover the associations between words in a text, the BERT method uses a transformer. There are two mechanisms in a transformer- an encoder and a decoder²⁸, but just the encoder is needed for BERT. BERT takes a two-way approach and systematically scans the input text to teach itself the meaning of words in their context. The encoder takes as input a series of tokens that have been vectorized. The vectors are then fed into the neural network, which produces a series of vectors that reflect the input. A word's output vector changes depending on the sentence in which it appears. A word's vector might vary depending on the context in which it appears; for instance, "like" in "He likes to play cricket" has a different vector than "like" in "His face turned red like a tomato." The approach begins with a phase of text processing before moving on to the model-building phase. The steps that BERT takes to process text are discussed in the next section²⁸.

Text processing: The BERT model represents input text in accordance with a prescribed set of principles. Additionally, this factor contributes to the improved performance of the model. The input embedding in BERT consists of an amalgam of three distinct types of embeddings²⁸.

Position embeddings (PEs): To learn the order-related information in the embeddings, PEs are used. PEs are used to restore information about the order that is lost in transformers. BERT develops distinct PEs specifically for each point in the input sequence. BERT possesses the capability to convey the positional information of words inside a sentence by utilizing PEs. This enables BERT to effectively capture and represent the sequence or order of words.

Sentence embeddings (SEs): Additionally, to assist the model in distinguishing between the first and second sentences, BERT learns an embedding that is unique to each of them. It is also capable of accepting paired sentences as inputs for activities such as QA.

Token embeddings (TEs): TEs are taught for each token in the WordPiece token vocabulary. The WordPiece token vocabulary comprises sub-word units derived from words found within the corpus. As an illustrative instance, this vocabulary collection will encompass all conceivable sub-words of the term Question, including Questio, Questi, and so on.

A token's input representation is built by adding together its embeddings at the segment and position levels. Because of this, it is an extensive embedding approach that provides a wealth of information to the model. Figure 4⁹ depicts the embeddings of the BERT model.

Figure 4: BERT embeddings. The figure shows how input embeddings are created for a transformer model. It begins with an input sequence: [CLS] the [MASK] sky is cloudy [SEP] it will rain [SEP]. Each token in the sequence receives its own embedding, such as E_the or E_[MASK]. Then, sentence embeddings are added for each sentence, such as E_A for the first and E_B for the second. Positional embeddings, labelled E₀ to E₉, are also included to indicate the position of each token. All of these are combined to form the final input embeddings. This figure has been modified from⁹. Please click here to view a larger version of this figure.

QAS design using BERT: For illustrative purposes, examine this question in conjunction with a paragraph extracted from a Wikipedia entry on Football League²⁸.

Question: Where was the Football League founded?

Passage: In 1888, the Football League was founded in England, becoming the first of many professional football competitions. During the 20th century, several of the various kinds of football grew to become some of the most popular team sports in the world.

Answer: England

The BERT model utilizes token extraction from both the question and context, subsequently combining them into a unified input. As previously stated, the process begins with the utilization of a [CLS] token, which serves as an indicator for the commencement of a sentence. Additionally, a [SEP] separator is employed to distinctly separate the question and passage. In addition to the [SEP] token, BERT incorporates SEs to distinguish between the question and the passage²⁸ containing the answer. BERT uses two SEs, one dedicated to the question and another to the passage, to establish a clear distinction between them. The embeddings are subsequently combined with a one-hot representation²⁸ of tokens to differentiate between the question and the passage. This process is illustrated in Figure 5.

Figure 5: BERT input representation. The figure illustrates how input embeddings are generated for a BERT-based QAS. It starts with the token [CLS], then the question How many? [SEP], and the passage BERT large is, each with its sentence embeddings (A for the question, B for the passage). Token, sentence, and positional embeddings are combined to make the final input. Please click here to view a larger version of this figure.

Subsequently, the combined embedded representation²⁸ of the question and context is utilized as input in the BERT model. The final hidden layer of BERT is modified to use SoftMax to generate probability distributions. These distributions determine the start and end indices of a substring within the input text sentence, which represents an answer, as depicted in Figure 6, for a visual representation.

Figure 6: BERT processing workflow for QA. The figure shows how the BERT model works for QA. It presents an input sequence that includes a question, marked by a classification token [CLS] at the start and a [SEP] separator token at the end, followed by a context, separated by another [SEP]. The tokens in this sequence are turned into embeddings. The model then predicts the start and end positions of the answer within the passage. Please click here to view a larger version of this figure.

Vector database (VD): A VD^17,18 is a specialized system for efficiently storing, managing, indexing, and querying high-dimensional vector representations of data. Vectors are often generated from deep learning models and encapsulate semantic or contextual information about the data. Each dimension of a vector represents a specific feature. Embeddings are numerical representations of objects such as text, images, videos, and audio. These embeddings are used in various applications, including Machine Learning (ML), NLP, recommendation systems, computer vision, and IR. VDs facilitate effective similarity searches and semantic querying by grouping similar objects close together in the vector space. Facebook AI Similarity Search (FAISS)²⁹ is a prominent VD used in this study to identify the most relevant context for user queries. Table 2 outlines the main characteristics of VDs and their use cases.

Feature	Description
High-Dimensional Data	Handles data with hundreds or thousands of dimensions.
Approximate Nearest Neighbour (ANN)	Enables fast similarity searches by approximating distances.
Scalability	Supports large-scale datasets with billions of vectors.
Integration	Often integrates with AI/ML frameworks and tools for seamless workflows.
Real-Time Queries	Provides low-latency vector searches for interactive applications.

Table 2: Main characteristics of VD. The table highlights important features of VD. These include handling high-dimensional data, running fast similarity searches using ANN algorithms, scaling to large datasets, working with AI and ML tools, and supporting quick, real-time queries for interactive use.

Performance evaluation metrics: The SCD-QA system was evaluated using two primary metrics: EM score¹⁴ and model Latency¹⁵. The EM score, a standard metric in QA studies, assesses prediction accuracy by measuring the proportion of predicted answers that exactly match the ground truth. For a set of N questions, the EM score is formally defined in Equation 1 as follows:

  where    (1)

This metric assigns a score of 1 for an exact match with the reference answer, and 0 for any difference.

The Latency metric quantifies the total processing time required by the system to generate a response to an individual query. This metric is calculated as the mean elapsed time across all queries, as presented in Equation 2:

(2)

where T_starti  and T_endi are the timestamps marking the start and end of processing the i^th query, respectively. Latency is reported in s and captures the system's responsiveness, encompassing all stages from input processing to answer generation.

Method

In order to implement the SCD-QA, the following test studies are incorporated in this paper.

SCD-QA dataset: A proposed dataset³⁰ is introduced for the implementation of the SCD-QA system. This dataset is derived from a text corpus containing Ph.D. rules for 2022⁸, for student guidelines from NIT Arunachal Pradesh, India. To establish the SCD-QA system, it is necessary to generate a JSON file that will encompass all pertinent information in a precise format. The dataset is generated from the text corpus using the Haystack annotation tool (version 2.18.1)³¹, an open-source framework. This tool facilitates the creation of the SCD-QA dataset in the style of the SQuAD¹⁴. The steps to create a SQuAD-type annotated dataset from the PDF file are shown in Figure 7, and the structure of our dataset in SQuAD style is illustrated in Figure 8.

Figure 7: Steps to create an annotated dataset from a PDF file. The figure illustrates a stepwise workflow for creating a SQuAD-style annotated dataset using Haystack tools. It outlines the process from extracting text from PDFs, cleaning and splitting it into passages, manually annotating question-answer pairs, storing the annotations in SQuAD JSON format, and finally exporting the dataset for model training. Please click here to view a larger version of this figure.

Figure 8: Structure of factoid QA dataset. The figure displays a JSON data structure that represents a paragraph and a QA pair from a dataset. It has a paragraphs section, which contains a set of questions and answers. One question asks, What is the minimum mark required to take admission to Ph.D. Science? Each question has an ID and an array of answers. The answer includes a document ID, question ID, the text 60% marks, the starting position of the answer, and an unspecified answer category. The is_impossible flag is set to false. Please click here to view a larger version of this figure.

The dataset is structured as a list of dictionaries, where each dictionary contains key fields such as data, paragraphs, question, answer_id, document_id, question_id, text, answer_start, answer_end, is_impossible, and context.

data: It contains the overall question-answering information.
paragraphs: A particular context, along with its questions and answers.
question: A particular question.
answer_id: A unique identification number for each answer text.
document_id: A unique identification number for each context.
question_id: A unique identification number for each question.
text: The answer text.
answer_start: The answer starting location of the correct answer in context.
answer_end: The answer ending location of the correct answer in context.
is_impossible: It tells whether the answer to the asked question is available in the context or not.
context: The text corpus from which an answer can be found.
All 80 questions in the proposed dataset follow the factoid and non-factoid question format. Samples of some types of framed questions are shown in Table 3.

Questions

Who is PS?

What is the font size of the state-of-the-art document?

How many days are in Maternity Leave?

Table 3: Sample question from the dataset. The table shows examples of two types of questions: the first and second are factoid questions, while the third is a non-factoid question.

FAISS: FAISS (version faiss_cpu-1.9.0)^29,32, developed by Facebook AI Research (FAIR), is an open-source library that facilitates efficient similarity search and clustering of dense vectors. It is specifically engineered to manage large-scale, high-dimensional data effectively. This system facilitates rapid and scalable ANN searches within datasets comprising millions or billions of vectors. It is widely utilized in applications related to embeddings, including NLP, recommendation systems, and image or video retrieval. FAISS offers multiple indexing methods, including Flat (brute force), Inverted File (IVF), Hierarchical Navigable Small World graphs (HNSW), and product quantization (PQ). These methods enable users to optimize search performance according to data size, dimensionality, and hardware specifications. In this study, flat indexing is used, which performs a brute-force search using the L2 (Euclidean) distance to identify the nearest neighbors. The system's scalability supports both CPU and GPU implementations, enabling high-performance computations across extensive datasets. Additionally, it provides flexibility to optimize the balance between speed and accuracy based on specific application requirements. FAISS is frequently used in NLP to assess semantic similarity in embeddings, such as BERT or Word2Vec. FAISS is utilized in QAS to store and manage vector representations of documents or sentences. It performs approximate ANN searches³³ to retrieve the most relevant context for user questions, enhancing semantic search with embeddings from transformer models such as BERT. FAISS is a fundamental tool in AI and ML workflows due to its versatility.

BERT-large-uncased-whole-word-masking-finetuned- SQuAD model: The present work utilizes a pre-trained language model known as BERT-large-uncased-whole-word-masking finetuned-squad⁹ to develop a factoid QAS using the dataset proposed in the study. The BERT model underwent pre-training on BookCorpus, a dataset of 11,038 unpublished books⁹, as well as English Wikipedia, with the exception of lists, tables, and headings. The model in question is uncased, meaning that it does not distinguish between the words english and English. The model is a pre-trained transformer model that has been trained on a substantial amount of English data using a self-supervised approach. The model was pre-trained exclusively on raw texts, without any human annotations. This allows it to leverage a large amount of publicly accessible data. The pre-training process involves automatically generating inputs and labels from the provided texts. The model was trained with two specific objectives⁹:

MLM: The process involves randomly masking 15%⁹ of the words in the input sentence. The model then processes the entire masked sentence and predicts the masked words. This approach diverges from conventional Recurrent Neural Networks (RNNs), which typically process words sequentially, and from autoregressive models such as GPT, which employ internal masking of future tokens. The functionality enables the model to acquire a bidirectional representation of the sentence.

NSP: During the pretraining phase, the model combines two disguised sentences as inputs. In some cases, these sentences are adjacent in the original text, indicating a direct relationship. In other cases, they are not, meaning there is no original proximity or context linking them. The subsequent step involves the model predicting whether the two sentences are logically coherent.

The present model is characterized by the subsequent configuration: the model architecture consists of 24 layers, with a hidden dimension of 1024. It utilizes 16 attention heads and has a total of 336 million parameters⁹.

WordPiece is used to tokenize the texts with a 30,000-word vocabulary size in the pre-processing steps. The model's inputs would then look like this: [CLS] Sentence A [SEP] Sentence B [SEP]. The only requirement is that the total length of the combined sentences is less than 512 tokens⁹. The specific pre-trained model yields the subsequent output: the F1 score achieved is 93.15%, whereas the precise match score is 86.91%⁹.

Distilbert/distilbert-base-cased-distilled-squad model: The DistilBERT¹⁰ model is a more compact, rapid, and efficient variant of the BERT⁹ model, developed to maintain the majority of BERT's semantic ability to comprehend while being less resource-intensive and more appropriate for practical applications with restricted computational capacity. The version distilbert-base-cased-distilled-squad has been fine-tuned on the SQuAD¹⁴ for QA tasks. The model utilizes the transformer architecture, including a decreased size of 66 million parameters in contrast to BERT's 110 million, while maintaining 97% of BERT's efficacy in language comprehension. DistilBERT reaches this using a method known as Knowledge Distillation, which conveys information from the larger BERT model to the more compact DistilBERT model. Because of its smaller size, it can infer information more quickly and use less memory, which makes it perfect for resource-constrained applications like mobile or edge devices. The model, carefully fine-tuned on the SQuAD 1.1 dataset, demonstrates exceptional proficiency in extractive QA tasks, whereby the objective is to locate a segment of text from a specified context that answers a question. DistilBERT achieves approximately 85% of BERT's F1 score on the SQuAD leaderboard and retains a significant portion of BERT's linguistic capacity, despite its reduced size. This model is an exceptionally efficient solution for production-level QA jobs, optimizing both performance and computational efficiency.

Deepset/roberta-base-squad2: The deepset/roberta-base-squad2 model^12,13 is a fine-tuned variant of the RoBERTa architecture. It is specifically designed for the SQuAD¹⁴ 2.0 dataset, which includes both answered and unanswerable questions. This improved RoBERTa framework eliminates BERT's⁹ NSP jobs. It also uses dynamic masking during training to boost efficiency and performance in NL comprehension tasks. The model has 125 million parameters and uses a bidirectional transformer. This allows it to acquire contextual information from both directions and excel at extractive QA tasks.

Fine-tuning on SQuAD 2.0 enables the model to identify the correct response span within a given context and to recognize when there is no answer. It uses a Byte Pair Encoding (BPE) tokenizer that handles sub-word tokenization and preserves case sensitivity. This improves its ability to process uncommon and complex words. The model performs well, achieving roughly 85%-90% F1 and 80%-85% EM. Its ability to spot unanswerable questions and its effective deployment make it valuable for customer service, knowledge retrieval, and virtual assistants.

The most effective way to deploy SQuAD-fine-tuned BERT models for QA is to use the Hugging Face Transformers pipeline³⁴. This framework streamlines tokenization, input formatting, model loading, and output post-processing. These models are widely recognized for their effectiveness in extractive QA, where the answer is a text span directly retrieved from the given context. To guide implementation, the following procedure outlines the steps for executing BERT models.

Input and setup: This process requires four main inputs and setup parameters: the model name, specified as a string identifier from the Hugging Face Hub (for example, google-bert/bert-large-uncased-whole-word-masking-finetuned-squad, distilbert/distilbert-base-cased-distilled-squad, or deepset/roberta-base-squad2); the question (Q), provided as a string containing the query; and the context (C), a string representing the relevant text or passage. The primary tool used is the high-level pipeline function from the Hugging Face transformers library (version 4.57.0) in Python (version 3.12.12).

Process: Execution (Hugging Face Pipeline): The process consists of six primary steps and employs the Hugging Face pipeline to address the complexity of the QA task:

Install Library: Begin by installing the required library with the command pip install transformers. This installation enables access to the models and the streamlined pipeline functionality.

Import Pipeline: Import the pipeline function using: from transformers import pipeline.

Initialize QA Pipeline: Initialize the QA pipeline using qa_pipeline = pipeline("question-answering", model=""). This command downloads the model and tokenizer, making them ready for inference.

Define Input: Set question = ... and context = ... to prepare the model's data.

Run Inference: Pass the question and context to the pipeline with result = qa_pipeline(question=question, context=context). The model processes the input and provides an answer.

Extract Answer: Get the answer string from the output dictionary using: answer = result['answer'].

Output: The process produces several output parameters in the following order: Answer (the extracted text span from the context, presented as a string), Score (a floating-point value quantifying the model's confidence in its prediction), and Start and End indices (integers specifying the character positions of the answer span within the context).

Sentence-transformers/all-MiniLM-L6-v2: The all-MiniLM-L6-v2³⁵ is a pre-trained sentence transformer from the Sentence-Transformers library (version 5.1.1)³⁶. It is optimized for efficient, precise text embedding. Developed on Microsoft's MiniLM framework, it includes six transformer layers and 384-dimensional embeddings. This design balances performance and computational proficiency, making it suitable for real-time applications. The model was trained on over a billion phrase pairs from datasets like SNLI, MultiNLI, STS benchmarks, and web-crawled data. As a result, it demonstrates proficiency in semantic similarity, grouping, and search-related tasks. Because of its small size (~22MB) and enhanced inference performance, all-MiniLM-L6-v2 simplifies applications such as semantic search, duplication detection, and text categorization. Although it is lightweight, it delivers strong accuracy on benchmarks such as STS-B and SICK-R, making it an effective choice for both scalable and limited-resource scenarios. The workflow for generating the embedding using Sentence-Transformer is depicted in Figure 9 below.

Figure 9: Step of the embedding process using the sentence transformer model. The figure illustrates the workflow for generating text embeddings using the sentence transformers framework. It outlines the process from importing libraries and loading a pretrained model to preparing text data, generating embeddings, converting them into NumPy arrays, and storing them for downstream tasks like indexing or similarity search. Please click here to view a larger version of this figure.

Proposed algorithm: This study outlines a suggested methodology in Algorithm 1 for the development of the SeCD-based SCD-QA system, capable of addressing both factoid and non-factoid questions asked by users.

ALGORITHM 1: Algorithm of the proposed SeCD-based SCD-QA system
Input: Set of raw context file (C), and the User's query (Q).
Output: The selected optimal transformer-based LLM (M'), and the response generated by M'.
Pre-processing Contexts: Annotate the raw context set C to create a structured dataset in D_SQuAD format.
D_SQuAD = f_annonate (C)
Generate Context Embeddings: Use a Sentence-Transformer f_embed to convert each context c  D_SQuAD into an embedding e^c.

Store Embeddings: Store E_c in a vector database V for efficient similarity search.

Generate Query Embedding: Transform the user's query Q into an embedding e_q using the same Sentence Transformer.

Context Retrieval: Retrieve the most relevant context embedding  from the database V based on similarity to e_q.

where sim(e_q,e_c), is the similarity function.
Pass Context to LLMs: Feed the retrieved context  into 3 transformer-based LLMs: Google-BERT (M₁), DistilBERT (M₂), RoBERTa (M₃) and a traditional model: TF-IDF+Cosine Similarity (M₄)

Model Evaluation: Compare the responses {R₁,R₂,R₃,R₄} using an evaluation function f_eval, which scores each response.

Select the Best Model: Identify the model M' with the highest evaluation score S'

Generate Final Output: Use M' to generate the final response R based on 

End

The proposed algorithm process outlines a systematic approach (Figure 10) for choosing an ideal transformer-based LLM to address user questions. It incorporates preprocessing, embedding-based context retrieval, and multi-model assessment to guarantee high-quality and contextually relevant answers. Below, the step-by-step process is explained for clarity:

Data Preparation and Preprocessing: The first phase involves the processing of raw textual data-

Input: The raw text files⁸ are adapted into the system.

Annotation Tool³¹: The input is transformed into a structured dataset in SQuAD¹⁴ format. This step involves creating QA pairs. It also organizes relevant textual data into well-defined context blocks.

Output: The dataset is now ready to create embeddings.

Context Embedding Generation: To enable efficient and scalable context retrieval, a trained Sentence-Transformer^35,36 model is applied to the annotated dataset. This transformer converts text into high-dimensional embeddings that capture semantic meaning. The embeddings are stored in VD, FAISS^29,32 to enable fast similarity-based searches.

User Query Processing: When a user submits a question, the question is processed by the same Sentence Transformer^35,36. This generates a corresponding embedding in the same vector space^29,32 as the context embeddings. This design ensures the query can be matched with relevant context entries.

Context Retrieval Using Vector Similarity: Using a similarity metric (e.g., cosine similarity), the system compares the query embedding with stored context embeddings. The system selects the most relevant context based on the highest similarity score. This ensures that only pertinent context is passed to downstream models. The process reduces computational overhead and improves relevance.

Model Evaluation Across Multiple LLMs: The selected context is evaluated by three transformer-based LLMs: Google-BERT²⁸, DistilBERT¹⁰, and RoBERTa¹². It is also evaluated by a traditional keyword-based TF-IDF³⁷ with a cosine similarity model. Each model processes the context and generates a response to the user's question.

Comparative Evaluation: The responses from the four LLM models are evaluated using two key metrics. First, semantic matching is assessed by EM¹⁴. Second, model Latency is analyzed by the mean reaction times¹⁵.

Model Selection and Final Output: The best-performing model is selected as the appropriate LLM for the user's question. The final response from the selected model is considered. This ensures a balance between computational efficiency and response quality.

Figure 10: Workflow of the SCD-QA system using transformer-based models. The figure shows how the SCD-QA system works. First, a raw context file is processed by an annotation tool to create a dataset in SQuAD format. A sentence transformer then generates embeddings, which are stored in a FAISS vector database. When a user asks a question, the system creates an embedding for it and selects the most relevant context. It compares three transformer-based models- Google-BERT, DistilBERT, and RoBERTa, and one traditional TFIDF + cosine similarity score -and uses the best-performing one to provide the answer. Please click here to view a larger version of this figure.

Results

The SCD-QA system implementation used four LLMs and a dataset of 80 factoid and non-factoid questions. Processing was performed in Google Colab, utilizing NVIDIA Tesla T4 GPUs (Driver Version: 550.54.15, CUDA Version: 12.4) with 12.7GB system RAM, 15GB GPU RAM, and 112.6GB disk space. Model performance was evaluated using two metrics: EM¹⁴ for semantic matching and model latency¹⁵ for reaction times.

Discussion

This study introduces the SCD-QA system, which leverages transformer-based language models to deliver precise, context-aware responses from structured institutional documents. The system's workflow consists of: (1) data preprocessing; (2) embedding generation using the state-of-the-art sentence transformer, all-MiniLM-L6-v2; (3) similarity-based retrieval through FAISS; and (4) comprehensive model evaluation. The pipeline was tested on the SCD-QA dataset, which includes 80 carefully annotated, answerable factoid and non-...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
all-MiniLM-L6-v2	Microsoft	version 5.1.1	Pre-trained sentence transformer model for generating high-dimensional embeddings. Converts text into embeddings for context retrieval.
Annotation Tool	Haystack	Version 2.18.1	Platform for structuring raw text into SQuAD format. Prepares dataset by creating QA pairs and context blocks.
DistilBERT Model	Hugging Face	NA	Lightweight transformer-based LLM with reduced computational requirements. Evaluated for comparison, offers lower latency but reduced accuracy.
FAISS VD	Facebook	faiss_cpu-1.9.0	Scalable VD for similarity-based searches. Stores and retrieves high-dimensional embeddings for efficient query matching.
Google-BERT Model	Google	NA	Pre-trained transformer-based LLM with bidirectional context modeling. Primary model for generating accurate responses to factoid and non-factoid queries.
NVIDIA Tesla T4 GPUs	NVIDIA	Driver Version: 550.54.15 CUDA Version: 12.4	GPUs with 15 GB GPU RAM for model inference. Provides computational power for processing and evaluating LLMs.
Python	Python Software Foundation	version 3.12.12	Python is a leading programming language in the field of Natural Language Processing (NLP) because of its straightforward syntax, comprehensive libraries, and strong community support. It supports a wide range of NLP tasks, including fundamental text preprocessing and complex machine learning implementations.
RoBERTa Model	Deepset	NA	Optimized transformer-based LLM with enhanced training strategies. Evaluated for comparison, balances accuracy and latency.
SCD-QA dataset as SQuAD  Format	NA	Version 2.0	Standardized format for question-answer pairs. Structures dataset for compatibility with transformer models.