Closed Domain Semantic Question Answering System as a Use Case of Transformer-Based BERT Models

Summary

This study designs an adaptive learning system for extracting knowledge from text to answer questions, comparing Google-BERT, DistilBERT, RoBERTa, and TF-IDF model, using cosine similarity, on latency and semantic generalization. Google-BERT performs best, though the system remains sensitive to spelling errors, emphasizing the need for strong preprocessing for practical application.

Abstract

To extract knowledge from text files and answer users' questions by finding the right information in context, an adaptive learning system, such as a Question Answering System (QAS), is designed for this purpose. This focus encourages further study on direct-answer systems and the use of large-scale tests for Question Answering (QA) tasks. To facilitate this, a Semantic Closed-domain QA (SCD-QA) dataset, which encompasses both factoid and non-factoid questions, is employed in conjunction with pre-trained transformer models. In this study, the ability to make inferences is measured and compared between three pre-trained transformer models, like Google-BERT, DistilBERT, and RoBERTa, on the SQuAD dataset, and a classic keyword-based TF-IDF model with cosine similarity. Results show Google-BERT performs best, with a mean Exact Match (EM) score of 90.0 and a mean latency of 1.27 s. The system also performs well on questions with synonyms, showing a strong understanding of meaning. But it performs inadequately on questions with spelling errors, indicating it is sensitive to misspellings and requires better early processing in use.

Introduction

Natural Language Processing (NLP) is a subdomain of Artificial Intelligence (AI) where QAS can be constructed, allowing computer systems to automatically generate answers to questions¹. NLP primarily focuses on the comprehension and interpretation of written language in a manner that emulates the cognitive processes of the human brain². Engaging in these tasks establishes NLP as a pivotal technique in constructing a system capable of generating human-like responses.

QAS, which responds to the user's questions, is one example of how NLP is used³. QAS is a study area that....

Protocol

Comprehensive Theoretical Basis

QAS Component: The QAS framework consists of three segments, as shown in Figure 3: i) Question processing module (QPM), ii) Document processing module (DPM), and iii) Answer extraction and formulation module (AEFM). The system receives questions that fall into two main categories: Factoid and Non-Factoid. Factoid questions typically use interrogative words like what, where, when, or who, while non-Factoid questions use words such as how and why.

DPM: From the provided list, the user may select a specific passage. Next,....

Results

The SCD-QA system implementation used four LLMs and a dataset of 80 factoid and non-factoid questions. Processing was performed in Google Colab, utilizing NVIDIA Tesla T4 GPUs (Driver Version: 550.54.15, CUDA Version: 12.4) with 12.7GB system RAM, 15GB GPU RAM, and 112.6GB disk space. Model performance was evaluated using two metrics: EM¹⁴ for semantic matching and model latency¹⁵ for reaction times.

Discussion

This study introduces the SCD-QA system, which leverages transformer-based language models to deliver precise, context-aware responses from structured institutional documents. The system's workflow consists of: (1) data preprocessing; (2) embedding generation using the state-of-the-art sentence transformer, all-MiniLM-L6-v2; (3) similarity-based retrieval through FAISS; and (4) comprehensive model evaluation. The pipeline was tested on the SCD-QA dataset, which includes 80 carefully annotated, answerable factoid and non-.......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
all-MiniLM-L6-v2	Microsoft	version 5.1.1	Pre-trained sentence transformer model for generating high-dimensional embeddings. Converts text into embeddings for context retrieval.
Annotation Tool	Haystack	Version 2.18.1	Platform for structuring raw text into SQuAD format. Prepares dataset by creating QA pairs and context blocks.
DistilBERT Model	Hugging Face	NA	Lightweight transformer-based LLM with reduced computational requirements. Evaluated for comparison, offers lower latency but reduced accuracy.
FAISS VD	Facebook	faiss_cpu-1.9.0	Scalable VD for similarity-based searches. Stores and retrieves high-dimensional embeddings for efficient query matching.
Google-BERT Model	Google	NA	Pre-trained transformer-based LLM with bidirectional context modeling. Primary model for generating accurate responses to factoid and non-factoid queries.
NVIDIA Tesla T4 GPUs	NVIDIA	Driver Version: 550.54.15 CUDA Version: 12.4	GPUs with 15 GB GPU RAM for model inference. Provides computational power for processing and evaluating LLMs.
Python	Python Software Foundation	version 3.12.12	Python is a leading programming language in the field of Natural Language Processing (NLP) because of its straightforward syntax, comprehensive libraries, and strong community support. It supports a wide range of NLP tasks, including fundamental text preprocessing and complex machine learning implementations.
RoBERTa Model	Deepset	NA	Optimized transformer-based LLM with enhanced training strategies. Evaluated for comparison, balances accuracy and latency.
SCD-QA dataset as SQuAD  Format	NA	Version 2.0	Standardized format for question-answer pairs. Structures dataset for compatibility with transformer models.