Development and implementation of a spoken question answering system for Bangla using large language models

Abstract

This study explores the development and implementation of a spoken question answering system for Bangla, using the latest advancements in deep learning. To achieve this, this research addresses two key aspects: text-based QA and spoken QA. For the text-based phase, we fine-tuned and evaluated several LLMs including mBERT, Bangla-BERT, RoBERTa on the SQuAD_bn datast. We also evaluated the performance of GPT-4o, Llama 3 by calculating Zero-shot and Few-shot performance. Notably, the GPT-4o with some limitations achieved state-of-the art results on this dataset by outperforming the existing models. A detailed error analysis revealed the limitations was from the dataset inconsistencies. Facing the lack of a Bangla spoken QA dataset, we created a synthesized dataset called Spoken_SQuAD_bn, derived from the SQuAD_bn dataset using the Google Cloud Text-to-Speech API. We benchmarked this new dataset using Automatic-Speech-Recognition (ASR) followed by LLMs, using the Audio Overlapping Score (AOS) metric along with the EM and F1. It showed a significant performance drop because of the ASR error propagation, highlighting the challenges of spoken QA in Bangla. This work establishes a foundation of Bangla spoken-QA by demonstrating the potential as well as the limitations of LLMs in this domain and provides a valuable benchmark dataset for future works.