Bangla Dialect Classification and Standardization Using Traditional and Transformer-Based Approaches on a Custom Multi-Regional Corpus

Abstract

Bangla language is one of the most spoken languages in this world but one of the low resource languages in Natural Language Processing (NLP). The challenge is complicated by the existence of several regional dialects like Sylhet, Chittagong, Barishal, Noakhali and Khulna which are quite different from Standard Bangla. This thesis completes the dialect classification and conversion of dialects to standard Bangla dialects using a customized multi-regional corpus, utilizing traditional machine learning models and transformer-based models. A corpus was constructed involving 23,440 dialect and standard sentence pairs from 5 major dialects. Following processes like cleaning, normalization, and dataset splitting, the corpus was used for model training using traditional machine learning models, that is SVM, NB, LR, RF, and advance transformer architectures, that is BanglaBERT, mBERT, MuRIL and XLM-R for classification, and LSTM baseline, BanglaT5, mBART-50, mT5 for standardization. Evaluation used a large variety of metrics: Accuracy, Precision, recall, F1-score for classification, and BLEU, ROUGE-L, METEOR, chrF, TER, and Exact match for standardization. While SVM showed the best accuracy of 81.1%, MuRIL and XLM-R achieved up to 92.4% with macro-F1 of more than 0.92. For indication of the standardization, the mBART50 achieved BLEU = 0.78, ROUGE-L = 0.89, METEOR = 0.87, and Exact Match = 65.6%. A user-friendly Gradio interface has also been created to make the system accessible to any users. This study add a new dialectal corpus, a large study on traditional and transformer models, and build an NLP tool like other models. The result shows us that advanced transformer-based model is appropriate for dialect diversity of bangla and it can help us to create a way for a standardized digital communication in Bangla.