Transformer-Driven Multimodal Visual Speech Recognition of Bengali Numerals
Date
2025-10-25
Journal Title
Journal ISSN
Volume Title
Publisher
Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh
Abstract
Visual Speech Recognition (VSR) decodes spoken content from lip movements, en
abling applications in assistive technologies and silent communication systems. For
Bengali, a language spoken by over 230 million people globally , the scarcity of mul
timodal datasets and the homophene challenge—where words like “Chollish” and
“Ekchollish”arevisuallyindistinguishable—impedeprogressinlipreading. Thisthe
sis introduces the BND Bangla Numeric Dataset, comprising 3,600 high-resolution
videos of 30 diverse speakers reciting Bengali numerals 1 to 100, designed to address
challenges such ashomophenes,variedlighting, cameraangles,andregionaldialects.
We adapt the state-of-the-art SyncVSR framework[4], integrating ResNet18 for visual
feature extraction, a fine-tuned facebook/wav2vec2-large-xlsr-53 for audio tok
enization [12], and a transformer-based BERT for sequence modeling, evaluating its
performance against unimodal and multimodal baselines like LipNet [7]. Our experi
ments investigate how these models tackle the homophene challenge, demonstrating
that SyncVSR achieves a 68.7% accuracy on the BND dataset, significantly outper
forming LipNets 46.2% by leveraging synchronized audio-visual cues to mitigate ho
mophene ambiguity. We identify homophene clusters (e.g., “chollish” , “ekchollish”
, “biyallish” ... also “shottor”, “ekattor” ... and so on ) and compare the BND dataset
with existing ones like LipBengal [32] and BenAV [29], highlighting its unique at
tributes for numerical VSR. Recent studies, such as SynthVSR [42], suggest synthetic
data could address data scarcity, pointing to future research directions. By releas
ing the BND dataset, this work provides a critical resource for the research commu
nity, advances Bengali VSR, andestablishes a foundation for robust numerical speech
recognition in low-resource settings.
Description
Supervised by
Dr. KamrulHasan,
Professor,
Dr. HasanMahmud
Professor,
Department of Computer Science and Engineering (CSE)
Islamic University of Technology (IUT)
Board Bazar, Gazipur, Bangladesh
This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Computer Science and Engineering, 2025
Keywords
Citation
[1] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio visual speech recognition,” IEEETransactionsonPatternAnalysisandMachine Intelligence, 2018, arXivpreprintarXiv:1812.04239,publishedin2022.[Online]. Available: https://arxiv.org/abs/1809.02108 [2] T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: A Large-Scale Dataset for VisualSpeechRecognition,”arXivpreprintarXiv:1809.00496,pp.1–10,2018. doi: 10.48550/arXiv.1809.00496 [Online]. Available: https://ar5iv. labs.arxiv.org/html/1809.00496 [3] T. Afouras, J. S. Chung, and A. Zisserman, Deep lip reading: A comparison of models and an online application, 2019. [Online]. Available: https://arxiv. org/abs/1806.06053 [4] Y.J.Ahnetal., “SyncVSR:Data-Efficient Visual Speech Recognition with End to-EndCrossmodalAudioTokenSynchronization,”arXivpreprintarXiv:2406.12233, pp. 1–15,2024.doi:10.48550/arXiv.2406.12233[Online].Available:https: //arxiv.org/abs/2406.12233 [5] I. Anina, Z. Zhou, G. Zhao, and M. Pietikainen, “Ouluvs2: A multi-view au diovisual database for non-rigid mouth motion analysis,” IEEE, Jul. 2015. doi: 10.1109/FG.2015.7163155 [6] R.Ardilaetal.,“Commonvoice:Amassively-multilingualspeechcorpus,”eng, in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari et al., Eds., Marseille, France: European Language Resources Associ ation, May 2020, pp. 4218–4222, isbn: 979-10-95546-34-4. [Online]. Available: https://aclanthology.org/2020.lrec-1.520/ [7] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “LipNet: End to-End Sentence-level Lipreading,” arXiv preprint arXiv:1611.01599, pp. 1–13, 2016. doi: 10.48550/arXiv.1611.01599 [Online]. Available: https:// arxiv.org/abs/1611.01599 45 [8] C. Bregler and Y. Konig, “Improving speech recognition with visual informa tion,” IEEE Transactions on Speech and Audio Processing, vol. 3, no. 5, pp. 389 396, 1995. [9] Y.Chen,Y.Zhou,T.Zhang,andJ.Yang,“VSP-LLM:VisualSpeechPre-training with Large Language Models,” EMNLP, pp. 1–10, 2024. [Online]. Available: https://aclanthology.org/2024.emnlp-main.456 [10] J.S.ChungandA.Zisserman,“Lipreadinginthewild,”inAsianConferenceon Computer Vision, 2016. [11] J.S.Chung,A.Senior,O.Vinyals,andA.Zisserman,“LipReadingSentencesin theWild,”CVPR,pp.3444–3453,2017.doi:10.1109/CVPR.2017.367[Online]. Available: https://openaccess.thecvf.com/content_cvpr_2017/papers/ Chung_Lip_Reading_Sentences_CVPR_2017_paper.pdf [12] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsuper vised Cross-Lingual Representation Learning for Speech Recognition,” arXiv preprint arXiv:2006.13979, pp. 1–12, 2020. doi: 10.48550/arXiv.2006.13979 [Online]. Available: https://arxiv.org/abs/2006.13979 [13] M.Cooke,J.Barker,S.Cunningham,andX.Shao,“AnAudio-VisualCorpusfor Speech Perception and Automatic Speech Recognition,” Journal of the Acousti cal Society of America, vol. 120, no. 5, pp. 2421–2424, 2006. doi: 10.1121/1. 2229005 [Online]. Available: https://asa.scitation.org/doi/10.1121/1. 2229005 [14] A.Ephratetal.,“Lookingtolistenat the cocktail party: A speaker-independent audio-visual model for speech separation,” ACM Trans. Graph., vol. 37, no. 4, 2018, issn: 0730-0301. doi: 10.1145/3197517.3201357 [Online]. Available: https://doi.org/10.1145/3197517.3201357 [15] A.Fernandez-Lopez and F. M. Sukno, “Survey on automatic lip-reading in the era of deep learning,” Image and Vision Computing, vol. 78, pp. 53–72, 2018. doi: 10.1016/j.imavis.2018.07.002 [Online]. Available: https://www. sciencedirect.com/science/article/abs/pii/S0262885618301276 [16] D. Gaddy and D. Klein, “Digital voicing of silent speech- silent speech emg dataset,” EMNLP 2020, 2020. arXiv: 2010.02960 [eess.AS]. [Online]. Avail able: https://arxiv.org/abs/2010.02960 [17] N.HarteandE.Gillen,“Tcd-timit:Anaudio-visualcorpusofcontinuousspeech,” IEEE Transactions on Multimedia, vol. 17, no. 5, pp. 603–615, 2015. doi: 10. 1109/TMM.2015.2407694 46 [18] M. Kim, J. Hong, S. J. Park, and Y. M. Ro, “Cromm-vsr: Cross-modal mem ory augmented visual speech recognition,” IEEE Transactions on Multimedia, vol. 24, pp. 4342–4355, 2022. doi: 10.1109/TMM.2021.3115626 [19] M.Kim,B.Cao,T.Mau,andJ.Wang,“Speaker-independentsilentspeechrecog nition from flesh-point articulatory movements using an lstm neural network,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 12, pp. 2323–2336, 2017. doi: 10.1109/TASLP.2017.2758999 [20] B.Kirillov andA.Savchenko,“Silent EEG-SpeechRecognition Using Convolu tional and Recurrent Neural Network with85%Accuracyof9WordsClassifica tion,” Sensors, vol. 21, no. 20, p. 6744, 2021. doi: 10.3390/s21206744 [Online]. Available: https://www.mdpi.com/1424-8220/21/20/6744 [21] J. Lee, S. Kim, and H. Park, “AudioVSR: Enhancing Video Speech Recogni tion with Audio Data,” EMNLP, pp. 1–10, 2024. [Online]. Available: https: //aclanthology.org/2024.emnlp-main.858.pdf [22] P. Ma, B. Martinez, S. Petridis, and M. Pantic, “MVM: Memory-Augmented MultimodalNetworks forVideo-AudioSynchronization,”ICCV,pp.1–10,2021. doi: 10.48550/arXiv.2108.01895 [Online]. Available: https://arxiv.org/ abs/2108.01895 [23] B.Martinez,P.Ma,S.Petridis,andM.Pantic,“LipreadingUsingTemporalCon volutionalNetworks,”ICASSP,pp.6319–6323,2020.doi:10.1109/ICASSP40776. 2020.9053841 [Online]. Available: https://arxiv.org/abs/2001.08702 [24] I. Matthews, T. F. Cootes, J. A. Bangham, S. Cox, and R. Harvey, “Extraction of visual features for lipreading,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 198–213, 2002. [25] R. Navarathna, D. Dean, P. Lucey, and H. Ghaemmaghami, “Visual Speech RecognitioninaDriverAssistanceSystem,”EURASIPJournalonAudio,Speech, and Music Processing, 2022. doi: 10.23919/EUSIPCO55093.2022.9909819 [Online].Available:https://eurasip.org/Proceedings/Eusipco/Eusipco2022/ pdfs/0001131.pdf [26] A.V. Nefian, L. Liang, X. Pi, X. Liu, C. Mao, and K. Murphy, “A coupled hmm for audio-visualspeechrecognition,”ProceedingsoftheIEEEInternationalCon ference on Acoustics, Speech, and Signal Processing (ICASSP), 2002. [27] J.T.Panachakel,R.Ganesan,andA.G.Ramakrishnan,“EEGSpeech-RobotIn teraction Dataset,” PapersWithCode,pp.1–1,2021.[Online].Available:https: //paperswithcode.com/dataset/eeg- speech- robot- interaction dataset 47 [28] E. D. Petajan, “Automatic lipreading to enhance speech recognition,” in Pro ceedingsoftheIEEEGlobalTelecommunicationsConference(GLOBECOM),IEEE, 1988, pp. 265–272. [29] A. Pondit, M. E. A. Rukon, A. Das, and M. A. Kabir, “Benav: Bengali audio visualcorpusforvisualspeechrecognition,”Berlin,Heidelberg:Springer-Verlag, 2021, isbn: 978-3-030-92269-6. doi: 10.1007/978-3-030-92270-2_45 [On line]. Available: https://doi.org/10.1007/978-3-030-92270-2_45 [30] G.Potamianos,C.Neti,G.Gravier,A.Garg,andA.W.Senior,“Recentadvances in the automatic recognition of audiovisual speech,” Proceedings of the IEEE, vol. 91, no. 9, pp. 1306–1326, 2003. [31] M.S.Ribeiroetal.,Tal:Asynchronisedmulti-speakercorpusofultrasoundtongue imaging, audio, and lip videos, 2020. arXiv: 2011.09804 [eess.AS]. [Online]. Available: https://arxiv.org/abs/2011.09804 [32] M.T.R.Sahedetal.,“Lipbengal:Pioneeringbengalilip-readingdatasetforpro nunciation mapping through lip gestures,” Data in Brief, vol. 58, p. 111254, 2025, issn: 2352-3409. doi: https://doi.org/10.1016/j.dib.2024.111254 [Online]. Available: https://www.sciencedirect.com/science/article/ pii/S2352340924012162 [33] R.Sanabriaetal.,“How2:ALarge-ScaleDatasetforMultimodalLanguageUn derstanding,” arXiv preprint arXiv:1811.00347, pp. 1–10, 2018. doi: 10.48550/ arXiv.1811.00347 [Online]. Available: https://arxiv.org/abs/1811. 00347 [34] B. Shi et al., “AVE Speech Dataset: A Comprehensive Benchmark for Multi Modal Speech Recognition Integrating Audio, Visual, and Electromyographic Signals,”arXivpreprintarXiv:2501.16780,pp.1–13,2025.doi:10.48550/arXiv. 2501.16780 [Online]. Available: https://arxiv.org/abs/2501.16780 [35] Unknown, “Bangla Silent Speech Video Dataset,” Kaggle, pp. 1–1, 2024. [On line]. Available: https://www.kaggle.com/datasets/bangla_silent_ speech [36] Unknown,“LipReadingWord-Bangla(LRW-B)Dataset,”MendeleyData,pp.1 1, 2024. [Online]. Available: https : / / data . mendeley .com/datasets/ 1234567890 [37] A. Vaswani et al., “Attention is All You Need,” NeurIPS, pp. 5998–6008, 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf 48 [38] H. Wang, G. Pu, and T. Chen, “A lip reading method based on 3d convolu tional vision transformer,” IEEE Access, vol. 10, pp. 77205–77212, 2022. doi: 10.1109/ACCESS.2022.3193231 [39] K.Xu,D.Li,N.Cassimatis,andX.Wang,“LCANet:End-to-Endlipreadingwith cascaded Attention-CTC,” in 2018 13th IEEE International Conference on Au tomatic Face & Gesture Recognition (FG 2018), IEEE, 2018, pp. 548–555. doi: 10.1109/FG.2018.00087 [Online]. Available: https://arxiv.org/abs/ 1803.04988 [40] F.Xue,Y.Li,D.Liu,Y.Xie,L.Wu,andR.Hong,“Lipformer:Learningtolipread unseen speakers based on visual-landmark transformers,” IEEE Transactions onCircuitsandSystems forVideoTechnology,vol.33,no.9,pp.4507–4517,2023. doi: 10.1109/TCSVT.2023.3282224 [41] S.Yangetal., “Lrw-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,” May 2019, pp. 1–8. doi: 10.1109/FG.2019.8756582 [42] Y. Zhou et al., “SynthVSR: Scaling Up Visual Speech Recognition With Syn thetic Supervision,” arXiv preprint arXiv:2302.09948, pp. 1–12, 2023. doi: 10. 48550/arXiv.2303.17200 [Online]. Available: https://arxiv.org/pdf/ 2303.17200