Zero-Shot Evaluation of Large Language Models on Medical Query Summarization Tasks
Date
2024-07-03
Journal Title
Journal ISSN
Volume Title
Publisher
Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh
Abstract
Online health consultations are becoming increasingly popular as a way for patients to
discuss their medical health inquiries. In Bangladesh, patients are also using online health
care solutions and thus providing medical queries in Bangla language. The COVID-19
pandemic has accelerated the use of these platforms, leading to a significant influx of
questions and placing a heavy burden on the limited number of healthcare professionals available to respond. Text summarization offers a promising solution by condensing
Bangla medical queries to highlight only the essential information needed for answers.
This not only reduces the time healthcare professionals spend parsing unnecessary details
but also serves as a crucial step toward developing automated medical question-answering
systems. This research presents a comprehensive zero-shot evaluation of several state-ofthe-art Bangla and multilingual text generation models on the task of summarizing Bangla Consumer Health Questions (CHQs). The models we evaluated include BanglaT5, mT5, GPT-3.5, and GPT-4. The evaluation was conducted using ‘BanglaCHQ-Summ,’ which is
currently the only available dataset specifically designed for summarizing Bangla CHQs,
comprising 2350 pairs of questions and their corresponding summaries. The study aimed
to determine which model performs best in terms of accurately and concisely summarizing Bangla medical queries. Among the models tested, GPT-4 demonstrated superior
performance, achieving a BERTScore of 90.25%
Description
Supervised by
Dr. Md Moniruzzaman,
Assistant Professor,
Department of Computer Science and Engineering (CSE)
Islamic University of Technology (IUT)
Board Bazar, Gazipur, Bangladesh
This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Software Engineering, 2024
Keywords
NLP, LLM, ROUGE, BLEU, CHQ
Citation
[1] Q. A. R. Adib and S. B. Alam, “Bnclinical-sum: Benchmarking datasets for bangla long & short clinical dialogue summarization,” Ph.D. dissertation, Brac University, 2024. [2] S. Akter, A. S. Asa, M. P. Uddin, M. D. Hossain, S. K. Roy, and M. I. Afjal, “An extractive text summarization technique for bengali document (s) using k-means clustering algorithm,” in 2017 ieee international conference on imaging, vision & pattern recognition (icivpr), IEEE, 2017, pp. 1–6. [3] A. Ben Abacha and D. Demner-Fushman, “On the summarization of consumer health questions,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds., Flo rence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 2228–2234. DOI: 10.18653/v1/P19-1215. [Online]. Available: https://aclanthology. org/P19-1215. [4] A. Bhattacharjee, T. Hasan, W. U. Ahmad, and R. Shahriyar, “BanglaNLG and BanglaT5: Benchmarks and resources for evaluating low-resource natural language generation in Bangla,” in Findings of the Association for Computational Linguis tics: EACL 2023, A. Vlachos and I. Augenstein, Eds., Dubrovnik, Croatia: Asso ciation for Computational Linguistics, May 2023, pp. 726–735. DOI: 10.18653/ v1/2023.findings-eacl.54. [Online]. Available: https://aclanthology. org/2023.findings-eacl.54. [5] M. Gambhir and V. Gupta, “Recent automatic text summarization techniques: A survey,” Artificial Intelligence Review, vol. 47, pp. 1–66, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:24465182. [6] M. M. Haque, S. Pervin, A. Hossain, and Z. Begum, “Approaches and trends of automatic bangla text summarization: Challenges and opportunities,” IJTD, vol. 11, no. 4, pp. 1–17, 2020. DOI: 10.4018/IJTD.20201001.oa. [Online]. Available: https://doi.org/10.4018/IJTD.20201001.oa. [7] S. M. A. I. Hayat, A. Das, and M. M. Hoque, “Abstractive bengali text summa rization using transformer-based learning,” in 2023 6th International Conference 27 on Electrical Information and Communication Technology (EICT), 2023, pp. 1–6. DOI: 10.1109/EICT61409.2023.10427906. [8] Y. E. Işıkdemir, “Nlp transformers: Analysis of llms and traditional approaches for enhanced text summarization,” Eskişehir Osmangazi Üniversitesi Mühendislik ve Mimarlık Fakültesi Dergisi, vol. 32, no. 1, pp. 1140–1151, 2024. [9] M. Kabir, M. S. Islam, M. T. R. Laskar, M. T. Nayeem, M. S. Bari, and E. Hoque, “Benllmeval: A comprehensive evaluation into the potentials and pitfalls of large language models on bengali nlp,” arXiv preprint arXiv:2309.13173, 2023. [10] A. Khan, F. Kamal, M. A. Chowdhury, T. Ahmed, M. T. R. Laskar, and S. Ahmed, “BanglaCHQ-summ: An abstractive summarization dataset for medical queries in Bangla conversational speech,” in Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), F. Alam, S. Kar, S. A. Chowdhury, F. Sadeque, and R. Amin, Eds., Singapore: Association for Computational Linguistics, Dec. 2023, pp. 85–93. DOI: 10.18653/v1/2023.banglalp-1.10. [Online]. Available: https://aclanthology.org/2023.banglalp-1.10. [11] Y. Kumar, K. A. Kaur, and S. Kaur, “Study of automatic text summarization ap proaches in different languages,” Artificial Intelligence Review, vol. 54, pp. 5897– 5929, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID: 233936657. [12] M. Moradi and N. Ghadiri, “Text summarization in the biomedical domain,” ArXiv, vol. abs/1908.02285, 2019. [Online]. Available: https://api.semanticscholar. org/CorpusID:199472647. [13] M. F. Mridha, A. A. Lima, K. Nur, S. C. Das, M. Hasan, and M. M. Kabir, “A survey of automatic text summarization: Progress, process and challenges,” IEEE Access, vol. 9, pp. 156 043–156 070, 2021. DOI: 10.1109/ACCESS.2021.3129786. [14] S. R. Razu, T. Yasmin, T. B. Arif, et al., “Challenges faced by healthcare pro fessionals during the covid-19 pandemic: A qualitative inquiry from bangladesh,” Frontiers in public health, vol. 9, p. 647 315, 2021. [15] A. Sarkar and M. S. Hossen, “Automatic bangla text summarization using term fre quency and semantic similarity approach,” in 2018 21st International Conference of Computer and Information Technology (ICCIT), 2018, pp. 1–6. DOI: 10.1109/ ICCITECHN.2018.8631934. [16] A. Sarker, Y.-C. Yang, M. A. Al-garadi, and A. Abbas, “A light-weight text sum marization system for fast access to medical evidence,” Frontiers in Digital Health, vol. 2, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID: 227257712. 28 [17] M. A. I. Talukder, S. Abujar, A. K. M. Masum, F. Faisal, and S. A. Hossain, “Ben gali abstractive text summarization using sequence to sequence rnns,” in 2019 10th International Conference on Computing, Communication and Networking Tech nologies (ICCCNT), 2019, pp. 1–5. DOI: 10.1109/ICCCNT45670.2019.8944839. [18] A. Trewartha, N. Walker, H. Huo, et al., “Quantifying the advantage of domain specific pre-training on named entity recognition tasks in materials science,” Pat terns, vol. 3, p. 100 488, Apr. 2022. DOI: 10.1016/j.patter.2022.100488. [19] S. Yadav, D. Gupta, and D. Demner-Fushman, Chq-summ: A dataset for consumer healthcare question summarization, Jun. 2022. DOI: 10.48550/arXiv.2206. 06581.