Leveraging LLMs for Coverage Analysis from High to Low-Level Requirements
Date
2025-10-25
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh
Abstract
The increasing capabilities of Large Language Models (LLMs) have opened new av
enues for automating critical tasks in software and systems engineering, particularly
in the generation and analysis of requirements. This thesis investigates the poten
tial of LLMs to generate Low-Level Requirements (LLRs) from existing High-Level
Requirements (HLRs), aiming to evaluate how effectively these models can replicate
human-derived requirements. The study employs multiple state-of-the-art LLMs to
generateLLRsusingvariousprompting strategies,followedbyasystematicevaluation
of coverage and traceability through expert validation and alignment with established
academic criteria.
The findings of this research provide insights into the strengths and limitations of
LLMs in capturing detailed, actionable system specifications. While LLMs demon
strate significant potential for automating certain aspects of requirements engineer
ing, humanexpertiseremainsessentialforensuringcompleteness, accuracy, andcon
textual relevance. By highlighting areas where automation can enhance efficiency
and where human oversight is necessary, this work contributes to a deeper under
standing of LLM capabilities in requirements engineering and offers practical recom
mendations for integratingthesemodels intoprofessionalpracticetoimproverequire
ment traceability, consistency, and quality.
Description
Supervised by
Mr. Shohel Ahmed,
Assistant Professor,
Department of Computer Science and Engineering (CSE)
Islamic University of Technology (IUT)
Board Bazar, Gazipur, Bangladesh
This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Software Engineering, 2025
Keywords
Citation
[1] AI@Meta,“Llama3modelcard,” AI@Meta, 2024. [Online]. Available: https: //github.com/meta-llama/llama3/blob/main/MODEL_CARD.md [2] A. Askell et al., “A general language assistant as a laboratory for alignment,” arXiv preprint arXiv:2112.00861, 2021. [3] J. Cleland-Huang, A. Czauderna, M. Gibiec, and J. Emenecker, “A machine learningapproachfor tracingregulatorycodestoproductspecificrequirements,” in Proceedings of the 32nd ACM/IEEE International Conference on Software En gineering- Volume 1, ser. ICSE ’10, Cape Town, South Africa: Association for ComputingMachinery,2010,pp.155–164,isbn:9781605587196.doi:10.1145/ 1806799.1806825 [Online]. Available: https://doi.org/10.1145/1806799. 1806825 [4] J. Cleland-Huang, O. Gotel, J. H. Hayes, P. Mäder, and A. Zisman, “Software traceability: Trends and future directions,” Future of Software Engineering Pro ceedings, 2014. [Online]. Available: https://api.semanticscholar.org/ CorpusID:7224437 [5] J. Cleland-Huang, A. Zisman, and O. Gotel, Software and Systems Traceability. DePaule University, Oct. 2012, pp. 1–491, isbn: 978-1-4471-2238-8. doi: 10. 1007/978-1-4471-2239-5 [6] C.Community, Center of excellence for software & systems traceability, [Online; accessed 27-April-2025]. [Online]. Available: http://coest.org/ [7] DeepSeek-AIetal.,Deepseekllm:Scalingopen-sourcelanguagemodelswithlongter mism,2024.arXiv:2401.02954[cs.CL].[Online].Available:https://arxiv. org/abs/2401.02954 45 [8] C. Dorn et al., “Supporting quality assurance with automated process-centric quality constraints checking,” in IEEE/ACM 43rd International Conference on SoftwareEngineering(ICSE),May2021,pp.1298–1310.doi:10.1109/ICSE43902. 2021.00118 [9] S. Ezzini, S. Abualhaija, C. Arora, and M. Sabetzadeh, “Ai-based question an sweringassistanceforanalyzingnatural-languagerequirements,”inIEEE/ACM 45thInternationalConferenceonSoftwareEngineering(ICSE),May2023,pp.1277 1289. doi: 10.1109/ICSE48619.2023.00113 [10] D. Falessi, J. Roll, J. Guo, and J. Cleland-Huang, “Leveraging historical asso ciations between requirements and source code to identify impacted classes,” IEEETransactionsonSoftwareEngineering,vol.46,pp.420–441,2018.[Online]. Available: https://api.semanticscholar.org/CorpusID:52047122 [11] A. Fan et al., “Large language models for software engineering: Survey and open problems,” in IEEE/ACM International Conference on Software Engineer ing, May 2023, pp. 31–53. doi: 10.1109/ICSE-FoSE59343.2023.00008 [12] O. Gotel et al., Traceability Fundamentals, I. Cleland-Huang, G. J., O., and A. Zisman, Eds. Software and Systems Traceability, Springer, Berlin, 2012, pp. 3 22. [13] O.GotelandA.C.W.Finkelstein,“Ananalysisoftherequirementstraceability problem,” Proceedings of IEEE International Conference on Requirements Engi neering,pp.94–101,1994.[Online].Available:https://api.semanticscholar. org/CorpusID:5870868 [14] J. H. Hayes, A. Dekhtyar, and S. K. Sundaram, “Advancing candidate link gen eration for requirements tracing: The study of methods,” IEEE Trans. Software Eng., vol. 32, no. 1, pp. 4–19, 2006. doi: 10.1109/TSE.2006.3 [Online]. Avail able: http://dx.doi.org/10.1109/TSE.2006.3 [15] L.W.HayesJaneHuffmanDekhtyarAlex, “Astudy of methods for textual sat isfaction assessment,” 10.1007/s10664-012-9198-8, 2013. doi: https://doi. org/10.1007/s10664-012-9198-8 46 [16] E.Holbrook,J.Hayes,andA.Dekhtyar,“Towardautomatingrequirementssat isfaction assessment,” in RequirementsEngineeringConference,2009, Oct. 2009, pp. 149–158. doi: 10.1109/RE.2009.10 [17] G.Inc., Gemini2.0 flash, [Online; accessed 27-April-2025]. [Online]. Available: https://cloud.google.com/vertex-ai/generative-ai/docs/models/ gemini/2-0-flash [18] M.Kassab, C. Neill, and P. Laplante, State of practice in requirements engineer ing: Contemporary data- innovations in systems and software engineering, Apr. 2014. [Online]. Available: https://link.springer.com/article/10.1007/ s11334-014-0232-4#citeas [19] T. Kojima, S. ( Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language mod els are zero-shot reasoners,” in Advances in Neural Information Processing Sys tems, S. Koyejo, S. Mohamed,A.Agarwal,D.Belgrave, K.Cho,andA.Oh,Eds., vol. 35, Curran Associates, Inc., 2022, pp. 22199–22213. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/ 8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf [20] B. Larson, J. Hatcliff, and P. Chalin, “Open source patient-controlled analgesic pump requirements documentation,” in Software Engineering in Health Care (SEHC), 2013 5th International Workshop, May 2013, pp. 28–34. doi: 10.1109/ SEHC.2013.6602474 [21] D. Luitel, S. Hassani, and M. Sabetzadeh, “Improving requirements complete ness: Automated assistance through large language models,” Requirements En gineering, vol. 29, pp. 1–23, Mar. 2024. doi: 10.1007/s00766-024-00416-3 [22] F. A. C. Pinheiro, “Requirements traceability,” in Perspectives on Software Re quirements, J. C. S. do Prado Leite and J. H. Doorn, Eds. Boston, MA: Springer US, 2004, pp. 91–113, isbn: 978-1-4615-0465-8. doi: 10.1007/978-1-4615 0465-8_5 [Online]. Available: https://doi.org/10.1007/978-1-4615 0465-8_5 [23] A.-R.Preda,C.Mayr-Dorn,A.Mashkoor,andA.Egyed,“Supportinghigh-level to low-level requirements coverage reviewing with large language models,” in Proceedings ofthe21stInternationalConferenceonMiningSoftwareRepositories, ser. MSR ’24, Lisbon, Portugal: Association for Computing Machinery, 2024, 47 pp. 242–253, isbn: 9798400705878. doi: 10.1145/3643991.3644922 [Online]. Available: https://doi.org/10.1145/3643991.3644922 [24] A. Rodriguez, K. Dearstyne, and J. Cleland-Huang, “Prompts matter: Insights and strategies for prompt engineering in automated software traceability,” Jul. 2023. doi: 10.48550/arXiv.2308.00229 [25] Y.ShinandJ.Cleland-Huang, “A comparative evaluation of two user feedback techniques for requirements trace retrieval,” in Proceedings of the 27th Annual ACMSymposiumonAppliedComputing,ser.SAC’12,Trento,Italy:Association for ComputingMachinery,2012,pp.1069–1074,isbn:9781450308571.doi:10. 1145/2245276.2231943 [Online]. Available: https://doi.org/10.1145/ 2245276.2231943 [26] B.Wei,“Requirementsareallyouneed:Fromrequirementstocodewithllms,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE), 2024, pp. 416–422. doi: 10.1109/RE59067.2024.00049 [27] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Infor mation Processing Systems, ser. NIPS ’22, New Orleans, LA, USA: Curran Asso ciates Inc., 2022, isbn: 9781713871088. [28] L. Zhao et al., “Natural language processing for requirements engineering: A systematic mapping study,” ACM Comput. Surv., vol. 54, no. 3, Apr. 2021, issn: 0360-0300. doi: 10.1145/3444689 [Online]. Available: https://doi.org/10. 1145/3444689