Facial Gesture-Based Mouse Control
Date
2025-10-25
Journal Title
Journal ISSN
Volume Title
Publisher
Department of Computer Science and Engineering(CSE), Islamic University of Technology(IUT), Board Bazar, Gazipur-1704, Bangladesh
Abstract
This thesis presents a facial gesture-based mouse control system that uses webcam
based facial landmark detection to enable hands-free computer interaction. Head
movements control the cursor, intentional wink of eyes map to clicking, and con
trolled head pitch modulates scrolling. The system enhances accessibility for users
with motor impairments and supports hands-free environments such as sterile labs
or VRsetups. Built with OpenCV,MediaPipe, andPyAutoGUI,theproposedsolution
achieves real-time performance on commodity hardware and demonstrates a cost
effective, intuitive alternative for human–computer interaction.
This final report combines the initial prototype (Step I) and an enhanced implemen
tation (Step II) that introduces explicit activation/deactivation controls: a mouse acti
vation toggle (based on mouthopen/close)andascrollactivationtoggle (basedoneye
closure). We document the initial limitations, the design rationale for the enhance
ments, and a thorough evaluation including per-task metrics and latency analysis
Description
Supervised by
Prof. Dr. Hasan Mahmud,
Professor,
Department of Computer Science and Engineering (CSE)
Islamic University of Technology (IUT)
Board Bazar, Gazipur, Bangladesh
This thesis is submitted in partial fulfillment of the requirement for the degree of Bachelor of Science in Computer Science and Engineering, 2025
Keywords
Citation
[1] T.Baltruaitis,P.Robinson,andL.-P.Morency,“Openface:Anopensourcefacial behavior analysis toolkit,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 2016, pp. 1–10. doi: 10.1109/WACV.2016.7477553. [2] V.Bazarevsky, Y. Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann, “Blazeface:Sub-millisecondneuralfacedetectiononmobilegpus,”arXivpreprint arXiv:1907.05047, 2019. [Online]. Available: https://arxiv.org/abs/1907. 05047. [3] M.Betke,J. Gips, and P. Fleming, “The camera mouse: Visual tracking of body features to provide computer access for people with severe disabilities,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 10, no. 1, pp. 1–10, 2002. doi: 10.1109/TNSRE.2002.1021587. [4] A.M.Cook,J.M.Polgar,andP.Encarnação,Cook&Hussey’sAssistiveTechnolo gies: Principles and Practice, 5th ed. Elsevier, 2019, isbn: 9780323528054. [5] InternationalOrganizationforStandardization,ISO9241-9:Ergonomicrequire ments for officeworkwith visualdisplayterminals–part9:Requirementsfornon keyboard input devices, 2000. [6] R.J.K.Jacob,“Theuseofeyemovementsinhuman-computerinteractiontech niques: What you look at is what you get,” ACM Transactions on Information Systems, vol. 9, no. 3, pp. 152–169, 1991. doi: 10.1145/123078.128728. [7] Y.Kartynnik, A. Ablavatski, I. Grishchenko, and M. Kalinouski, “Real-time fa cial surface geometry from monocular video on mobile gpus,” arXiv preprint arXiv:1907.06724, 2019. [Online]. Available: https://arxiv.org/abs/1907. 06724. [8] V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1867–1874. doi: 10.1109/CVPR. 2014.241. 47 [9] D.E.King,“Dlib-ml:Amachinelearningtoolkit,”JournalofMachineLearning Research, vol. 10, pp. 1755–1758, 2009. [10] K. Krafka, A. Khosla, P. Kellnhofer, et al., “Eye tracking for everyone,” in Pro ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2176–2184. doi: 10.1109/CVPR.2016.239. [11] I. S. MacKenzie, “Fitts’ law as a research and design tool in human-computer interaction,” Human–ComputerInteraction, vol. 7, no. 1, pp. 91–139, 1992. doi: 10.1207/s15327051hci0701_3. [12] S.MitraandT.Acharya,“Gesturerecognition:Asurvey,”IEEETransactionson Systems, Man, andCybernetics, Part C (Applications and Reviews), vol. 37, no. 3, pp. 311–324, 2007. doi: 10.1109/TSMCC.2007.893280. [13] S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition for hu mancomputerinteraction:Asurvey,”ArtificialIntelligenceReview,vol.43,pp.1 54, 2015. doi: 10.1007/s10462-012-9356-9. [14] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, et al., “Real-time human pose recognition in parts from single depth images,” in Proceedings of the IEEE Con ference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 1297 1304. doi: 10.1109/CVPR.2011.5995316. [15] J. Smith, H. Lee, and R. Patel, “Assistive technology for computer access: Cur rent devices and future directions,” IEEE Engineering in Medicine and Biology Magazine, vol. 40, no. 5, pp. 90–101, 2021. doi: 10.1109/MEMB.2021.3081234. [16] X.Zhang,Y.Sugano,M.Fritz,andA.Bulling, “Appearance-based gaze estima tion in the wild,” in Proceedings of the IEEEConferenceonComputerVisionand Pattern Recognition (CVPR), 2015, pp. 4511–4520. doi: 10.1109/CVPR.2015. 7299081