EEG and eye tracking feature contextualization based affective computing through LightGBM enabled stacked ensembling of LLMs

Abstract

Affective Computing plays a crucial role in analyzing and interpreting human emotional states. Electroencephalography (EEG) signals is widely regarded as a reliable modality due to their direct measurement of brain activity. However, traditional deep learning models used in EEG-based affective computing face vital challenges, such as high inter-subject variability, reliance on extensive feature engineering, and difficulties in integrating multimodal information such as eye-tracking data. To address these limitations, a novel preprocessing pipeline was developed in this study which encodes high-dimensional EEG features along with the eye-tracking features into structured contextualized sequences, making them compatible with Large Language Model (LLM) architectures. Additionally, an ensemble learning framework was incorporated to effectively integrate these heterogeneous modalities. This research explores the potential of LLMs in affective computing which have demonstrated their ability to process structured text representations utilizing their pretraining on diverse and extensive corpora. This adaptability was harnessed by transforming EEG and eye-tracking data into structured and contextualized string representations. The experimental results demonstrate the effectiveness of our approach, in which FLAN-T5, GPT-2, and BERT achieved classification accuracies of 86%, 92%, and 90%, respectively on the EEG dataset. Furthermore, the integration of EEG and eye-tracking data using a Stacked Ensemble method with LightGBM as the meta-model led to a significant improvement, achieving a final classification accuracy of 96% which indicates the benefits of combining different types of data for affective computing.