From: 2024-05-31 09:00 to 10:00
Place: Seminar Room M 3170-73 at Dept. of Automatic Control, LTH
Contact: bo [dot] bernhardsson [at] control [dot] lth [dot] se

Date & Time: May 31st, 09:00 - 10:00
Location: Seminar Room M 3170-73 at Dept. of Automatic Control, LTH 
Author: Alessandro Celoria and Valentín López
Title: An ASR-based Hybrid Approach for Auditory Attention Decoding
Supervisor: Bo Bernhardsson, Emina Alickovic and Martin Skoglund. 
Examiner: Pontus Giselsson

Abstract: Auditory Attention Decoding (AAD) aims to determine the focus of a listener's attention in scenarios with multiple overlapping speakers, known as the "Cocktail Party Problem." This thesis investigates AAD using Whisper, a transformer-based Automatic Speech Recognition (ASR) system capable of extracting linguistic and semantic information from its latent encoder layers. Two AAD methodologies are proposed: first, a forward pipeline that utilizes Whisper for pre-processing audio stimuli in conjunction with a Temporal Response Function (TRF) model for predicting Electroencephalography (EEG) responses. Second, a hybrid approach aims to identify the speech source by applying Canonical Correlation Analysis (CCA) and its neural network variant, Deep Canonical Correlation Analysis (DCCA), to Whisper's latent encoder layers and EEG signals. The performance of these methodologies is compared across fixed decision window lengths, assessing selective attention capabilities with limited information to highlight Whisper's enhanced performance when combined with CCA methods. Additionally, an electrode-wise analysis is performed, restricting the number of electrodes to specific cerebral cortex regions, as a step towards the development of neurosteered hearing aid devices.