Entropy-tuned Confidence Score Ensembles for Multimodal Sentiment Analysis

Hiemstra, Tim

Entropy-tuned Confidence Score Ensembles for Multimodal Sentiment Analysis

Files

Hiemstra, T. s-1020403-MSc-MKI92-Thesis-2024.pdf (655.65 KB)

Authors

Hiemstra, Tim

Issue Date

2024-10-31

Language

en

URI

https://theses.ubn.ru.nl/handle/123456789/18907

Abstract

This thesis investigates the use of confidence score ensembles to enhance multimodal sentiment analysis (MSA), with a focus on spoken reviews and video blogs. Specifically, it examines two main research questions: (1) Do confidence score ensembles improve MSA performance? and (2) Does entropy tuning enhance the performance of these ensembles? To explore these questions, pre-trained models for audio (wav2vec2.0 XLS R) and text (DeBERTaV3) were fine-tuned on two datasets: MELD and CMU-MOSI. We employ various strategies for combining confidence scores, including selecting the highest confidence and applying weighted averages based on equal weights, F1 scores, and a reciprocal formula emphasizing higher F1 scores. To enhance ensemble performance, entropy tuning was applied using three similarity measures: minimizing mean difference (MD), percentile differences (PD), and Kolmogorov Smirnov distance (KSD). We do not find significant evidence for either research question. Our findings show that, while confidence score ensembles improved F1 scores on MELD in 8/16 cases, their performance on CMU-MOSI was more limited, with only 1/16 showing improvement. Entropy tuning showed potential, particularly on CMU-MOSI, with 2/3 methods producing more consistent results than untuned ensembles. However, performance differences between modalities and data complexity may have hindered greater improvements. Overall, this thesis highlights the potential of entropy tuning in ensemble methods, particularly for future advancements in MSA applications. With larger ensembles and diverse datasets, entropy tuning could offer significant im provements in model performance and further refine multimodal sentiment analysis approaches.