Fusing acoustic and linguistic parameters for multilingual emotion recognition

Abstract

Emotion recognition from speech can be used for detection of customer problems in call centers, agent performance monitoring, improving automatic speech recognition accuracies, enhancing human robot as well as human machine interaction. In this thesis two di erent spontaneous databases are investigated in terms of binary emotion classi cation. On Turkish call center dataset (CCD) which consists of human-human dialogs, emotion recognition problem is de ned on angry and non-angry classes. On Fau Aibo dataset (FAD) which is composed of recordings of children playing with a pet robot, the negative and idle classes are considered. For extracting acoustic information we have implemented Support Vector Machines with utterance level features and Gaussian Mixture Models with frame level features. In terms of language modeling we compared word based, stem-only and stem+ending structures using manual transcriptions. Stem+ending based system resulted in the highest accuracies on CCD whereas the word based LM performed the best on FAD. This can be mainly attributed to the agglutinative nature of Turkish language. When we fused the acoustic and LM classi ers using a Multi Layer Perceptron (MLP) we could achieve 89% and 69% correct detection of both classes for CCD and FAD respectively.