Human - computer interaction platform for the hearing impaired in healthcare and finance applications

Abstract

In this thesis, we propose a human-computer interaction platform for the hearing impaired, that would be used in hospitals and banks. In order to develop such a system, we collected BosphorusSign, a Turkish Sign Language corpus in health and nance domains, by consulting sign language linguists, native users and domain specialists. Using a subset of the collected corpus, we have designed a prototype system, which we called HospiSign, that is aimed to help the Deaf in their hospital visits. The HospiSign platform guides its users through a tree-based activity diagram by asking speci c questions and requiring the users to answer from the given options. In order to recognize signs that are given as answers to the interaction platform, we proposed using hand position, hand shape, hand movement and upper body pose features to represent signs. To model the temporal aspect of the signs we used Dynamic Time Warping and Temporal Templates. The classi cation of the signs are done using k- Nearest Neighbors and Random Decision Forest classi ers. We conducted experiments on a subset of BosphorusSign and evaluated the e ectiveness of the system in terms of features, temporal modeling techniques and classi cation methods. In our experiments, the combination of hand position and hand movement features yielded the highest recognition performance while both of the temporal modeling and classi cation methods gave competitive results. Moreover, we investigated the e ects of using a tree-based activity diagram and found the approach to not only increase the recognition performance, but also ease the adaptation of the users to the system. Furthermore, we investigated domain adaptation and facial landmark localization techniques and examined their applicability to the gesture and sign language recognition tasks.