Query-by-sing sistem for Turkish sign language broadcasts

Abstract

Sign based query search system is a specialized type of query by example search system. The main objective of this study is to locate visual query sign from large video dataset. This study is a baseline of a query by sign search system for deaf and mute people to make their access to audiovisual video easy. Hand gesture recognition and retrieval is an open problem and there is no exact solution. In this thesis, our approach proposes a method uses one of the latest successful method OpenPose and subsequeunce dynamic time warping-based retrieval task. We use Turkish Sign Language Broadcast as a dataset and it is processed offline using OpenPose. Hand signs in the dataset and query signs are represented by feature vectors. Feature vectors are a combination of positions of finger configurations, unit motion vector and appearance based shape and texture characteristics are which are used to calculate similarity for query match ing. Cosine metric is used to measure distance to analyze similarity between searched query and all subsequences in the dataset. Different sized segmented windows from dataset are used to compare retrieval performance. Experimental results indicate that the proposed method is promising for further studies in query-by-sign search systems. Moreover DTW is combined with and windowing approach improves the performance. Performance of the system is measured by precision at 10 calculation. Number of successful retrievals from top 10 result give us the performance of the system.