Cooperative sign language tutoring: a multiagent approach

Abstract

Sign language is the natural means of communication for the hearing-impaired. Sign languages are based on signs, which are a combination of hand gestures, facial expressions, and head movements. Teaching these visual languages to others is an important, but a di cult task. Sign languages can be learned e ectively only with frequent feedback from an expert in the eld. The expert needs to watch a performed sign, and decide whether the sign has been performed well based on her previous knowledge about the sign. The experts role can be imitated by an automatic system, which uses a training set as its knowledgebase to train a classi er that can decide whether the performed sign is correct. However, when the system does not have enough previous knowledge about a given sign, the decision will not be accurate. Accordingly, we propose a multiagent architecture in which agents represent sign language experts, and they cooperate with each other to decide on the correct classi- cation of performed signs. We apply di erent cooperation strategies and test their performances in varying environments. We further study the robustness of our strategies. Our results also show that the best performing strategy is our proposal, the Bayesian modeling strategy. Further, through analysis of the multiagent system, we discover inherent properties of sign languages, such as the existence of dialects.