Appearance-based cognition of objects pointed out by human hands

Abstract

In this thesis, human-guided appearance-based object cognition in robots is addressed. Here, the robot observes the human pointing to the object of interest based on the incoming RGB-D data and then either recognizes or learns this object as needed. This problem is of interest because the associated learning problem does not require objects and their labels to be provided externally as is the case with supervised learning or learned objects can be more human-intuitive since the robot is not completely on its own as is the case with unsupervised learning. We propose a complete end-to-end system consisting of three stages: First, the robot determines the pointing direction. For this, it first finds hands and humans in the in coming RGB image via exploiting a state-of-the-art CNN-based detector. Following, it finds the point cloud object corresponding to the hand segment through applying a density-based segmentation algorithm on the RGB-D data and then estimates the 3D pointing direction vector from the implicit geometry of the 3D hand segment. We also introduce a RGB-D data set with varied robot-human distances and pointing gesture directions - due to the unavail ability of such a data set. In the second stage, the robot determines the targeted object based on the 3D pointing direction. For this, it determines a set of candidate point cloud objects and then selects the object that is most likely to be targeted. The final stage is either to recog nize the target object or to learn it as necessary. In this, its objects’ memory that is organized hierarchically plays a key role. In the latter case, the new object class is added to the memory using an unsupervised learning algorithm. To the best of our knowledge, the proposed sys tem is the first end-to-end system in which the robot’s reasoning is completely autonomous. The advantages of the proposed approach are as follows: i) Applicability in a wide-range of robot-human interactions regardless of human proximity and background variability; ii) Ability to continue learning new object classes through interaction with humans.