Semantic scene analysis through visual exploration and learning

Abstract

This thesis is concerned with the construction of object maps and their usage by a mobile robot endowed with a RGB pan-tilt camera. Such a representation is an important part of semantic scene analysis. However, it is a challenging task as the robot needs to be capable of object detection, scene exploration, object recognition and object mapping. We propose a series of approaches that address these problems so that the robot can construct an object map completely on its own. First, a novel approach that enables the robot to achieve both object detection and scene exploration simultaneously is proposed. In this approach, camera movements are guided by the so-far discovered object candidates. In parallel, the robot generates object candidates by tracking segments and determining spatio-temporally coherent ones. Following, an approach in which the generated object candidates are used in learning and recogni tion of object categories. In this approach, the robot has an evolving long-term object categories memory where the knowledge of learned object categories is organized in a hierarchical structure with each terminal node corresponding to an object category. The robot evolves its long-term object categories memory on its own through accu mulating the unrecognized object candidates in its working object candidates memory. It waits for its working object candidates memory to fill up and then determines the new object categories which are then added to its long-term object categories mem ory. As such, the robot is able to construct an egocentric map of objects around it. In the case of revisiting a scene, a validation method that is based on comparing the object map of the current scene with those previously constructed is presented. All the proposed approaches are experimentally evaluated using visual data obtained from a mobile robot.