Constructin semantic place representations via object discovery and visual exploration

Abstract

The goal of this thesis is to develop a Semantic Place Model (SPM) using a mobile robot with a pan-tilt camera. Here, a semantic place model is de ned to be an egocentric representation of objects and their spatial relations in the S2-space. The construction of SPM consists of two parts. Firstly, the object discovery and visual exploration are used in a coupled manner. The visual exploration is driven by the object discovery via constructing a family of arti cial potential function using the properties of object candidates. The object candidates are determined via spatiotemporally coherent segments based on a previous work on semantic place recognition in our laboratory. The bene t of this method is that it enables robot to generate its own video input which is crucial for autonomous mobile robots. In the second part, we propose a novel approach to constructing semantic place models. First, object candidates are assigned labels via employing a convolutional neural network based recognition system. The robot then uses the object labels along with their spatial relations to construct the semantic scene model as an attributed graph. It then merges the constructed scene model with its existing semantic place model in order to combine object information from di erent viewing directions and locations to get a complete representation of a place. In this way, the robot is able to cover a larger area. The resulting models are evaluated in a series of on-robot experiments.