Indoor visual understanding with RGB-D images using deep neural network

Abstract

We created a visual understanding pipeline for house robots with Red-Green-Blue Depth (RGB-D) sensors. Our pipeline consists of three components; (i) an RGB-D object detection network,(ii)asimpletrackingalgorithm,and(iii)a3D semantic mapping module. Our constraint is running the whole system in real-time. Most RGB-D object detection networksdonothavesuchconstraintandcannotruninreal-timebecausetheyrequirecostly preprocessing methods. Instead of the costly methods, we seek ways to make raw depth data more useful by a cheap preprocessing technique of RGB overlaying on the depth data. After adding depth branches into two state-of-the-art RGB object detection networks we compared performances of feeding raw depth input and RGB overlaid depth input on the SUN RGB-D dataset. The results of using only one type of input show that our overlaying method gets 0.9% better mean average precision (mAP) than feeding the network with raw depth data. SSD with decision level fused depth network increased the mAP around 2% compared to RGB only SSD. We collected a small household object detection dataset to test thetrackingmethodcombinedwiththeobjectdetector. Weusedmedianflowtrackingonthe objectboxesdetectedbytheobjectdetector,whichincreasedthemAPoftheobjectdetection network by around 5% on our dataset. Our final module consists of 3D semantic mapping which we used Robot Operating System node of the Real-time Appearance-Based Mapping library for the 3D mapping part. The semantic information of the objects are created by the object detector network combined with the tracker. Although the object detector network proposes 2D bounding boxes around the objects, labeling these pixels and seeing the object from different angles allow us to create 3D maps with object labels.