Object and relation-centric reasoning of action effects in push manipulation tasks

Abstract

In complex robotic systems, prediction of effects is a challenging problem when the number of objects varies, especially in the presence of rich and various interactions among these objects. To be able to model such systems, the representation of data should be able to su ciently encode multiple objects and interactions between them. In this thesis, we rst show our initial research on e ect prediction on objects with various shapes. Then we propose a Graph Neural Network based framework, Belief Regulated Dual Propagation Networks (BRDPN), a general-purpose learnable physics engine. Our framework consists of two complementary components, a physics predictor and a belief regulator. While the former predicts the future states of the object(s) manipulated by the robot, the latter constantly corrects the robot's knowledge regarding the objects and their relations. Through our experiments, in complex environments consisting of di erent shaped objects and articulation types we have shown that by using this framework, the robot can reliably predict the consequences of its actions in object trajectory level and exploit its own interaction experience to correct its belief about the state of the environment. Furthermore, we have shown that we can use this framework in tool manipulation and planning.