Bidirectional multi-step prediction with affordances

Abstract

Affordances are action possibilities of an object, directly perceived by agents based on their capabilities. Affordances are learned from goal-free exploration of the agent’s capabilities through observing the effects of their actions on objects in an en vironment. The agent can then use the learned affordances to make plans to reach a goal since the agent knows which actions on a certain object are possible and which action results in the desired effect. The affordance principle is also followed in robotics to learn to distinguish which actions in the repertoire of a robot are applicable to an object in its environment. This information can then be utilized in goal- directed plan ning, either directly or with the aim of reducing the search space for possible solutions. In this work, the problem of making multi-step predictions for object manipulation is investigated in the continuous domain. Several types of actions are defined in a robot’s repertoire, and the interactions of the robot with a number of objects possessing differ ing qualities in a tabletop setting are recorded. Relative distance quantities are used for representing actions and effects which allow generalizability, alongside a top-down centered depth image of the object. This data is used to train a model which can be conditioned on actions to predict the effects, conditioned on effects to predict the applied actions, or conditioned on both to predict the actions and effects. By using a planner on top of this model, the capacity to chain together a correct sequence of actions for an object to reach the desired goal position is achieved. The model is ver ified in experiments, generating and executing reasonable plans efficiently. Setting it apart from previous work, using continuous effect and actions enables the planner to find solutions to configurations that were not observed in training using partial action executions.