Neural Task Programming, Learning to Generalize Across Hierarchical Tasks
[ ]Feifei Li has published their work on how to have the robot learn to pick and place the objects by itself. The paper is available at arxiv.
The abstract: — In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction. NTP takes as input a task specification (e.g., video demonstration of a task) and recursively decomposes it into finer sub-task specifications. These specifications are fed to a hierarchical neural program, where bottomlevel programs are callable subroutines that interact with the environment. We validate our method in three robot manipulation tasks. NTP achieves strong generalization across sequential tasks that exhibit hierarchal and compositional structures. The experimental results show that NTP learns to generalize well towards unseen tasks with increasing lengths, variable topologies, and changing objectives. stanfordvl.github.io/ntp/
Fig. 1: (top) At test time, NTP instantiates a task-conditional policy (a neural program) that performs the specified task by interpreting a demonstration of a task. The policy interacts with the environment through robot APIs. (bottom) We evaluate NTP on Block Stacking (A,B), Object Sorting (C, D) and Table Clean-up (Figure 8) tasks in both simulated and real environment.
Fig. 2: Neural Task Programming (NTP): Given an input program, a task specification, and the current environment observation, a NTP model predicts the sub-level program to run, the subsequence of the task specification that the sub-level program should take as input, and if the current program should stop.
Detailed network.