Interactive (and Collaborative) Robot Programming using Language
Programming robots to carry out desired tasks is difficult and time-consuming. This PhD project focuses on collaborative and instructional dialogue agents to help human operators program robot tasks.
In this collaborative scenario, a human operator converses with an AI agent to explain the steps that are to be performed, using high-level references and abstractions that make sense to the human, as opposed to simple verbal instructions corresponding to rudimentary robot movements. The AI agent must interpret the high-level instructions and translate these into a robot program based on advanced AI path planning and goal planning capabilities. Importantly, the human operator’s perception of the shared environment may differ from that of the AI’s sensors, leading to scenarios where instructions may be ambiguous, and their disambiguation requires a collaborative dialogue.
The PhD candidate will research and develop methods to situate a collaborative dialogue agent, focusing on the core research question of how to tackle ambiguities in instruction-to-code translation within a grounded robotics scenario. The project will lead to the development of a system – tools, algorithms and data – that allows a human operator to interactively program a robot using language. The research will lead to a better understanding of the design space in human-robot collaboration and will generate an effective example of collaborative intelligence based on both human and robot capabilities. The PhD candidate will be expected to publish this research in relevant high-quality peer-reviewed journals and conferences.
The project will be part of the Data61’s collaborative intelligence (CINTEL) Future Science Platform (FSP) that will develop the science that enables human intelligence and technology to work together across multiple domains, driving sustainable productivity growth and improving both the quantity and quality of jobs for human workers.
Associated Researchers