Using AR to Define Manipulation Contact Tasks from Single Demonstrations

We use an AR headset with hand tracking to extract the affordance of a manipulation demonstration. The object affordance is modeled as an affordance primitive. We use AugRE to connect and localize the users with mobile manipulators and TeMoto to manage the robot actions to navigate to, plan, and  execute. This system does not limit the robot to just imitating the users demonstration, rather the robot is able to manipulate the object however a user wants (e.g. clockwise, counterclockwise, 30 degrees, 300 degrees, etc.) with just one demonstration from the user.

A corresponding paper was submitted for review and acceptance into the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

Music by
Artist: Droid Bishop
Track: In Search of Gemini