Time-Lapse Light Field Photography for Perceiving Non-Lambertian Scenes

Authors: Stefanie Tellex, John Oberlin

Robust robotic perception and manipulation of household objects requires theability to detect, localize and manipulate a wide variety of objects, which maybe mirror reflective like polished metal or shiny like smooth plastic(non-Lambertian); for example picking a metal fork out of a sink full ofrunning water or screwing a metal nut onto a bolt. Existing perceptualapproaches based on photographs only take into account the average intensity oflight at each pixel, discarding information about the angle from which thatlight approached, which limits their ability to account for non-Lambertianobjects and scenes. To address this problem, we demonstrate time-lapse lightfield photography with an eye-in-hand camera of a manipulator robot. Aneye-in-hand robot can capture both the intensity of rays, as in a conventionalphotograph, as well as the direction of the rays. We present a formal modelfor robotic light-field photography that fits into a probabilistic roboticsframework. Using this model, we perform 3D reconstruction with a monocularcamera by finding approximate maximum-likelihood estimates, corresponding to asoftware lens with a very wide baseline. This information can be used todetect, localize and manipulate non-Lambertian objects in non-Lambertianscenes: our approach enables Baxter robot to pick a shiny metal fork out of asink filled with running water 24/25 times, as well as to localize objects wellenough to screw a nut onto a quarter inch bolt. The techniques in this paperpoint the way toward new approaches to robotic perception that leverage arobot’s ability to move its camera to infer the state of the external world.