MIT Researchers Present A new computer vision system turns any shiny object into a camera of sorts: Enabling an observer to see around corners or beyond obstructions

Valuable and often concealed information about one’s immediate surroundings can be gleaned from an object’s reflection. By repurposing them as cameras, one can do previously inconceivable image feats, such as looking through walls or up into the sky. This is challenging because several factors influence reflections, including the object’s geometry, the material’s qualities, the 3D environment, and the observer’s viewpoint. By internally deconstructing the object’s geometry and radiance from the specular radiance being reflected onto it, humans can derive depth and semantic clues about the occluded portions in the surroundings.

Computer vision researchers at MIT and Rice have developed a method of using reflections to produce images of the real environment. Using reflections, they transform shiny objects into “cameras,” giving the impression that the user is gazing at the world through the “lenses” of commonplace items like a ceramic coffee cup or a metallic paperweight.

The method used by the researchers involves transforming shiny objects of undetermined geometry into radiance-field cameras. The main idea is to use the object’s surface as a digital sensor to record reflected light from the surrounding environment in two dimensions.

Researchers demonstrate that novel view synthesis, the rendering of novel views that are only directly visible to the glossy object in the scene but not to the observer, is possible thanks to recovering the environment’s radiance fields. Furthermore, we can picture occluders created by nearby objects in the scene using the radiance field. The method developed by the researchers is taught from start to finish using many photographs of the object to simultaneously estimate its geometry, diffuse radiance, and the radiance field of its 5D environment.

The research aims to separate the object from its reflections so that the object may “see” the world as if it were a camera and record its surroundings. Computer vision has struggled with reflections for some time because they are a distorted 2D representation of a 3D scene whose shape is unknown.

Researchers model the object’s surface as a virtual sensor, collecting the 2D projection of the 5D environment radiance field around the object to create a 3D representation of the world as the thing sees it. Most of the environment’s radiance field is obscured except via the object’s reflections. Beyond field-of-view, novel-view synthesis, or the rendering of novel views that are only directly visible to the glossy object in the scene but not to the observer, is made possible by the use of environment radiance fields, which also allow for depth and radiance estimation from the object to its surroundings.

In summing up, the team did the following:

• They demonstrate how implicit surfaces can be transformed into virtual sensors with the ability to capture 3D images of their environments using only virtual cones.
• Together, they calculate the object’s 5D ambient radiance field and estimate its diffuse radiance.
• They demonstrate how to use the light field of the surrounding environment to generate fresh viewpoints invisible to the human eye.

This project aims to reconstruct the 5D radiance field of the surroundings from many photographs of a shiny item whose shape and albedo are unknown. Glare from reflective surfaces reveals scene elements outside the frame of view. Specifically, the surface normals and curvature of the glossy object determine how the observer’s images are mapped onto the real world.

Researchers may need more accurate information on the object’s shape or the reflected reality, contributing to the distortion. It’s also possible that the glossy object’s color and texture will blend in with the reflections. Furthermore, it isn’t easy to discern depth in reflected scenes since reflections are two-dimensional projections of a three-dimensional environment.

The team of researchers overcame these obstacles. They begin by photographing the shiny object from various angles, catching a variety of reflections. Orca (Objects such as Radiance-Field Cameras) is the acronym for their three-stage process.

Orca can record multiview reflections by imaging the object from various angles, which are then used to estimate the depth between the glossy object and other objects in the scene and the shape of the glossy object itself. More information about the strength and direction of light rays coming from and hitting each point in the image is captured by ORCa’s 5D radiance field model. Orca can make more precise depth estimates thanks to the data in this 5D radiance field. Because the scene is displayed as a 5D radiance field rather than a 2D image, the user can see details that corners or other obstacles would otherwise obscure. Researchers explain that once ORCa has collected the 5D radiance field, the user can position a virtual camera wherever in the area and generate the synthetic image the camera would produce. The user might also alter the appearance of an item, say from ceramic to metallic, or incorporate virtual things into the scene.

By expanding the definition of the radiance field beyond the traditional direct-line-of-sight radiance field, researchers can open new avenues of inquiry into the environment and the objects inside it. Using projected virtual views and depth, the work can open up possibilities in virtual item insertion and 3D perception, such as extrapolating information from outside the camera’s field of vision.

Check out the Paper and Project Page. Don’t forget to join our 22k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at [email protected]

🚀 Check Out 100’s AI Tools in AI Tools Club