Apple Launches SHARP a Fast 2D Photograph to 3D Scene Model
Apple has published its findings and released a new model, SHARP, that can turn a single 2D photo into a photorealistic 3D scene in under a second, allowing the user to tilt their viewing angle in the generated scene with near-real-time restoration of depth, scale, and distance.
Working of SHARP Technology
It works using a 3D Gaussian method, which equates to millions of small points of colored lights in space. In contrast to conventions that require many images of a subject from multiple vantage points, Apple's neural network predicts the entire 3D scene in one shot using one image, thus eliminating the traditionally long optimization steps.
This ability comes from training the model on a large synthetic-to-real dataset. By this training, SHARP has learned common depth and geometrical patterns, which allows it to first estimate the distances in a new photo and then quickly build up a plausible 3D environment.
Performance and Limitations
Apple report on the orders of magnitude from quality improvement in the 3D generation in contrast to prior methods, with a concomitant reduction in time to generate the scene. The model does compromise, though.
- Works well with images that are quite close.
- It can't create or add-on features that were not visible in the original 2D photo.
Availability and Community Use
SHARP is now available to developers and researchers on GitHub. Users have started to experiment with the tech and apply it on very diverse images. Examples online include the conversion of personal photos, scenery, racing cars, and even an image of the surface of Venus into interactive 3D scenes.
