DeepMind AI Can Make 3D Models, Scenes From Pictures

Google-owned DeepMind has created an AI program that’s able to generate simple 3D objects and scenes after being fed nothing more than a two-dimensional picture of those objects and scenes, or in some cases a few pictures from different angles, if the object or scene is a bit more complex. Thus far, it’s been able to render low-polygon 3D objects with multiple faces, along with mazes and scenery populated by things like spheres and tubes. The technology is a type of computer vision that’s based on neural networking, and is able to build on prior knowledge in order to make and compile complex observations about things around it.

Called the Generative Query Network, this new type of AI program is made to be spatially aware by way of general training ahead of time. It uses that knowledge to build expectations about where objects may be in a 3D plane based on where those objects are in a picture, and where other objects are in relation to them. The AI can be fed an isometric picture of a scene, and then generate a 3D equivalent, including modelling all the objects in the scene by drawing on its knowledge to fill in the gaps for unseen sides of objects, or things that may be behind walls or around corners in some cases. It can also take in top-down 2D representations of scenes and put out a 3D maze or environment. The AI’s building knowledge and basic expectations mean that it does not need a human to label objects in the environment.

This technology is extremely limited for now, but it holds serious potential. It could be used to build 3D maps for navigation, for example, or could be used to make 3D scans of objects and translate any object in a picture into a 3D model, for example. Naturally, i could also be beneficial for self-driving cars. The tech is not quite there at this stage, and will require more time to build before it can be used for anything practical. If DeepMind’s previous exploits are any indication, this technology will continue to evolve, and will eventually find its way into Google products in some form.