A machine learning model ported into Google’s TensorFlow open-source AI framework has been paired up with the Unity game development engine to create a fully functional motion capture system out of a cheap webcam. Essentially, the AI model takes the image captured by the webcam and processes it to figure out where a person’s limbs should be at any given time. A program built in Unity, meanwhile, extrapolates that data into a 3D space to build a functional motion capture skeleton that can interact with digital elements on the screen. To put it as simply as possible, the AI program allows almost any camera to work the same magic that Microsoft’s infrared-enabled, depth-sensing Kinect camera does.
This particular demo created by programmers Or Fleisher and Dror Ayalon utilizes a Unity program built with a TensorFlow port of an existing AI program made by a different team of programmers. The original program and the dataset captured for training were created by a team of eight researchers and made into a closed demo called VNect, with its components being available for download. Fleisher and Ayalon’s port into TensorFlow and creation of a demo in Unity will in turn make the AI model and the programs to make it work vastly more accessible. While Fleisher and Ayalon did not create or release the model and program, they did make it much easier for content creators to have access to it, potentially sparking a mass expansion of cheap, readily-available augmented reality solutions.
Ayalon previously worked at a number of startups, including calling app Viber. Fleisher, meanwhile, has a good amount of augmented and virtual reality work under his belt, including an application somewhat similar to this one that projects augmented reality images of film subjects into a 3D space. A demo of that project was shown online, using footage from the Quentin Tarantino film Pulp Fiction. Projects like this one push the boundaries of what artificial intelligence and augmented reality can do together, with AI being used to analyze and extrapolate data for input into an AR environment. Fleisher has a history of projects using AI, VR, and AR, and has worked with Ayalon in the past on creative endeavors centered around the arts.