Google is working on one of the main problems with sharing VR experiences through mixed reality videos – the obscuring of users’ faces. Seeing the upper half of a user’s face that the visor obscures can be very helpful in making a mixed reality sharing experience more engaging, but even if present, the cameras inside a VR headset do not always produce an optimal image. Google’s latest project captures a VR user’s face beforehand, maps and models their gaze, and uses the model and the information about the VR experience at hand to create a transparent overlay of a VR headset.
Google uses special tracking techniques to achieve this effect. For starters, the VR headset used by the company has a special marker that makes it easier to track on video. The real-time synchronization that is required to create any mixed reality video is also utilized in order to help determine where a user should be looking and to accurately and seamlessly insert the missing parts of their face where they should be. This example uses an HTC Vive unit modified with eye tracking hardware, which allows Google’s engineers to know in which direction to point the composite image of the user’s eyes at the right moment.
Of course, Google’s model is far from perfect; without an eye-tracking headset, it cannot accurately show where a user is looking. It also cannot show elements of facial expression happening inside the headset in real time and requires a complicated setup procedure. All of this means that such technology won’t be available to the masses for some time, and when it is, the results may not be all that great without special hardware and software considerations. Google’s attempt is commendable, however, and creates a fairly lifelike effect. Detecting and showing the actual face instead of using a recreation would require sophisticated cameras installed on the inside of a VR headset, which is impractical and can cause issues with both the user experience and the produced video. Still, should the technology take off, things like VR gaming, social VR applications, and enterprise VR applications all stand to benefit.
