360 degree video, like any other VR medium, suffers from its fair share of flaws. A major one is VR sickness, a malady that just about every form of VR media has on at least some level. With 360 degree video, not only does the user control the experience, but the video moves as it pleases, which can be at odds with the way that the viewer is looking. One of the biggest reasons for this cross-contamination, so to speak, is shaking. Human hands shoot shaky video; there’s no way around that. The same can be said of human-mounted 360 degree cameras. Aside from professional fare on tripods, just about every VR-friendly video out there will be quite shaky, making those who tend toward VR sickness a bit queasy. Facebook is looking to fix that.
Their engineers are working on a special set of algorithms that can be applied to just about any 360-degree video and fix a number of issues, but the biggest one is shakiness. A shaky 360 degree video is extrapolated into a cube map to make motion tracking a bit easier. Once Facebook’s algorithm has figured out what in the video is moving because it’s supposed to be and what is moving because of a shaky camera, it’s put through a special “deformed rotation” model to make erratically moving objects sync up with the background a bit more, then the whole thing is systematically stabilized. The entire process takes about 20 milliseconds for a single frame in most videos in 1920 x 1080 resolution. Since frames are on screen for 30 milliseconds, the processing could be done in real time, with a bit of tweaking and a slight delay on the video.
Once the video is stabilized and culled down to a decent working resolution, a number of things can be done to it that would either not work or come out badly with a non-stabilized 360-degree video. One of these things is hyperlapsing, or making a super-fast time lapse of an otherwise long video so that it can fit in with other content, be digested quicker, or turn a long bike ride or hike into a quick, bite-sized taste of the journey. Hyperlapsing is built right into Facebook’s stabilizing algorithm cluster, and more functions are coming in the future as the feature begins to roll out to all users.
