This week Google, via its Research Blog, announced YouTube creators are gaining the ability to swap out backgrounds in videos. Similar to how ‘green screens’ are used, this is designed to afford video makers the option to quickly change the background to create more engaging videos, and generally speaking, as often as they want.
The blog post on this does go into quite a lot of technical detail on how this feature has been made possible, although the crux of it all is machine learning. As Google explains the technology uses semantic segmentation – understanding an image at the pixel level – to identify individual features (lips, hair, mouth, and so on). These become defined as ‘foreground’ elements and then the software is able to effectively separate those aspects from the background, creating two distinct layers. From here on, the process is fairly straightforward (at the user level) with creators able to remove the current background layer and replace it with a new artificial one. Much in the same way Photoshop is used to remove backgrounds. Of course, one of the fundamental differences between removing the background from a video (compared to a still image) is movement. And while this is a fairly significant issue Google has seemingly overcome this by applying the technology at a frame by-frame level. For example, Google states its original goal was to ensure the software was capable of doing this at a minimum of 30 frames per second. In testing, Google reports it was able to achieve over 100 frames per second with Apple’s iPhone 7, and 40 frames per second using its own Pixel 2 smartphone.
Due to the current limitations of the technology, the blog post does state how this process has been largely created with mobile in mind and therefore is more aimed at creating content for YouTube on mobile. Which is where the issue of availability comes in. As Google not only explains this is in a very early beta stage (after all, it was announced on the company’s research blog) but is also only available to those who make use of YouTube ‘Stories.’ Which, in itself, is a very limited subset of YouTube users. Google does say, however, its ambition is to make the feature more widely available through its “Augmented Reality services” in due course. In the meantime, below is a brief example of how the end result looks when used with Stories.