Researchers from three different universities in the United States have created an artificial intelligence program called Composition Retrieval And Fusion Network, CRAFT for short, which has been trained to take descriptions of scenes involving characters from popular older cartoon The Flintstones and piece together videos. CRAFT is able to understand natural language descriptions by parsing them for known objects and descriptors of activity and location. Once it has an idea of how to piece a scene together, it can take bits and pieces from different Flintstones episodes and put them into a coherent scene, as seen in the attached gallery.
As seen in the YouTube video embedded below, the system is far from perfect. While it can parse objects, determine entities by their bounding boxes, and retrieve the right background most of the time, it can also fail in a few different ways. Sometimes, it grabs the right ingredients for a scene, but something goes wrong, such as a background not moving with walking characters, or a character on the phone standing far away from the corded phone’s wall unit while talking on it. Complex requests with multiple seldom-used entities can result in a catastrophic failure, wherein CRAFT fails to parse a request and ends up spitting out a scene full of glitches and other nonsense.
CRAFT was made to demonstrate that AI can be taught to use multiple conventions of their knowledge base in conjunction in order to synthesize an output that’s in a different form than its input. CRAFT is not creative per se, but its talents lie in figuring out how best to frame a scene, where to place everything and everyone depending on the background and the sizes of entities involved, and how to make actors and entities fulfill their prescribed tasks in the most logically consistent way possible. While it is still far from perfect and, at this stage, mostly useful for fooling around, the implications of such an AI getting to this level are numerous. Future AI programs could build upon what CRAFT proved in order to provide new use cases for AI and even advance toward humanlike AI programs.
