Arr, me hearties! Google's wondrous Lumiere be showin' us the future o' creatin' wee AI videos, mateys!
2024-01-26
Arr, me hearties! This wondrous AI be settin' sails on a voyage to create life-like moving pictures! It be providin' ye with special tools to forge yer own tales on the silver screen. Avast! Set sail on this grand adventure and let yer imagination run wild!
Google has developed a new AI model called Lumiere that is capable of generating high-quality videos from text. The previous text-to-video generation model, Imagen Video, produced videos with unrealistic and nightmarish creatures, but Lumiere has greatly improved the quality. The videos created by Lumiere are more realistic, with accurate representations of sea turtles, fur on animals, and genuine smiles on people. The movement in Lumiere videos is also smooth and fluid, unlike other generative AIs. Lumiere achieves this lifelike appearance by using the Space-Time U-Net architecture (STUNet), which allows it to understand the objects in a video, their movements, and changes, resulting in smooth and realistic videos.In addition to text-to-video generation, Lumiere has other features, such as support for multimodality. Users can upload source images or videos to Lumiere and edit them according to their specifications. For example, an image of the "Girl with a Pearl Earring" can be turned into a video where she smiles instead of staring blankly. Lumiere also has an ability called Cinemagraph, which animates highlighted portions of pictures. It can make a butterfly flap its wings while the surrounding flowers remain still. Lumiere also offers Video Inpainting, which allows users to edit portions of video clips, such as changing the color of a dress or altering the appearance of a car.
Currently, it is unclear whether Lumiere will be launched to the public or implemented as a new service. It could potentially be integrated into future Pixel phones as an evolution of the Magic Editor. However, Lumiere still has some issues, such as jerky animations and distorted limbs. The research paper on Lumiere can be found on Cornell University's arXiv website, but it is a dense read. Overall, Lumiere represents an impressive advancement in text-to-video generation, with its ability to create high-quality and realistic content.