V2A by Google DeepMind

V2A by Google DeepMind is a video-to-audio generation system designed for creative professionals and filmmakers who need to generate high-quality audio for their videos. The system combines video pixels with natural language text prompts to generate rich soundscapes for the on-screen action.
V2A works by encoding video input into a compressed representation, then using a diffusion model to iteratively refine the audio from random noise, guided by the visual input and natural language prompts. The system can also generate speech from input transcripts and synchronize it with characters' lip movements, although this may not always be perfect due to the lack of conditioning on transcripts.
Filmmakers and video editors can get the most value from V2A by using it to generate soundtracks for their videos, including dramatic scores, realistic sound effects, and dialogue that matches the characters and tone of the video. This can save time and effort compared to manually creating or searching for audio tracks, and can also enable new creative possibilities such as rapid experimentation with different audio outputs.