TA-V2A: Textually assisted video-to-audio generation

Y You, X Wu, T Qu - ICASSP 2025-2025 IEEE International …, 2025 - ieeexplore.ieee.org
ICASSP 2025-2025 IEEE International Conference on Acoustics …, 2025ieeexplore.ieee.org
As artificial intelligence-generated content (AIGC) continues to evolve, video-to-audio (V2A)
generation has emerged as a key area with promising applications in multimedia editing,
augmented reality, and automated content creation. While Transformer and Diffusion models
have advanced audio generation, a significant challenge persists in extracting precise
semantic information from videos, as current models often lose sequential context by relying
solely on frame-based features. To address this, we present TA-V2A, a method that …
As artificial intelligence-generated content (AIGC) continues to evolve, video-to-audio (V2A) generation has emerged as a key area with promising applications in multimedia editing, augmented reality, and automated content creation. While Transformer and Diffusion models have advanced audio generation, a significant challenge persists in extracting precise semantic information from videos, as current models often lose sequential context by relying solely on frame-based features. To address this, we present TA-V2A, a method that integrates language, audio, and video features to improve semantic representation in latent space. By incorporating large language models for enhanced video comprehension, our approach leverages text guidance to enrich semantic expression. Our diffusion model-based system utilizes automated text modulation to enhance inference quality and efficiency, providing personalized control through text-guided interfaces. This integration enhances semantic expression while ensuring temporal alignment, leading to more accurate and coherent video-to-audio generation.
ieeexplore.ieee.org