runway logo cmu logo neurips 2021 logo uist 2023 logo

Soundify: Matching Sound Effects to Video

David Chuan-En Lin1, Anastasis Germanidis2, Cristóbal Valenzuela2, Yining Shi2, Nikolas Martelaro1

1Carnegie Mellon University, 2Runway

📄 UIST Paper📄 NeurIPS Paper📝 Citation (BibTeX)

Teaser image

Soundify assist users in matching sound effects and ambients to video and helps dynamically adjust panning and volume by localizing "sound emitters".


In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Example Results

Soundify can localize sound emitters in videos containing multiple audio categories.

Demo Video