Audio source separation for various video contents

We aim to separate voices and background sounds without delay by learning them individually from the input audio. This supports functions that make voices easier to hear or adjust the volume of background sounds while watching videos.

When watching a video, loud background sounds can make voices difficult to hear. This study proposes separating voices from background sounds so their volumes can be adjusted independently. Separating a mixture into its individual sounds is called audio source separation.

The method learns the characteristics of voices and background sounds separately from the incoming audio and estimates each sound. Instead of examining the audio from the entire video in advance, it learns and separates sounds as they arrive, aiming to operate without delay.

In our checks, the method required little processing, worked without delay on a range of devices, and allowed viewers to hear its effects immediately. It also estimated voices and background sounds accurately across different kinds of video content. We aim to use it for functions such as making speech easier to hear, reducing background sound, adjusting the sense of presence in sports videos, and supporting singing practice.

• 広畑 誠, 小野 利幸, 西山 正志,
  多様な映像コンテンツに対応した遅延なし音源分離技術,
  日本音響学会 春季研究発表会, pp. 1-4, March, 2013.
• 西山 正志, 広畑 誠, 小野 利幸,
  声と背景音のボリュームバランス調整に向けた音源分離, 情処研報 CVIM187-46, pp. 1-5, May, 2013.