40% decrease in editing workload by employing image recognition AI and audio signal processing for analysis and adjustment

Challenges:

The Suntory 10,000 People's Ninth is a music concert hosted by MBS since 1983, where 10,000 people sing Beethoven's Ninth Symphony.

Since 2020, due to measures to prevent the spread of COVID-19, the participation of general chorus groups at the venue has been suspended, and instead, approximately 10,000 submitted chorus videos are played on the venue's screen.

However, because the shooting environments for the submitted videos vary, the long editing time required for the large volume of over 10,000 video data was a challenge.

Solutions:

Initially, Monstarlab divided the videos into audio and visual elements and explored the optimal technologies to be used for each.

For audio adjustment, the start of singing varied in each video, which made it time-consuming to synchronize the timing across all videos. Therefore, we sought a solution using multiple audio signal processing techniques.

As a result of our technical investigation, we successfully identified the singing part from the audio waveform using Cross-correlation and cut out noise mixed in during recording by using Signal processing in conjunction. Furthermore, we repeatedly tested hypotheses to align the start of singing by identifying a certain sound range, in order to adjust for differences in recording environments and vocal volumes.

For video cropping, we utilized image recognition AI to align the angle of view with the subject in all videos. Because photos that happened to be in the background were sometimes recognized depending on the shooting environment, we made it possible to capture the range of movement of the person within a certain area, enabling high-precision extraction.