Recording the Aachen Symphony Orchestra in 3D

The em32 Eigenmike® is a spatial microphone array that uses 32 electret microphones embedded in a sphere of just 8 cm diameter. Sound field capturing applying the Eigenmike® is based on spherical harmonic decomposition and allows the conserving and reproduction of a spatial acoustic experience. One question is to which extent such recording techniques help to replace complex microphoning arrangements and elaborate mixing.

The 3D audio recording was accompanied by a 360° camera system in the 4K version. The Kodak SP360 4K BK5 "Dual Pro Pack" camera system was used in a first approach. It allows to create a full sphere video by stitching two back-to-back fisheye cameras with little overlap. The 3D audio and 360° video content can be reviewed in the IKS|Lab using different audio formats with up to 36 speakers and a virtual reality (VR) headset, on a PC, or even mobile on a smartphone.

The microphone and camera systems were used in different configurations and positions inside and in front of the orchestra. The combination of 3D audio with 360° video recording is a relatively new technology. As a result, it is associated with new measurement tasks and challenges. The huge amount of recorded data was subsequently processed and will be made available to all partners for their own demonstration purposes.

Binaural signals require a special binaural recording technology with the HEAD acoustics HMS II.3 dummy head. The simplified replica of a human head captures shadowing and diffraction properties as well as interaural differences in the recorded signals. Playback using headphones virtually transfers listeners into the spatial scene and conveys a listening impression almost like being there, including spatial localization of individual sound sources.

Furthermore, the dummy head was equipped with a GoPro HERO 4 camera to document its view field. The additional visual impression offers the listener a better orientation in the acoustic scene. The dummy head has been placed at different locations within and in front of the orchestra. With these recordings, the listener experiences the music in the middle of the symphony orchestra.

In a next step, it is planned to use more dummy heads and GoPro cameras to create a simultaneous multi-channel audio recording with the respective videos such that in one recorded scene we can switch between different spatial perspectives - acoustically and visually.

For supporting reference measurements, supplementary individual conventional Beyerdynamic MM1 measurement microphones and a stereo microphone setup were used.

In addition, the em32 Eigenmike® was combined with the stereo microphone system in a special arrangement. The recorded data, amongst other things, will be the basis for a student project that investigates to which extend a virtual stereo microphone arrangement can be generated from a Higher Order Ambisonics (HOA) sound field representation.

The results of the first field experiments were so promising that we got the opportunity to join the Aachen Symphony Orchestra performing "Ein deutsches Requiem" by Johannes Brahms in Aachen Cathedral on 5th May 2017. To keep the recording equipment as compact as possible, only the em32 Eigenmike® and the 360° camera system Kodak SP360 4K-BK5 were used at this occasion.

On June, 1^st 2017 we performed an especially elaborative recording during the rehearsal for the event "Music is it!" conducted by Justus Thorau with compositions from Strawinsky, Grieg, Beethoven, Vivaldi, Márquez, and Gulda. Kindly supported by HEAD acoustics we were able to apply in total five immersive audio (and video) recording systems - four artificial heads (one HMS  II.3 and three HMS IV.1) with four GoPro HERO4 cameras and our 3D audio recording system with interactive 360° video. All these systems captured the sound in the midst of the orchestra at different positions.

On July, 10th 2017 we had the opportunity to record Gustav Mahler's 9th Symphony in the Eurogress. As in the Aachen cathedral before, we just used the em32 Eigenmike®/360°-camera recording system to keep the impact for the orchestra as small as possible.

The bright, uniform illumination of the performance area and the proper positioning of the recording unit have led to the best results so far. This recording profited definitely from all our experience gained out of previous measurement campaigns in the orchestra. Due to the wide dynamic range of Mahler’s composition, the choice of the recording level is a challenging task. It is always tricky to find an approriate compromise for the recording level to avoid overload effects on one side and to minimize the impact of the noise floor on the other side. Also the orientation of the 360°-camera in the scene might severly influence the quality of the visual impression. The light, the distance to the musicians and the position of the stitching edge have to be considered.

In this special concert the interaction of individual solo instruments at different positions in the orchestra led to a unique spatial sound experience, which can now be reproduced impressively in our IKS|Lab with 36 Neumann KH120D speakers and the HTC Vive headset.

After finishing the recordings with four artificial heads and the corresponding GoPro HERO4 cameras in June, one of the next workpackages was to identify sound passages for demonstration purposes and to cut and edit the different audio and video tracks. Our research assistants Maximilian Kentgens and Stefan Kühl were professionally supported by Justus Thorau, the conductor from the Aachen Symphony Orchestra.

The finished video takes the listener to four different positions within the orchestra. The listening experience of the binaural video even surprised us and will be published soon.

 On request of the music director Justus Thorau we recorded the 1. Symphony concert of the season 2017/2018. Equipped only with our 360° camera and the Eigenmike® we aimed to capture the complex and massive sound of Symphony No. 2 by Gustav Mahler, known as the Resurrection Symphony. With an ensemble of 250 persons - musicians, two soloists, four choirs, and an offstage brass and percussion part producing a "sound of distance" – it was a challenge for both the audio and the video recording. The high dynamic range of the Symphony and the recording at only one position in the orchestra will definitely not lead to optimal results. However, this will be exactly the question to be answered. How to deal with adverse and complex audio conditions, i.e., by appropriate processing the Higher Order Ambisonics format? 

IKS and WDR recorded the Aachen Symphony Orchestra playing,

• The Hebrides, Op.26 by Felix Mendelssohn Bartholdy,
• the Clarinet Concerto by Magnus Lindberg, and
• the Symphony No. 5 in E-flat major, Op. 82, by Jean Sibelius

with the 3D recording unit Eigenmike and a classical multi-microphone setup.

The WDR installed a typical orchestral recording setup consisting of a main microphone system in combination with spot microphones for each instrument group and solo instrument. The main microphone system was realized with a series of spaced Schoeps microphones with MK 2S capsules in front of the stage. They were arranged as an A-B time-of-arrival pair supported by two additional microphones on the left and right to broaden the stereo panorama.

Thanks to the generous support of the WDR, in addition to the signals from our 3D recording unit (Eigenmike), about 30 further soundtracks from the microphones distributed in the orchestra are now available for our research projects - dealing with questions of coding / decoding or the manipulation of the sound scene. Furthermore, the recorded stereo mix of WDR serves us as a professional reference.

To kick off the new concert series "Orchester hautnah" of the Aachen Symphony Orchestra, the Institute has set up an exhibition booth in the foyer of the Eurogress to present the visitors new optical and sound perspectives. With different demos and the new Youtube playlist we presented our recent work to the general public.