The complex mutual interaction between human visual perception and hearing demands combined examinations of 360° video and spatial audio systems for Virtual Reality (VR) applications. Therefore, we present a joint audio-visual end-to-end chain from spatial recording to immersive reproduction with full rotational three degrees of freedom (3DOF). The audio-subsystem is based on Higher Order Ambisonics (HOA) obtained by Spherical Microphone Array (SMA) recordings, while the video is captured with a 360° camera rig. A spherical multi-loudspeaker setup for audio in conjunction with a VR head-mounted video display is used to reproduce a scene as close as possible to the original scene with regard to the perceptual modalities of the user. A database of immersive content as a basis for future research in spatial signal processing was set up by recording several rehearsals and concerts of the Aachen Symphony Orchestra. The data was used for a qualitative assessment of the eligibility of the proposed end-to-end system. A discussion shows the potential and limitations of the approach. Therein, we highlight the importance of coherent audio and video to achieve a high degree of immersion with VR recordings.
(See document for exact affiliation information.)
https://aes2.org/publications/elibrary-page/?id=19803
(2941KB)
Click to purchase paper as a non-member or login as an AES member. If your company or school subscribes to the E-Library then switch to the institutional version. If you are not an AES member Join the AES. If you need to check your member status, login to the Member Portal.
