12
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Scene-Aware Audio for 360\textdegree{} Videos

      Preprint
      , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Although 360\textdegree{} cameras ease the capture of panoramic footage, it remains challenging to add realistic 360\textdegree{} audio that blends into the captured scene and is synchronized with the camera motion. We present a method for adding scene-aware spatial audio to 360\textdegree{} videos in typical indoor scenes, using only a conventional mono-channel microphone and a speaker. We observe that the late reverberation of a room's impulse response is usually diffuse spatially and directionally. Exploiting this fact, we propose a method that synthesizes the directional impulse response between any source and listening locations by combining a synthesized early reverberation part and a measured late reverberation tail. The early reverberation is simulated using a geometric acoustic simulation and then enhanced using a frequency modulation method to capture room resonances. The late reverberation is extracted from a recorded impulse response, with a carefully chosen time duration that separates out the late reverberation from the early reverberation. In our validations, we show that our synthesized spatial audio matches closely with recordings using ambisonic microphones. Lastly, we demonstrate the strength of our method in several applications.

          Related collections

          Most cited references15

          • Record: found
          • Abstract: found
          • Article: not found

          Statistics of natural reverberation enable perceptual separation of sound and space.

          In everyday listening, sound reaches our ears directly from a source as well as indirectly via reflections known as reverberation. Reverberation profoundly distorts the sound from a source, yet humans can both identify sound sources and distinguish environments from the resulting sound, via mechanisms that remain unclear. The core computational challenge is that the acoustic signatures of the source and environment are combined in a single signal received by the ear. Here we ask whether our recognition of sound sources and spaces reflects an ability to separate their effects and whether any such separation is enabled by statistical regularities of real-world reverberation. To first determine whether such statistical regularities exist, we measured impulse responses (IRs) of 271 spaces sampled from the distribution encountered by humans during daily life. The sampled spaces were diverse, but their IRs were tightly constrained, exhibiting exponential decay at frequency-dependent rates: Mid frequencies reverberated longest whereas higher and lower frequencies decayed more rapidly, presumably due to absorptive properties of materials and air. To test whether humans leverage these regularities, we manipulated IR decay characteristics in simulated reverberant audio. Listeners could discriminate sound sources and environments from these signals, but their abilities degraded when reverberation characteristics deviated from those of real-world environments. Subjectively, atypical IRs were mistaken for sound sources. The results suggest the brain separates sound into contributions from the source and the environment, constrained by a prior on natural reverberation. This separation process may contribute to robust recognition while providing information about spaces around us.
            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Overview of geometrical room acoustic modeling techniques

              Bookmark
              • Record: found
              • Abstract: not found
              • Conference Proceedings: not found

              Modeling acoustics in virtual environments using the uniform theory of diffraction

                Bookmark

                Author and article information

                Journal
                12 May 2018
                Article
                10.1145/3197517.3201391
                1805.04792
                eccd5567-3562-440d-91e6-62206a033661

                http://arxiv.org/licenses/nonexclusive-distrib/1.0/

                History
                Custom metadata
                SIGGRAPH 2018, Technical Papers, 12 pages, 17 figures, http://www.cs.columbia.edu/cg/360audio/
                cs.GR cs.CV cs.ET cs.SD eess.AS

                Computer vision & Pattern recognition,Electrical engineering,Graphics & Multimedia design,General computer science

                Comments

                Comment on this article