Creating a framework for audio/visual interaction requires a robust and extremely flexible audio engine to sit within the Unity environment. It should provide complete freedom to develop custom synthesisers and effects chains to suit any given scenario, and apply them fluently to sonify any aspect of the virtual world.
Immersion
The video above demonstrates our initial implementation of a simple granular synthesiser in Unity3D, where grains can be individually spatialised by attaching them to GameObjects. As the distance between the GameObjects and the user’s head increases, there is a volume and frequency roll-off, much like the standard work-flow in Untiy3D. However, to achieve the level of sonic immersion we require, we have implemented the Oculus Spatializer plugin, which allows quick deployment of volumetric audio sources and head tracking. While this plugin doesn’t fit within our “all open-source” project model, we have chosen to use this to begin our project to save time. We plan to work on a custom spatialisation solution in the future.
Additionally, you might notice a doppler effect as the grains fly past the user’s head, a function provided by default within the Unity3D 3D sound implementation. This provides the pitching up of approaching sounds and pitching down receding sounds.
I believe that to enable a truly novel expressive instrument within VR, natural interactions with virtual objects should provide a familiar, yet magical realisation of audio, visual and spatial congruency. In this environment, the sonic and visual outputs act as though it were a single substance, at the behest of the artist’s exploration and manipulation. Academic works published over 2 decades ago by Golan Levin and Axel Mulder provide valuable insights into early approaches of audio/visual performance interfaces, and both serve as important influences to some of the techniques developed in TEKH:2. You’ll most likely find us referencing their work in future posts.
Audio Engine
Since this form of audio/visual interaction is not widely indulged within even the most esoteric of academic fields, let alone realised within a game engine such as Unity, we needed to “roll our own” audio solution to satisfy the project’s unique requirements.
What We’ve Tried
Two audio frameworks were considered before deciding on our current approach:
I’ll spare you from an extensive review and summarise our decision not to utilise either of these approaches in our project. While they are both very powerful frameworks (especially JUCE):
- They use C++ (a language we’re not fluent with).
- They produce plugins that exist only within the audio mixer system of Unity3D, instead of being deployed as GameObjects within the environment.
- This means each instance of every audio source would require a custom spatialisation pipeline for the audio to exist within the environment. Without this, the level of sonic immersion would not be satisfied.
- Finally, they demand an additional SDK to use, which complicates distribution and maintenance.
What We’ve Settled On
Due to the limitations of these existing audio systems, we have decided to create our own synthesiser and DSP pipeline directly within C# using Unity3D’s OnAudioFilterRead DSP chain. This allows us to have complete control over the audio system, while allowing us to spatialise the audio sources by attaching them to GameObjects.
Manual Eisl’s “Granulator” demonstrates a simple example of a powerful synthesis paradigm (called Granular Synthesis) that leverages Unity’s OnAudioFilterRead chain. Importantly, this code forms the basis of our own granular synthesis engine used here, which I will expand on in future posts.
Thanks for reading our first development journal entry. We’re looking forward to sharing more very soon.