How to mix in immersive audio
Rob Lawrence unravels the mysterious art of mixing in immersive audio and highlights the key benefits, challenges, and opportunities for those looking to enter the market.
The concept of height channel arrays and ‘engulfing’ cinema audiences in sound is not a new one. Yet the idea of working with multichannel formats for some mixing engineers is daunting. This concern can partly be explained by a wide gap between what is known and what is published. Spatial audio tools and concepts are developing apace. The Dolby Atmos and Auro-3D formats are rapidly being adopted in India, China, Russia, and the US by cinema exhibitors eager to provide their audiences with premium entertainment experiences. Despite many of the technical hurdles that have yet to be overcome, understanding the tools and the process of mixing in immersive audio can give audio mixers a competitive advantage.
Challenges
Re-recording is a large-format mixing discipline. The time to perform the final mix, according to re-recording mixer Gilbert Lake, can be anything from one to 10 weeks depending on the scale of the film. With this in mind, re-recording mixers have the challenge of mixing for sizeable audiences who occupy large auditoriums where individual members of the audience will be situated differently. When working in Dolby Atmos, Lake finds himself working in “a reference cinema” environment “with a mixing desk in the middle of it”. Unsurprisingly, mixing for film is not always performed in an ideal auditorium. Re-recording mixers occasionally find themselves working in smaller sized mixing suites subject to production schedules and production budget. Ideally, the final stages of a mix will happen, according to Gareth Llewellyn, Auro-3D re-recording mixer for Galaxy Studios, “in the best room you can be in” yet “you have to make the mix as coherent as possible for everyone in the room”.
While having experience in legacy formats such as 5.1 and 7.1 is helpful, not only to understand how elements may translate for the “fold-down” of formats but also when sound object panning is concerned, Llewellyn warns that if you get too carried away and draw attention to the speakers, “you’ve failed in the art of film mixing”.
Llewellyn says that from his experience “an Auro-3D workflow is not that difficult to implement, and it allows you tremendous opportunities to augment a traditional mix or to take a film’s sound into whole new creative areas”. Speak to any sound consultant or re-recording mixer who works in immersive audio and they will likely tell you that modern mixes for film are more about creating a sense of immersion, ambience, and a sense of ‘being there’ rather than the novel opportunity of whizzing sounds around and overhead. While all formats provide an opportunity to manipulate sound elements in space, there is always a danger of removing the audience from the screen. Llewellyn suggests that immersion is a matter of practice and mastery and is “not gimmicky when it’s done well”. However, he warns that despite the fun of using “new toys on big action scenes… object panning rockets, bullets, fly-bys [and] putting aircraft in the ceiling…” as a re-recording mixer you have to be aware of what may distract an audience from the story being told as “it’s not all about the loud bits”.
Gareth Llewellyn, Auro-3D re-recording mixer for Galaxy Studios
Benefits
The recent evolution of easy-to-learn spatial audio software means tools that provide easy access to these new formats are now available yet the art of creating exciting and convincing mixes is as challenging as it has ever been. According to Llewellyn the advantage of spatial audio mixing is that “you can fill in the gaps… it allows you to colour in the spaces… you’re not reinventing the stereo image”.
Understandably technical concerns among audio engineers when reproducing audio through multiple loudspeaker arrays will include phase alignment, timbral distortion, and sound coloration. Llewellyn quickly points out that a benefit of mixing with a multichannel array is that the process actually solves a lot of problems that stereo once had. “It makes sound more pleasant, more realistic, and more relaxing,” he adds.
According to Wilfried Van Baelen, CEO of Galaxy Studios, the home of Auro-3D, this is partly due to a reduced level of mental processing as the ears receive more organic information. New Audio Technology’s Tom Ammerman prefers the stereo versions of his mixes when they originate in an immersive mix. His theory is that he has “much more space” for sound object placement to work with and so can work much faster.
Finding a place to start
While mixing in music and film are subtly different techniques, there are underlying principles that when applied can quickly deliver convincing results. For example, prior to mixing, almost all immersive audio mixers agree that existing (monophonic and stereophonic) recording techniques are valid sound sources. This implies that natively (spatial audio) recorded material is not always necessary. For mix engineers willing to invest their own time researching these tools, their prior experience will support their endeavours in terms of intuitively understanding how multiple channels will collapse.
When looking for a place to start, Ammerman adds: “I never start in stereo… I always start in 3D.” Lake, who has worked on The Hobbit and District 9, starts by spending his time premixing the effects and/or dialogue. As with any audio production mix, the mixer will need to make subjective decisions based on the source material and in collaboration with the sound supervisors and sound editors. Lake says they will often have had “a long-running discourse with the director as they build up the elements of the soundtrack”.
Llewellyn adds: “The best approach is to have the original sound team available, with their source material, and you simply add the 3D tools.” In such scenarios, often involving AMS Neve DFC or Harrison consoles, mixers can build on a 7.1 mix by adding reverbs and reflections in the height channel arrays. In some instances, certain sound elements are simply panned up to the height channels. Such ideal scenarios are not always achieved due to production budgets, deadlines, and the availability of the original sound team: specialists are often booked months ahead for other projects. A more common approach is to start with using the original Foley, ambience, special effect, music, and dialogue stems including options. Stems options are dedicated mono or surround versions of sound objects which provide the mixer with choice.
Llewellyn says the ideal approach to DAW workflow is to begin with the immersive audio mix and have a bussing structure that allows easy monitoring and development of the smaller formats in parallel with the immersive mix. Your approach will also depend on whether or not the mix is a hybrid mix (ie console and DAW) or a pure ‘in the box’ (DAW) mix (eg, using Pro Tools). Llewellyn says working in this way can provide a scalable platform from which to configure panning for additional formats further down the line. Lake adds that, to begin with, the team on The Hobbit had to find complicated workarounds to deal with the new Atmos format, but console manufacturers have been quick to integrate controls for 9.1 array and object panning. Additionally new software tools offer 3D linking: a process where the ambient effects respond to the original panning choices.
Creative decisions often include whether or not to use the height channel information, for example when collapsing natively recorded music from 13.1 or 11.1 to 7.1 and 5.1. Ammerman suggests starting with summing the height and rear channels at their respective positions (along the horizontal plane) with a level reduction in the surround channels (including the height channels) by 3dB. Once the tools have been better understood manipulating reverb, equalisation, and panning is relatively intuitive to learn. “Your prior experience in stereo and 5.1 is enough to inform you,” Llewellyn informs.
In film, 5.1 and 7.1 are still the final versions most often approved by those responsible for the delivery of the final film release. Lake suggests “some directors give you creative control” whereas others “have very set ideas as to what they want to hear”.
Beyond 7.1 there is still caution and hesitation exercised among film makers to commit fully to immersive audio mixes. The cause? Such formats are still relatively new and potentially risky particularly given the large production budgets involved. Llewellyn adds: “Immersive audio is still an unknown quantity… until people see the ball rolling, there is inertia…” While the number of film exhibitors and production releases capable of reproducing an immersive audio experience is notably increasing, convincing production houses to pursue immersive audio film releases is still, in their own terms, risky business.
Conclusion
Immersive audio is new territory for enthusiasts and professionals to explore and advance their skills and experience. Experimenting with new workflows and software tools can potentially provide mixers with a competitive edge. Getting started ought not to be as daunting as one might expect despite some of the initial technical and practical hurdles that need to be overcome. The advantages of modern software, and the fact that almost any source material is suitable, provide new dimensions for artistic expression, or film director interpretation, while providing opportunities to thrill audiences with enhanced listening experiences. Even broader opportunities exist when gaming, headphone entertainment, and broadcast audio are considered.
If in any doubt where to start, Llewellyn’s advice is clear: “Hear the formats… do some tests, do some recordings, set up [speaker arrays] and listen to them.” Lake suggests talking to people in the industry and listening to mixes – start with establishing your own sensibilities. Wherever your chosen starting point, what is evident is that if you can find the time and employ your own creativity, the source material, the know-how, the audiences, and the technology exist. It’s simply a case of having a go.
