Here's how musicians and tech companies are teaming up to reduce audio latency

Musicians desperate to work together in harmony while kept apart by COVID-19 lockdowns have been collaborating with tech companies to shave milliseconds from delays on their online connections, driving innovations that will transcend music.

To recreate the experience of performing together live when artists and audiences are apart, a collective effort has been underway to reduce the lag between a sound being produced and being heard, known as latency.

In rehearsals for Rossini's Barber of Seville, the San Francisco Opera has used a test version of a device called Aloha, developed by Stockholm-based Elk OS., in partnership with Ericsson, Vodafone and Verizon.

The pocket-sized device cuts the lag from around 600 milliseconds, which would make two performers sound out of sync, to roughly 20 milliseconds, which is no greater than if the performers were in the same room.

Also read: Facebook joins the audio race to compete with Clubhouse, Twitter

"That was shocking to me that technology has advanced that far," said soprano Anne-Marie MacIntosh, who is performing in the production scheduled to open to socially-distanced, drive-in audiences on Friday. Terrified of throat infections, singers tend to be germaphobic at the best of times, which makes digital technology like the Aloha attractive with or without a pandemic. "You can be in a separate space and still have a rehearsal and still do it safely and not worry about potentially getting someone else sick," said MacIntosh.

While innovations such as the Aloha can work with the existing internet infrastructure, the rollout of 5G, which promises speeds 10 to 20 times faster than 4G wireless networks, could spur even more dramatic advances.

'SCRATCHING THE SURFACE'

The likes of Verizon and Ericsson intend to take advantage of 5G to overcome latency issues for a wide range of industries. Digital surgery, self-driving cars, gaming and virtual reality are some of the areas likely to benefit.

"We are just scratching the surface of how transformative this technology is going to be," said Nikki Palmer, chief product development officer at Verizon.

Matthew Shilvock, general director of the San Francisco Opera, said technology had enabled the upcoming Rossini production to be more experimental and transform the backstage environment into part of the set.

As singers no longer need to be on stage to sing together and audiences can be anywhere, there is also the potential for spectators, however remote, to be involved in the live moment that makes performances exciting.

"I think there's a new hunger, there's a new curiosity that has developed around digital content," Shilvock said. Elk was founded by Michele Benincaso, a violin-maker who trained at a specialist school in Cremona, Italy. He began working in his Stockholm basement around six years ago with the dream of making musicians as digitally connected as other professionals. "What the pandemic has told us and is still telling us, it's time for music and musicians to move into the digital world," he said. "The final goal is engaging the fan base in a completely new way."

REMOTE SURGERY

While many musicians are eager to return to performances in packed concert halls, Benincaso said that the benefits of reducing latency to levels seen when performers are in the same room would be felt long after the end of COVID-19 lockdowns.

For example, it can cut carbon emissions as well as budgets by eliminating journeys, reduce the need for drummers to lug unwieldy kit to recording sessions, and allow audiences to play with professional musicians in innovative ways.

Benincaso said he was also working with major bands from outside the field of classical music, but could not name them because of confidentiality agreements.

He said the San Francisco Opera experiments had put Aloha on track for a commercial release in October.

Read more: Apple Music pays artists an average of $0.01 per stream

Beyond music, an early 5G-driven change is likely to be in the sphere of leisure and entertainment, as amusement parks and shopping malls could use low-latency networks to provide an augmented reality experience for visitors.

Further into the future, developers are aiming to reach a sufficiently minute and stable latency to enable a surgeon to perform an operation at great distance from a patient, using robotic arms. In that scenario, even the briefest spike into a higher latency, or greater lag, could be a matter of life and death. "There cannot be a sudden change when you are doing a critical manoeuvre," said Jan Söderström, head of Technology Office Silicon Valley at Ericsson.

There is also the question of how to make all this pay, which probably means slicing up the network, so that the surgeons or musicians for whom timing is of the essence, might pay more for a guarantee of stable low latency. "That's not current but that's the course, the model that we foresee," Söderström said.