Interview with Dr Andreas Wilzeck, Chair of the LIPS Consortium, Manager Spectrum Policy and Standards at Sennheiser

Dr Wilzeck, during your chairmanship of the LIPS consortium, you have seen the research topic of the LIPS project – networked events with participants in separate locations – gaining an unforeseen topicality with COVID-19…

AW: When the project started in 2018 there was no thought of current developments, and our focus was exclusively on finding out how we can improve things for our customers, in our markets. As such, the LIPS project had never envisioned replacing real-life events. We were rather looking at complementing them, at giving them more artistic freedom. As a research project, we do not develop products. However, we can provide inspiration as to what solutions could look like – also in the current crisis.

Could you briefly describe the LIPS project to our readers? What were the aims of the research consortium?

AW: We had asked ourselves how we can network two locations in such a way that the participants at these two separate locations do not notice any difference to being in the same place. How do you achieve an immersive experience like that? Let me give you an example of an idea and a solution that were driven by the Gottfried Wilhelm Leibniz Universität Hannover: There’s a conference room at location A and a conference room at location B. One wall of these rooms is basically a large screen, and immersive technologies as well as video and audio alignment ensure that the participants at both locations feel as if they are together in one room and participating in the very same conference. Another application would be a regional council meeting, where a “windowed reality” would be generated for all the individual village councils that participate. It is questions like these that the LIPS project tried to answer.

What would be a typical application for music and concerts?

AW: Networked concerts would be a fantastic option, with different bands playing together. Suppose there is a concert in the city of Frankfurt, and another one happening at the same time in the city of Mannheim. What if the bands or the orchestras would suddenly play a piece together? Or we have a famous artist, who has been a mentor to a newcomer, and he or she virtually appears on stage during a concert of the newbie? The buzz word is Networked Music Performance. So, basically we wanted to test and find out which workflows or parts thereof can be improved with new technologies in the future.

Covid-19 has certainly shown the usefulness of the ideas and solutions that the LIPS project partners worked on. But how did the pandemic change your cooperation, the way you worked together?

AW: Due to COVID-19, we suddenly had to forgo physical project meetings. You suddenly notice that something important is missing: There is no direct exchange with people, no chat over a cup of coffee. For a research project, it is always very bad if all possibilities for interaction are purely virtual. It’s the same as with music events. Everybody who experienced them misses them sadly, and understands that, despite all innovation, a virtual event can only be a small compensation for the real thing. On the other hand, the corona situation meant that we could use part of our own research results and greatly enjoy video and audio conferencing. It was very interesting to see how your own project and your own life suddenly become a use case…

Many different stakeholders have cooperated in the LIPS project. What are the biggest challenges in a research project of this size?

AW: One of the biggest challenges of such a project – and particularly at the beginning of a project – is that you need to find a common language, especially when you’re coming from different industries and backgrounds. We had the big advantage that the LIPS project developed from an earlier project, so large parts of the consortium had already worked together and knew each other. Nonetheless, there were discussions about the direction our research should take – it’s quite normal that everybody wants to see the ideas for their own industry explored further. Luckily enough, we had already agreed half-way through the project that we should work in the direction of laboratory and field tests. So we had one joint approach.

What do you take away from the collaboration?

AW: For me, this collaboration has once again confirmed how important research projects like LIPS are in bringing together industry and research institutions and in bringing together different industries – in our case audio and video production – and fostering cooperation. That is a really essential function of such research projects. At the same time, it is infinitely exciting to see how these very different industries and institutions contribute their technical expertise and join forces to ponder how things can be improved. Platform thinking is emerging, so to speak, and that’s great to see. On the one hand, you have this strong focus on practical applications, but on the other hand, you’re always looking to the possible future. I really enjoyed supporting this consortium from the definition to the successful field test.

The pandemic has given a big boost to virtual collaboration and there are now many solutions for digital meetings and networked music making – so what’s special about LIPS?

AW: There are indeed solutions that already allow musicians to play together, for example. However, this is still at a level where a lot of manual work is required from the musicians – moreover, this is work in areas where they are not at home. They suddenly have to deal intensively with IT, it’s not like just plugging in the mic somewhere and they’re done. There are now very many solutions that are basically advanced video telephony solutions but these in turn have the disadvantage of a very high variance in IT hardware. Not every operating system handles audio equally well.

However, a musician is very dependent on the latencies not jittering, i.e. not being sometimes in the range of 10 ms and then suddenly in the range of 30 ms. This back and forth drives a musician crazy relatively quickly. He or she needs as constant a transmission path as possible to play along with other musicians.

That’s why it is important to minimize the factors that cause variance in audio transmission – no matter what kind of network and no matter what kind of technology are used. That’s one of the reasons why we decided in the consortium to use a fiber-optic connection in order to test the ideal case. At the same time, it allowed us to specifically simulate all (interference) effects that one would encounter under real-life conditions. And a great deal of field test work will certainly still be necessary in order to be able to better evaluate what is common in the so-called wide-area networks, and to see what can actually be expected or needs to be prepared for. Because in the end, it really should be as easy as the musician only plugging in a cable and that’s it.

We have heard a lot about audio – what about video?

AW: Video is on a totally different level as regards a) data rate and b) latency. This is basically due to the frame rate of the video picture. Typically, many applications delay the audio signal on purpose so that it matches the picture, making it lip-sync. Within the LIPS project, we deliberately dispensed with delaying the audio. We accepted that video and audio will not match fully at all times in order to achieve a considerably lower latency of the audio signal – because this is a more important and more valuable benefit for the musician. On the other hand, when watching TV, you would certainly want your audio to be lip-sync.

So video is actually of lesser importance to musicians?

AW: That’s right. It’s very interesting indeed that a musician is not dependent at all on being able to see his or her fellow-musician. It is more critical to be able to hear them optimally. Of course, video does help with any non-verbal communication, for example with gestures to signal the other “there’s a problem” or “let’s start.”. This is also important. What we worked on intensively with the universities was to create some kind of global metronome, which would enable the musicians to keep time, because the clock is directly in their ear no matter where they are. Such aspects developed during the project, too.

The topic of 5G played a role too in the team’s research – can you tell us more?

AW: We looked at 5G from various perspectives. One aspect were audience services. How can we use 5G as a platform, for example at concerts, to provide additional services to the audience? We also examined what importance 5G might have for a production overall and for wireless production in particular. What we need to bear in mind here is that 5G’s paramount purpose is to ensure telephony and data communication. A constant challenge is to actually have 5G at a given location as network coverage will still need to evolve.

5G will certainly have some relevance when it comes to connecting to a glass-fiber network when you are located at a few kilometers from a fiber-optical interconnection point. This is known as “last mile connectivity”. 5G promises exciting options when it has been rolled out to provide audience services. Today, many concert goers film the event with their smartphone or stream it live via Instagram or YouTube. This is not necessarily a desired use of the smartphone… So we took a look at more meaningful uses of 5G, such as enhancing the listening experience for the concert-goer no matter where he or she is located in the auditorium. It was ideas like these that we examined.

You attended the concluding workshop of the project remotely – how did you like the experience?

AW: Yes, I participated from my sofa, listening with my Sennheiser headphones. I fully enjoyed the concert and really liked how immersive everything felt. You really had this spherical, spatial experience – I was totally thrilled!