Meta wants virtual landscapes to feel like real life
Meta and a group of researchers from the University of Texas at Austin (UT Austin) are working on bringing realistic audio to the Metaverse.
as Kristen Garuman, director of research at Meta AI, explains (opens in new tab)There’s more to augmented and virtual reality (AR and VR, respectively) than just visuals. Audio plays a very important role in making the world come alive. Garuman says, “Audio is shaped by the environment that [it’s] In.” There are many factors that affect how sound behaves like the geometry of the room, what is in said room, and how far a person is from the source.
To achieve this, Meta plans to use AR glasses to record both audio and video from one location, then using a set of three AI models, transform and clean up the recording so that it feels like It’s happening in front of you when you drive it back. At home. The AI will take into account the room you are in so that it matches the environment.
Looking at the projects, it appears that Meta is focusing on AR glasses. Meta’s plan for VR headsets includes mimicking the visuals and sounds of an environment like a concert, so it feels like you’re there in person.
We asked Meta how people can hear better audio. Will people need a pair of headphones to hear or will it come from a headset? We didn’t get any response.
We also asked Meta how developers can get hold of these AI models. They are designed to be open source so that third-party developers can work on the technology, but Meta did not provide any further details.
AI. converted by
The question is, how can the Meta AR record audio on a pair of glasses and whether it reflects a new setting.
The first solution is known as AViTAR which is a . Is “Visual Acoustic Matching Models.” (opens in new tab) It is the AI that alters the audio to match a new environment. Meta cites the example of a mother recording her child’s dance recital in an auditorium with a pair of AR glasses.
One of the researchers claims that the mother in question could take that recording and play it back at home where the AI would morph the audio. It will scan the environment, noticing any obstacles in a room, and singing sounds occurring in front of it with the same glasses. The researcher says that the audio will come from the glasses.
is there to help clean up the audio Visually Informed Deverberation (opens in new tab), Basically, it removes the distracting action from the clip. Example given is recording a violin concert at a train station, taking it home, and having the AI clean up the clip so that you hear nothing but the music.
The final AI model is visualvoice (opens in new tab), which uses a combination of visual and audio signals to differentiate voices from other noises. Imagine that you are recording a video of two people arguing. This AI will isolate a voice so that you can understand them while silencing everything else. Meta explains that visual cues are important because in order to understand certain nuances and know who is speaking, AI needs to see who is talking.
With regards to visuals, Meta says they plan to bring in video and other cues to further enhance the AI-powered audio. Since this technology is still in its early stages of development, it’s unknown when Meta will bring AI to the Quest headset near you.
Be sure to read our latest review on Oculus Quest 2 If you are thinking of buying one. Spoiler alert: we love it.