Mark Zuckerberg’s vision for the metaverse was meant to reimagine how we interact with each other and the world, providing us with an immersive world where we could seamlessly combine digital and physical information.
The parent company, renamed Meta along the way, had begun introducing headsets and reimagining everyday computing with its Project Orion augmented-reality glasses.
Now, however, Meta is making deep budget cuts to its Reality Labs division, which could see around 10% of the 15,000 employees working on the metaverse and related projects lose their jobs. Meta’s chief technology officer Andrew Bosworth confirmed the cuts to staff in a memo on January 13.
But my years of research with colleagues suggest this apparent U-turn is far from the end of the road for the technology. In search of commercial applications that stretch beyond gaming, it is, though, likely to signal a shift from virtual reality (VR) to less-immersive ways to merge the digital and real worlds.
This augmented reality approach has already been realised through products such as Microsoft HoloLens, which present virtual information within an optical see-through display.
Such augmented reality devices give the illusion of virtual information appearing in physical 3D space. They can also allow users to interact through both gestures and gaze, using integrated hand- and eye-tracking technology.
The problem with virtual reality
After decades of research and development, VR technology is unquestionably a real product serving real needs. State-of-the-art headsets provide users with impressive immersive 3D environments along with integrated robust hand and eye tracking. Beyond gaming, virtual reality is used to train medical doctors, engineers, pilots, and many others.
But there is a conflict when it comes to more general, day-to-day applications. I and many others believe that with the advent of AI, new interfaces will be needed beyond the mobile phone to control and benefit from the applications in the work and home. At the same time, it is clear from our research that many people find VR headsets just too immersive, unsettling and impractical to use.
In a two-week user study in 2022, we compared working in virtual reality for an entire working week – five days in a row, eight hours each day – against a baseline of performing the same work using a standard setup with a regular display, external keyboard and mouse.
In this study, we asked 16 volunteers to do their ordinary office work, such as word processing, programming, creating spreadsheets, and so on. The headline result was that users could work in virtual reality for an entire workweek – but there were lots of issues in doing so.
Study participants using VR experienced a higher perceived workload as well as lower usability, lower perceived productivity, higher frustration, lower wellbeing, higher anxiety, a greater experience of simulator sickness and higher visual fatigue. In short, VR yielded worse outcomes on all key metrics.
Despite these results, in the interviews participants commented that they could see themselves using VR if headsets were lighter and if exposure to virtual reality was limited to a few hours at most.
In a follow-up research paper in 2024, we examined the video evidence we had collected in the study in detail. It showed what participants did while wearing the headset – adjusting it, managing the cable when it got in the way, eating and drinking by lifting up the headset halfway, receiving phone calls, and rubbing their faces.
Our analysis showed people did gradually get used to the VR headsets. Overall, participants adjusted their headset about 40% less frequently towards the end of the workweek, and removed the headset approximately 30% less frequently.
This tells us it is possible to work in virtual reality as we normally work with a physical desktop, keyboard and mouse. But if we arrange it so the VR setup replicates our ordinary setup then VR, unsurprisingly, performs worse. We are asking a virtual environment to perfectly replicate our physical work environment, which is impossible.
More importantly, it tells us something about trade-offs. Virtual reality provides a fully immersive virtual environment that transports users to completely different virtual worlds. But this has to be balanced against negative qualities such as poor ergonomics, nausea and fatigue.
Superhuman powers
For any form of extended reality – from augmented-reality smart glasses to something much more ambitious – to achieve mainstream success, it needs to provide more positive than negative qualities in relation to devices we are already familiar with, such as laptops, tablets and phones.
The solution, I believe, is to be bold and reimagine extended reality – not as a transplantation or extension of devices we already use in our daily lives, but as a medium that bestows us superhuman powers. In particular, it can enable us to seamlessly interact with computing systems in the 3D space around us.
In physical reality, you have to select a tool to use it: you pick up a spraycan, then push a button to spray-paint. In a desktop interface, you click the spraycan icon and can thereafter spray-paint using a mouse. But in extended reality, there is no need to first select the tool in order to use it – you can just do it with hand gestures.
Simply by forming your hand as if you were holding a spraycan and pushing down your index finger to spray, the system can automatically recognise that you wish to use the spraycan tool. It will then allow you to spray-paint the digital items, modulated by your index finger pushing down a virtual spraycan button.
Extended reality can also provide a medium for interacting with personal robotics by, for example, showing the robot’s future movements in 3D space in front of us. As artificial intelligence becomes increasingly embedded in our physical reality, this will become more important.
Ultimately, any vision of a metaverse (not just Zuckerberg’s version) will only succeed if it goes beyond current user interfaces. Extended reality must embrace the fact that it allows a seamless blending of virtual and physical information within our 3D world.
