Apple's Upcoming Mixed-Reality Headset: How It Will Work

Apple Inc.'s mixed-reality headset is a ambitious attempt to create a 3D version of the iPhone's operating system, with eye- and hand-tracking systems that could set the technology apart from rival products. The headset is a long-anticipated product from Apple, and it is clear that the company has put a lot of thought into its design and features.

Apple is set to release a new VR device later this year, called the Reality Pro. This device will take a new approach to virtual meetings and immersive video, and is seen as a direct challenge to the current VR industry leader, Meta Platforms Inc. This is a high-stakes move for Apple, as they are expanding into a new product category for the first time since 2015. They will need to impress consumers in order to be successful.

Apple is entering an uncertain market with a premium-priced product. The company’s 1,000-person-plus Technology Development Group has spent more than seven years on the project, and Apple is counting on it to become a new revenue source. With sales growth poised to stall this year, the company is hoping that this new product will help boost its bottom line.

Virtual reality has been a challenge for the biggest technology companies. Some projections have the industry topping $100 billion by the end of the decade, but headsets are still seen as niche items. Meta has lost billions of dollars on its efforts to develop VR technology.

Apple is looking to bring something new to the table with its upcoming product. According to people familiar with the product, its eye- and hand-tracking capabilities will be a major selling point. The device is expected to cost roughly twice the price of rival devices, but it will offer advanced FaceTime-based videoconferencing and meeting room features.

The headset will also be able to show immersive video content, serve as an external display for a connected Mac, and replicate many functions of iPhones and iPads.

The headset will have several external cameras that can analyze a user’s hands, as well as sensors within the gadget’s housing to read eyes. That way, the wearer can control the device by looking at an on-screen item – whether it’s a button, app icon or list entry – to select it.

Other headsets typically rely on a hand controller, but this one uses a different approach. To activate the task, users simply need to pinch their thumb and index finger together. This makes it much more convenient and user-friendly.

Apple's new headset will use both virtual and augmented reality, just like Meta's latest headset. With VR, users will see images and content within the goggles. AR will overlay digital content on top of real-world views.

The new headset will feature two ultra-high-resolution displays, developed in partnership with Sony Group Corp. These will provide an immersive VR experience, while a set of external cameras will enable an AR "pass-through mode." This will allow users to see the real world through the cameras positioned on the headset. For users who wear prescription glasses, Apple will offer custom lenses that sit within the enclosure itself.

The device will have a Digital Crown that lets users switch between VR and AR. When in VR, the wearer is fully immersed. When AR is enabled, the content fades back and becomes surrounded by the user’s real environment. This is expected to be a highlight of the product.

Although the headset is still months away from being released, some features could still be canceled or changed, according to sources. An Apple spokeswoman declined to comment.

The headset's FaceTime software will realistically render a user's face and full body in virtual reality. This will allow two people - each with an Apple headset - to communicate and feel like they're in the same room. The technology differs from virtual meeting rooms on Meta's headset, which creates a more cartoon-like avatar of the user.

The headset will only support realistic avatars during one-on-one video chats because of the immense processing power necessary for the feature. It will still allow for FaceTime sessions with several people, but additional users will be displayed as an icon or Memoji - Apple's customized emoji.

Apple is planning to unveil a new device as early as this spring, though the schedule could still shift, according to sources. This would allow the company to discuss the product at its annual conference for software developers in June and then release it later this year.

Apple plans to start production of its new mixed-reality headset in February in China, with a possible launch in the US later in the year. The company is also working on a cheaper version of the device, which could be released in 2024 or 2025.

Apple anticipates that it will sell approximately 1 million units of its new headset within the first year. This is in comparison to more than 200 million units for the iPhone, which is the Cupertino, California-based company's most lucrative product. In a rare move, it is also not planning to make a profit on the initial version - even at the high price - which indicates that the company is taking a long-term view of the platform.

The new device will feature immersive video watching as a core feature. Apple has discussed developing VR content for the platform with about half a dozen media partners, including Walt Disney Co. and Dolby Laboratories Inc. And the tech giant is working to update its own Apple TV+ material to work with the headset. As part of the push, Apple bought streaming company NextVR in 2020, aiming to create sports content in VR.

Apple is planning for the headset to have a dedicated video-watching feature that can make viewers feel like they’re seeing a movie on a giant screen in another environment, such as a desert or outer space. However, while the headset’s video will be immersive, its speakers will be less powerful. So users will need to wear AirPods earbuds to get full spatial audio — a surround-sound effect.

A mixed-reality headset would allow users to interact with digital content in a more realistic way, potentially decreasing reliance on the iPhone for sales.

The device will also have productivity features, including the ability to serve as an external monitor for a Mac. With that feature, users will be able to see their Mac’s display in virtual reality but still control the computer with their trackpad or mouse and physical keyboard. This will be a great productivity boost for Mac users who want to take advantage of the immersive power of VR without sacrificing the ability to use their computer in a traditional way.

The headset's operating system, xrOS, will have many of the same features as an iPhone and iPad but in a 3D environment. That includes the Safari web browser, photos, mail, messages and the calendar app. It will also have apps for the company's services, such as the App Store to install third-party software, Apple TV+, music and podcasts. The company is working on health-tracking functions as well.

The headset's interface will be familiar to Apple users, as it will be nearly identical to that of the iPhone and iPad. The home screen will feature a grid of icons that can be reorganized, with users being able to pin widgets such as the weather, calendar appointments, email and stock-market performance among their app icons.

When users need to input text, they can use the Siri voice assistant or rely on an iPhone, Mac or iPad keyboard. However, an iPhone is not required for operation, as the company is developing technology that will let users type in midair with their hands. This feature is unlikely to be ready for the initial launch, but it is something that the company is working on.

Apple has created its own gaming engine to power VR games, and third-party developers are expected to create popular VR experiences for the iPhone. In 2017, Apple released ARKit and other tools to help developers prepare augmented reality experiences for the iPhone. This has helped set the stage for programmers to build apps, games and services for the headset.

Apple's new device will feature a variation of the M2 chip found in the company's latest Macs, as well as a dedicated processor for graphics and mixed-reality experiences. That second chip will be called the Reality Processor, according to trademark applications filed by Apple.

Apple made the decision to move the battery from inside the headset to an external pack in order to address the issue of overheating. The external pack rests in a user’s pocket and connects to the headset via a cable. Another change is the inclusion of a cooling fan, similar to those found on high-end Macs.

The headset can last for around two hours on a single battery pack, which is on par with similar products on the market. However, the battery itself is quite large – roughly the size of two iPhone 14 Pro Maxes stacked on top of each other. Some internal prototypes for software development have a built-in battery that charges via USB-C, though.

Meta's batteries are on the back of the headset, which helps to balance the device on a person's head. This may be less comfortable for users than Apple's approach, especially if they are watching a movie while wearing the headset. Some testers have complained that the product can be cumbersome, according to the people.

The battery life on this laptop is shorter than other similar models on the market, which could create some inconvenience for users. If users want to watch multiple movies or play games for hours at a time, they may need to buy multiple batteries and frequently swap them out.

Apple has acknowledged the challenges the company is facing internally and is trying to set realistic expectations for the product. One benefit of the device, according to Apple, is that it could spur customers to visit Apple retail stores - not necessarily to buy the product, but to try it out. Apple believes that customers may then purchase another device, such as an iPad or AirPods.

To show off its new headset, Apple is creating a dedicated demonstration area within its retail stores. This is similar to what the company did when it launched the Apple Watch, which is now a central part of its $41 billion division.

The initial headset will be made from aluminum, glass and cushions, and will be reminiscent of Apple's $550 AirPods Max headphones. The product will have a curved screen on the front that can outwardly show a wearer's eyes, with speakers on the sides and a headband that helps fit the device around a user's head.

This will be different from the design of most competitor products, which are made mostly of plastic and have multiple straps to secure the device to the wearer.

The eye and hand tracking feature of the Apple headset may be one of its most memorable elements. Apple has a history of including groundbreaking interfaces in its products that set them apart from competitors. With the iPod, it was the click wheel. With the iPhone and iPad, it was the multitouch approach. And with the Apple Watch, it was the Digital Crown.

Apple is hoping that the headset's sci-fi-like interface will make it a winner.