visionOS 2 Also Improves Apple Vision Pro's Hand Tracking & Scene Understanding

visionOS 2 brings major improvements to hand tracking on Apple Vision Pro and expands the headset’s understanding of the environment around you.

visionOS 2 Officially Announced For Apple Vision Pro
visionOS 2 introduces a number of new features and improvements across the operating system.

For more on the user-facing new features and improvements in visionOS 2, see our roundup here.

Hand Tracking Improvements

In our review of Apple Vision Pro we criticized the fact that developers received a very low update-rate for hand tracking, making games like Super Fruit Ninja and Synth Riders feel sluggish and unresponsive.

With visionOS 2 Apple is bringing two significant improvements to hand tracking.

The first improvement is increased update rate. On visionOS 1 the system provided developers with new hand tracking poses at 30Hz. On visionOS 2 hand tracking poses are provided at the display refresh rate, typically 90Hz.



0:00
/0:12



The other improvement is prediction, which is optional for developers. Enabling prediction will provide extrapolated hand poses representing where the hand should be by the time each frame is displayed. This results in lower effective latency, but with the tradeoff of accuracy.

Apple recommends developers use prediction to attach virtual objects to your hands, but not when fluidity is important such as 3D drawing where you want the lines to be smooth.

Scene Understanding Improvements

Since launch Apple Vision Pro has provided developers with a continuously updated 3D mesh of the environment, generated by the LiDAR sensor. But building useful apps that take advantage of this requires understanding what exists within this mesh.

With visionOS 2 the headset now has Room Tracking. The system will provide developers with precise geometry of the walls and floor of each room, even supporting non-rectangular walls.

When entering a new room, the system will inform the developer and start providing geometry only for the new room. Apple says a developer could use this to have a virtual pet greet you when you enter your bedroom, for example.



0:00
/0:28



Beyond just walls and floors, visionOS 2 also brings improvements to plane detection.

While visionOS 1 could only detect horizontal and vertical planes, visionOS 2 can detect slanted planes too.



0:00
/0:17



These improvements should allow visionOS developers to build apps more aware of your physical environment that adapt to the space you’re in and anchor to real world surfaces.

Resources