In the patent application, Apple notes that virtual reality (VR), augmented reality (AR) and mixed reality (MR) are becoming increasingly popular due to their ability to change the user’s perception of the world. For example, VR, AR and MR are used for learning purposes, game purposes, content creation purposes, social media and interaction purposes or the like.
These technologies stand out in the user̵
These technologies are becoming more common due to, for example, miniaturization of hardware components, improvements in hardware performance, and improvements in software efficiency. As an example, a user can experience VR content using a head-mounted device (HMD) that encloses the user’s field of view and is connected to a computer. As another example, a user may experience AR content by wearing an HMD that still allows the user to see the surroundings (eg Glasses with optical transparency).
As an example, HMD tracks the user’s eyes as an input for interactive AR / VR element. As another example, eye tracking is also used for contextual awareness of the user’s gaze direction or fixation point. As another example, eye tracking is used to perform smooth reproduction to reduce the resolution of focus outside the content to reduce power consumption.
Apple says that eye tracking is a useful input for the AR / VR experience. However, eye tracking is often achieved with the use of LEDs attached to the HMD. This increases the hardware cost and footprint associated with HMD. Apple thinks content-built glitter is a better solution
Here is the summary of the invention: “In one implementation, a method includes: synthesizing an AR / VR content stream by inserting a number of glitter provided for eye tracking into one or more content frames of the AR / VR content stream; to display, via one or more AR / VR monitors, the AR / VR content stream to an HMD user; providing, via the image sensor, light intensity data corresponding to the one or more content frames of the AR / VR content stream including the number of glitter, the light intensity data including a projection of an eye to the user of the HMD who has projected there on the plurality of glitter; and determining an orientation of the eye of the HMD user based on the light intensity data.