Apple updates 2 patents relating to a future interactive vehicle head-up display and virtual paper used in mixed reality environments
Today, the United States Patent and Trademark Office released two patent updates from Apple. The first concerns a future vehicle interactive head-up display that a user could use through air gestures to control various interfaces, including games, in autonomous or semi-autonomous vehicles. The second patent relates to the use of Virtual Paper in AR/VR environments, giving virtual paper many dimensions beyond real-world paper.
Apple updates two interesting patents
Future vehicle head-up displays interacting with driver hand gestures
Aspects of Apple’s invention involve projecting an interactive scene onto a surface from a protruding object. In a particular embodiment, the interactive scene is projected from a vehicle and can be used by the vehicle to provide a scene or an image with which a user can interact by various gestures detected by the system.
Additionally, the interactive scene may be customized according to one or more system-determined preferences, such as user preferences, system preferences, or preferences obtained through feedback from similar systems. Based on one or more user inputs (such as user gestures received at the system level), the projected scene may be modified or new scenes may be projected.
Additionally, control of certain aspects of the vehicle (such as unlocking doors, starting the engine, etc.) can be controlled through the interactive scene and sensed user gestures.
Apple’s patent FIG. 1 below is a diagram of a system for a vehicle to project an interactive scene onto a surface for interaction with a user of the projection system. In a particular embodiment, the projection system #100 is integrated or otherwise associated with a vehicle, and more particularly an autonomous vehicle to allow a user of the projection system to interact with the vehicle to control one or more aspects of the vehicle.
Figure 3 of the above Apple patent is a diagram illustrating the projection of a welcoming interactive scene from a vehicle; FIG. 4 is a diagram illustrating the projection of an interactive numeric or telephone keypad. In another example, a LIDAR system or another type of depth measurement device such as a RADAR system can be used to detect the user’s position in the scene; FIG. 5 is a diagram illustrating the projection of an interactive game interface scene from a vehicle.
You can view our original patent report from 2018 here which was first discovered in Europe for more graphics and details. The Apple patent was granted in September 2021 under the number 11,117,535. Apple’s updated patent published today interestingly removes the original 23 patent claims for a single patent claim listed below:
New patent claim #1: “A system for projecting an image onto a surface in the vicinity of a vehicle, the system comprising: a presence detector which detects the presence of a user at a distance threshold from a projection system integrated into a vehicle, the presence detector comprising a proximity sensor and a recognition system for determining an identity of the detected user; a projection controller in communication with the presence detector for receiving the determined identity of the detected user; and a image projector in communication with the projection controller, wherein the projection controller transmits one or more instructions to the image projector to display an interactive image on the surface near the vehicle based at least on the identity of detected user.
A driver (or passenger) using gestures to interact with the head-up display on a windshield or passenger window was worked out in more detail in a patent granted in 2021 that we’ve covered here.
Virtual paper in AR/VR environments
Apple obviously covered this invention in a May 2022 report titled “Apple has been awarded a patent relating to virtual paper that will be used in augmented and virtual reality environments”. Apple’s patent relates to the future use of Virtual Paper on devices that will be able to wrinkle, flip, bend, bend while keeping the graphics intact in different positions as the virtual paper changes angle and perspective. The virtual paper could include mixed reality content in 2D, 3D and animation where the image could constantly change depending on a particular state it is in. It’s a pretty cool thing that’s happening now.
FIG patent. 4A under the 2D text “Matrix” (#410) appears to be floating on the surface of the front of the virtual paper (#405), and the 3D chicken (#420) appears to be inside the virtual paper.
As such, the 2D “Matrix” text appears to be in front of (e.g., superimposed) the 3D chicken from the user’s perspective. As a result, the 2D “Matrix” text obscures part of the 3D chicken crest inside the virtual paper.
Apple’s patent FIG. 4B illustrates the reverse side of the virtual paper with the chicken, the outlined surface (#405) appears to be translucent or semi-transparent, so we can see the outline of a blurry chicken image with just the tip of the beak present in it. far right of the image to show you that the graphics follow the movement of the virtual paper.
The patent then goes 10 miles deep into lighting, raster images, spheres, pixel depths that might throw the average reader off.
As shown in Figs. 4C-4E above, upon transforming the display of the virtual paper, the display of the virtual content associated with the virtual paper also transforms in response to the transformation of the outline of the virtual paper. For example, parts of the 3D chicken protrude from one side of the virtual paper when the virtual paper or virtual page rotates.
Today’s patent update adds 20 new patent claims to better protect the patent from competitors and patent trolls. Below are just 2 of the 20 new demands:
New patent claim #1: “A method comprising: on a device comprising one or more processors, a non-transitory memory and one or more displays: displaying a bounded region based on a first set of world coordinates, wherein the contents of the bounded region include a stereoscopic virtual object displayed in accordance with a second set of world coordinates; receiving input directed to the content; and moving the content within a perimeter of a first side of the bounded region in accordance with the input, comprising: moving the virtual object stereoscopic within the perimeter of the first side of the bounded region according to the input; and animating the stereoscopic virtual object according to the second set of world coordinates.
New patent claim #12: “A device comprising: one or more processors; non-transitory memory; one or more screens; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: display a bounded region based on a first set of world coordinates, the content at the within the bounded region comprising a stereoscopic virtual object displayed in accordance with a second set of world coordinates; receive content-directed input; and moving the content within a perimeter of a first side of the bounded region according to the input, comprising: moving the stereoscopic virtual object within the perimeter of the first side of the bounded region according to the input; and animating the stereoscopic virtual object according to the second set of world coordinates.
You can view our report on patents granted in May 2022 for more details and graphs here and/or review Apple’s updated patent filing 20220245913 to review the full patent and the remaining 18 new patent claims here.