A brand new patent for PSVR 2, discovered by MP1st, suggests that Sony’s next-generation VR headset for PlayStation 5 will support facial tracking for more immersive experiences. The patent, which was filed on the USPTO website, is the latest in a series of patents that have been filed by Sony Interactive Entertainment. This means that the patent will be tied into the PlayStation hardware platform, rather than used in other parts of Sony’s wider business.
A multi-function patient, the first part of the registration describes how the PSVR 2 headset might be able to track a player’s facial movements through a sophisticated internal camera array.
In the example provided, the technology was able to record the minor facial movements such as a widening of the eyes or a flaring of the nostrils and then replicate such movements to other users who aren’t wearing the headset. This potentially means that in a first-person shooter, one player could surprise another and in doing so, would see a look of shock, surprise or horror come across the facial features of that player in real-time.
On this point, the patent reads thus:
“A method for rendering a virtual reality (VR) scene viewable via a head-mounted display (HMD) is provided. The method includes detecting eye gaze of a user using one or more eye gaze sensors disposed in a display housing of the HMD. And, capturing images of a mouth of the user using one or more cameras disposed on the HMD, wherein the images of the mouth include movements of the mouth.
Then, the method includes generating a virtual face of the user. The virtual face includes virtual eye movement obtained from the eye gaze of the user and virtual mouth movement obtained from said captured images of the mouth. The method includes presenting an avatar of the user in the VR scene with the virtual face. The avatar of the user is viewable by another user having access to view the VR scene from a perspective that enables viewing of the avatar having the virtual face of the user. Facial expressions and movements of the mouth of the user wearing the HMD are viewable by said other user, and the virtual face of the user is presented without the HMD.”
Through the supplied diagram examples which were filed with the patent, you can get a relatively good idea of how this technology functions under the hood, so to speak.
Ultimately, would this technology would seem to mean is that more than ever before, when you go online with a PSVR title, your avatar will bear a much greater resemblance to you. From the eyes to the nose, to the mouth and even your cheekbones, PSVR 2 games could offer up a whole new level of immersion when it comes to player character models that we just haven’t seen yet.
On this topic, the patent goes into lengthy detail as seen below:
“The following implementations of the present disclosure provide methods, systems, computer-readable media, and cloud systems, for rendering virtual reality (VR) views into VR scenes for presentation to a head-mounted display (HMD). One method includes sensing the position of a nose of the user when the HMD is worn by the user. The method includes identifying a model of the nose of the user based on the position that is sensed.
The model of the nose having a dimension that is based on the position of the nose of the user when the HMD is worn. The method further includes rendering images to a screen of the HMD to present the VR scenes. The images being augmented to include nose image data from the model of the nose. In one example, the HMD is configured to capture facial feature expressions which are used to generate avatar faces of the user, and convey facial expressions and/or emotion.
The embodiments described herein, therefore, enable systems of the HMD to track features of the user space, while wearing the HMD. The features can include identifying characteristics of the user’s nose and using that information to render at least a portion of the user’s nose in images presented to the screens of the HMD for viewing the virtual-reality content.
By presenting images of the user’s nose in the images of the scenes being viewed, the images or partial images of the user’s nose will resemble the views that a user typically sees when not wearing an HMD. As such, the user’s brain viewing images will be accustomed to seeing at least part of the user’s own nose in the scenes, and the user’s brain will automatically filter out the presence of the nose.
However, by providing the user’s nose or partial views of the user’s nose in the scenes, it is possible for the user to feel a more realistic experience when viewing the virtual-reality content. If the user’s nose is not provided, at least partially in the images, the user’s appearance or perception of the virtual-reality scenes will be less than natural.
As such, by providing this integration into the images, in locations where the user is expecting to see at least part of the user’s nose, the views into the virtual-reality content will be more natural, and as expected by the user’s natural brain reactions to filter out the nose. Embodiments described herein present methods for identifying the nose, characterizing the nose physical characteristics, and then generating models of the nose for presentation at least partially in the scenes.”
As MP1st also notes, this sort of technology isn’t quite so far fetched as you might think either, as Facebook has been working on its own implementation of similar tech for a while now as seen in the video at the bottom of this article.
This patent comes hot on the heels of another PSVR 2 patent which outlines how Sony’s next generation PSVR headset for PlayStation 5 will use advanced, Valve Index style controllers to provide full on finger tracking and improved ergonomics, because let’s face it – the humble PlayStation Move controller really isn’t fit for purpose at this point.
What sorts of applications do you see for PSVR 2’s facial recognition tech? Sound off in the comments!
Source: USPTO via MP1st