8 ms·
TransparentHMD: Revealing the HMD User’s Face to Bystanders (2017) [pdf]
- billconan 3y agoI prefer no outward-facing display if that can make Vision Pro cheaper.
- ramboldio 3y agoI have to try it out to see whether it's worth it. But since the display can be fairly low-resolution, I don't except that it adds a lot of cost. Weight would be a bigger concern to me..
- mickdarling 3y agoMy concern is primarily power draw. For a device with only a 2 hour battery, every erg matters.
- wahnfrieden 3y agoIt uses slow refresh like new iphones
- Geee 3y agoIt's not on all the time. Only when you interact with people.
- gnicholas 3y agoHm, doesn't it use the outward-facing display at all times, to display your status (whether you're fully immersed, are able to see others, or are recording a video, etc.?) I'm sure there's more processing required to show the eyes, but it's still illuminating the display when it's showing these other states. These could be done via simple LED indicators, saving some energy.
- ricardobeat 3y agoConsidering the amount of processing, rendering, sensors, eye tracking and 10x higher resolution displays all happening inside the headset, it's unlikely that the external display consumes enough power to affect battery life significantly. It would probably only save a few minutes at most.
- dangus 3y agoWhile the displays are significant areas of power draw, for this product I don't think they're the low-hanging fruit. This is a device that's performing a whole bunch of computation in real time: video, lidar, infrared, eye tracking, and then it has to drive two 4K+ displays, one for each eye, with 3D-accelerated content being rendered at all times. Go to an AR web page on your phone and start playing around with 3D objects in your space and you'll notice your phone getting significantly warmer and drawing more power. The Vision Pro is doing this literally all the time. Also, the camera and sensor work that is tracking your eyes has to happen whether or not there is an outward-facing display. Apple makes a watch with a display that is always on, and their phones have high resolution OLEDs that can stay on for over 13 hours (iPhone 14 Pro Max 150 nits brightness doing continuous 5G web browsing).
- dangus 3y agoHow heavy is an OLED panel? Isn't it like a piece of flexible plastic? Early reviewers seem to say that the metal construction of the Vision Pro seems to be contributing a lot to its weight. Most other headsets are all plastic.
- woah 3y agoApple isn't generally in the habit of cutting corners
- jackmott42 3y agoThey cut headphone jacks to save space. No reason you can't cut a pointless outer display to save space too!
- wahahah 3y agoThe 7 had space for a jack (see StrangeParts retrofitting one); they removed it purely for sleekness (and money).
- ec109685 3y agoThe first iPhone only had 2G, the first iPad was super slow, and first AppleWatch had unusable apps. They’re always evaluating if they can cut a corner.
- drcode 3y agoin apple's eyes, that would make you a VR zombie apple has decided VR zombies hurt their brand & they won't allow it
- layer8 3y agoThis is funny, because to my eyes their main marketing image for the Vision Pro (dark-skinned woman) makes a face like a dazed zombie.
- billconan 3y agothe world is biased against us introverts by forcing us to socialize. you see, we wear a noise cancelling headset to pretend to be working, in order to avoid unwanted socialization.
- haswell 3y agoI think that more than anything, the inclusion of this feature is a hint about where Apple intends to take this product, and that they want to send a crystal clear message that this device is meant for interacting with other people. And it seems this is such an important aspect of the product that they're willing to reduce the addressable market from a cost perspective. This, to me, is what makes this product intriguing. And it makes me think that Apple's real goal is something closer to a pair of glasses, and they just know they can't get there without a long series of iterations.
- JohnFen 3y ago> that they want to send a crystal clear message that this device is meant for interacting with other people. But that display increases the "creepy factor" by orders of magnitude.
- haswell 3y agoThis product isn't in the public's hands yet, so I think it's a bit early to conclude that it's categorically creepy. But even if it ends up being a bit creepy, a) I think Apple is well aware of that and b) again I think this highlights how important they think it is to send the message from day 1 that solving for isolation is a top priority and intrinsic to their end-goal.
- JohnFen 3y ago> This product isn't in the public's hands yet, so I think it's a bit early to conclude that it's categorically creepy I was just going by the materials that Apple has released. It's true that I haven't personally seen the device in action in real life. But what Apple has shown certainly triggers a "creepy factor" in me. Whether or not this is an issue for many people, and whether or not Apple will address it, isn't really relevant. What I've seen right now creeps me out a bit.
- haswell 3y ago> Whether or not this is an issue for many people, and whether or not Apple will address it, isn't really relevant Why/how is this not relevant in a broader sense? Or are you just saying this isn’t relevant to you personally?
- dangus 3y agoI think it's one of the most important features of the design. Apple is trying to get VR/AR users out in public so that it can be a mainstream device. In the long run, adding a second screen isn't that expensive, and the cameras that capture the video of your eyes already have to be inside the system to perform eye tracking. If smartphone manufacturers can make folding phones with second screens for under $1000 I think that the outward-facing display is not the lowest hanging fruit for cost reduction.
- ladberg 3y agoThis is not the same as the Vision Pro's display. This paper describes tracking a single other person and displaying a perspective-correct rendering for them, but the Vision Pro displays a perspective-correct rendering for many viewpoints at once using a lenticular screen. Apple's solution works for >1 people at the same time and doesn't require any external tracking (though it's already doing the external tracking regardless), at the cost of lower resolution and only being correct in one dimension vs two.
- ramboldio 3y agoAdding multiple viewpoints is actually sth Meta first proposed, based on the paper from above: https://research.facebook.com/blog/2021/08/display-systems-research-reverse-passthrough-vr/ https://research.facebook.com/blog/2021/08/display-systems-r...
- jessriedel 3y agoMore detail from that post: > There are several established ways to display 3D images. For this research, we used a microlens-array light field display because it’s thin, simple to construct, and based on existing consumer LCD technology. These displays use a tiny grid of lenses that send light from different LCD pixels out in different directions, with the effect that an observer sees a different image when looking at the display from different directions. The perspective of the images shift naturally so that any number of people in the room can look at the light field display and see the correct perspective for their location. > As with any early stage research prototype, this hardware still carries significant limitations: First, the viewing angle can’t be too severe, and second, the prototype can only show objects in sharp focus that are within a few centimeters of the physical screen surface. Conversations take place face-to-face, which naturally limits reverse passthrough viewing angles. And the wearer’s face is only a few centimeters from the physical screen surface, so the technology works well for this case — and will work even better if VR headsets continue to shrink in size, using methods such as holographic optics.
- jessriedel 3y agoDo you know where one could read more about Apple's technique? I don't know much about lenticular displays or why the trick only works in one direction (presumably the horizontal one).
- CharlesW 3y agoHey @ramboldio, as one of the authors of the paper, do you have insider knowledge that Apple got the idea from your paper vs. Facebook's "bizarre 'reverse passthrough'"¹ prototype from 2021? Is there a licensing arrangement? (Just curious, it's a really interesting idea in any case!) ¹ https://www.laptopmag.com/news/facebooks-bizarre-reverse-passthrough-vr-headset-wants-to-show-off-your-eyes https://www.laptopmag.com/news/facebooks-bizarre-reverse-pas...
- ramboldio 3y agono insider knowledge, I just know that the work from facebook cited our work: https://research.facebook.com/blog/2021/08/display-systems-research-reverse-passthrough-vr/ https://research.facebook.com/blog/2021/08/display-systems-r... They also add the display that would work with different angles. So it looks like, maybe Apple implemented Meta's research. The timeline could work. For completeness, there is also another paper "FrontFace" proposing a similar idea that was published around the same time: https://dl.acm.org/doi/10.1145/3098279.3098548 https://dl.acm.org/doi/10.1145/3098279.3098548
- CharlesW 3y agoThanks for the info! It must feel great to see this becoming a reality, and I hope it benefits you and your partners professionally.
- ramboldio 3y agoYes, feels awesome, thanks!
- ladberg 3y agoIf you're suggesting that Apple implemented Meta's research starting as it was published on 8/2021, then that timeline absolutely does not work.
- deleted 3y ago[deleted]
- AndrewKemendo 3y agoHaving patented technology for see through AR display in 2016 that is cited by Apple [1] and knowing how crazy hard it is, it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology instead of trying to solve the hard see through AR problems. [1]https://patents.google.com/patent/US10757400B2/en https://patents.google.com/patent/US10757400B2/en
- warning26 3y agoI'm quite certain they're still trying to solve the hard see-through AR problems, and are hoping to release a future Vision headset with a true see-through display. But otherwise I agree, it makes sense for them to focus on what they can do best with current tech as a stopgap. With current see-through HMD tech, AR ends up incredibly disappointing. (See also: Hololens & Magic Leap's limited FOV)
- mensetmanusman 3y agoThey are still asking suppliers to solve the technology, yes.
- AndrewKemendo 3y agoRight, I think it's going to come eventually but it's really really really hard
- samwillis 3y agoMy understanding is that occlusion in "see through" AR is an unsolved problem, everything looks ghostly and somewhat semitransparent. Until someone solves that I suspect the re-projection method is the only viable option.
- AndrewKemendo 3y agoYes "projecting black" is effectively impossible - every once in a while someone from Magic Leap would claim they had solved it but everyone knew it was bullshit
- DonHopkins 3y agoIt should be touch sensitive so it can detect when somebody pokes you in the eye.
- ramboldio 3y agoLike this? https://gugenheimer.com/?portfolio=facetouch-enabling-touch-interaction-in-display-fixed-uis-for-mobile-virtual-reality https://gugenheimer.com/?portfolio=facetouch-enabling-touch-...
- TastyLamps 3y agoGoogle did something similar in 2017 (although it's overlaying your WHOLE FACE) on the headset and only when viewed through a camera: https://blog.google/products/google-ar-vr/google-research-and-daydream-labs-seeing-eye-eye-mixed-reality/ https://blog.google/products/google-ar-vr/google-research-an...
- gfodor 3y agoAnyone who thought reprojection was the solve for AR considered this solution since you'd have to find a way to simulate glass. The first consumer grade passthrough VR was the GearVR in 2015 or so, so I don't think the idea was originally conceived this late.
- Demmme 3y agoIs this verified? U do like the LMU after all I'm in Munich but this though is more obvious than magic.
- ramboldio 3y agoWhat I can verify: To the best of my knowledge, this is the first work that proposes putting a photorealistic, perspective corrected face on a VR headset. "FrontFace" (https://dl.acm.org/doi/10.1145/3098279.3098548 https://dl.acm.org/doi/10.1145/3098279.3098548) is the first work that proposes putting eyes on a display on VR to "lower the communication barrier".
- deanCommie 3y agoI have extremely mixed feelings about the fact that Apple seems to have "solved" AR by faking it through VR, and that means that with some iteration, a battery improvement, and a price drop, this could truly be the next revolutionary device that people can't help but want to use. And I am scared of what it means for our society when "eye contact" no longer means a direct connection in person, but an indirect one through 2 cameras and 2 screens. Obviously this is old hat for Facetime, and all remote collaboration. But in-person too?
- deanCommie 3y agoI have extremely mixed feelings about the fact that Apple seems to have "solved" AR by faking it through VR, and that means that with some iteration, a battery improvement, and a price drop, this could truly be the next revolutionary device that people can't help but want to use. And I am scared of what it means for our society when "eye contact" no longer means a direct connection in person, but an indirect one through 2 cameras and 2 screens. Obviously this is old hat for Facetime, and all remote collaboration. But in-person too?
- marcell 3y agoHow does this technology handle multiple bystanders looking at the display from different angles?