7 ms·
Hi HN, I'm Sten, one of the creators. We built this because we wanted 3D experiences without needing a VR headset. The approach: use your webcam to track wher
by il_nets 11mo ago
Hi HN, I'm Sten, one of the creators.
We built this because we wanted 3D experiences without needing a VR headset.
The approach: use your webcam to track where you're looking, and adjust
the 3D perspective to match.
Demo: https://portality.io/dragoncourtyard/ https://portality.io/dragoncourtyard/
(Allow camera, move your head left/right)
It creates motion parallax - the same depth cue your brain uses when you
look around a real object. Feels like looking through a window instead of
at a flat screen.
Known limitations:
- Only works for one viewer at a time
- Needs decent lighting
- Currently WebGL only
We're still figuring out where this is genuinely useful vs just a novelty.
Gaming seems promising, also exploring education and product visualization.
Happy to answer questions!
- jampekka 11mo agoThe demo gives just a blank screen on Android Firefox, Kiwi and Chrome.
- kgarten 11mo agoIt works for me, just needs a time to load.
- aname2 11mo agohow long on what internet connection? i m on 1min and counting on 50mbit. but maybe it doesnt work on ubuntu 24 + firefox? should be webgl capable though. ok took around 2min for me to load, then works.
- Tade0 11mo agoDevtools say 40.23 MB / 13 MB transferred. Even on 1000Mbps it needed a moment. Works for me on Ubuntu 24 and Firefox.
- jampekka 11mo agoAh, thanks! Got it now on a better connection. A loading indicator would ease confusion.
- il_nets 11mo agoIt seems to vary a lot depending on device. It's just a "basic demo" for now, but yes good idea, thanks!
- socalgal2 11mo agoI don't know if you designed it for a specific monitor but, feedback. It tried using it on my M1 Mac. First thing, there is no loading indicator and it takes too long to start so I though it was broken a few times before I realized I had to wait longer. Second thing, although it was clearly tracking my head and moving the camera it did not make me feel like I'm looking into a window into a 3d scene behind the monitor. These kinds of demos have been around before. I don't know why some work better than others. some others: https://discourse.threejs.org/t/parallax-effect-using-face-tracking/16625 https://discourse.threejs.org/t/parallax-effect-using-face-t... https://www.anxious-bored.com/blog/2018/2/25/theparallaxview-illusion-of-depth-by-3d-head-tracking-on-iphone-x https://www.anxious-bored.com/blog/2018/2/25/theparallaxview...
- svantana 11mo agoIsn't it because the webcam FOV is unknown, which is needed to estimate distance from pixel (along with face size, but that should be less variant). The three.js demo had the strength parameter that can be used to calibrate. The iphone app is pre-calibrated for the most common devices, I believe.
- il_nets 11mo agoThanks for sharing all this feedback! And we will check out these links, appreciate you sharing it!
- president_zippy 11mo agoIt was definitely useful and appreciated on the "New" Nintendo 3DS XL, which also used a camera to track your eye movements and adjust the divergence accordingly. I hate the fact that Nintendo abandoned this technology because experiencing Ocarina of Time and Star Fox 64 in 3D was world-changing to me. I'd say I'm not the only one who misses this technology in games, because a used New 3DS XL costs at least $200 on eBay right now, which is more than what I paid new. I always thought 3D would combine really nicely with a ray traced graphics full of bright colors and reflections, similar to all those ray tracing demos with dozens of glossy marbles.
- Auracle 11mo agoSamsung recently released a monitor with similar technology, I believe, as a FYI.
- nxobject 11mo agoI'm surprised there's still a market for non-VR consumer 3D! I remember the post-Avatar rush of 3D-related products that never quite panned out.
- kevindamm 11mo agoI remember the 3D glasses that you could plug into the Sega Master System in the mid-80s. They took what would be interlaced frames and rendered them to different eyes instead (which made the version getting shown on the connected TV pretty trippy too). And then there was the time travel arcade game (also by Sega) that used a kind of Pepper's Ghost effect to give the appearance of 3D without glasses. That was in the early 90s. I think the idea of 3D displays keeps resurfacing because there's always a chance that the tech has caught up to people's dreams, and VR displays sure have brought the latency down a lot but even the lightest headsets are still pretty uncomfortable after extended use. Maybe in another few generations... but it will still feel limiting until we have holodeck-style environments IMO.
- 11mo ago
- LordHeini 11mo agoThe obvious use case would be to replace the clunky head tracking systems which are often used in simulator games. Systems like trackir, which require dedicated hardware.
- dTal 11mo agoYou can do this today with OpenTrack: https://github.com/opentrack/opentrack https://github.com/opentrack/opentrack Also, TrackIR is just an IR webcam, IR leds, and a hat with reflectors. You can DIY the exact same setup easily with OpenTrack, but OpenTrack also has a neural net webcam-only tracker which is, AFAIK, pretty much state of the art. At any rate it works incredibly robustly. Actually I have already used it to implement the same idea as the post, with the added feature of anaglyph (red/blue) glasses 3D. The way I did it, I put an entire lightfield into a texture and rendered it with a shader. Then I just piped the output of OpenTrack directly into the shader and Robert, c'est votre proverbial oncle. The latency isn't quite up to VR standard (the old term for this is "fishtank VR"), but it's still quite convincing if you don't move your head too fast.
- jojobas 11mo agoTrackir is just a camera with an infrared led.
- trenchpilgrim 11mo agoThere's already a wide variety of Opentrack plugins that use everything from off the shelf webcams to DIY infrared trackers to an iPhone app and FaceID/AirPods.
- Tade0 11mo agoI can confirm that is works decently well with a sunny roof window in the background, which is normally enough for people to complain that my face is too dark. 8yo me, who instinctively tried to look behind the display's field of view during intense gaming sessions, would appreciate this feature very much. My belief is that if it shifted the pov to a lesser degree than in the demo, people generally wouldn't notice, but still subconsciously register this as a more immersive experience. I'm also glad that the web version doesn't try to cook my laptop - good work.
- il_nets 11mo agoThanks! If you click "Menu" and then "Settings" you can play around with e.g. the sensitivity. Ideally we'd automatically optimize the calibration according to for example what device you are using, but that's something we would do a bit more long-term. Appreciate it!
- utopcell 11mo agodefinitely laggy, but works even under low lighting conditions and a camera that is not facing straight forward.
- il_nets 11mo agoMay I ask what device and browser you are using?
- toss1 11mo agoVery cool! I can see this quite useful for educational demonstrations of physics situations and mechanical systems (complex gearing, etc.). Also maybe for product simulations/demonstrations in the design phase — take output from CAD files and make a nice little 3D demo. Maybe have an "inertia(?)" setting that makes it keep turning when you move far enough off center, as if you were continuing to walk around it. The single-viewer limitation seems obvious and fundamental, and maybe a bit problematic for the above use cases, such as when showing something to a small group of people. One key may be to take steps to ensure it robustly locks onto and follows only one face/set of eyes. It would be jarring to have it locking onto different faces as conditions or positions subtly change.
- il_nets 11mo agoGood ideas - we've considered these as well actually! The intertia-idea wouldn't be too difficult to implement, but its usefulness would probably depend on the application area. Yep exactly. Usually it locks onto one person's face but it can also jump around, so there are still optimizations we can do there - but generally it's supposed to be for one person. If you compare to VR headsets, two people can't wear the same VR headset anyway!
- nine_k 11mo agoThis is fun! But I see it showing me a close-up view (smaller FOV) if I move my head back, and a wider view (larger FOV) if I bring my head closer. This is the opposite to what I'd expect: if I bring my head nearer to the screen, I should see a more detail, closer (narrower FOV).
- Wowfunhappy 11mo agoYour expectation doesn't match real life! Try it with a window. If you bring your head closer to the window, your FOV increases, ie you can see more of the scene outside the window.
- nine_k 11mo agoBut there is no window. There is a box with a dragon in it. When I bring my eyes closer to the box, I expect to see more of the box, and more of the dragon. You can try it with any real box; substitute any appropriate object for the dragon. Pick your favorite 3D environment that features walls and dragons; notice how both grow, covering more of your FOV. This is what I expect to see as I draw closer to the screen.
- marcodiego 11mo agoHow about contributing that to Godot?
- il_nets 11mo agoDefinitely! Our current focus is on Unity, because that's what we're most used to, but we'd also build the solution for at least Unreal and Godot as well!
- zipy124 11mo agoThe latency is unfortunately bad enough that it prevents me from getting any depth illusion :( I've seen other implementations where it works really well though.
- il_nets 11mo agoMay I ask what device you are trying it on? In our experience, it can vary depending on what device - something we're looking to improve on further down the line. Do you remember which other implementations you've seen that worked really well?
- arnarbi 11mo agoBack in college (~2008) we implemented this with a 7 foot tall back-projected screen and a couple of Wii remotes after seeing Johnny Lee’s video. The nice thing with that screen was that you could stand so close to it you couldn’t really see the edges. We had as many people come test as we could, and we found that 90% of them didn’t get a sense of depth, likely because it lacked stereo-vision cues. It only worked for folks with some form of monocular vision, incl myself, who were used to relying primarily on other cues like parallax.