16 ms·

Show HN: DIY virtual HDMI monitor using "AR" glasses

112 points by blensor 1y ago

I am making a virtual HDMI monitor using Viture Pro XR glasses and an SBC ( currently OrangePi 5 Plus because it has HDMI-in ). What it does is map the frames from the HDMI input onto a virtual display that is controlled by the IMU data from the glasses ( 3DOF only ). I've put AR in quotes in the title because many won't view those display glasses as true AR but by tracking the head movement it comes close. I am trying to build kind of a "low cost" version of a virtual screen that acts like a monitor and can be connected to anything that has an HDMI output I started off using the official Viture SDK to interact with the glasses but have since switched to a reverse engineered implementation of the protocol because their SDK is not available for ARM Here is a video showing the first version: https://www.youtube.com/watch?v=D6w5kAA22Ts https://www.youtube.com/watch?v=D6w5kAA22Ts Big caveat: Performance still needs to improve a lot because the whole frame reading/converting is completely unoptimized for now. What other solutions do exist out there? * Streaming the computer screen to a headset like Meta Quest/Vision Pro * Connecting a HDMI capture dongle to the Meta Quest directly * XReal Beam ( basically the same as this project but official and for XReal glasses ) And for the obvious question, why I am not use something like a Quest or Vision Pro 1. Comfort 2. Price 3. Comfort After using those display glasses over HMDs it's hard to convince myself to use a headset for productivity again

stavros 1y ago
I'm actually wearing the Xreal Air 2 right now, and yes, this is basically exactly what their app does. Personally, I only need one screen, so I'm using the glasses without the app, but this is great if you want a virtual second display somewhere in the room! Very well done.
- k__ 1y ago
  The Xreal Air 2 Ultra look pretty interesting. After using both 3DoF and 6DoF for years, I just can't imagine using a 3DoF headset for more than watching videos or simple writing tasks.
  - sitkack 1y ago
    Just so it is clear, the "Xreal Air 2 Ultra" appear to have 6DoF tracking
    - k__ 1y ago
      Yes, but the previous versions didn't.
- blensor 1y ago
  I'm actually a bit jealous about the XReals. Back when I bought the Viture I wanted to buy XReal first but they had such strange shipping policies ( they shipped to every county around me but not to Austria ) so I went for the Viture.
  - stavros 1y ago
    The Xreals are the only ones I've tried, so I don't know how they compare, but I have trouble with fit. I have trouble in general, because of my nose bridge being too wide, but even when I put the glasses right against my forehead, I still can't get the entire screen in focus, the edges are blurry. I really wish the screen were smaller and more to the center, then they'd be perfect. I guess it depends on whether you want them for games (the important stuff is usually in the centrer) or for work (the important stuff is usually at the edges). I really like the focal distance, though, it's around 5m away and pretty comfortable for the eyes.
    - blensor 1y ago
      With the viture I have the problem that the bottom of the screen is almost always not visible. I initially used the accessibility feature that let's me zoom into the desktop with mouse follow moving the viewport but that gets old quickly, which is one reason why I started that project. I'm also planning to add other "display" modes like a curved monitor and in theory we could connect multiple monitors using several hdmi capture devices but that's probably even less useful
      - gavinray 1y ago
        I also have this problem with the Viture glasses. Do you know if this accessibility feature is available when using it with phones?
        blensor 1y ago
        I haven't found a way to do it on the phone unfortunately which is really annoying. You could in theory use the approach I posted here and use a USB to HDMI adapter to connect the phone to device and it would do the same thing it does with a regular PC but that loses all the benefits of being small and portable if you have to carry around all that additional stuff
- mentos 1y ago
  I’m staying at a friends and using my laptop and a second monitor of his to work. I’ve wondered if the Xreal Air 2 would be better than a large monitor. My hunch is that having a dedicated smaller monitor not on my face is more comfortable and productive than a large virtual one. From your experience would you agree?
  - stavros 1y ago
    Hm, no, I much prefer the Xreal to the laptop's screen (except for the blurry edges). I don't care about the monitor size, only the resolution, but I'd rather have the monitor always be straight ahead in my vision (no matter how my head is tilted), and I really like the Xreal for that. As I said, though, I always only use a single monitor anyway, so YMMV.
    - 0xffff2 1y ago
      I haven't previously heard of Xreal. Looking at their glasses, it seems like they're all 1080p. I find it very difficult to deal with text on a regular 1080p monitor after having uses 4k monitors for a while. Is there something about the way the glasses work that makes 1080p glasses equivalent/better than at least a 1440p monitor in terms of clarity?
      - stavros 1y ago
        Well, 1080p is 1080p, what counts is density. The display looks huge on the glasses, so density isn't great, but it doesn't bother me personally. YMMV, though.
      - einsteinx2 1y ago
        I have them and they actually look worse than a real 1080P screen because the edges are out of focus. I know some people do, but I could never use them for real work. They’re ok for videos and video games but not amazing. If they were 4K they’d be significantly more usable but for me the number 1 issue is the edge blur and general clarity issues which I guess are lens related.
- KeplerBoy 1y ago
  Is the xreal usable outdoors in bright conditions?
  - stavros 1y ago
    Yes, I live in Greece and it's fine. The dark cover is pretty dark.
deivid 1y ago
Cool project, but I'm missing what does this "remapping" do that the glasses can't do by themselves? I see the glasses are connected (via usb-c?) to the OrangePi. Is this project converting HDMI signal to the Viture protocol? if so, can't you run a screen grabber directly on your machine and send it to the glasses?
- blensor 1y ago
  If you connect the glasses to your computer you get a fixed screen in front of your eyes (like those display glasses back in the day that promised to give you a huge screen), which is fine for video viewing but if you want to use it for productivity you want to be able to look at certain parts of your screen vs. straining your eyes to glimpse the edges. That's where this comes in, it allows you to look around the virtual screen with the added bonus that you can connect it to almost anything and behave the same not just your PC
xeonmc 1y ago
If you have 3dof tracking only, consider making the displayed image to have no stereodisparity, this way it is optically correct to being a screen projected onto the celestial sphere. This is also how reflex sights works on fighter jets and guns.
- blensor 1y ago
  I have to look into that, thanks. I've been thinking about combining it with our work on the reverse engineered HTC Vive Ultimate Tracker stuff here https://youtube.com/shorts/U4vhEPQw-Uo https://youtube.com/shorts/U4vhEPQw-Uo to get 6DOF But realistically I ( and nobody else ) won't strap the tracker to my head all the time
LonelyWolfe 1y ago
Just venting and ranting about AR glasses here: One thing about AR glasses that I don't get so far: Why do ALL of the AR glasses use only 1080p displays per eye whereas all the other VR solutions that also have a display right in front of your eye have even bigger resolutions? I really do want AR glasses that can act as an alternative to a physical display. But they're always 1080p for some reason. The companies I've seen making them appear to me to keep fixing important issues and adding really useful features - but seem to avoid touching anything that would improve the resolution. Edit : There's a part of me that believes the situation here is a lot like how Linus Tech Tips described the Dashcam situation.
- k__ 1y ago
  Because the glasses are much smaller than headsets, I'd assume.
- riedel 1y ago
  One of the reason is IMHO that the optical resolution with those lense designs does not get better than 1080p. It seems even that resolution is blurry at the edges. So anything beyond probably would be a waste without totally different optics.
  - LonelyWolfe 1y ago
    This feels like the most probable reason. Likely a big redesign would be needed for larger resolutions.
- blensor 1y ago
  Not disagreeing with the resolution but I assume for most last-gen use cases it didn't make much sense because you also need to be able to drive that somehow and you can be happy that you get a 2 Lane display port interface up until the more recent phones. So if you have high res displays that no one can realistically use for the intended mobile cinema use case then you are wasting money and energy
- spookie 1y ago
  AR solutions value mobility over fidelity. This has numerous repercussions, leading to those displays.
amelius 1y ago
I looked at those Viture glasses, but the pitch of the glasses (how far the eyes are apart) looks strangely low. Perhaps this works in the Asian market, but it just seems inadequate for the Western world.
- gavinray 1y ago
  I own them, bought just recently. The eyes have these circular wheels near your eyebrow above the frame that lets you independently adjust the distance for each eye, if that's what you're talking about.
  - amelius 1y ago
    Good to know, but does that move the arms too? Some people have bigger heads than others.
    - gavinray 1y ago
      It does not, but for what it's worth, I have been told I have quite a large head and they fit comfortably.
      - amelius 1y ago
        Ok, one more question: does the wheel also make the black glasses move? Because I'd imagine that if only the internal optics move to the sides, you get to have less protection from external light on the sides of your view.
      - blensor 1y ago
        I think they suit bigger heads as well is because the hinge on the stems can go slightly beyond 90°
    - deleted 1y ago
      [deleted]
  - meta-level 1y ago
unsupp0rted 1y ago
I, for one, would like to see AR glasses that use e-ink. Black and white with a slow refresh rate. Would be perfect for my use-cases.
- blensor 1y ago
  you still would need a backlight but it's an interesting idea
  - unsupp0rted 1y ago
    Would be cool to wear a "light panel" on your sleeve. Outdoors you see the e-ink fine. Indoors you look down at your illuminated sleeve to see the e-ink in your glasses.
  - sleepybrett 1y ago
    You can't backlight an eink display. They are opaque, by design. Thats why they make such great ebook displays, so you can take them outside and still read them. Eink displays require .. i guess 'forelight'. Light that reflects off the display front.
    - blensor 1y ago
      I didn't know that, I just wonder how the kindle with light works though. Is that light coming from the front too?
      - sleepybrett 1y ago
        frontlit
      - ashdksnndck 1y ago
        If you look closely at a backlit kindle screen, you can see that the e-ink pixels are a decent distance behind the surface. It’s more noticeable if there are scratches or dust on the screen and you rotate the display. The surface reflects the light from the sides back to the e-ink.
sitkack 1y ago
I think what is not clear from your project description is that your AR glasses don't provide an environment locked virtual screen, meaning a screen that stays fixed in space while your head moves. The native screen of your glasses provide a screen that is fixed relative to the glasses, like having a display locked to your face. Some AR glasses do provide this feature, so it overcomes a product flaw in AR glasses and will at some point not be necessary. Your project could also allow for other features that the AR glass manufacturer didn't think of, or has gated behind upgrades and product segmentation. Is my assessment correct?
- andrewmcwatters 1y ago
  It's also a Viture-specific project, but I suspect if the author wanted, this could expand to other video display glasses in the future and he could simply rename the project.
  - blensor 1y ago
    Spot on. And yes expanding it to other glasses ( which are popping up quite a lot recently ) would for sure be possible
- blensor 1y ago
  Yes that's correct, and it could be expanded to other features yes, one thing I've been mulling over is mult monitor input which you don't even get on devices that support what I did here natively
- msgodel 1y ago
  I use AR glasses heavily for work and other things. I'm actually typing this on a pair right now. I've never understood what the application for environment locked screens is other than novelty/marketing. My glasses provide enough sensor data to implement this but I just can't be bothered.
  - veilrap 1y ago
    I'm curious how/what you use the screens via AR for if you're not using environment locked screens. Particularly in a productivity/work environment. Unless I'm misunderstanding the feature, it seems like enironment locked screens allows for more natural usage and interactions with the screens in the virtual space? My experience is mostly with VR/AR products like Oculus has been mostly with environment locked AR information.
skeptrune 1y ago
Wow, $409 getting you a AR display that's viable to hack on is crazy. I can't believe how approachable hardware is for tinkerers some time.
verdverm 1y ago
https://immersed.com https://immersed.com is a great piece of software that gives you multiple virtual screens. It supports all major OS and VR devices, with their own https://visor.com https://visor.com headset coming soon (tm)
wkat4242 1y ago
Nice! I wish this worked on my XReal Air :( It does have IMU sensors because their own app (nebula) uses them, and even does 3D effects. By the way I find my Quest 3 way more comfortable than the Xreal. The problem with the xreal is, the prescription inlays are too small so I can see only a part of the display sharp unless I put in lenses every time. And it is kinda heavy on my nose and ears. the Quest is super nice, has a wider FoV. But it's heavier and bulkier to bring along of course. PS I had no idea you could connect an HDMI dongle to the quest. I'll check that out.
- blensor 1y ago
  If I had XReal I could look into that too, it's probably pretty similar to how it works for the Viture. Regarding the HDMI on Quest 3, I believe it was sometime last year when they made that possible https://www.meta.com/help/quest/1350679463006443 https://www.meta.com/help/quest/1350679463006443
  - wkat4242 1y ago
    Yeah I bet it's real similar. But I understand. Thanks for the heads up about that Quest thing, I actually have some of those adapters so I will give it a try.
benhoff 1y ago
Did you fix the op5plus HDMI in driver? Thing was unstable like 8-12 months ago
- blensor 1y ago
  So far I haven't had any issue but I haven't done long tests yet. I'm using BredOS not their official OrangePI images though
Aaron2222 1y ago
> What other solutions do exist out there? Xreal One and One Pro have this built into the glasses themselves. The Xreal Eye camera is meant to add on-glassed 6DOF tracking as well.

112 points by blensor 1y ago

stavros 1y ago

I'm actually wearing the Xreal Air 2 right now, and yes, this is basically exactly what their app does. Personally, I only need one screen, so I'm using the glasses without the app, but this is great if you want a virtual second display somewhere in the room! Very well done.

k__ 1y ago
The Xreal Air 2 Ultra look pretty interesting. After using both 3DoF and 6DoF for years, I just can't imagine using a 3DoF headset for more than watching videos or simple writing tasks.
- sitkack 1y ago
  Just so it is clear, the "Xreal Air 2 Ultra" appear to have 6DoF tracking
  - k__ 1y ago
    Yes, but the previous versions didn't.
blensor 1y ago
I'm actually a bit jealous about the XReals. Back when I bought the Viture I wanted to buy XReal first but they had such strange shipping policies ( they shipped to every county around me but not to Austria ) so I went for the Viture.
- stavros 1y ago
  The Xreals are the only ones I've tried, so I don't know how they compare, but I have trouble with fit. I have trouble in general, because of my nose bridge being too wide, but even when I put the glasses right against my forehead, I still can't get the entire screen in focus, the edges are blurry. I really wish the screen were smaller and more to the center, then they'd be perfect. I guess it depends on whether you want them for games (the important stuff is usually in the centrer) or for work (the important stuff is usually at the edges). I really like the focal distance, though, it's around 5m away and pretty comfortable for the eyes.
  - blensor 1y ago
    With the viture I have the problem that the bottom of the screen is almost always not visible. I initially used the accessibility feature that let's me zoom into the desktop with mouse follow moving the viewport but that gets old quickly, which is one reason why I started that project. I'm also planning to add other "display" modes like a curved monitor and in theory we could connect multiple monitors using several hdmi capture devices but that's probably even less useful
    - gavinray 1y ago
      I also have this problem with the Viture glasses. Do you know if this accessibility feature is available when using it with phones?
      - blensor 1y ago
        I haven't found a way to do it on the phone unfortunately which is really annoying. You could in theory use the approach I posted here and use a USB to HDMI adapter to connect the phone to device and it would do the same thing it does with a regular PC but that loses all the benefits of being small and portable if you have to carry around all that additional stuff
mentos 1y ago
I’m staying at a friends and using my laptop and a second monitor of his to work. I’ve wondered if the Xreal Air 2 would be better than a large monitor. My hunch is that having a dedicated smaller monitor not on my face is more comfortable and productive than a large virtual one. From your experience would you agree?
- stavros 1y ago
  Hm, no, I much prefer the Xreal to the laptop's screen (except for the blurry edges). I don't care about the monitor size, only the resolution, but I'd rather have the monitor always be straight ahead in my vision (no matter how my head is tilted), and I really like the Xreal for that. As I said, though, I always only use a single monitor anyway, so YMMV.
  - 0xffff2 1y ago
    I haven't previously heard of Xreal. Looking at their glasses, it seems like they're all 1080p. I find it very difficult to deal with text on a regular 1080p monitor after having uses 4k monitors for a while. Is there something about the way the glasses work that makes 1080p glasses equivalent/better than at least a 1440p monitor in terms of clarity?
    - stavros 1y ago
      Well, 1080p is 1080p, what counts is density. The display looks huge on the glasses, so density isn't great, but it doesn't bother me personally. YMMV, though.
    - einsteinx2 1y ago
      I have them and they actually look worse than a real 1080P screen because the edges are out of focus. I know some people do, but I could never use them for real work. They’re ok for videos and video games but not amazing. If they were 4K they’d be significantly more usable but for me the number 1 issue is the edge blur and general clarity issues which I guess are lens related.
KeplerBoy 1y ago
Is the xreal usable outdoors in bright conditions?
- stavros 1y ago
  Yes, I live in Greece and it's fine. The dark cover is pretty dark.

deivid 1y ago

Cool project, but I'm missing what does this "remapping" do that the glasses can't do by themselves? I see the glasses are connected (via usb-c?) to the OrangePi. Is this project converting HDMI signal to the Viture protocol? if so, can't you run a screen grabber directly on your machine and send it to the glasses?

blensor 1y ago
If you connect the glasses to your computer you get a fixed screen in front of your eyes (like those display glasses back in the day that promised to give you a huge screen), which is fine for video viewing but if you want to use it for productivity you want to be able to look at certain parts of your screen vs. straining your eyes to glimpse the edges. That's where this comes in, it allows you to look around the virtual screen with the added bonus that you can connect it to almost anything and behave the same not just your PC

xeonmc 1y ago

If you have 3dof tracking only, consider making the displayed image to have no stereodisparity, this way it is optically correct to being a screen projected onto the celestial sphere. This is also how reflex sights works on fighter jets and guns.

blensor 1y ago
I have to look into that, thanks. I've been thinking about combining it with our work on the reverse engineered HTC Vive Ultimate Tracker stuff here https://youtube.com/shorts/U4vhEPQw-Uo https://youtube.com/shorts/U4vhEPQw-Uo to get 6DOF But realistically I ( and nobody else ) won't strap the tracker to my head all the time

LonelyWolfe 1y ago

Just venting and ranting about AR glasses here: One thing about AR glasses that I don't get so far: Why do ALL of the AR glasses use only 1080p displays per eye whereas all the other VR solutions that also have a display right in front of your eye have even bigger resolutions? I really do want AR glasses that can act as an alternative to a physical display. But they're always 1080p for some reason. The companies I've seen making them appear to me to keep fixing important issues and adding really useful features - but seem to avoid touching anything that would improve the resolution. Edit : There's a part of me that believes the situation here is a lot like how Linus Tech Tips described the Dashcam situation.

k__ 1y ago
Because the glasses are much smaller than headsets, I'd assume.
riedel 1y ago
One of the reason is IMHO that the optical resolution with those lense designs does not get better than 1080p. It seems even that resolution is blurry at the edges. So anything beyond probably would be a waste without totally different optics.
- LonelyWolfe 1y ago
  This feels like the most probable reason. Likely a big redesign would be needed for larger resolutions.
blensor 1y ago
Not disagreeing with the resolution but I assume for most last-gen use cases it didn't make much sense because you also need to be able to drive that somehow and you can be happy that you get a 2 Lane display port interface up until the more recent phones. So if you have high res displays that no one can realistically use for the intended mobile cinema use case then you are wasting money and energy
spookie 1y ago
AR solutions value mobility over fidelity. This has numerous repercussions, leading to those displays.

amelius 1y ago

I looked at those Viture glasses, but the pitch of the glasses (how far the eyes are apart) looks strangely low. Perhaps this works in the Asian market, but it just seems inadequate for the Western world.

gavinray 1y ago
I own them, bought just recently. The eyes have these circular wheels near your eyebrow above the frame that lets you independently adjust the distance for each eye, if that's what you're talking about.
- amelius 1y ago
  Good to know, but does that move the arms too? Some people have bigger heads than others.
  - gavinray 1y ago
    It does not, but for what it's worth, I have been told I have quite a large head and they fit comfortably.
    - amelius 1y ago
      Ok, one more question: does the wheel also make the black glasses move? Because I'd imagine that if only the internal optics move to the sides, you get to have less protection from external light on the sides of your view.
    - blensor 1y ago
      I think they suit bigger heads as well is because the hinge on the stems can go slightly beyond 90°
  - deleted 1y ago
    [deleted]
- meta-level 1y ago

unsupp0rted 1y ago

I, for one, would like to see AR glasses that use e-ink. Black and white with a slow refresh rate. Would be perfect for my use-cases.

blensor 1y ago
you still would need a backlight but it's an interesting idea
- unsupp0rted 1y ago
  Would be cool to wear a "light panel" on your sleeve. Outdoors you see the e-ink fine. Indoors you look down at your illuminated sleeve to see the e-ink in your glasses.
- sleepybrett 1y ago
  You can't backlight an eink display. They are opaque, by design. Thats why they make such great ebook displays, so you can take them outside and still read them. Eink displays require .. i guess 'forelight'. Light that reflects off the display front.
  - blensor 1y ago
    I didn't know that, I just wonder how the kindle with light works though. Is that light coming from the front too?
    - sleepybrett 1y ago
      frontlit
    - ashdksnndck 1y ago
      If you look closely at a backlit kindle screen, you can see that the e-ink pixels are a decent distance behind the surface. It’s more noticeable if there are scratches or dust on the screen and you rotate the display. The surface reflects the light from the sides back to the e-ink.

sitkack 1y ago

I think what is not clear from your project description is that your AR glasses don't provide an environment locked virtual screen, meaning a screen that stays fixed in space while your head moves. The native screen of your glasses provide a screen that is fixed relative to the glasses, like having a display locked to your face. Some AR glasses do provide this feature, so it overcomes a product flaw in AR glasses and will at some point not be necessary. Your project could also allow for other features that the AR glass manufacturer didn't think of, or has gated behind upgrades and product segmentation. Is my assessment correct?

andrewmcwatters 1y ago
It's also a Viture-specific project, but I suspect if the author wanted, this could expand to other video display glasses in the future and he could simply rename the project.
- blensor 1y ago
  Spot on. And yes expanding it to other glasses ( which are popping up quite a lot recently ) would for sure be possible
blensor 1y ago
Yes that's correct, and it could be expanded to other features yes, one thing I've been mulling over is mult monitor input which you don't even get on devices that support what I did here natively
msgodel 1y ago
I use AR glasses heavily for work and other things. I'm actually typing this on a pair right now. I've never understood what the application for environment locked screens is other than novelty/marketing. My glasses provide enough sensor data to implement this but I just can't be bothered.
- veilrap 1y ago
  I'm curious how/what you use the screens via AR for if you're not using environment locked screens. Particularly in a productivity/work environment. Unless I'm misunderstanding the feature, it seems like enironment locked screens allows for more natural usage and interactions with the screens in the virtual space? My experience is mostly with VR/AR products like Oculus has been mostly with environment locked AR information.

skeptrune 1y ago

Wow, $409 getting you a AR display that's viable to hack on is crazy. I can't believe how approachable hardware is for tinkerers some time.

verdverm 1y ago

https://immersed.com https://immersed.com is a great piece of software that gives you multiple virtual screens. It supports all major OS and VR devices, with their own https://visor.com https://visor.com headset coming soon (tm)

wkat4242 1y ago

Nice! I wish this worked on my XReal Air :( It does have IMU sensors because their own app (nebula) uses them, and even does 3D effects. By the way I find my Quest 3 way more comfortable than the Xreal. The problem with the xreal is, the prescription inlays are too small so I can see only a part of the display sharp unless I put in lenses every time. And it is kinda heavy on my nose and ears. the Quest is super nice, has a wider FoV. But it's heavier and bulkier to bring along of course. PS I had no idea you could connect an HDMI dongle to the quest. I'll check that out.

blensor 1y ago
If I had XReal I could look into that too, it's probably pretty similar to how it works for the Viture. Regarding the HDMI on Quest 3, I believe it was sometime last year when they made that possible https://www.meta.com/help/quest/1350679463006443 https://www.meta.com/help/quest/1350679463006443
- wkat4242 1y ago
  Yeah I bet it's real similar. But I understand. Thanks for the heads up about that Quest thing, I actually have some of those adapters so I will give it a try.

benhoff 1y ago

Did you fix the op5plus HDMI in driver? Thing was unstable like 8-12 months ago

blensor 1y ago
So far I haven't had any issue but I haven't done long tests yet. I'm using BredOS not their official OrangePI images though

Aaron2222 1y ago

> What other solutions do exist out there? Xreal One and One Pro have this built into the glasses themselves. The Xreal Eye camera is meant to add on-glassed 6DOF tracking as well.