6 ms·

With less than an inch of separation, is the sense of depth even perceptible?

by ticklemyelmo 3y ago

varjag 3y ago
As the two cameras have very different focal lengths you get pronounced parallax effect that can be exploited in post.
- ricardobeat 3y ago
  Two focal lengths at the same physical distance to the subject have exactly the same perspective (i.e. if you crop them to the same area they will look the same). There is no extra information to be had from that.
  - varjag 3y ago
    The depth information that can be obtained from differences in angular position/size of objects within cameras' FOV. There's a reason a photo taken with a 28mm doesn't look the same as with a 50 a few steps back.
    - simondw 3y ago
      But one of the lenses isn't a few steps back from the other.
    - FPGAhacker 3y ago
      A cropped 28mm is indistinguishable from a 50mm or any other larger size, relative photo site receptor size notwithstanding.
      - deleted 3y ago
        [deleted]
      - deleted 3y ago
        [deleted]
    - ricardobeat 3y ago
      > a few steps back Exactly. The steps back change perspective, not the lenses. That’s what I was trying to say above. In the iPhone both lenses are at the same distance to the subject.
      - deleted 3y ago
        [deleted]
abracadaniel 3y ago
Early reviews indicate it is, as some reviewers have had access to spatial video taken from a phone, but I’m not sure if those were ideal conditions or just ad-hoc.
Reubend 3y ago
Pure speculation: when combined with the LIDAR depth sensor, the two cameras probably don't need as much physical separation to accurately create a depth map. The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
- TaylorAlexander 3y ago
  I was wondering about the use of the lidar sensor. Notably they do not say they are using it, but maybe they just wanted to keep it simple? Idk seems weird not to use lidar but also seems weird not to mention it if they are using it.
- aYsY4dDQ2NrcNzA 3y ago
  3D movies have existed for decades, without adjustable viewing angles.
  - mlyle 3y ago
    But if you have eyes 50mm apart, and source material from cameras 15mm apart (plus other depth information), you'll need to in-paint a small amount where your eyes could see "around" something and the cameras can't.
    - coldtea 3y ago
      Or you can make do with 15mm-apart worth of "around"? It's still more "spatial"/3D than a regular (single lens) image. Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
      - mlyle 3y ago
        > Or you can make do with 15mm-apart worth of "around"? You need to move the close objects further apart in left/right than they are in the camera. Then you need to fill the newly empty areas with something. > Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video. This doesn't affect anything.
golergka 3y ago
I would expect some ML magic in the image processing pipeline that would make it pop out.
klausa 3y ago
One of those lenses is a Ultra-Wide though, with a _very_ different FoV than the other one.
- brookst 3y ago
  It uses a crop from the center. Not sure if that crop has same FOV as the other lens though. I’d expect so?
0xDEF 3y ago
One of the two cameras is the ultra-wide camera so it gets some additional parallax and visual information than just the separation to the other camera.
- ryandamm 3y ago
  That’s not how parallax works. The wider field of view of the ultra wide camera will show some of the scene that the other camera doesn’t see, but over overlapping parts of the scene the parallax is a strict function of the location of the two lenses’ entrance windows.
ynniv 3y ago
They're probably fusing both lenses with the Lidar and some other tricks to reliably compute a dense surface. That would explain their suggestion not to move the camera very much, as that would cause a large portion of the mesh to be rebuilt. A blogger exported what appears to be two side by side videos, so maybe the view really is narrow or reconstruction happens at playback. There might also be Lidar data in there that he didn't notice. Apple bought C3 Technologies a decade ago, and they use this technique to fuse photos from low flying charters to produce the 3d view in Apple Maps. [ Paper: https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstract https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstra... ] [ Coverage: https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3d-mapping-company-c3-technologies-looking-to-take-ios-maps-to-the-next-level/ https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3... ] [ Similar: https://web.stanford.edu/class/ee367/Winter2021/projects/report_blankenberg.pdf https://web.stanford.edu/class/ee367/Winter2021/projects/rep... ]

With less than an inch of separation, is the sense of depth even perceptible?

varjag 3y ago

As the two cameras have very different focal lengths you get pronounced parallax effect that can be exploited in post.

ricardobeat 3y ago
Two focal lengths at the same physical distance to the subject have exactly the same perspective (i.e. if you crop them to the same area they will look the same). There is no extra information to be had from that.
- varjag 3y ago
  The depth information that can be obtained from differences in angular position/size of objects within cameras' FOV. There's a reason a photo taken with a 28mm doesn't look the same as with a 50 a few steps back.
  - simondw 3y ago
    But one of the lenses isn't a few steps back from the other.
  - FPGAhacker 3y ago
    A cropped 28mm is indistinguishable from a 50mm or any other larger size, relative photo site receptor size notwithstanding.
    - deleted 3y ago
      [deleted]
    - deleted 3y ago
      [deleted]
  - ricardobeat 3y ago
    > a few steps back Exactly. The steps back change perspective, not the lenses. That’s what I was trying to say above. In the iPhone both lenses are at the same distance to the subject.
    - deleted 3y ago
      [deleted]

abracadaniel 3y ago

Early reviews indicate it is, as some reviewers have had access to spatial video taken from a phone, but I’m not sure if those were ideal conditions or just ad-hoc.

Reubend 3y ago

Pure speculation: when combined with the LIDAR depth sensor, the two cameras probably don't need as much physical separation to accurately create a depth map. The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.

TaylorAlexander 3y ago
I was wondering about the use of the lidar sensor. Notably they do not say they are using it, but maybe they just wanted to keep it simple? Idk seems weird not to use lidar but also seems weird not to mention it if they are using it.
aYsY4dDQ2NrcNzA 3y ago
3D movies have existed for decades, without adjustable viewing angles.
- mlyle 3y ago
  But if you have eyes 50mm apart, and source material from cameras 15mm apart (plus other depth information), you'll need to in-paint a small amount where your eyes could see "around" something and the cameras can't.
  - coldtea 3y ago
    Or you can make do with 15mm-apart worth of "around"? It's still more "spatial"/3D than a regular (single lens) image. Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
    - mlyle 3y ago
      > Or you can make do with 15mm-apart worth of "around"? You need to move the close objects further apart in left/right than they are in the camera. Then you need to fill the newly empty areas with something. > Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video. This doesn't affect anything.

golergka 3y ago

I would expect some ML magic in the image processing pipeline that would make it pop out.

klausa 3y ago

One of those lenses is a Ultra-Wide though, with a _very_ different FoV than the other one.

brookst 3y ago
It uses a crop from the center. Not sure if that crop has same FOV as the other lens though. I’d expect so?

0xDEF 3y ago

One of the two cameras is the ultra-wide camera so it gets some additional parallax and visual information than just the separation to the other camera.

ryandamm 3y ago
That’s not how parallax works. The wider field of view of the ultra wide camera will show some of the scene that the other camera doesn’t see, but over overlapping parts of the scene the parallax is a strict function of the location of the two lenses’ entrance windows.

ynniv 3y ago

They're probably fusing both lenses with the Lidar and some other tricks to reliably compute a dense surface. That would explain their suggestion not to move the camera very much, as that would cause a large portion of the mesh to be rebuilt. A blogger exported what appears to be two side by side videos, so maybe the view really is narrow or reconstruction happens at playback. There might also be Lidar data in there that he didn't notice. Apple bought C3 Technologies a decade ago, and they use this technique to fuse photos from low flying charters to produce the 3d view in Apple Maps. [ Paper: https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstract https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstra... ] [ Coverage: https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3d-mapping-company-c3-technologies-looking-to-take-ios-maps-to-the-next-level/ https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3... ] [ Similar: https://web.stanford.edu/class/ee367/Winter2021/projects/report_blankenberg.pdf https://web.stanford.edu/class/ee367/Winter2021/projects/rep... ]