8 ms·

If anyone is interested in this field, we need your help with real-time reconstruction. Currently a huge challenge is in real-time reconstruction. Approaches i

If anyone is interested in this field, we need your help with real-time reconstruction. Currently a huge challenge is in real-time reconstruction. Approaches involve estimating point clouds from images then optimizing splats on those. Other approaches are using SLAM like LiDAR to have distance of points to the camera but then still optimizing on that. Optimization is producing good results but takes iterations that are not suitable for real-time. Pixel-wise splat estimation with iPhone LiDAR could produce good results but need help and expertise

heliophobicdude 3y ago
If anyone is up to the challenge, Apple provides an excellent sample project available for download with low level access and operations on the LiDAR points here and shading all for an iPhone app: https://developer.apple.com/documentation/avfoundation/additional_data_capture/capturing_depth_using_the_lidar_camera https://developer.apple.com/documentation/avfoundation/addit...
dwallin 3y ago
Is this paper achieving what you mean: https://arxiv.org/pdf/2311.16728.pdf https://arxiv.org/pdf/2311.16728.pdf Caveat, this field is far outside my wheelhouse. So if not, I'd love to understand more about the nuances here.
- markisus 3y ago
  I don’t think this is realtime reconstruction, meaning that changes in the scene will not be represented in the reconstructed map immediately.
  - smusamashah 3y ago
    The Splatter Image paper recently featured on HN seems to do that though https://news.ycombinator.com/item?id=38720226 https://news.ycombinator.com/item?id=38720226
londons_explore 3y ago
You say you need help... But doesn't this paper solve the problem? Ie. 60fps and quality looks pretty much perfect to me. What's next to do?
- brookman64k 3y ago
  The paper says that training / reconstruction takes 4 hours. While inference runs at 60fps reconstruction is far from real-time.
countWSS 3y ago
Perhaps synthetic generation could help? something like InseRF or Instruct-NeRF2NeRF could be used to generate components and variations.
smusamashah 3y ago
How about Splhow about https://szymanowiczs.github.io/splatter-image https://szymanowiczs.github.io/splatter-image it works for single image to 3d but claims that adding more images improves the quality

heliophobicdude 3y ago

If anyone is up to the challenge, Apple provides an excellent sample project available for download with low level access and operations on the LiDAR points here and shading all for an iPhone app: https://developer.apple.com/documentation/avfoundation/additional_data_capture/capturing_depth_using_the_lidar_camera https://developer.apple.com/documentation/avfoundation/addit...

dwallin 3y ago

Is this paper achieving what you mean: https://arxiv.org/pdf/2311.16728.pdf https://arxiv.org/pdf/2311.16728.pdf Caveat, this field is far outside my wheelhouse. So if not, I'd love to understand more about the nuances here.

markisus 3y ago
I don’t think this is realtime reconstruction, meaning that changes in the scene will not be represented in the reconstructed map immediately.
- smusamashah 3y ago
  The Splatter Image paper recently featured on HN seems to do that though https://news.ycombinator.com/item?id=38720226 https://news.ycombinator.com/item?id=38720226

londons_explore 3y ago

You say you need help... But doesn't this paper solve the problem? Ie. 60fps and quality looks pretty much perfect to me. What's next to do?

brookman64k 3y ago
The paper says that training / reconstruction takes 4 hours. While inference runs at 60fps reconstruction is far from real-time.

countWSS 3y ago

Perhaps synthetic generation could help? something like InseRF or Instruct-NeRF2NeRF could be used to generate components and variations.

smusamashah 3y ago

How about Splhow about https://szymanowiczs.github.io/splatter-image https://szymanowiczs.github.io/splatter-image it works for single image to 3d but claims that adding more images improves the quality