6 ms·

Show HN: 3DGS.cpp – performant, cross platform Gaussian Splatting with Vulkan

50 points by 0x02A 3y ago

[deleted]

deleted 3y ago
[deleted]
Cloudef 3y ago
I wonder if this technique could be used to bring back PS1 esque pre-rendered background style games but with actual depth
- blarg1 3y ago
  Probably, I saw one where they took scenes from movies and recreated 3D versions of them.
- 0x02A 3y ago
  The original paper doesn't work well with few-shot learning. I'm assuming that there is only one camera angle for each pre-rendered background. For single image to 3D, check out DreamGaussian. [1] [1] https://arxiv.org/pdf/2309.16653.pdf https://arxiv.org/pdf/2309.16653.pdf
  - Cloudef 3y ago
    Pre-rendered in this context means you still have access to the full 3d scene so you can generate the full gaussian splatting model from that. The benefit here would be to lower the cost of rendering that complicated scene. There is a game called Fantasian (by the former FF dev) that uses real-life dioramas as backgrounds, I bet this tech would've been perfect fit for that too.
    - 0x02A 3y ago
      Oh I see. That's a pretty cool use case. Not sure how GS would perform on non-photorealistic scenarios, but certainly worth a try.
w-m 3y ago
Great project, it's nice to see so many people are building stuff with Gaussian Splatting. Just the other day I went through the whole list of known viewers on MrNeRF's awesome 3DGS resources, to find one that runs on a MacBook. I'm working on compressing 3D scenes by sorting Gaussians into 2D grids [0], and I wanted a native viewer that I could for experiments on the go.. perhaps as an alternative backend to the CUDA one in my colleague's exploratory Python viewer [1]. VulkanSplatting was the only one I could get to compile and run on my Intel MacBook. Unfortunately the feeble Intel GPU isn't able to display even the Lego scene at an interactive framerate. Do you think there's performance headroom, and that it will become possible in the future, or should I give up trying to run this on an Intel MBP? [0]: https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/ https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/ [1]: https://github.com/Florian-Barthel/gaussian_viewer https://github.com/Florian-Barthel/gaussian_viewer
- 0x02A 3y ago
  Thanks for trying it out! I haven't had the opportunity to benchmark this on an Intel Macbook. Were you able to see which kernel takes the most time? There should be a performance graph if you have the GUI enabled. For my Apple Silicon benchmarks, the main bottleneck is the parallel radix sort that sorts the Gaussians by tile and depth. I used a some shaders from a sorting library, but it has some performance gaps with SOTA parallel sort algorithms. I think fixing this would give a 1.5x overall performance boost and maybe 3x on Macbooks. Also the wave size isn't tuned for different GPUs. Another area of improvement is better management of the shared memory. Right now, we just let the driver manage it as the L1 cache. However, we could manage it manually and group Gaussian retrievals together for the same tile. This is what the official implementation does. Although 3DGS is the first radiance field with SOTA quality that runs in real-time, I think it's still quite heavy. Due to the explicit representation of the scene, a lot of operations are memory bound. If you can't get an interactive frame rate right now, it's unlikely the improvements will make a material difference. Hopefully that's where your work on compression comes in and solves the problem :)
  - w-m 3y ago
    For some reason the GUI is not showing up for me in 3DGS.cpp. (I checked out the repo, made a build folder and built with Ninja, then launched ./apps/viewer/vulkan_splatting_viewer). I still have a VulkanSplatting build from this Wednesday though. In VulkanSplatting, when looking at the Lego scene from above with the default window size, I'm getting just below 1 ms for the sorting kernel and just above 1 ms for "render", everything else is too small in the graph to register. But it only displays at a handful of fps, so it seems quite some time goes unaccounted for. Maybe spinning up Instruments could give some more insights into what's happening? I tried `cmake -G Xcode` to have that setup easily, but the Xcode CMake generation fails with CMake Error in src/shaders/CMakeLists.txt: The custom command generating 3DGS.cpp/build-xcode/shaders/shaders.h is attached to multiple targets: shaders xcode_shaders but none of these is a common dependency of the other(s). This is not allowed by the Xcode "new build system".
0x02A 3y ago
Hey HN, this project started out when I was exploring rendering radiance fields in real-time for standalone VR and AR headsets. I was frustrated by the lack of performant, yet non-CUDA, implementations. Also, this would be a good excuse to learn about compute pipelines in Vulkan. Right now, the renderer runs on Windows, Linux, macOS, iOS, and visionOS (as an iPad app). OpenXR support and an immersive visionOS app are coming soon. Training is also WIP. As we're seeing the industry adopt research in Gaussian Splatting at a fast pace, I hope this makes it easier for folks implementing Gaussian Splatting or variants in their products. Would love to hear your feedback! For more context, see previous HN discussions on Gaussian Splatting: https://news.ycombinator.com/item?id=39120016 https://news.ycombinator.com/item?id=39120016 https://news.ycombinator.com/item?id=38576974 https://news.ycombinator.com/item?id=38576974
- heliophobicdude 3y ago
  Thank you so much for open sourcing this! I've started a project to initialize each pixel in an image into a mesh of GS. I used a depth map to unproject them into space. I've been very curious what a few training iterations would do to optimize my scenes. The original 3DGS implementation is not accessible on my hardware at the moment! I really look forward to your training implementation!
  - 0x02A 3y ago
    That's super interesting. Are you trying to do single-view image to 3D?
    - heliophobicdude 3y ago
      Kinda! My ultimate goal is to be able to join multiple 3D reconstructions from a set of iPhones all in different positions! Like hemisphere. The single-view image to 3D is pretty accurate but across lots of iPhones.
      - 0x02A 3y ago
        Does colmap fail on your dataset?
pantalaimon 3y ago
I'd love to try it out, but where can I get the scene files?
- w-m 3y ago
  The original 3DGS paper offers a download of a few scenes (14 GB): https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/dat... The link is taken from the projects' GitHub page: https://github.com/graphdeco-inria/gaussian-splatting https://github.com/graphdeco-inria/gaussian-splatting
pzo 3y ago
This looks interesting. Little sad that even though it supports iOS, the license is LGPL 2.1 - seems like a dealbreaker for anyone who would like to release any project based on it to App Store. Unless EU will use bigger stick on Apple and we finally get some alternative to App Store, there is not easy way for user to swap dynamically linked libraries.
- jcelerier 3y ago
  ? There are already multiple software using the (L)GPL on the AppStore. https://en.m.wikipedia.org/wiki/List_of_free_and_open-source_iOS_applications https://en.m.wikipedia.org/wiki/List_of_free_and_open-source... Hell, webkit which is the base of all browsing on iOS is LGPL.

50 points by 0x02A 3y ago

[deleted]

deleted 3y ago

[deleted]

Cloudef 3y ago

I wonder if this technique could be used to bring back PS1 esque pre-rendered background style games but with actual depth

blarg1 3y ago
Probably, I saw one where they took scenes from movies and recreated 3D versions of them.
0x02A 3y ago
The original paper doesn't work well with few-shot learning. I'm assuming that there is only one camera angle for each pre-rendered background. For single image to 3D, check out DreamGaussian. [1] [1] https://arxiv.org/pdf/2309.16653.pdf https://arxiv.org/pdf/2309.16653.pdf
- Cloudef 3y ago
  Pre-rendered in this context means you still have access to the full 3d scene so you can generate the full gaussian splatting model from that. The benefit here would be to lower the cost of rendering that complicated scene. There is a game called Fantasian (by the former FF dev) that uses real-life dioramas as backgrounds, I bet this tech would've been perfect fit for that too.
  - 0x02A 3y ago
    Oh I see. That's a pretty cool use case. Not sure how GS would perform on non-photorealistic scenarios, but certainly worth a try.

w-m 3y ago

Great project, it's nice to see so many people are building stuff with Gaussian Splatting. Just the other day I went through the whole list of known viewers on MrNeRF's awesome 3DGS resources, to find one that runs on a MacBook. I'm working on compressing 3D scenes by sorting Gaussians into 2D grids [0], and I wanted a native viewer that I could for experiments on the go.. perhaps as an alternative backend to the CUDA one in my colleague's exploratory Python viewer [1]. VulkanSplatting was the only one I could get to compile and run on my Intel MacBook. Unfortunately the feeble Intel GPU isn't able to display even the Lego scene at an interactive framerate. Do you think there's performance headroom, and that it will become possible in the future, or should I give up trying to run this on an Intel MBP? [0]: https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/ https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/ [1]: https://github.com/Florian-Barthel/gaussian_viewer https://github.com/Florian-Barthel/gaussian_viewer

0x02A 3y ago
Thanks for trying it out! I haven't had the opportunity to benchmark this on an Intel Macbook. Were you able to see which kernel takes the most time? There should be a performance graph if you have the GUI enabled. For my Apple Silicon benchmarks, the main bottleneck is the parallel radix sort that sorts the Gaussians by tile and depth. I used a some shaders from a sorting library, but it has some performance gaps with SOTA parallel sort algorithms. I think fixing this would give a 1.5x overall performance boost and maybe 3x on Macbooks. Also the wave size isn't tuned for different GPUs. Another area of improvement is better management of the shared memory. Right now, we just let the driver manage it as the L1 cache. However, we could manage it manually and group Gaussian retrievals together for the same tile. This is what the official implementation does. Although 3DGS is the first radiance field with SOTA quality that runs in real-time, I think it's still quite heavy. Due to the explicit representation of the scene, a lot of operations are memory bound. If you can't get an interactive frame rate right now, it's unlikely the improvements will make a material difference. Hopefully that's where your work on compression comes in and solves the problem :)
- w-m 3y ago
  For some reason the GUI is not showing up for me in 3DGS.cpp. (I checked out the repo, made a build folder and built with Ninja, then launched ./apps/viewer/vulkan_splatting_viewer). I still have a VulkanSplatting build from this Wednesday though. In VulkanSplatting, when looking at the Lego scene from above with the default window size, I'm getting just below 1 ms for the sorting kernel and just above 1 ms for "render", everything else is too small in the graph to register. But it only displays at a handful of fps, so it seems quite some time goes unaccounted for. Maybe spinning up Instruments could give some more insights into what's happening? I tried `cmake -G Xcode` to have that setup easily, but the Xcode CMake generation fails with CMake Error in src/shaders/CMakeLists.txt: The custom command generating 3DGS.cpp/build-xcode/shaders/shaders.h is attached to multiple targets: shaders xcode_shaders but none of these is a common dependency of the other(s). This is not allowed by the Xcode "new build system".

0x02A 3y ago

Hey HN, this project started out when I was exploring rendering radiance fields in real-time for standalone VR and AR headsets. I was frustrated by the lack of performant, yet non-CUDA, implementations. Also, this would be a good excuse to learn about compute pipelines in Vulkan. Right now, the renderer runs on Windows, Linux, macOS, iOS, and visionOS (as an iPad app). OpenXR support and an immersive visionOS app are coming soon. Training is also WIP. As we're seeing the industry adopt research in Gaussian Splatting at a fast pace, I hope this makes it easier for folks implementing Gaussian Splatting or variants in their products. Would love to hear your feedback! For more context, see previous HN discussions on Gaussian Splatting: https://news.ycombinator.com/item?id=39120016 https://news.ycombinator.com/item?id=39120016 https://news.ycombinator.com/item?id=38576974 https://news.ycombinator.com/item?id=38576974

heliophobicdude 3y ago
Thank you so much for open sourcing this! I've started a project to initialize each pixel in an image into a mesh of GS. I used a depth map to unproject them into space. I've been very curious what a few training iterations would do to optimize my scenes. The original 3DGS implementation is not accessible on my hardware at the moment! I really look forward to your training implementation!
- 0x02A 3y ago
  That's super interesting. Are you trying to do single-view image to 3D?
  - heliophobicdude 3y ago
    Kinda! My ultimate goal is to be able to join multiple 3D reconstructions from a set of iPhones all in different positions! Like hemisphere. The single-view image to 3D is pretty accurate but across lots of iPhones.
    - 0x02A 3y ago
      Does colmap fail on your dataset?

pantalaimon 3y ago

I'd love to try it out, but where can I get the scene files?

w-m 3y ago
The original 3DGS paper offers a download of a few scenes (14 GB): https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/dat... The link is taken from the projects' GitHub page: https://github.com/graphdeco-inria/gaussian-splatting https://github.com/graphdeco-inria/gaussian-splatting

pzo 3y ago

This looks interesting. Little sad that even though it supports iOS, the license is LGPL 2.1 - seems like a dealbreaker for anyone who would like to release any project based on it to App Store. Unless EU will use bigger stick on Apple and we finally get some alternative to App Store, there is not easy way for user to swap dynamically linked libraries.

jcelerier 3y ago
? There are already multiple software using the (L)GPL on the AppStore. https://en.m.wikipedia.org/wiki/List_of_free_and_open-source_iOS_applications https://en.m.wikipedia.org/wiki/List_of_free_and_open-source... Hell, webkit which is the base of all browsing on iOS is LGPL.