8 ms·

Show HN: I've made a Monte-Carlo raytracer for glTF scenes in WebGPU

131 points by lisyarus 2y ago

This is a GPU "software" raytracer (i.e. using manual ray-scene intersections and not RTX) written using the WebGPU API that renders glTF scenes. It supports many materials, textures, material & normal mapping, and heavily relies on multiple importance sampling to speed up convergence.

deleted 2y ago
[deleted]
modeless 2y ago
Do you have a link that runs in the browser?
- lisyarus 2y ago
  Nope, this project is desktop-only
  - firtoz 2y ago
    [flagged]
  - modeless 2y ago
    You should try building it with Emscripten. SDL2 is supported.
crazygringo 2y ago
This is a completely side question, but just because it always astonishes me how "real" raytraced scenes can look in terms of lighting, but it's too complex/slow for video games. How far have we gotten in terms of training AI models on raytraced lighting, to simulate it but fast enough for video games? Training an AI not on rendered scenes from any particular viewpoint, but rather on how light and shadows would be "baked into" textures? Because what raytracing excels at is the overall realism of diffuse light. And it seems like the kind of thing AI would be good at learning? I've always though, e.g. when looking at the shadows trees cast, I couldn't care less if the each leaf shape in the shadow is accurate or entirely hallucinated. The important things seem to be a combination of the overall light diffusion, combined with correct nearby shadow shapes for objects. Which is seems AI would excel at?
- Etheryte 2y ago
  At any reasonable quality, AI is even more expensive than raytracing. A simple intuition for this is the fact that you can easily run a raytracer on consumer hardware, even if at low FPS, meanwhile you need a beefy setup to run most AI models and they still take a while.
  - toshinoriyagi 2y ago
    While some very large models may need beefy hardware, there are multiple forms of deep learning used for similar purposes: Nvidia's DLSS is a neural network that upscales images so that games may be rendered quickly at lower resolutions, and than upscaled to the display resolution in less total time than rendering natively at the display resolution. Nvidia's DLDSR downscales a greater-than-native resolution image faster than typical downscaling algorithms used in DSR. Nvidia's RTX HDR is a post-processing filter that takes an sRGB image and converts it to HDR. So, it is very likely that a model that converts rasterized images to raytraced versions is possible, and fast. The most likely road block is the lack of a quality dataset for training such a model. Not all games have ray tracing, and even fewer have quality implementations.
    - mywittyname 2y ago
      > So, it is very likely that a model that converts rasterized images to raytraced versions is possible, and fast. How would this even work and not just be a DLSS derivative? The magic of ray tracing is the ability to render light sources and reflections that are not in the scene. So where is the information coming from that the algorithm would use to place and draw the lights, shadows, reflections, etc? I'm not asking to be snarky. I can usually "get there from here" when it comes to theoretical technology, but I can't work out how a raster image would contain enough data to allow for accurate ray tracing to be applied for objects whose effects are only included due to ray tracing.
    - jsheard 2y ago
      To be clear DLSS is a very different beast than your typical AI upscaler, it uses the principle of temporal reuse where real samples from previous frames are combined with samples from the current frame in order to converge towards a higher resolution over time. It's not guessing new samples out of thin air, just guessing whether old samples are still usable, which is why DLSS is so fast and accurate compared to general purpose AI upscalers and why you can't use DLSS on images or videos.
omolobo 2y ago
It's a mega-kernel, so you'll get poor occupancy past the first bounce. A better strategy is to shoot, sort, and repeat, which then also allows you to squeeze in an adaptive sampler in the middle. > // No idea where negative values come from :( I don't know, but: > newRay.origin += sign(dot(newRay.direction, geometryNormal)) * geometryNormal * 1e-4; The new origin should be along the reflected ray, not along the direction of the normal. This line basically adds the normal (with a sign) to the origin (intersection point), which seems odd. Poor's man way to find where the negatives come from is to max(0,...) stuff until you find it.
- lisyarus 2y ago
  > It's a mega-kernel, so you'll get poor occupancy past the first bounce Sure! If you look into the to-do list, there's a "wavefront path tracer" entry :) > new origin should be along the reflected ray I've found that doing it the way I'm doing it works better for preventing self-intersections. Might be worth investigating, though.
  - omolobo 2y ago
    It probably works better when the reflected ray is almost tangent to the surface. But that should be an epsilon case.
- TomClabault 2y ago
  > A better strategy is to shoot, sort, and repeat Do we have good sorting strategy whose costs are amortized yet? Meister 2020 (https://meistdan.github.io/publications/raysorting/paper.pdf https://meistdan.github.io/publications/raysorting/paper.pdf) shows that the hard part is actually to hide the cost of the sorting. > squeeze in an adaptive sampler in the middle. Can you expand on that? How does that work? I only know of adaptive sampling in screen space where you shoot more or less rays to certain pixels based on their estimated variance so far.
  - TomClabault 2y ago
    After reading this paper a bit more it seems that the it focuses on simple scenes and simple materials only, which a bit unfortunate. This is exactly where ray reordering overhead is going to be the most problematic. They also do talk about the potential of ray reordering for complex scenes and complex materials in the paper (because reordering helps with shading divergence since "all" reordered rays are pretty much going to hit the same material). So maybe ray reordering isn't dead just yet. Probably would have to try that at some point...
deleted 2y ago
[deleted]
pjmlp 2y ago
WebGPU projects that don't provide browser examples are kind of strange, then better use Vulkan or whatever.
- lisyarus 2y ago
  See my answer to artemonster above.
- sspiff 2y ago
  WegGPU is a way nicer HAL if you're not an experienced graphics engineer. So even if you only target desktops, it's a valid choice. On the web, WebGPU is only supported by Chrome-based browser engines at this point, and a lot of software developers us Firefox (and don't really like encouraging a browser monoculture), so it doesn't make a ton of sense to target browser based WebGPU for some people at this point.
  - pjmlp 2y ago
    The answer is middleware engine, all of them with much nicer tooling available, without the constraints of a browser sandboxing design, for 2017 graphics APIs minimum common denominator.
  - lisyarus 2y ago
    It's not as much about experience as it is about trade-offs. I've worked a lot with Vulkan and it's an incredible API, but when you're working alone and you don't have the goal of squeezing 250% performance out of your GPU on dozens of different GPU architectures, your performance becomes pretty much independent of a specific graphics API (unless your API doesn't support some stuff like multi-draw-indirect, etc).
artemonster 2y ago
> "GPU "software" raytracer" > WebGPU > this project is desktop-only Boss, I am confused, boss.
- lisyarus 2y ago
  I'm using WebGPU as a nice modern graphics API that is at the same time much more user-friendly and easier to use compared to e.g. Vulkan. I'm using a desktop implementation of WebGPU called wgpu, via it's C bindings called wgpu-native. My browser doesn't support WebGPU properly yet, so I don't really care about running this thing in browser.
  - firtoz 2y ago
    That's a fascinating approach. And it gets me a bit sad about the state of WebGPU, however hopefully that'll be resolved soon... I also on Linux am impatiently waiting for WebGPU to be supported on my browser.
bezdomniy 2y ago
Very cool. I did a similar project with wgpu in Rust - https://github.com/bezdomniy/Rengin https://github.com/bezdomniy/Rengin nice to find your projects to see where I can improve!
bella964 2y ago
[dead]

131 points by lisyarus 2y ago

deleted 2y ago

[deleted]

modeless 2y ago

Do you have a link that runs in the browser?

lisyarus 2y ago
Nope, this project is desktop-only
- firtoz 2y ago
  [flagged]
- modeless 2y ago
  You should try building it with Emscripten. SDL2 is supported.

crazygringo 2y ago

This is a completely side question, but just because it always astonishes me how "real" raytraced scenes can look in terms of lighting, but it's too complex/slow for video games. How far have we gotten in terms of training AI models on raytraced lighting, to simulate it but fast enough for video games? Training an AI not on rendered scenes from any particular viewpoint, but rather on how light and shadows would be "baked into" textures? Because what raytracing excels at is the overall realism of diffuse light. And it seems like the kind of thing AI would be good at learning? I've always though, e.g. when looking at the shadows trees cast, I couldn't care less if the each leaf shape in the shadow is accurate or entirely hallucinated. The important things seem to be a combination of the overall light diffusion, combined with correct nearby shadow shapes for objects. Which is seems AI would excel at?

Etheryte 2y ago
At any reasonable quality, AI is even more expensive than raytracing. A simple intuition for this is the fact that you can easily run a raytracer on consumer hardware, even if at low FPS, meanwhile you need a beefy setup to run most AI models and they still take a while.
- toshinoriyagi 2y ago
  While some very large models may need beefy hardware, there are multiple forms of deep learning used for similar purposes: Nvidia's DLSS is a neural network that upscales images so that games may be rendered quickly at lower resolutions, and than upscaled to the display resolution in less total time than rendering natively at the display resolution. Nvidia's DLDSR downscales a greater-than-native resolution image faster than typical downscaling algorithms used in DSR. Nvidia's RTX HDR is a post-processing filter that takes an sRGB image and converts it to HDR. So, it is very likely that a model that converts rasterized images to raytraced versions is possible, and fast. The most likely road block is the lack of a quality dataset for training such a model. Not all games have ray tracing, and even fewer have quality implementations.
  - mywittyname 2y ago
    > So, it is very likely that a model that converts rasterized images to raytraced versions is possible, and fast. How would this even work and not just be a DLSS derivative? The magic of ray tracing is the ability to render light sources and reflections that are not in the scene. So where is the information coming from that the algorithm would use to place and draw the lights, shadows, reflections, etc? I'm not asking to be snarky. I can usually "get there from here" when it comes to theoretical technology, but I can't work out how a raster image would contain enough data to allow for accurate ray tracing to be applied for objects whose effects are only included due to ray tracing.
  - jsheard 2y ago
    To be clear DLSS is a very different beast than your typical AI upscaler, it uses the principle of temporal reuse where real samples from previous frames are combined with samples from the current frame in order to converge towards a higher resolution over time. It's not guessing new samples out of thin air, just guessing whether old samples are still usable, which is why DLSS is so fast and accurate compared to general purpose AI upscalers and why you can't use DLSS on images or videos.

omolobo 2y ago

It's a mega-kernel, so you'll get poor occupancy past the first bounce. A better strategy is to shoot, sort, and repeat, which then also allows you to squeeze in an adaptive sampler in the middle. > // No idea where negative values come from :( I don't know, but: > newRay.origin += sign(dot(newRay.direction, geometryNormal)) * geometryNormal * 1e-4; The new origin should be along the reflected ray, not along the direction of the normal. This line basically adds the normal (with a sign) to the origin (intersection point), which seems odd. Poor's man way to find where the negatives come from is to max(0,...) stuff until you find it.

lisyarus 2y ago
> It's a mega-kernel, so you'll get poor occupancy past the first bounce Sure! If you look into the to-do list, there's a "wavefront path tracer" entry :) > new origin should be along the reflected ray I've found that doing it the way I'm doing it works better for preventing self-intersections. Might be worth investigating, though.
- omolobo 2y ago
  It probably works better when the reflected ray is almost tangent to the surface. But that should be an epsilon case.
TomClabault 2y ago
> A better strategy is to shoot, sort, and repeat Do we have good sorting strategy whose costs are amortized yet? Meister 2020 (https://meistdan.github.io/publications/raysorting/paper.pdf https://meistdan.github.io/publications/raysorting/paper.pdf) shows that the hard part is actually to hide the cost of the sorting. > squeeze in an adaptive sampler in the middle. Can you expand on that? How does that work? I only know of adaptive sampling in screen space where you shoot more or less rays to certain pixels based on their estimated variance so far.
- TomClabault 2y ago
  After reading this paper a bit more it seems that the it focuses on simple scenes and simple materials only, which a bit unfortunate. This is exactly where ray reordering overhead is going to be the most problematic. They also do talk about the potential of ray reordering for complex scenes and complex materials in the paper (because reordering helps with shading divergence since "all" reordered rays are pretty much going to hit the same material). So maybe ray reordering isn't dead just yet. Probably would have to try that at some point...

deleted 2y ago

[deleted]

pjmlp 2y ago

WebGPU projects that don't provide browser examples are kind of strange, then better use Vulkan or whatever.

lisyarus 2y ago
See my answer to artemonster above.
sspiff 2y ago
WegGPU is a way nicer HAL if you're not an experienced graphics engineer. So even if you only target desktops, it's a valid choice. On the web, WebGPU is only supported by Chrome-based browser engines at this point, and a lot of software developers us Firefox (and don't really like encouraging a browser monoculture), so it doesn't make a ton of sense to target browser based WebGPU for some people at this point.
- pjmlp 2y ago
  The answer is middleware engine, all of them with much nicer tooling available, without the constraints of a browser sandboxing design, for 2017 graphics APIs minimum common denominator.
- lisyarus 2y ago
  It's not as much about experience as it is about trade-offs. I've worked a lot with Vulkan and it's an incredible API, but when you're working alone and you don't have the goal of squeezing 250% performance out of your GPU on dozens of different GPU architectures, your performance becomes pretty much independent of a specific graphics API (unless your API doesn't support some stuff like multi-draw-indirect, etc).

artemonster 2y ago

> "GPU "software" raytracer" > WebGPU > this project is desktop-only Boss, I am confused, boss.

lisyarus 2y ago
I'm using WebGPU as a nice modern graphics API that is at the same time much more user-friendly and easier to use compared to e.g. Vulkan. I'm using a desktop implementation of WebGPU called wgpu, via it's C bindings called wgpu-native. My browser doesn't support WebGPU properly yet, so I don't really care about running this thing in browser.
- firtoz 2y ago
  That's a fascinating approach. And it gets me a bit sad about the state of WebGPU, however hopefully that'll be resolved soon... I also on Linux am impatiently waiting for WebGPU to be supported on my browser.

bezdomniy 2y ago

Very cool. I did a similar project with wgpu in Rust - https://github.com/bezdomniy/Rengin https://github.com/bezdomniy/Rengin nice to find your projects to see where I can improve!

bella964 2y ago

[dead]