duckworthd

5 ms·

1.
by duckworthd 3y ago
Thank you :)
2.
by duckworthd 3y ago
I hope so, but it'll be a good while before we can release anything. We have tight dependencies to other not-yet-OSS libraries, and until they're released, our won't work either.
3.
by duckworthd 3y ago
Maybe! That's the seed of a completely different research paper :)
4.
by duckworthd 3y ago
The big difference is access to fisheye lenses a burst mode that can be run for minutes at a time, and the ability to minimize the amount of camera post processing. In principle, the capture could be done with a smartphone, but the experien
5.
by duckworthd 3y ago
You're looking at something called a "neural radiance field" backed by a sparse, low resolution voxel grid and a dense high resolution triplane grid. That's a bit of a word soup, but you can think of it like a glowing fo
6.
by duckworthd 3y ago
What's worked well for me: Find a way to put what AI/ML on your critical path. Think of it like learning a new language: classes, lessons, and watching TV helps, but nothing works like full-on immersion. In the context of AI/
7.
by duckworthd 3y ago
Yes, I hope so! But it'll take at least a few months of work. We have some tight dependencies to not-yet-open-sourced code, and until that's released, any code we put out will be dead on arrival. In the meantime, feel free to expl
8.
by duckworthd 3y ago
This should absolutely be possible! The hard part is making it look natural: NeRF models (including SMERF) have no explicit materials or lighting. That means that any character inserted into the game will look out of place.
9.
by duckworthd 3y ago
Oof, there's a lot of machinery here. It depends a lot on your academic background. I'd recommend starting with a tutorial on neural radiance fields, aka NeRF, ( https://sites.google.com/berkeley.edu/nerf-tutor
10.
by duckworthd 3y ago
In terms of the live viewer, there's actually no limit on footprint size. 300 m^2 is simply the biggest indoor capture we had!
11.
by duckworthd 3y ago
Oops, I need to update the license files. Our code is released under the Apache 2.0 license, as in this repo: https://github.com/google-research/google-research/blob/mast...
12.
by duckworthd 3y ago
The viewer biases towards assets closer to user's camera (otherwise you'd have to load the whole scene!). We tried training SMERF with a larger number of smaller submodels, but at some point, it becomes too onerous to train and qu
13.
by duckworthd 3y ago
Thanks for the feedback! I agree, we could do better with the movement UX. A challenge for another day.
14.
by duckworthd 3y ago
You're probably thinking of 3D Gaussian Splatting (3DGS), another fantastic approach to real-time novel view synthesis. There's tons of fantastic work being built on 3DGS right now, and the dust has yet to settle with respect to w
15.
by duckworthd 3y ago
Block-NeRF is a predecessor work that helped inspire SMERF, in fact! https://waymo.com/research/block-nerf/
16.
by duckworthd 3y ago
I hope to release the code in the new year, but we have some big dependencies that need to be released worse. In the meantime, you can already begin hacking on the live viewer, https://github.com/smerf-3d/smerf-3d.githu
17.
by duckworthd 3y ago
3D understanding as a field is very much in its infancy. Good work is being done in this area, but we've got a long ways to go yet. SMERF is all about "view synthesis" -- rendering realistic images -- with no attempt at seman
18.
by duckworthd 3y ago
Interesting effect. It does look very voxel-y. I'm not a video game developer at heart, so I can only guess how it was implemented. I doubt NeRF models were involved, but I wouldn't be surprised if some sort of voxel discretizatio
19.
by duckworthd 3y ago
> Do you need position data to go along with the photos or just the photos? Short answer: Yes. Long answer: Yes, but it can typically be derived from images. Structure-from-motion methods are typically used to derive lens and position in
20.
by duckworthd 3y ago
Update: Code for the web viewer is here, https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...
21.
by duckworthd 3y ago
I can't say. I'm not familiar with BD in Cyberpunk.
22.
by duckworthd 3y ago
I won't say too much about this, but the amount of buzz around articles these days is more of "research today" sort of thing. Top conferences like CVPR receives thousands of submissions each year, and there's a lot of
23.
by duckworthd 3y ago
We unfortunately haven't tested our web viewer in Firefox. Let us know which platform you're running and we'll do our best to take a look in the new year (holiday vacation!). In the meantime, give it a shot in a Webkit- or Ch
24.
by duckworthd 3y ago
I hope to release code in the new year, but it'll take a while. The codebase is heavily wired into other not-yet-open-sourced libraries, and it'll take a while to disentangle them.
25.
by duckworthd 3y ago
Collaboration is a thing at the Big G :)
26.
by duckworthd 3y ago
Be careful with this one! Luma's offering requires that the camera follow the recorded video path. Our method lets the camera go wherever you desire!
27.
by duckworthd 3y ago
Nice discovery :). Check the developer console: it'll tell you.
28.
by duckworthd 3y ago
Thank you!
29.
by duckworthd 3y ago
All these details and more in our technical paper! In short: SMERF training takes much longer, SMERF rendering is nearly as fast as 3DGS when a CUDA GPU is available, and quality is visibly higher than 3DGS on large scenes and slightly high
30.
by duckworthd 3y ago
Not yet, as far as I'm aware. The current flow involves a DSLR for capture, COLMAP for camera parameter estimation, one codebase for training a teacher model, our codebase for training SMERF, and our web viewer for rendering models. So

More ›