5 ms·

Hi, I'm a developer at NexOptic[0] and we are a company that was deeply inspired by this paper when it was first published. We had a lot of early success when a

by y7r4m 6y ago

Hi, I'm a developer at NexOptic[0] and we are a company that was deeply inspired by this paper when it was first published. We had a lot of early success when attempting to replicate the results on our own and ended up running with it, and extending it into our own product line under our ALIIS brand of AI powered solutions. For those curious, our current approach differs in some very significant ways to the author's implementation, such as performing our denoising and enhancement on a raw bayer -> raw bayer basis with a separate pipeline for tone mapping, white-balance, and HDR enhancement. As well, we explored a fair amount of different architectures for the CNN and came to the conclusion that a heavily mixed multi-resolution layering solution produces superior results. As other commentators have pointed out, the most interesting part of it is really coming to terms that, as war1025 pointed out, "The message has an entropy limit, but the message isn't the whole dataset." It is incredibly powerful what can be accomplished with even extraordinarily noisy information as long as one has a extremely "knowledge packed" prior. If anyone has any questions about our research in this space, please feel free to ask. [0] https://nexoptic.com/artificialintelligence/ https://nexoptic.com/artificialintelligence/

dbeardsl 6y ago
Sounds like you have taken this pretty far, do you have any example outputs? The only one I found via your website was a PDF with a low res image with no context.
- y7r4m 6y ago
  Sure, we have a short deck[0] that gives an intro to our noise reduction, and also here is a folder[1] that shows off a calibration target we captured with a actual camera (20ms, f22) in low-light conditions: (original, 100x gain, 100x gain + ALIIS) We also have some more raw data[2] where there is the original bayer data available as .npy files with 40db analog gain applied, however I think the calibration targets show off what we are able to accomplish more dramatically. Finally, we have a short youtube video[3] that shows off how it works when applied to video. [0] https://www.dropbox.com/s/0bm4dpxhn35vkhe/ALLIS_Investor_Intro_Q12020.pdf https://www.dropbox.com/s/0bm4dpxhn35vkhe/ALLIS_Investor_Int... [1] https://www.dropbox.com/sh/k861saentyq1cs6/AADmO7X_L49nUkEI_taE7e8Sa https://www.dropbox.com/sh/k861saentyq1cs6/AADmO7X_L49nUkEI_... [2] https://www.dropbox.com/sh/fv8omdf4fbx59m9/AABDnf6sdvv7rtIml-li_Go0a https://www.dropbox.com/sh/fv8omdf4fbx59m9/AABDnf6sdvv7rtIml... [3] https://youtu.be/99Cq1bWCmMM https://youtu.be/99Cq1bWCmMM
randyrand 6y ago
It would be really cool if you could feed the network a photo with flash that it could use for gathering more information, but then recreated a photo without flash from the non-flash raw. Often flash is not the look people are going for, but would be okay with the flash firing in order to improve the non-flash photo.
- y7r4m 6y ago
  Absolutely! We recently rebranded our AI solutions from ALLIS (Advanced Low Light Imagine Solution) to ALIIS (All Light Intelligent Imaging Solution) specifically because we are beginning to branch out to handle use cases such as this! As a proof of concept that this task can be tackled directly, a quick search brought up "DeepFlash: Turning a Flash Selfie into a Studio Portrait"[0] Beyond denoising, we are already running experiments with very promising results on haze, lens flare, and reflection removal; super resolution; region adaptive white balancing; single exposure HDR; and a fair bit more. One of the other cooler things we are doing is putting together a unified SDK where our algorithms and neural nets will be able to run pretty much anywhere, on any hardware, using transparent backend switching. (e.g. CPU, GPU, TPU, NPU, DSP, other accelerator ASICs, etc..) [0] https://arxiv.org/abs/1901.04252 https://arxiv.org/abs/1901.04252
  - exikyut 6y ago
    Before reading your reply to OP's comment I got to thinking about how the super-resolution process and flash photography might interact (https://news.ycombinator.com/item?id=22905317 https://news.ycombinator.com/item?id=22905317). I get the impression you left the point I got to a long time ago :)
  - dmos62 6y ago
    DeepFlash: Turning a Flash Selfie into a Studio Portrait https://www.youtube.com/watch?v=enLmReROhc8 https://www.youtube.com/watch?v=enLmReROhc8
- exikyut 6y ago
  The way I mistakenly initially parsed this comment gave rise to a potentially-dumb idea/question: What would happen if you - begin capturing video (unsure of fps) on a phone-quality sensor in a near-dark environment - pulse the phone's flash LED(s) like you're taking a photo - do super-resolution on the resulting video to extract a photo... - ...while factoring in the decay in brightness/saturation in consecutive video frames produced by the flash pulse? I vaguely recall reading somewhere that oversaturated photos have more signal in them and are easier to fix than undersaturated. Hmm. IIRC super-resolution worked with 30fps source video for better quality; I wonder if 60fps or 120fps source video would produce better brightness decay data, or whether super-resolution could actually help extract more signal out of the decay sequence too. On the other hand, I'm not sure if super-resolution fundamentally requires largely consistent brightness in order to work as well as it does. :/ Perhaps individual networks could be trained/tuned to specific slices/windows of the brightness gradient. I also wonder if it would be useful to factor the superresolution process into each of the brightness-specific stages or just to do it at the end.

by y7r4m 6y ago

dbeardsl 6y ago

Sounds like you have taken this pretty far, do you have any example outputs? The only one I found via your website was a PDF with a low res image with no context.

y7r4m 6y ago
Sure, we have a short deck[0] that gives an intro to our noise reduction, and also here is a folder[1] that shows off a calibration target we captured with a actual camera (20ms, f22) in low-light conditions: (original, 100x gain, 100x gain + ALIIS) We also have some more raw data[2] where there is the original bayer data available as .npy files with 40db analog gain applied, however I think the calibration targets show off what we are able to accomplish more dramatically. Finally, we have a short youtube video[3] that shows off how it works when applied to video. [0] https://www.dropbox.com/s/0bm4dpxhn35vkhe/ALLIS_Investor_Intro_Q12020.pdf https://www.dropbox.com/s/0bm4dpxhn35vkhe/ALLIS_Investor_Int... [1] https://www.dropbox.com/sh/k861saentyq1cs6/AADmO7X_L49nUkEI_taE7e8Sa https://www.dropbox.com/sh/k861saentyq1cs6/AADmO7X_L49nUkEI_... [2] https://www.dropbox.com/sh/fv8omdf4fbx59m9/AABDnf6sdvv7rtIml-li_Go0a https://www.dropbox.com/sh/fv8omdf4fbx59m9/AABDnf6sdvv7rtIml... [3] https://youtu.be/99Cq1bWCmMM https://youtu.be/99Cq1bWCmMM

randyrand 6y ago

It would be really cool if you could feed the network a photo with flash that it could use for gathering more information, but then recreated a photo without flash from the non-flash raw. Often flash is not the look people are going for, but would be okay with the flash firing in order to improve the non-flash photo.

y7r4m 6y ago
Absolutely! We recently rebranded our AI solutions from ALLIS (Advanced Low Light Imagine Solution) to ALIIS (All Light Intelligent Imaging Solution) specifically because we are beginning to branch out to handle use cases such as this! As a proof of concept that this task can be tackled directly, a quick search brought up "DeepFlash: Turning a Flash Selfie into a Studio Portrait"[0] Beyond denoising, we are already running experiments with very promising results on haze, lens flare, and reflection removal; super resolution; region adaptive white balancing; single exposure HDR; and a fair bit more. One of the other cooler things we are doing is putting together a unified SDK where our algorithms and neural nets will be able to run pretty much anywhere, on any hardware, using transparent backend switching. (e.g. CPU, GPU, TPU, NPU, DSP, other accelerator ASICs, etc..) [0] https://arxiv.org/abs/1901.04252 https://arxiv.org/abs/1901.04252
- exikyut 6y ago
  Before reading your reply to OP's comment I got to thinking about how the super-resolution process and flash photography might interact (https://news.ycombinator.com/item?id=22905317 https://news.ycombinator.com/item?id=22905317). I get the impression you left the point I got to a long time ago :)
- dmos62 6y ago
  DeepFlash: Turning a Flash Selfie into a Studio Portrait https://www.youtube.com/watch?v=enLmReROhc8 https://www.youtube.com/watch?v=enLmReROhc8
exikyut 6y ago
The way I mistakenly initially parsed this comment gave rise to a potentially-dumb idea/question: What would happen if you - begin capturing video (unsure of fps) on a phone-quality sensor in a near-dark environment - pulse the phone's flash LED(s) like you're taking a photo - do super-resolution on the resulting video to extract a photo... - ...while factoring in the decay in brightness/saturation in consecutive video frames produced by the flash pulse? I vaguely recall reading somewhere that oversaturated photos have more signal in them and are easier to fix than undersaturated. Hmm. IIRC super-resolution worked with 30fps source video for better quality; I wonder if 60fps or 120fps source video would produce better brightness decay data, or whether super-resolution could actually help extract more signal out of the decay sequence too. On the other hand, I'm not sure if super-resolution fundamentally requires largely consistent brightness in order to work as well as it does. :/ Perhaps individual networks could be trained/tuned to specific slices/windows of the brightness gradient. I also wonder if it would be useful to factor the superresolution process into each of the brightness-specific stages or just to do it at the end.