7 ms·

Show HN: I made a computer vision addon for Blender

268 points by morroida 6y ago

dr_zoidberg 6y ago
Ok, this turned out to be far more interesting than the title here reads like. The little abstract at the top is far more informative: > A Blender user-interface to generate synthetic ground truth data (benchmarks) for Computer Vision applications. And it lets you make stereo images, depth maps, segmentation masks, surface normals and optical flow data from the rendered animation, and export it all in .npz numpy format. Quite interesting project.
tim44 6y ago
Nerb here. I can't say what this does but build a frame for ai comparison? Blender doesn't need so much eyes anyways? Buuuuut it doesnt have them either in this way?
blensor 6y ago
Wouldn't a Show HN post make more sense here? I noticed that they tend to fare better (keep the interest active for longer)
daenz 6y ago
Very cool! I've done something similar for improving an OCR system on crinkled paper[0]. Blender is a powerful and totally underutilized tool for this kind of work 0. https://www.arwmoffat.com/work/synthetic-training-data https://www.arwmoffat.com/work/synthetic-training-data
- morroida 6y ago
  Uou this is awesome! And it's very nicely presented in the website. I'm wondering how you mapped from the UV to the 3D model. I would like to add that feature to the addon.
  - daenz 6y ago
    It's been awhile since I've looked at the code, but take a look at the code around this https://github.com/amoffat/metabrite-receipt-tests/blob/master/blender/receipts.py#L269-L285 https://github.com/amoffat/metabrite-receipt-tests/blob/mast... for mapping from UV space to image space TLDR: using a KD-tree, I find the face containing the UV coordinate. Then I transform the UV coordinate to barycentric coordinates within that containing face, then put that barycentric coordinate through the local -> world -> view -> perspective transform matrices
    - emcq 6y ago
      A common approach in rendering engines to convert screen space coordinates to objects is to render a second image with light and shadow disabled where the color uniquely maps to an id. You then can uniquely identify 24 bits worth of objects without needing to maintain a KD tree.
- technicolorwhat 6y ago
  What the heck. This is beyond awesome I totally want to try it out
- modeless 6y ago
  I've thought about doing this myself! Did it end up improving the OCR system for real world images?
  - daenz 6y ago
    The startup ran out of money before we could find out :) It was sort of a skunkworks project.
- chrisseaton 6y ago
  Does using synthetic training data introduce any problems? How do you ensure your synthetic data matches real data?
morroida 6y ago
So many likes, was not expecting that! I will be presenting this work tomorrow at MICCAI and then I will post my presentation link in the README of the repository!
- punnerud 6y ago
  Thank you for your work! Looking forward to the presentation. I had to look up MICCAI. To others: 23rd INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING & COMPUTER ASSISTED INTERVENTION (4-8 OCTOBER 2020) https://www.miccai2020.org/en https://www.miccai2020.org/en
  - morroida 6y ago
    done! added to the GitHub README [presentation video](https://imperialcollegelondon.box.com/s/cg54pddsf2pkx4ngf4pg96c7o1wm23rx https://imperialcollegelondon.box.com/s/cg54pddsf2pkx4ngf4pg...)
    - punnerud 6y ago
      Love it! Should maybe add a link in addition to the video "image". Was not intuitive if it is a an image from a video or a link to a video (I am not used to video "preview" on Github).
      - morroida 6y ago
        Thank you! Yeah, I found that "hack" of the "image-video" in StackOverflow. Will add the link too as suggested
- morroida 6y ago
  Guys I won the best paper award!
doctoboggan 6y ago
This is awesome! I’ve been working with blender scripts a lot lately for my side project where I generate jewelry for 3D printing (https://lulimjewelry.com https://lulimjewelry.com) It’s an incredibly powerful tool, IMO one of the best large open source applications. I’ve learned some good ideas by reading the plug-in here, thank you!
- bluejellybean 6y ago
  Really cool! I just sent you an email (I love your contact info on HN btw!)
btparker 6y ago
Excellent! For a while my job entailed this very thing: creating synthetic data for computer vision, and I used Blender as well! You've done a great job.
santaclaus 6y ago
Very cool! Just the other day I was trying to set Blender’s camera based on a standard 3x4 computer vision KRT matrix, and it is surprisingly a pain in the ass —- I wish more of these graphics CAD packages (Blender, Houdini, Maya) made it easier to deal with vision data.
- morroida 6y ago
  I agree, these tools should have an official computer vision module since so many people are using synthetic data these days.
bluejellybean 6y ago
Very, very cool work, I fully expect to use this exact project in the future! To those of you who always thought this stuff looked neat but never tried it out, and to those who may have used blender in the past and gave up, I would HIGHLY encourage you to try again with the latest version of the software. Although there is still a bit of a learning curve, there have been massive improvements have to the software suite. The ability to code audio/visual in blender is just incredible. I describe it like this: Imagine yourself as someone trying to code an image that looks like a tree in machine code. Then imagine your partner comes over, sees what you are working on, and hands you Python and a fully setup IDE, it's like being given literal magic. I downloaded the latest version earlier this year to write some basic AI simulations (cube wars!) and to create models for my 3D-printer. Like when I first learned to code, the fun of the machine totally sucked me and I got completely off-task from my original goal. Lately I've been working on two things with the same lines of code, music videos and simulated walks through forests(Think the movie Avatar). With only a few hundred lines of code I am able to generate infinite forest trails in which you can walk (or fly a drone-style camera) through, synced to music that is generated by the AI-mushrooms WITHIN in the scene itself! Literally was able to go from 0 to highly visually engaging trippy music videos in the last year with minimial musical production experience and with no music-video production background. The ease in which you are able to generate things via code is stunning and the limits feel completely boundless.
- canada_dry 6y ago
  Similarly, when my niece recently got married I was tasked with putting together the photo/video show for the event. I chose Blender because it's easy to use and loaded with image and video capability. Plus, it gave me a good reason to come upto speed on the latest version.
amelius 6y ago
One question: why does it need to run inside Blender instead of just using its graphical primitives as a library?
- morroida 6y ago
  That's a good idea, it would make it faster for sure.
DonCopal 6y ago
Is this face motion tracking using a camera to apply to 3D models?
tantalor 6y ago
"synthetic ground truth"... isn't that an oxymoron?
- notsuoh 6y ago
  Ground truth in this case means images with labels of some sort on parts of the image or objects in the image, not that they're real images themselves. So a cat in a photo with a label or mask on the cat would be "ground truth" on what part of the image a cat is in, or that there is a cat.
  - tantalor 6y ago
    But if it's CG cat then it's not a real cat, hence not "ground truth".
    - notsuoh 6y ago
      Haha, just pointing out how the term is used in the industry. If you want to really play with semantics then I would argue that even a photo of a real cat isn’t a real cat and thus not ground truth. But while the consequences of that might have the awesome effect of clearing the shelters, it’s not practical, so we just call any labeled pixels representing a cat to a fidelity good enough for our purposes ground truth. ;)
m_ke 6y ago
A few related projects: - https://github.com/DLR-RM/BlenderProc https://github.com/DLR-RM/BlenderProc - https://github.com/cheind/pytorch-blender https://github.com/cheind/pytorch-blender - https://github.com/DIYer22/bpycv https://github.com/DIYer22/bpycv - https://github.com/yuki-koyama/blender-cli-rendering https://github.com/yuki-koyama/blender-cli-rendering and youtube videos covering blender for synthetic data: - https://www.youtube.com/playlist?list=PLq7npTWbkgVAt4cnrsEzouM6kDKySnkRI https://www.youtube.com/playlist?list=PLq7npTWbkgVAt4cnrsEzo... - https://www.youtube.com/watch?v=EaXy-m2I5hs https://www.youtube.com/watch?v=EaXy-m2I5hs
- canada_dry 6y ago
  The difference is that none of the above examples actually process real image input - the Cartucho app utilizes stereo cam images.
  - A-Train 6y ago
    I dont think so. It generates stereo images from 3D (not takes them as an input) so it is exactly related to blenderproc and other tools.
    - canada_dry 6y ago
      Yes... you are correct. My mistake.

268 points by morroida 6y ago

Ok, this turned out to be far more interesting than the title here reads like. The little abstract at the top is far more informative: > A Blender user-interface to generate synthetic ground truth data (benchmarks) for Computer Vision applications. And it lets you make stereo images, depth maps, segmentation masks, surface normals and optical flow data from the rendered animation, and export it all in .npz numpy format. Quite interesting project.

tim44 6y ago

Nerb here. I can't say what this does but build a frame for ai comparison? Blender doesn't need so much eyes anyways? Buuuuut it doesnt have them either in this way?

blensor 6y ago

Wouldn't a Show HN post make more sense here? I noticed that they tend to fare better (keep the interest active for longer)

daenz 6y ago

Very cool! I've done something similar for improving an OCR system on crinkled paper[0]. Blender is a powerful and totally underutilized tool for this kind of work 0. https://www.arwmoffat.com/work/synthetic-training-data https://www.arwmoffat.com/work/synthetic-training-data

morroida 6y ago
Uou this is awesome! And it's very nicely presented in the website. I'm wondering how you mapped from the UV to the 3D model. I would like to add that feature to the addon.
- daenz 6y ago
  It's been awhile since I've looked at the code, but take a look at the code around this https://github.com/amoffat/metabrite-receipt-tests/blob/master/blender/receipts.py#L269-L285 https://github.com/amoffat/metabrite-receipt-tests/blob/mast... for mapping from UV space to image space TLDR: using a KD-tree, I find the face containing the UV coordinate. Then I transform the UV coordinate to barycentric coordinates within that containing face, then put that barycentric coordinate through the local -> world -> view -> perspective transform matrices
  - emcq 6y ago
    A common approach in rendering engines to convert screen space coordinates to objects is to render a second image with light and shadow disabled where the color uniquely maps to an id. You then can uniquely identify 24 bits worth of objects without needing to maintain a KD tree.
technicolorwhat 6y ago
What the heck. This is beyond awesome I totally want to try it out
modeless 6y ago
I've thought about doing this myself! Did it end up improving the OCR system for real world images?
- daenz 6y ago
  The startup ran out of money before we could find out :) It was sort of a skunkworks project.
chrisseaton 6y ago
Does using synthetic training data introduce any problems? How do you ensure your synthetic data matches real data?

morroida 6y ago

So many likes, was not expecting that! I will be presenting this work tomorrow at MICCAI and then I will post my presentation link in the README of the repository!

punnerud 6y ago
Thank you for your work! Looking forward to the presentation. I had to look up MICCAI. To others: 23rd INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING & COMPUTER ASSISTED INTERVENTION (4-8 OCTOBER 2020) https://www.miccai2020.org/en https://www.miccai2020.org/en
- morroida 6y ago
  done! added to the GitHub README [presentation video](https://imperialcollegelondon.box.com/s/cg54pddsf2pkx4ngf4pg96c7o1wm23rx https://imperialcollegelondon.box.com/s/cg54pddsf2pkx4ngf4pg...)
  - punnerud 6y ago
    Love it! Should maybe add a link in addition to the video "image". Was not intuitive if it is a an image from a video or a link to a video (I am not used to video "preview" on Github).
    - morroida 6y ago
      Thank you! Yeah, I found that "hack" of the "image-video" in StackOverflow. Will add the link too as suggested
morroida 6y ago
Guys I won the best paper award!

doctoboggan 6y ago

This is awesome! I’ve been working with blender scripts a lot lately for my side project where I generate jewelry for 3D printing (https://lulimjewelry.com https://lulimjewelry.com) It’s an incredibly powerful tool, IMO one of the best large open source applications. I’ve learned some good ideas by reading the plug-in here, thank you!

bluejellybean 6y ago
Really cool! I just sent you an email (I love your contact info on HN btw!)

btparker 6y ago

Excellent! For a while my job entailed this very thing: creating synthetic data for computer vision, and I used Blender as well! You've done a great job.

santaclaus 6y ago

Very cool! Just the other day I was trying to set Blender’s camera based on a standard 3x4 computer vision KRT matrix, and it is surprisingly a pain in the ass —- I wish more of these graphics CAD packages (Blender, Houdini, Maya) made it easier to deal with vision data.

morroida 6y ago
I agree, these tools should have an official computer vision module since so many people are using synthetic data these days.

bluejellybean 6y ago

Very, very cool work, I fully expect to use this exact project in the future! To those of you who always thought this stuff looked neat but never tried it out, and to those who may have used blender in the past and gave up, I would HIGHLY encourage you to try again with the latest version of the software. Although there is still a bit of a learning curve, there have been massive improvements have to the software suite. The ability to code audio/visual in blender is just incredible. I describe it like this: Imagine yourself as someone trying to code an image that looks like a tree in machine code. Then imagine your partner comes over, sees what you are working on, and hands you Python and a fully setup IDE, it's like being given literal magic. I downloaded the latest version earlier this year to write some basic AI simulations (cube wars!) and to create models for my 3D-printer. Like when I first learned to code, the fun of the machine totally sucked me and I got completely off-task from my original goal. Lately I've been working on two things with the same lines of code, music videos and simulated walks through forests(Think the movie Avatar). With only a few hundred lines of code I am able to generate infinite forest trails in which you can walk (or fly a drone-style camera) through, synced to music that is generated by the AI-mushrooms WITHIN in the scene itself! Literally was able to go from 0 to highly visually engaging trippy music videos in the last year with minimial musical production experience and with no music-video production background. The ease in which you are able to generate things via code is stunning and the limits feel completely boundless.

canada_dry 6y ago
Similarly, when my niece recently got married I was tasked with putting together the photo/video show for the event. I chose Blender because it's easy to use and loaded with image and video capability. Plus, it gave me a good reason to come upto speed on the latest version.

amelius 6y ago

One question: why does it need to run inside Blender instead of just using its graphical primitives as a library?

morroida 6y ago
That's a good idea, it would make it faster for sure.

DonCopal 6y ago

Is this face motion tracking using a camera to apply to 3D models?

tantalor 6y ago

"synthetic ground truth"... isn't that an oxymoron?

notsuoh 6y ago
Ground truth in this case means images with labels of some sort on parts of the image or objects in the image, not that they're real images themselves. So a cat in a photo with a label or mask on the cat would be "ground truth" on what part of the image a cat is in, or that there is a cat.
- tantalor 6y ago
  But if it's CG cat then it's not a real cat, hence not "ground truth".
  - notsuoh 6y ago
    Haha, just pointing out how the term is used in the industry. If you want to really play with semantics then I would argue that even a photo of a real cat isn’t a real cat and thus not ground truth. But while the consequences of that might have the awesome effect of clearing the shelters, it’s not practical, so we just call any labeled pixels representing a cat to a fidelity good enough for our purposes ground truth. ;)

m_ke 6y ago

A few related projects: - https://github.com/DLR-RM/BlenderProc https://github.com/DLR-RM/BlenderProc - https://github.com/cheind/pytorch-blender https://github.com/cheind/pytorch-blender - https://github.com/DIYer22/bpycv https://github.com/DIYer22/bpycv - https://github.com/yuki-koyama/blender-cli-rendering https://github.com/yuki-koyama/blender-cli-rendering and youtube videos covering blender for synthetic data: - https://www.youtube.com/playlist?list=PLq7npTWbkgVAt4cnrsEzouM6kDKySnkRI https://www.youtube.com/playlist?list=PLq7npTWbkgVAt4cnrsEzo... - https://www.youtube.com/watch?v=EaXy-m2I5hs https://www.youtube.com/watch?v=EaXy-m2I5hs

canada_dry 6y ago
The difference is that none of the above examples actually process real image input - the Cartucho app utilizes stereo cam images.
- A-Train 6y ago
  I dont think so. It generates stereo images from 3D (not takes them as an input) so it is exactly related to blenderproc and other tools.
  - canada_dry 6y ago
    Yes... you are correct. My mistake.