5 ms·

Stitching – A Python package for fast and robust Image Stitching (Panoramas)

155 points by LukasWeber 4y ago

_aavaa_ 4y ago
Would be nice to include some sort of performance benchmarks that compare it to the alternatives.
- LukasWeber 4y ago
  what do you think should be the alternatives it should be compared to?
  - Derbasti 4y ago
    Perhaps hugin
  - _aavaa_ 4y ago
    Depends on what the package is supposed to be used for. If its meant for photographers to create panoramas, then I think a comparison to: Photoshop, Hugin (or PTGUI for gpu acceleration), and Microsoft ICE would be a good benchmark.
deleted 4y ago
[deleted]
est 4y ago
How is this compared with Microsoft ICE?
- bmh 4y ago
  Microsoft seems to have discontinued ICE a long time ago. As far as I know, the state of the art in panorama stitching is PTGui. I was working on mobile panorama stitching last year, and one of my datasets had a kitchen wall that was almost purely white, so very little detail for the classic feature algorithms such as SIFT and ORB to cling to. The OpenCV stitching pipeline, which is built on these (and RANSAC), didn't do very well when matching these walls. But PTGui was amazing on this data - it would find just a tiny number of very high quality feature points to match (eg 3 or 4), and produce a perfect panorama. In addition, it's really fast. I was very impressed.
  - PetitPrince 4y ago
    True, but ICE is free while PT GUI is 150€. And compared to the open source darling Hugin it has also a less fussy UI.
    - ggm 4y ago
      Hugin hasn't worked for me on macos well for a couple of years.
      - actionfromafar 4y ago
        I never got it work with my images. (Flatbed mosaic.)
    - _aavaa_ 4y ago
      The fussy UI is nothing. The real benefit is the gpu acceleration. That alone is worth the money.
  - folli 4y ago
    So how did you solve the mobile panorama stitching? You ran PTGui on a backend server and uploaded the user images there?
    - bmh 4y ago
      I started from the OpenCV stitching_detailed example, and worked from there. That example runs pretty much out of the box on Android.
      - folli 4y ago
        Cool, thanks!
  - LukasWeber 4y ago
    A simple GUI would be great to let the user select high quality 3-4 matches (reference points) between the image pairs. Then the rest of the stitching pipeline could stay the same.
kwon-young 4y ago
In my case, my dad needed a Gui for stitching microscope images and he told me that none of the panorama stitching Guis were working for him. I found out that the out of the box opencv Stitcher [1] class is perfect for that but nobody seems to have made a GUI out of it. So, I've spent two days making an extremely simple proof of concept of a Qt Gui [2]. The majority of the time was spent trying to use GitHub Action to automatically build an executable... If you go see in the release section, there is a tag with a prebuilt binary. [1]: https://docs.opencv.org/4.x/d8/d19/tutorial_stitcher.html https://docs.opencv.org/4.x/d8/d19/tutorial_stitcher.html [2]: https://github.com/kwon-young/ImageStitcher https://github.com/kwon-young/ImageStitcher
- ahns 4y ago
  For microscopy images, have you tried ashlar? https://github.com/labsyspharm/ashlar https://github.com/labsyspharm/ashlar It may help to have the microscope stage positions available in the OME metadata.
  - kwon-young 4y ago
    I did not know of this project. I suppose it did not come up in my research sinve I was looking for simple Guis that my dad could use. His setup only has a simple microscope camera that can stream a video to an old windows xp pc with a hardware/software combination...
- teruakohatu 4y ago
  Just a suggestion, maybe create a new repo that is not a fork from a template. Just copy the files in, and then create a simple README that explains what it is and how to run it.
  - nextaccountic 4y ago
    So Github still doesn't have an unfork button.. but they can unfork the repo using a virtual assistant https://stackoverflow.com/questions/38831301/how-to-un-fork-the-github-repository/66470086#66470086 https://stackoverflow.com/questions/38831301/how-to-un-fork-... .. I don't get why the regular UI can't do this Anyway in some language ecosystems there are cli tools that make repositories from templates, and a benefit is that it prevents people from forking the template (which is appropriate if you are modifying the template like upgrading dependencies etc, but not if you are just making a new project using it)
StephenSmith 4y ago
I'd love someone with more CV experience to explain why this fails on my use case? (four image zip) https://drive.google.com/file/d/1gTZWlKmS4ZtEZDK8OjjLeGN_V0FAAlrx/view?usp=sharing https://drive.google.com/file/d/1gTZWlKmS4ZtEZDK8OjjLeGN_V0F...
- Mumps 4y ago
  (sorry don't have time to run through the OP script and check what you're getting for outputs, but...) - I'm not even entirely sure how these images would stitch together - between 1.jpeg and the rest, there is very little overlap. RANSAC likely won't find matchable points. - you're asking for a pretty severe homography of the scene.
  - StephenSmith 4y ago
    Thanks! That's what I was trying to understand.
- bmh 4y ago
  A panorama is built by stitching together many images taken from the same viewpoint. The 1st critical thing you need to know, is that the camera may not move. It may only rotate. If the camera moves a tiny bit, that's generally OK, but there will be stitching artifacts. To be really precise, the entrance pupil of the camera shouldn't move, but this is quite hard to get right. The 2nd thing, is that the stitching algorithms need to match images to each other, and in order to do this, the images must have large overlapping portions. This usually means that if you want to create a 360 degree panorama where you spin your camera all the way around, you'll have at least about 12 images. The minimum number of images depends on how wide your lens is. This is a great overview of panoramas: http://6.869.csail.mit.edu/fa17/lecture/lecture14sift_homography.pdf http://6.869.csail.mit.edu/fa17/lecture/lecture14sift_homogr... One more thing: It really helps to apply lens correction to your camera images, if you're trying to create a high quality panorama.
  - kurupt213 4y ago
    Nice explanation
O5vYtytb 4y ago
Wondering if anyone has examples/ideas for how to do this with video? I have two cameras (GoPro) that I use to record athletic games on a 20ft tall tripod, and I want to automatically stitch them together for easier analysis of games. I think I can use a tool like this to find the stitching parameters for the first frame and then apply it to all future frames, assuming the camera doesn't move?
- bmh 4y ago
  Yes, this would definitely work, so long as the two cameras have sufficient overlap. If you were to use OpenCV's stitching code, then you could run the first few phases of that on your initial photo pair, which determine the camera angles, and then save those angles. Then, for every video frame, you could skip the photo angle computation, and just run the imagine stitching logic. The stitching_detail code is very readable, and quite easy to hack for experimentation.
philipov 4y ago
Is there a way to get it to report progress while it's working? It's been running for 30 minutes and I can't tell if it's doing anything. There's been no output since it reported the files it was combining.

155 points by LukasWeber 4y ago

_aavaa_ 4y ago

Would be nice to include some sort of performance benchmarks that compare it to the alternatives.

LukasWeber 4y ago
what do you think should be the alternatives it should be compared to?
- Derbasti 4y ago
  Perhaps hugin
- _aavaa_ 4y ago
  Depends on what the package is supposed to be used for. If its meant for photographers to create panoramas, then I think a comparison to: Photoshop, Hugin (or PTGUI for gpu acceleration), and Microsoft ICE would be a good benchmark.

deleted 4y ago

[deleted]

est 4y ago

How is this compared with Microsoft ICE?

bmh 4y ago
Microsoft seems to have discontinued ICE a long time ago. As far as I know, the state of the art in panorama stitching is PTGui. I was working on mobile panorama stitching last year, and one of my datasets had a kitchen wall that was almost purely white, so very little detail for the classic feature algorithms such as SIFT and ORB to cling to. The OpenCV stitching pipeline, which is built on these (and RANSAC), didn't do very well when matching these walls. But PTGui was amazing on this data - it would find just a tiny number of very high quality feature points to match (eg 3 or 4), and produce a perfect panorama. In addition, it's really fast. I was very impressed.
- PetitPrince 4y ago
  True, but ICE is free while PT GUI is 150€. And compared to the open source darling Hugin it has also a less fussy UI.
  - ggm 4y ago
    Hugin hasn't worked for me on macos well for a couple of years.
    - actionfromafar 4y ago
      I never got it work with my images. (Flatbed mosaic.)
  - _aavaa_ 4y ago
    The fussy UI is nothing. The real benefit is the gpu acceleration. That alone is worth the money.
- folli 4y ago
  So how did you solve the mobile panorama stitching? You ran PTGui on a backend server and uploaded the user images there?
  - bmh 4y ago
    I started from the OpenCV stitching_detailed example, and worked from there. That example runs pretty much out of the box on Android.
    - folli 4y ago
      Cool, thanks!
- LukasWeber 4y ago
  A simple GUI would be great to let the user select high quality 3-4 matches (reference points) between the image pairs. Then the rest of the stitching pipeline could stay the same.

kwon-young 4y ago

In my case, my dad needed a Gui for stitching microscope images and he told me that none of the panorama stitching Guis were working for him. I found out that the out of the box opencv Stitcher [1] class is perfect for that but nobody seems to have made a GUI out of it. So, I've spent two days making an extremely simple proof of concept of a Qt Gui [2]. The majority of the time was spent trying to use GitHub Action to automatically build an executable... If you go see in the release section, there is a tag with a prebuilt binary. [1]: https://docs.opencv.org/4.x/d8/d19/tutorial_stitcher.html https://docs.opencv.org/4.x/d8/d19/tutorial_stitcher.html [2]: https://github.com/kwon-young/ImageStitcher https://github.com/kwon-young/ImageStitcher

ahns 4y ago
For microscopy images, have you tried ashlar? https://github.com/labsyspharm/ashlar https://github.com/labsyspharm/ashlar It may help to have the microscope stage positions available in the OME metadata.
- kwon-young 4y ago
  I did not know of this project. I suppose it did not come up in my research sinve I was looking for simple Guis that my dad could use. His setup only has a simple microscope camera that can stream a video to an old windows xp pc with a hardware/software combination...
teruakohatu 4y ago
Just a suggestion, maybe create a new repo that is not a fork from a template. Just copy the files in, and then create a simple README that explains what it is and how to run it.
- nextaccountic 4y ago
  So Github still doesn't have an unfork button.. but they can unfork the repo using a virtual assistant https://stackoverflow.com/questions/38831301/how-to-un-fork-the-github-repository/66470086#66470086 https://stackoverflow.com/questions/38831301/how-to-un-fork-... .. I don't get why the regular UI can't do this Anyway in some language ecosystems there are cli tools that make repositories from templates, and a benefit is that it prevents people from forking the template (which is appropriate if you are modifying the template like upgrading dependencies etc, but not if you are just making a new project using it)

StephenSmith 4y ago

I'd love someone with more CV experience to explain why this fails on my use case? (four image zip) https://drive.google.com/file/d/1gTZWlKmS4ZtEZDK8OjjLeGN_V0FAAlrx/view?usp=sharing https://drive.google.com/file/d/1gTZWlKmS4ZtEZDK8OjjLeGN_V0F...

Mumps 4y ago
(sorry don't have time to run through the OP script and check what you're getting for outputs, but...) - I'm not even entirely sure how these images would stitch together - between 1.jpeg and the rest, there is very little overlap. RANSAC likely won't find matchable points. - you're asking for a pretty severe homography of the scene.
- StephenSmith 4y ago
  Thanks! That's what I was trying to understand.
bmh 4y ago
A panorama is built by stitching together many images taken from the same viewpoint. The 1st critical thing you need to know, is that the camera may not move. It may only rotate. If the camera moves a tiny bit, that's generally OK, but there will be stitching artifacts. To be really precise, the entrance pupil of the camera shouldn't move, but this is quite hard to get right. The 2nd thing, is that the stitching algorithms need to match images to each other, and in order to do this, the images must have large overlapping portions. This usually means that if you want to create a 360 degree panorama where you spin your camera all the way around, you'll have at least about 12 images. The minimum number of images depends on how wide your lens is. This is a great overview of panoramas: http://6.869.csail.mit.edu/fa17/lecture/lecture14sift_homography.pdf http://6.869.csail.mit.edu/fa17/lecture/lecture14sift_homogr... One more thing: It really helps to apply lens correction to your camera images, if you're trying to create a high quality panorama.
- kurupt213 4y ago
  Nice explanation

O5vYtytb 4y ago

Wondering if anyone has examples/ideas for how to do this with video? I have two cameras (GoPro) that I use to record athletic games on a 20ft tall tripod, and I want to automatically stitch them together for easier analysis of games. I think I can use a tool like this to find the stitching parameters for the first frame and then apply it to all future frames, assuming the camera doesn't move?

bmh 4y ago
Yes, this would definitely work, so long as the two cameras have sufficient overlap. If you were to use OpenCV's stitching code, then you could run the first few phases of that on your initial photo pair, which determine the camera angles, and then save those angles. Then, for every video frame, you could skip the photo angle computation, and just run the imagine stitching logic. The stitching_detail code is very readable, and quite easy to hack for experimentation.

philipov 4y ago

Is there a way to get it to report progress while it's working? It's been running for 30 minutes and I can't tell if it's doing anything. There's been no output since it reported the files it was combining.