4 ms·

How the hell does the Seek by iNaturalist app work so well and also be small/performant enough to the job completely offline on a phone? You should really try i

by qz_kb 3y ago

How the hell does the Seek by iNaturalist app work so well and also be small/performant enough to the job completely offline on a phone? You should really try it out for IDing animals and plants if you haven't, it's like a real life pokedex. Have they released any information (e.g. a whitepaper?) about how the model works or how it was trained? The ability to classify things incrementally and phylogenetically makes it helpful to narrow down your own search even when it doesn't know the exact species. I've been surprised by it even IDing the insects that made specific galls on random leaves or plants.

xipho 3y ago
https://openaccess.thecvf.com/content_cvpr_2018/html/Van_Horn_The_INaturalist_Species_CVPR_2018_paper.html https://openaccess.thecvf.com/content_cvpr_2018/html/Van_Hor... https://openaccess.thecvf.com/content/CVPR2021/html/Van_Horn_Benchmarking_Representation_Learning_for_Natural_World_Image_Collections_CVPR_2021_paper.html https://openaccess.thecvf.com/content/CVPR2021/html/Van_Horn... Etc.
EGreg 3y ago
Probably because many people around the world participated in classifying what was posted? I am guessing. Please tell me if that is correct. How do they prevent false labels ?
- trash3 3y ago
  [dead]
- xipho 3y ago
  There are hundreds of thousands of "false' labels. Pictures can be classified many times.
  - EGreg 3y ago
    I always wondered how do you determine truth in such sites?
    - phito 3y ago
      You ask actual experts for identification
- nemo 3y ago
  Any observations can be submitted, but the observation has to be verified by a different observer. Most identifiers are folks with more experience identifying things locally, and the data quality is high. There's very little incentive to game the system and if something is misidentified other iNatters can also add identifications correcting mistakes which happens regularly - often various scientists/specialists tend to sweep observations in their taxa of note and correct issues. There's criteria for a "high quality" observation, including being verified. Only those observations that are high quality are used for training.
fogleman 3y ago
I reverse engineered their stuff a bit. I downloaded their Android APK and found a tensorflow lite model inside. I found that it accepts 299x299px RGB input and spits out probabilities/scores for about 25,000 species. The phylogenetic ranking is performed separately (outside of the model) based on thresholds (if it isn't confident enough about any species, it seems to only provide genus, family, etc.) They just have a CSV file that defines the taxonomic ranks of each species. I use it to automatically tag pictures that I take. I took up bird photography a few years ago and it's become a very serious hobby. I just run my Python script (which wraps their TF model) and it extracts JPG thumbnails from my RAW photos, automatically crops them based on EXIF data (regarding the focus point and the focus distance) and then feeds it into the model. This cropping was critical - I can't just throw the model a downsampled 45 megapixel image straight from the camera, usually the subject is too small in the frame. I store the results in a sqlite database. So now I can quickly pull up all photos of a given species, and even sort them by other EXIF values like focus distance. I pipe the results of arbitrary sqlite queries into my own custom RAW photo viewer and I can quickly browse the photos. (e.g. "Show me all Green Heron photos sorted by focus distance.") The species identification results aren't perfect, but they are very good. And I store the score in sql too, so I can know how confident the model was. One cool thing was that it revealed that I had photographed a Blackpoll Warbler in 2020 when I was a new and budding birder. I didn't think I had ever seen one. But I saw it listed in the program results, and was able to confirm by revisiting the photo. I don't know if they've changed anything recently. Judging by some of their code on GitHub, it looked like they were also working on considering location when determining species, but the model I found doesn't seem to do that. I can't tell you anything about how the model was actually trained, but this information may still be useful in understanding how the app operates. Of course, I haven't published any of this code because the model isn't my own work.
- datadrivenangel 3y ago
  That sounds like an awesome setup! Would you be willing to share your script with another bird photography enthusiast?
- kubrickslair 3y ago
  Thanks for sharing - I was curious too but didn’t delve in myself.
rahimnathwani 3y ago
I'd never heard of this app, but your description made me want to install it. When I googled it I was surprised at the app ratings: Apple: 4.8 Google Play: 3.4 The most common issue mentioned by negative Play store reviews is the camera not focusing on the right thing, and needing to try many different angles before something is recognized correctly. This is probably nothing to do with the underlying model, which I guess is the same on both platforms.
- Taek 3y ago
  Camera zoom is definitely annoying, there's no way to control how zoomed in it is. And yes, it often takes as much as a minute to identify a species, because you have to keep adjusting zoom and angle and trying to catch every important feature. That said, once you are used to it, it becomes less noticeable and just feels like part of the game.
  - rahimnathwani 3y ago
    I'm curious why this seems (from the reviews) to be an issue only on Android?
    - whatevertrevor 3y ago
      I tried this app for a while, and there's definitely some rough edges. My partner's phone was much quicker in recognizing plants and flowers, so after a while I gave up and we just used her phone instead. And then there's the issue of many plants being slightly mis-identified, many really common flowers being incorrectly identified. I don't really know how people get remotely close to wild animals to identify them, I had no luck with animal life with this and after a while I started mistrusting its results and cross referencing with google lens anyway.
- techterrier 3y ago
  our own app Birda has this issue too, most of the 1 star reviews are 'not the app I was looking for' or 'not a game'
bandergirl 3y ago
I wish I had the same experience as you. The vast majority of time I point at tree leaves in South East Asia it tells me it’s Dicots and it stops there. Only rarely I get the actual full classification; the last time it happened was for a common almond tree.
- burkaman 3y ago
  It's very bad at trees for some reason. Also mushrooms, but I thought that might be intentional so they don't get blamed for someone eating something poisonous that was misidentified. PlantNet often works better for trees.
  - s0rce 3y ago
    It seems to do well for trees for me in California.
  - alecst 3y ago
    My experience has been great with mushrooms, just to add another datapoint. I mean, it's often about as good as you can get by eye without breaking out the lab equipment.
  - joshvm 3y ago
    Trees are generally difficult to classify well with computer vision. It's hard for the models to establish context because at a scale where you can see the whole tree you tend to include lots of background. If you include a bark photo, it's often ambiguous if there's growth/stuff/weathering on top. Flowers tend to be good inputs. The training imagery is also really inconsistent in inaturalist and again for plants it's hard to establish context. These are mostly smartphone pictures that non experts have captured in the app. While someone might have verified that a plant is a particular species, because there isn't any foreground/background segmentation, there are often confounding objects in the frame. On top of that you only get a 300px input to classify from. With animals I'd say it's much more common for photographers to isolate the subject. There's also massive class imbalance in inat, a large number of the observations are things like mallards (ie common species in city parks). I guess the best solution would be to heavily incorporate geographic priors and expected species in the location (which I think is partly done already).

by qz_kb 3y ago

xipho 3y ago

https://openaccess.thecvf.com/content_cvpr_2018/html/Van_Horn_The_INaturalist_Species_CVPR_2018_paper.html https://openaccess.thecvf.com/content_cvpr_2018/html/Van_Hor... https://openaccess.thecvf.com/content/CVPR2021/html/Van_Horn_Benchmarking_Representation_Learning_for_Natural_World_Image_Collections_CVPR_2021_paper.html https://openaccess.thecvf.com/content/CVPR2021/html/Van_Horn... Etc.

EGreg 3y ago

Probably because many people around the world participated in classifying what was posted? I am guessing. Please tell me if that is correct. How do they prevent false labels ?

trash3 3y ago
[dead]
xipho 3y ago
There are hundreds of thousands of "false' labels. Pictures can be classified many times.
- EGreg 3y ago
  I always wondered how do you determine truth in such sites?
  - phito 3y ago
    You ask actual experts for identification
nemo 3y ago
Any observations can be submitted, but the observation has to be verified by a different observer. Most identifiers are folks with more experience identifying things locally, and the data quality is high. There's very little incentive to game the system and if something is misidentified other iNatters can also add identifications correcting mistakes which happens regularly - often various scientists/specialists tend to sweep observations in their taxa of note and correct issues. There's criteria for a "high quality" observation, including being verified. Only those observations that are high quality are used for training.

fogleman 3y ago

I reverse engineered their stuff a bit. I downloaded their Android APK and found a tensorflow lite model inside. I found that it accepts 299x299px RGB input and spits out probabilities/scores for about 25,000 species. The phylogenetic ranking is performed separately (outside of the model) based on thresholds (if it isn't confident enough about any species, it seems to only provide genus, family, etc.) They just have a CSV file that defines the taxonomic ranks of each species. I use it to automatically tag pictures that I take. I took up bird photography a few years ago and it's become a very serious hobby. I just run my Python script (which wraps their TF model) and it extracts JPG thumbnails from my RAW photos, automatically crops them based on EXIF data (regarding the focus point and the focus distance) and then feeds it into the model. This cropping was critical - I can't just throw the model a downsampled 45 megapixel image straight from the camera, usually the subject is too small in the frame. I store the results in a sqlite database. So now I can quickly pull up all photos of a given species, and even sort them by other EXIF values like focus distance. I pipe the results of arbitrary sqlite queries into my own custom RAW photo viewer and I can quickly browse the photos. (e.g. "Show me all Green Heron photos sorted by focus distance.") The species identification results aren't perfect, but they are very good. And I store the score in sql too, so I can know how confident the model was. One cool thing was that it revealed that I had photographed a Blackpoll Warbler in 2020 when I was a new and budding birder. I didn't think I had ever seen one. But I saw it listed in the program results, and was able to confirm by revisiting the photo. I don't know if they've changed anything recently. Judging by some of their code on GitHub, it looked like they were also working on considering location when determining species, but the model I found doesn't seem to do that. I can't tell you anything about how the model was actually trained, but this information may still be useful in understanding how the app operates. Of course, I haven't published any of this code because the model isn't my own work.

datadrivenangel 3y ago
That sounds like an awesome setup! Would you be willing to share your script with another bird photography enthusiast?
kubrickslair 3y ago
Thanks for sharing - I was curious too but didn’t delve in myself.

rahimnathwani 3y ago

I'd never heard of this app, but your description made me want to install it. When I googled it I was surprised at the app ratings: Apple: 4.8 Google Play: 3.4 The most common issue mentioned by negative Play store reviews is the camera not focusing on the right thing, and needing to try many different angles before something is recognized correctly. This is probably nothing to do with the underlying model, which I guess is the same on both platforms.

Taek 3y ago
Camera zoom is definitely annoying, there's no way to control how zoomed in it is. And yes, it often takes as much as a minute to identify a species, because you have to keep adjusting zoom and angle and trying to catch every important feature. That said, once you are used to it, it becomes less noticeable and just feels like part of the game.
- rahimnathwani 3y ago
  I'm curious why this seems (from the reviews) to be an issue only on Android?
  - whatevertrevor 3y ago
    I tried this app for a while, and there's definitely some rough edges. My partner's phone was much quicker in recognizing plants and flowers, so after a while I gave up and we just used her phone instead. And then there's the issue of many plants being slightly mis-identified, many really common flowers being incorrectly identified. I don't really know how people get remotely close to wild animals to identify them, I had no luck with animal life with this and after a while I started mistrusting its results and cross referencing with google lens anyway.
techterrier 3y ago
our own app Birda has this issue too, most of the 1 star reviews are 'not the app I was looking for' or 'not a game'

bandergirl 3y ago

I wish I had the same experience as you. The vast majority of time I point at tree leaves in South East Asia it tells me it’s Dicots and it stops there. Only rarely I get the actual full classification; the last time it happened was for a common almond tree.

burkaman 3y ago
It's very bad at trees for some reason. Also mushrooms, but I thought that might be intentional so they don't get blamed for someone eating something poisonous that was misidentified. PlantNet often works better for trees.
- s0rce 3y ago
  It seems to do well for trees for me in California.
- alecst 3y ago
  My experience has been great with mushrooms, just to add another datapoint. I mean, it's often about as good as you can get by eye without breaking out the lab equipment.
- joshvm 3y ago
  Trees are generally difficult to classify well with computer vision. It's hard for the models to establish context because at a scale where you can see the whole tree you tend to include lots of background. If you include a bark photo, it's often ambiguous if there's growth/stuff/weathering on top. Flowers tend to be good inputs. The training imagery is also really inconsistent in inaturalist and again for plants it's hard to establish context. These are mostly smartphone pictures that non experts have captured in the app. While someone might have verified that a plant is a particular species, because there isn't any foreground/background segmentation, there are often confounding objects in the frame. On top of that you only get a 300px input to classify from. With animals I'd say it's much more common for photographers to isolate the subject. There's also massive class imbalance in inat, a large number of the observations are things like mallards (ie common species in city parks). I guess the best solution would be to heavily incorporate geographic priors and expected species in the location (which I think is partly done already).