10 ms·

This is exactly what I needed and was thinking about building a similar app. But to be really useful and automated, there's a bit long way I'm afraid. The bigge

This is exactly what I needed and was thinking about building a similar app. But to be really useful and automated, there's a bit long way I'm afraid. The biggest deal breaker is the time spend for the data entry and updates, leaving a pen-and-paper / notes solution still the most easy to use and just visiting local groceries and shops. I was collecting receipts and playing a bit with scanning of these, but the amount of noise in the data and inconsistencies between shops were a bit pushing away (unless you want to dedicate time cleaning the data and training own tailored information extraction models). One of the options to solve the data entry could be: - good quality automatic scanning of receipts (not only individual barcodes) from shops using OCR possibly supported with image recognition for double-checking (can happen that products will be mis-labelled or without quantities, etc) - when ordering on-line, the receipt should be available, so should be also much easier Yet, not always one will have a meaningful receipt available... Solved the data entry and being able to predict own's supply needs would be also great to have a up-to date management of the inventory. Here are even more challenges on the tracking of the available goods at home, where these are and how many items (and in what state, expiration date, etc) would require most probably implementing different solutions from IoT (connected cameras, sensors, etc.). Then, having a connected home with own groceries supplies under control, one can then automate further the shopping process with feeding-back the information about own's demand to on-line groceries one is subscribed to. This can enable customer subscription plans, and for retailer keeping a possible continuous flow of goods. This could be really really useful especially for upcoming months, when it seems like we are expected to spend a bit more time at home rather than usual, hopefully not fighting in the local shops for the last rolls of the new white paper gold.

NamTaf 6y ago
Absolutely agree with everything you've said. I had dreamed of a Libib[1] for my kitchen, but knew I would have to do all the painful data adding and it was too much of a bother. The other big challenge I never resolved was how you'd account for e.g. using 1/4 cup of flour out of a bigger volume. Or taken to its extreme, cooking oil. How do you know how much your 'splash' is? You can't predict the remaining volume without a lot of fiddling to measure it and that defeats the purpose. In the end, I opted for manual databases too, but they're pain to keep up-to-date. I still think there's a lot of value in a database for all-or-nothing style ingredients, but it was enough to deter me. I'm glad someone is less lazy. [1]: https://www.libib.com/ https://www.libib.com/
- six2seven 6y ago
  > The other big challenge I never resolved was how you'd account for e.g. using 1/4 cup of flour out of a bigger volume. Or taken to its extreme, cooking oil. How do you know how much your 'splash' is? You can't predict the remaining volume without a lot of fiddling to measure it and that defeats the purpose. Definitely. Or, even, just taking some snacks to another room, and later placing it back to an 'intelligent' cupboard, let's say. In extreme cases, one may need be under a constant tracking at home. But, simplifying, another option may be to use an improved voice assistant when cooking. One would need then to interact with the VA to note and confirm the stuff used. Improved VA, in essence, would try to 'understand' the activities and interact, instead of being a passive one (as most of currently are), that one needs to say commands. This would require adding it access to camera and maybe other sensors available at 'smart' home. Definitely a lot of interesting and challenging problems to tackle not forgetting too about the user privacy part.
  - byproxy 6y ago
    My naive solution, assuming a sufficiently "smart" home, would be to have each shelf be an accurate scale. If it's sensitive enough, it should be able to detect the weight of what was consumed.
    - urzav 6y ago
      A shelf for each item?
  - deleted 6y ago
    [deleted]
  - urzav 6y ago
    What you are talking about is called taking inventory. Even if automated tracking is being performed, i.e., by scanning the boxes or containers when they are moved out and updating the quantity in the database, a periodic inventory audit is performed in the warehouse to reconcile the actual number of the product on the shelf with the one derived from reflecting any additions or subtractions of the product. In the household setting, the most realistic system are we r would be to keep track of the addition (purchase/replenishment) ołl the containers/packages of each product and subtraction when the entire container is used-up and discarded. Obviously, this system would require maintaining at least one spare container for each product and reorder replacement when the remaining inventory count is 1 or whatever number of stock of a specific product one would want to maintain. Partial expenditures (splashes or table spoons of cooking oil, pinch of silt, a number of eggs from the pack, cups of sugar or flour from the container, a serving of cereal from box etc.) of the product from the container wouldn’t be tracked. The inventory would only be updated when the container/package has been fished up or purchased and added to the home inventory. For those items which one wishes not stock more than one container or packaging, a solution would be to label such containers, so that you can re-order the product (via Alexa or Google Assistant or Siri) when the remaining amount of the product is below certain level, depending on the product. I.e., you wouldn’t stock more than one container of milk or eggs. But when you got below half full, you would know to reorder. If certain products are used exclusively by the kids, they should learn to talk to Alexa or let parents know that they would soon run out of the product (they would learn if they want to have the product available). If one would follow this system of continuous replenishment, it might be unnecessary to keep the track of the available stock/inventory. But updating of the inventory database could be done automatically, using designated commands to the voice assists which invoke IFTT engine action sequence in which the product is ordered or added to the weekly order or shopping list and updated in the inventory database as being ordered. Further, the delivery of the orders could be automatically detected by the IFTT email analysis extension, but for accessing the order in the notification and updating the inventory of items delivered in a given order, one would need to write a bit of the code, for each of the suppliers, i.e., Amazon, Jet, Target, Chewy etc. The high turnover items with no reserve (fresh groceries) do not justify keeping track of their inventory, just remember to reorder them or add to the shopping list and complete the recurring task of doing grocery shopping, of which the voice assistants and personal devices could be reminding you about.
jka 6y ago
If you were going to tackle the ingredient data-entry problem, what would your preferred system design be? On the other end of the system, I'm hoping to implement a way to bulk load recipes into grocy, using the open source recipe-scrapers[0] library. [0] - https://github.com/hhursev/recipe-scrapers https://github.com/hhursev/recipe-scrapers
- aindilis 6y ago
  There are also lots of MealMaster recipes available online at sites like this: http://thehoseys.org/buster/recipes.html http://thehoseys.org/buster/recipes.html
huffmsa 6y ago
Sound like you stole my notebook on this subject.
toss1 6y ago
Just discovered a potential workaround with one vendor. It turns out that with an Amazon Prime account and shopping at Whole Foods the entire history is available down the the SKU, quantity, and date. Discovered this when going to order a delivery for the first time in the COVID-19 outbreak - aside from the generic shopping selection, we can also pick from our own previous purchases. I haven't checked if there are other methods to access the data (e.g., with the app, or some history list), but there's the potential for at least some screen-scraping, and maybe they'll make the history available in a downloadable file if we pester them? It'd certainly help both this app and making their stores a bit 'stickier'.
zebnyc 6y ago
I would imagine that all this tracking & automation would hit have to address the concerns of security/privacy. Also, costs would simply explode making the whole solution a non-starter for a lot of households.
- deleted 6y ago
  [deleted]
chipperyman573 6y ago
If you shop at walmart, there is an option to add a receipt to your account by scanning the barcode in the app (or it shows up by itself when you use walmart pay). These can be retrieved online from anywhere (not just the app) and is how I do it, way easier than you think. It basically comes down to $(".icon-button-children").each((index, item) => ($(item).click())) // expand all items $(".LinesEllipsis ").each((index, item) => console.log(item.innerText)) // record the stuff on https://www.walmart.com/account https://www.walmart.com/account
- davchana 6y ago
  If you pay from Walmart Pay app, with any card, receipts get added to your account automatically.
- pc86 6y ago
  But then you have to shop at Walmart.
bauerd 6y ago
Re data entry: was thinking this through a lot recently. Basically what I'd like to build requires either (1) product recognition vision like Google Lens offers, or (2) a barcode scanner and an extensive barcode product database. Both can be combined of course. My solution would be to read frames from a smartphone's camera until a barcode is detected. This can be achieved with eg Firebase ML, on-device. If a barcode database lookup gives a product, put it on the in/out list. If not, send the frame to a product recognition vision service. This could be Google Cloud Vision AI, but they don't give you access to their product set that backs Google Lens. Finally, provide controls to adjust the number of items on the in/out list I thought about OCRing supermarket receipts too but these differ so greatly in layout etc. per country I figured it's not the way to go Edit: Problem you run into as not all of this can be done on-device is privacy concerns of coursem just my thoughts on what interface I'd like to have
- jka 6y ago
  The 'Open Food Facts' and 'Open Product Data' projects could be of interest for barcodes: https://world.openfoodfacts.org/ https://world.openfoodfacts.org/ https://product-open-data.com/ https://product-open-data.com/ I just tested Product Open Data using the barcode from a can of Coconut Milk from my cupboard, and it returned the correct product (including a photograph) and the name of the brand and their owner. Edit: updated to add Open Food Facts, given that Open Product Data appears no-longer-maintained
  - bauerd 6y ago
    Yeah tried it with a variety of products. It doesn't know about enough products, makes entry unreliable. Think of utmost importance here is that the entry works very reliably, if I have to fallback to manual entry on every fifth product, that sucks
    - jka 6y ago
      Hmm, yep. That's a problem, especially if there isn't a straightforward way to contribute missing products and corrections. Do you know of any alternative data sources? Edit: since writing these comments, I've learned that Product Open Data may be unmaintained[0]. [0] - https://discuss.okfn.org/t/can-we-resurrect-product-open-data-com/173/32 https://discuss.okfn.org/t/can-we-resurrect-product-open-dat...
starpilot 6y ago
With the image recognition, I think the dream might be: take a picture of your shopping cart or basket, recognize any product that is at no more than 50% occluded. Put a red square around any item not recognized, let the user pick it up and rotate it until it's recognized.
homarp 6y ago
there is https://github.com/Forceu/barcodebuddy https://github.com/Forceu/barcodebuddy that integrates with grocy: "If already in Grocys system, it will consume/add/open the product in Grocy. If an unknown barcode is passed, the product name will be looked up and a corresponding product can be chosen in the Web UI. Tags can be saved - if a new product contains the tag in the name, the product will be already preselected in the drop-down list."
cheez 6y ago
I would be OK if we can take a receipt, or an order email, dump it into the system and it takes best guesses. For automation, at least.
MidgetGourde 6y ago
Me too, was thinking about creating simple web app. for myself. Purely from the view of food wastage. I live alone and it's pretty difficult sometimes being able to plan meals when some food packs don't last. Freezable things are OK, but sometimes the salad goes in the bin. Tracking these things avoids purchasing any unnecessary items from the shop. Will check this out.
hackme1234 6y ago
Agree. This is good for hobbyists but a pen and paper solution is so much simpler.

NamTaf 6y ago

Absolutely agree with everything you've said. I had dreamed of a Libib[1] for my kitchen, but knew I would have to do all the painful data adding and it was too much of a bother. The other big challenge I never resolved was how you'd account for e.g. using 1/4 cup of flour out of a bigger volume. Or taken to its extreme, cooking oil. How do you know how much your 'splash' is? You can't predict the remaining volume without a lot of fiddling to measure it and that defeats the purpose. In the end, I opted for manual databases too, but they're pain to keep up-to-date. I still think there's a lot of value in a database for all-or-nothing style ingredients, but it was enough to deter me. I'm glad someone is less lazy. [1]: https://www.libib.com/ https://www.libib.com/

six2seven 6y ago
> The other big challenge I never resolved was how you'd account for e.g. using 1/4 cup of flour out of a bigger volume. Or taken to its extreme, cooking oil. How do you know how much your 'splash' is? You can't predict the remaining volume without a lot of fiddling to measure it and that defeats the purpose. Definitely. Or, even, just taking some snacks to another room, and later placing it back to an 'intelligent' cupboard, let's say. In extreme cases, one may need be under a constant tracking at home. But, simplifying, another option may be to use an improved voice assistant when cooking. One would need then to interact with the VA to note and confirm the stuff used. Improved VA, in essence, would try to 'understand' the activities and interact, instead of being a passive one (as most of currently are), that one needs to say commands. This would require adding it access to camera and maybe other sensors available at 'smart' home. Definitely a lot of interesting and challenging problems to tackle not forgetting too about the user privacy part.
- byproxy 6y ago
  My naive solution, assuming a sufficiently "smart" home, would be to have each shelf be an accurate scale. If it's sensitive enough, it should be able to detect the weight of what was consumed.
  - urzav 6y ago
    A shelf for each item?
- deleted 6y ago
  [deleted]
- urzav 6y ago
  What you are talking about is called taking inventory. Even if automated tracking is being performed, i.e., by scanning the boxes or containers when they are moved out and updating the quantity in the database, a periodic inventory audit is performed in the warehouse to reconcile the actual number of the product on the shelf with the one derived from reflecting any additions or subtractions of the product. In the household setting, the most realistic system are we r would be to keep track of the addition (purchase/replenishment) ołl the containers/packages of each product and subtraction when the entire container is used-up and discarded. Obviously, this system would require maintaining at least one spare container for each product and reorder replacement when the remaining inventory count is 1 or whatever number of stock of a specific product one would want to maintain. Partial expenditures (splashes or table spoons of cooking oil, pinch of silt, a number of eggs from the pack, cups of sugar or flour from the container, a serving of cereal from box etc.) of the product from the container wouldn’t be tracked. The inventory would only be updated when the container/package has been fished up or purchased and added to the home inventory. For those items which one wishes not stock more than one container or packaging, a solution would be to label such containers, so that you can re-order the product (via Alexa or Google Assistant or Siri) when the remaining amount of the product is below certain level, depending on the product. I.e., you wouldn’t stock more than one container of milk or eggs. But when you got below half full, you would know to reorder. If certain products are used exclusively by the kids, they should learn to talk to Alexa or let parents know that they would soon run out of the product (they would learn if they want to have the product available). If one would follow this system of continuous replenishment, it might be unnecessary to keep the track of the available stock/inventory. But updating of the inventory database could be done automatically, using designated commands to the voice assists which invoke IFTT engine action sequence in which the product is ordered or added to the weekly order or shopping list and updated in the inventory database as being ordered. Further, the delivery of the orders could be automatically detected by the IFTT email analysis extension, but for accessing the order in the notification and updating the inventory of items delivered in a given order, one would need to write a bit of the code, for each of the suppliers, i.e., Amazon, Jet, Target, Chewy etc. The high turnover items with no reserve (fresh groceries) do not justify keeping track of their inventory, just remember to reorder them or add to the shopping list and complete the recurring task of doing grocery shopping, of which the voice assistants and personal devices could be reminding you about.

jka 6y ago

If you were going to tackle the ingredient data-entry problem, what would your preferred system design be? On the other end of the system, I'm hoping to implement a way to bulk load recipes into grocy, using the open source recipe-scrapers[0] library. [0] - https://github.com/hhursev/recipe-scrapers https://github.com/hhursev/recipe-scrapers

aindilis 6y ago
There are also lots of MealMaster recipes available online at sites like this: http://thehoseys.org/buster/recipes.html http://thehoseys.org/buster/recipes.html

huffmsa 6y ago

Sound like you stole my notebook on this subject.

toss1 6y ago

Just discovered a potential workaround with one vendor. It turns out that with an Amazon Prime account and shopping at Whole Foods the entire history is available down the the SKU, quantity, and date. Discovered this when going to order a delivery for the first time in the COVID-19 outbreak - aside from the generic shopping selection, we can also pick from our own previous purchases. I haven't checked if there are other methods to access the data (e.g., with the app, or some history list), but there's the potential for at least some screen-scraping, and maybe they'll make the history available in a downloadable file if we pester them? It'd certainly help both this app and making their stores a bit 'stickier'.

zebnyc 6y ago

I would imagine that all this tracking & automation would hit have to address the concerns of security/privacy. Also, costs would simply explode making the whole solution a non-starter for a lot of households.

deleted 6y ago
[deleted]

chipperyman573 6y ago

If you shop at walmart, there is an option to add a receipt to your account by scanning the barcode in the app (or it shows up by itself when you use walmart pay). These can be retrieved online from anywhere (not just the app) and is how I do it, way easier than you think. It basically comes down to $(".icon-button-children").each((index, item) => ($(item).click())) // expand all items $(".LinesEllipsis ").each((index, item) => console.log(item.innerText)) // record the stuff on https://www.walmart.com/account https://www.walmart.com/account

davchana 6y ago
If you pay from Walmart Pay app, with any card, receipts get added to your account automatically.
pc86 6y ago
But then you have to shop at Walmart.

bauerd 6y ago

Re data entry: was thinking this through a lot recently. Basically what I'd like to build requires either (1) product recognition vision like Google Lens offers, or (2) a barcode scanner and an extensive barcode product database. Both can be combined of course. My solution would be to read frames from a smartphone's camera until a barcode is detected. This can be achieved with eg Firebase ML, on-device. If a barcode database lookup gives a product, put it on the in/out list. If not, send the frame to a product recognition vision service. This could be Google Cloud Vision AI, but they don't give you access to their product set that backs Google Lens. Finally, provide controls to adjust the number of items on the in/out list I thought about OCRing supermarket receipts too but these differ so greatly in layout etc. per country I figured it's not the way to go Edit: Problem you run into as not all of this can be done on-device is privacy concerns of coursem just my thoughts on what interface I'd like to have

jka 6y ago
The 'Open Food Facts' and 'Open Product Data' projects could be of interest for barcodes: https://world.openfoodfacts.org/ https://world.openfoodfacts.org/ https://product-open-data.com/ https://product-open-data.com/ I just tested Product Open Data using the barcode from a can of Coconut Milk from my cupboard, and it returned the correct product (including a photograph) and the name of the brand and their owner. Edit: updated to add Open Food Facts, given that Open Product Data appears no-longer-maintained
- bauerd 6y ago
  Yeah tried it with a variety of products. It doesn't know about enough products, makes entry unreliable. Think of utmost importance here is that the entry works very reliably, if I have to fallback to manual entry on every fifth product, that sucks
  - jka 6y ago
    Hmm, yep. That's a problem, especially if there isn't a straightforward way to contribute missing products and corrections. Do you know of any alternative data sources? Edit: since writing these comments, I've learned that Product Open Data may be unmaintained[0]. [0] - https://discuss.okfn.org/t/can-we-resurrect-product-open-data-com/173/32 https://discuss.okfn.org/t/can-we-resurrect-product-open-dat...

starpilot 6y ago

With the image recognition, I think the dream might be: take a picture of your shopping cart or basket, recognize any product that is at no more than 50% occluded. Put a red square around any item not recognized, let the user pick it up and rotate it until it's recognized.

homarp 6y ago

there is https://github.com/Forceu/barcodebuddy https://github.com/Forceu/barcodebuddy that integrates with grocy: "If already in Grocys system, it will consume/add/open the product in Grocy. If an unknown barcode is passed, the product name will be looked up and a corresponding product can be chosen in the Web UI. Tags can be saved - if a new product contains the tag in the name, the product will be already preselected in the drop-down list."

cheez 6y ago

I would be OK if we can take a receipt, or an order email, dump it into the system and it takes best guesses. For automation, at least.

MidgetGourde 6y ago

Me too, was thinking about creating simple web app. for myself. Purely from the view of food wastage. I live alone and it's pretty difficult sometimes being able to plan meals when some food packs don't last. Freezable things are OK, but sometimes the salad goes in the bin. Tracking these things avoids purchasing any unnecessary items from the shop. Will check this out.

hackme1234 6y ago

Agree. This is good for hobbyists but a pen and paper solution is so much simpler.