7 ms·

Show HN: OK-Robot: open, modular home robot framework for pick-and-drop anywhere

542 points by MahiShafiullah 3y ago

Hi all, excited to share our latest work, OK-Robot, which is an open and modular framework to perform navigation and manipulation with a robot assistant in practically any homes without having to teach the robot anything new! You can simply unbox the target robot, install OK-Robot, give it a "scan" (think a 60 second iPhone video), and start asking the robot to move arbitrary things from A to B. We already tested it out in 10 home environments in New York city, and one environment each in Pittsburgh and Fremont. We based everything off of the current best machine learning models, and so things don't quite work perfectly all the time, so we are hoping to build it together with the community! Our code is open: https://github.com/ok-robot/ok-robot https://github.com/ok-robot/ok-robot and we have a Discord server for discussion and support: https://discord.gg/wzzZJxqKYC https://discord.gg/wzzZJxqKYC If you are curious what works and what doesn't work, take a quick look at https://ok-robot.github.io/#analysis https://ok-robot.github.io/#analysis or read our paper for a detailed analysis: https://arxiv.org/abs/2401.12202 https://arxiv.org/abs/2401.12202 P.S.: while the code is open the project unfortunately isn't fully open source since one of our dependencies, AnyGrasp, has a closed-source, educational license. Apologize in advance, but we used it since that was the best grasping model we could have access to! Would love to hear more thoughts and feedback on this project!

mdfriefeld 3y ago
congrats on the awesome work!
- MahiShafiullah 3y ago
  Thank you!
footlose_3815 3y ago
This looks really cool, but I immediately think of the possibility of it starting a fire, and thinking everything's fine
alx__ 3y ago
This is rad. I would totally buy a 25k robot if I could train it to fold and put away my laundry (serious)
- deleted 3y ago
  [deleted]
- riedel 3y ago
  You might have to buy a second one to this one for folding [1] [1] https://pantor.github.io/speedfolding/ https://pantor.github.io/speedfolding/
  - MahiShafiullah 3y ago
    In fact, Hello Robot already shared a teleoperated demo of folding shirts! https://www.youtube.com/watch?v=QtG8nJ78x2M&t=180s https://www.youtube.com/watch?v=QtG8nJ78x2M&t=180s But yes, a second arm is needed.
- modeless 3y ago
  I would buy a $100k robot if it could do the laundry and the dishes and the cooking and clean up after the kids. In a heartbeat.
foreverobama 3y ago
[dead]
khnov 3y ago
It is opensource but still costs nearly 25k dollar. why is it that expensive ?
- RIMR 3y ago
  The software is open source. The hardware is proprietary and protected by patents.
  - dheera 3y ago
    I don't think patents are what is making this specific hardware expensive. Rather it's just a lack of market and supply chain scaling.
- deleted 3y ago
  [deleted]
- kscottz 3y ago
  Since when does open source mean cheap? Labor isn't free. Building custom PCBs and hardware in low quantity isn't cheap. Building, calibrating, and testing robots isn't cheap.
  - yjftsjthsd-h 3y ago
    Now in all fairness, open source tends to mean cheaper because it does reduce how much has to be invented in-house, and also (sometimes) because it lets you crowd source free labor. In software, that can lead to stuff getting completely built for free (or close) because the base costs are low and mostly consist of labor that some people might be willing to do for free. In hardware, it's likely that open source still reduces the costs, but... you can make a thousand copies of a library for free; making a thousand copies of a part is never going to be free.
  - tekknik 3y ago
    I’ve been getting back into robotics lately and one thing that’s rubbing me the wrong way is these days, with PCB way, everybody seems to be making their own boards. Why? Some boards are innovative, but how come everybody needs their own FOC controller? Can we get one project going and focus on that before adding yet another FOC controller, but this time with wireless!
dlivingston 3y ago
That's very cool. I have almost no experience with robotics, so excuse the silly questions: - How does it know what objects are? Does it use some sort of realtime object classifier neural net? What limitations are there here? - Does the robot know when it can't perform a request? I.e. if you ask it to move a large box or very heavy kettlebell? - How well does it do if the object is hidden or obscured? Does it go looking for it? What if it must move another object to get access to the requested one?
- fishbotics 3y ago
  Disclaimer: I'm not one of the authors, but I work in this area. You basically hit the nail on the head with these questions. This work is super cool, but you named a lot of the limitations with contemporary robot learning systems. 1. It's using an object classifier. It's described here (https://github.com/ok-robot/ok-robot/tree/main/ok-robot-navigation https://github.com/ok-robot/ok-robot/tree/main/ok-robot-navi...), but if I understanding it correctly basically they are using a ViT model (basically an image classification model) to do some labeling of images and projecting them onto a voxel grid. Then they are using language embeddings from CLIP to pair the language with the voxel grid. The limitations of this are that if they want this to run on the robot, they can't use the super huge versions of these models. While they could use a huge model on the cloud, that would introduce a lot of latency. 2. It almost certainly cannot identify invalid requests. There may be requests that are not covered by their language embeddings, in which case the robot would maybe do nothing. But it doesn't appear that this system has any knowledge of physics, other than the hardware limitations of the physical controller. 3. Hidden? Almost certainly wouldn't work. The voxel labeling relies on a module that labels the voxels and without visual info, it can't label them. Also, as far as I can tell, it doesn't appear to have very complex higher-order reasoning about, say, that a fork is in a drawer, which is in a kitchen, which is often in the back of a house. Partially obscured? That would be subject to the limitations of the visual classifier, so it might work. ViT is very good, but it probably depends on how obscured the object is.
  - ativzzz 3y ago
    > While they could use a huge model on the cloud, that would introduce a lot of latency. Will all the recent work to make gen. AI faster (see groq for LLM & fal.ai for stable diffusion), I wonder if the latency will become low enough to make this a non-issue or at least good enough
deleted 3y ago
[deleted]
jamesdwilson 3y ago
I'd love to see this be usable as potentially a mower and or vacuum/mop with different swappable components.
- bozhark 3y ago
  Make it micro. I want mini robots cleaning dust and debris, silently and out of my way. I don’t want macro bots getting in my way
  - observationist 3y ago
    I agree, micro bots would be best to handle the dirty jobs.
    - jamesdwilson 3y ago
      okay, so we all agree a Matryoshka doll like system similar to SD card and microSD card is appropriate then.
  - mikeegg1 3y ago
    That's what Zorg though in _Fifth Element_.
- happytiger 3y ago
  Not to counterpoint, but just for the sake of discussion, i kind of want the opposite but for maybe similar reasons. I want flexible robots that can replace my human labor. I don’t want specialist robots that are obligate specialists. Laundry, cooking, dishes, sweeping, vacuuming, and other constantly recurring tasks are what I would love to see automated not just a “robot that sweeps” like the market has been trying to sell me. Ever since I read the second shift book about the unpaid extra 40 hour week women work doing domestic tasks I’ve dreamed of robots replacing that for humanity. It’s a massive cost to people individually and humanity overall, and kind of a silent epidemic. It’s crazy but freeing up half of humanity from the drudge work of daily chores is one of the most obvious disruptive technology plays. I rarely hear people put the robot revolution in this context, but I very much think we should start doing so. Here’s a good overview for the uninitiated: https://www.americanprogress.org/article/unequal-division-labor/ https://www.americanprogress.org/article/unequal-division-la...
bshah_ 3y ago
The failures analysis is super well done, nice work! Curious what qualifies as hardware failure, e.g. there's 5 trials where the "Realsense gave bad depth", and how that's determined.
- MahiShafiullah 3y ago
  Thanks! We collect all the data and analyze it post-facto to see what may have caused the failure. For example, on the 5 trials you mentioned, the Realsense gave wrong depth on transparent or semi-transparent objects, and so the pointcloud generated from the robot's head camera was simply wrong.
Geisterde 3y ago
I like the presentation of this, heres 10 different environments and multiple videos of each.
fnordpiglet 3y ago
This is remarkable and could be life changing for the disabled, elderly, gamers, or profoundly lazy and their caretakers.
- marci 3y ago
  I forgot where I saw that, but generally, improving things for people with disabilities improves things for everyone, like making sidewalks wheelchair friendly helps parents with a stroller, or people carrying heavy stuff, walking with a cane, young children on bicycles, people who can't see well...
  - salviati 3y ago
    I heard this from Anna Martelli Ravenscroft in her presentation "Diversity as a Dependency" [0] [0] https://www.youtube.com/watch?v=wOpdDxJzNkw https://www.youtube.com/watch?v=wOpdDxJzNkw
  - sandworm101 3y ago
    >> improving things for people with disabilities improves things for everyone Everything has its limits. Many years ago I was involved in building a series of staircases in a rock climbing area inside a park. There were about a hundred steps in a handful of orientations to get from the parking lot over a rocky hill to the small valleys behind. The project was primarily to prevent trail erosion and falls. These steps weren't going to even have handrails. (Think 2x6 framed boxes filled with dirt and bolted to the rock.) Then someone in government said if we wanted to use donated money inside a park we would have to somehow make the project wheelchair accessible. All stop. Project over. No stairs were built. Access trail remained a mess. We were going to replicate these stairs from another climbing area in BC. There is no way to make such a thing wheelchair accessible. https://sonnybou.ca/ssbou2001/skaha01.jpg https://sonnybou.ca/ssbou2001/skaha01.jpg
    - developerDan 3y ago
      It’s so frustrating that city leaders can’t even try to use common sense. Where I live a parking requirement blocked a restaurant from being built and our city council publicly acknowledged that there isn’t enough space for parking and a building, but “that’s the law” so they blocked it. Lazy idiots.
hansonpeter 3y ago
[dead]
deleted 3y ago
[deleted]
taco_emoji 3y ago
I know nothing about robotics, but can someone ELI5 why the robot makes so many extraneous movements? E.g. the video that shows it moving Takis from the desk to the nightstand, it approaches the desk, and then the arm mechanism moves all the way down (an unnecessary maneuver), then rises again before reaching the level needed to pick up the Takis.
- throwup238 3y ago
  A lot of those movements are there to zero out the axes so that each movement starts from a known good position and orients itself against the camera. Usually there's a switch that, for example, senses when the body goes all the way to the bottom, which is the origin for the whole positioning system. Several other movements are for safety since it doesn't have a bunch of cameras and really complex logic for collision avoidance so it resets to a smaller profile between moving around. Since motors are capable of very precise movements and errors accumulate, this is a best practice when starting new movements. Humans instead have a complex hand-eye coordination system that trained all of our lives (and some people are better at it than others).
  - leptons 3y ago
    >A lot of those movements are there to zero out the axes so that each movement starts from a known good position and orients itself against the camera. Usually there's a switch that, for example, senses when the body goes all the way to the bottom, which is the origin for the whole positioning system. Ideally the "zeroing" should be done once when the robot "wakes up" or only once in a while, and there should be digital encoders on all motors, the position should always be known within a tiny margin of error, and not enough to cause a problem for positioning. At least that's how I'd do it, I'm not sure how they built this thing.
    - MahiShafiullah 3y ago
      It's always a trade-off! You could have more accurate sensors and motors that are more expensive, or you can have cheaper motors with no sensors and higher accumulated errors. Since this is more of a research project than a product, we went for a cheap robot with the slower-but-more-accurate approach.
chubs 3y ago
A friend is working on a slightly related project, I’m curious how they map out the room in voxels, anyone care to suggest how this is done?
- MahiShafiullah 3y ago
  The mapping process can be done with any RGB-D cameras, we use an iPhone pro but any apple devide with AR-Kit should work. Once we have a sequence of RGB-D images with associated camera poses, we can just backproject the pixels (and any associated information, like CLIP embeddings) using the depth into voxels.
btbuildem 3y ago
I've been watching this project for a while now, great progress! I envision an integration with a mobility aid (eg, a wheelchair) for someone with limited control over their limbs. Imagine a "smart" exoskeleton that can help with otherwise impossible tasks -- it could be a game-changer for so many people.
k4rli 3y ago
It's cool but what's the point for a normal person? Useful for warehouses and manufacturing but I don't see myself ever needing such things
- polygamous_bat 3y ago
  Are elderly or disabled people "normal" in your book? Do you see yourself or your loved ones growing old someday?
- MahiShafiullah 3y ago
  A large motivation behind this line of home-robot work for me is thinking about the elderly, people with disabilities, or busy parents who simply don't have enough time to do it all. I am personally hopeful that we can teach AI to take the jobs that no one wants rather than the jobs that everyone wants :)
rsync 3y ago
I very much want a stabilized platform vehicle that I can send point-to-point with a payload on it. So, a gyro-stabilized platform like a segway that I can send back and forth from point A to point B on a not-terrible-but-rough (walking path) route. I have tried to stay abreast of the options in the past and have never seen anything that matches this ... does anyone know if there is anything new that matches this use-case ? (the use-case is a tray of drinks and hors d'oeuvres that needs to go from one part of a property to another without spilling ... needs to be minimally all-terrain)
- thebruce87m 3y ago
  You’ve maybe seen these already in restaurants? https://www.pudurobotics.com/product/detail/bellabot https://www.pudurobotics.com/product/detail/bellabot Not sure I’ve seen them take drinks though, but definitely food.
  - KomoD 3y ago
    Some of these robots are kind of "jittery" from what I've seen, I've seen soup delivered without spilling but it might not work well with drinks (not on "rough" terrain obviously)
- fragmede 3y ago
  https://www.youtube.com/watch?v=VGzRfvgnS_s https://www.youtube.com/watch?v=VGzRfvgnS_s The technology's out there, just need to glue it all together.
- EnigmaFlare 3y ago
  You can carry drinks on a plate suspended on strings. Keeps g pointing towards the bottom of the glasses. No electronic stabilizing needed.
owenpalmer 3y ago
Isn't this the same as dobb-e? https://dobb-e.com/ https://dobb-e.com/
- ohnit 3y ago
  The projects look related and have an author in common. Both are mentioned on the website for robot that they used: https://hello-robot.com/stretch-embodied-ai https://hello-robot.com/stretch-embodied-ai
- MahiShafiullah 3y ago
  No, although it has some of the same people on the team (aka I'm the first author there, and my advisor is advising both projects :) ) The primary difference is that this is zero-shot (meaning the robot needs 0 (zero!) new data in a new home) but has only two skills (pick and drop); where Dobb-E can have many skills but will need you to give some demonstrations in a new home.
westmeal 3y ago
Is the title a reference to OK computer or is it just something you all came up with?
- MahiShafiullah 3y ago
  The title has multiple meanings, some credits definitely should go to OK computer/radiohead, but also "OK Google" for controlling a home assistant, open-knowledge (OK) models, etc.
CodeWriter23 3y ago
Back in the day, my friend would lament not having “closetgrep” to find the needed thingy stored in an overfilled closet.
- drivers99 3y ago
  I bought 40 little cardboard boxes (VATTENTRÅG - they are pretty cheap, and shallow enough so you don't have to go digging too much) from IKEA and started putting what was in each one in text files so I could literally do that (grep for things). I still need to catalog 38 out of the 40 boxes though so I'm reconsidering my strategy.
ctoth 3y ago
For solving long term tasks like finding things that aren't there, you can turn the annotated scene into a templated description and feed it to a large-enough model trained on interactive fiction. You are standing in a kitchen. Ahead of you to your right there is a large refrigerator with the handle on the right side. There is a set of cabinets to your left with a plate sitting on the counter above them. > get beer You don't see any beer here. << COT: I know that beer is often found in the fridge. I should try opening the refrigerator > open fridge Opening the refrigerator reveals 4 cans of beer. > get beer taken Obviously we're still several years from this working, but it's very exciting to consider. Interactive Fiction narrative fed by real sensors plus chain-of-thought blocks as internal monologue.
- garblegarble 3y ago
  Great, now we can teach robots to wander around rooms looking for things, saying "keys, keys, keys... where would I put keys?"
  - sillysaurusx 3y ago
    Get a Tile. I have one attached to my keys, and saying "hey Alexa, find my keys" has been really nice. We also have one taped to our remote, which turned out to be excellent since our couch constantly eats it. I just wish it lit up, but sound-only is fine. It would be really cool if the robot could just know where your keys are by attaching some kind of tile-type thing to it. If it already has a scan of your home, theoretically it could show a photo. But I have no idea if it’s possible to pinpoint an object via rfid.
  - ponderings 3y ago
    I can pick up and place objects myself if only I could remember where I put them. You could take video data and have fuzzy identification of objects moving around, then throw away the video and keep track of the objects, the blue floppy thing (gloves) and the metal shiny deforming things (keys) then have a more constructive dialog about the keys. A voice responding, what do the keys look like? Is there a blue square thing on the key ring? The less identifiable the object the funnier the discussion. What shirt? You have many shirts! Oh, the blue one, you have 4 of those, one in the sink, one behind the bed, one in the laundry basket, one in the closet. Oh the one with stripes! Why didn't you say so, it's behind the bed bro. It could also ask you if they are suppose to be on the outside in the front door after you close it.
Whooping7116 3y ago
[dead]
nxobject 3y ago
Take that, SHRDLU!
ghostly_s 3y ago
why are these general purpose robots always so slow? Intuitively we expect machines to be able to do tasks faster than humans, but even the 5x speed video is much slower than a human could do the task.
jacobsenscott 3y ago
It appears slow, but tests show it completes most tasks more quickly, accurately, and with less complaining, than most members of the gen z cohort.
- idiotsecant 3y ago
  Weird energy.
- DoktorDelta 3y ago
  It must not have anything better to do with its time
ingend88 3y ago
For a long time, I wanted to use a robot with a gripper to make tea. Is there any 6DOF robot available within a reasonable price of <$1000 to do so ?
rapjr9 3y ago
Robots like this will have a small market until they can handle obstacles. The cat toy that the cat left in the middle of the floor, the papers that an open window blew off the table, the toys the kids left scattered about, the pencil that rolled off the desk while you were away, the dirty laundry you left laying on the floor, the ridge between carpet and hardwood floors, doors left open or closed, and more. That means there may be several tasks that intervene before a primary task can accomplished (move the toys, pick up the papers, pick up the laundry, open the door). Some obstacles will semi-permanently block a wheeled robot, such as cables, things stacked that you don't want moved, furniture, a sleeping pet, stacked unopened packages from the mail, etc. I believe this means general purpose home robots can not have wheels, they must have legs, perhaps more than two legs for stability. It may sound weird but I think the ideal design might be somewhere between a large friendly spider and a dog. It's odd how robotics has mostly fallen into this idea that the world is two dimensional and flat. They've idealized away the really difficult problems of dealing with mobility in a 3D world. Note that everything this robot does involves only planar horizontal surfaces. Basically it looks like a person had to go through the rooms and clean them up for the robot to function. Roomba's have the same problem.
- iancmceachern 3y ago
  This same problem exists in the hospital setting on anything with wheels. They solve it there with "cord pushers". Basically cattle guards for stuff on the floor. Not every problem needs a complex solution. I think the brilliance of this project is in its simplicity, particularly in the robot design.
  - rapjr9 3y ago
    Take a look at these pictures: https://duckduckgo.com/?q=cluttered+old+persons+home&atb=v314-1&iax=images&ia=images https://duckduckgo.com/?q=cluttered+old+persons+home&atb=v31... I was thinking that people who live in an environment like this are most in need of a robot to help them.
    - iancmceachern 3y ago

mdfriefeld 3y ago

congrats on the awesome work!

MahiShafiullah 3y ago
Thank you!

footlose_3815 3y ago

This looks really cool, but I immediately think of the possibility of it starting a fire, and thinking everything's fine

alx__ 3y ago

This is rad. I would totally buy a 25k robot if I could train it to fold and put away my laundry (serious)

deleted 3y ago
[deleted]
riedel 3y ago
You might have to buy a second one to this one for folding [1] [1] https://pantor.github.io/speedfolding/ https://pantor.github.io/speedfolding/
- MahiShafiullah 3y ago
  In fact, Hello Robot already shared a teleoperated demo of folding shirts! https://www.youtube.com/watch?v=QtG8nJ78x2M&t=180s https://www.youtube.com/watch?v=QtG8nJ78x2M&t=180s But yes, a second arm is needed.
modeless 3y ago
I would buy a $100k robot if it could do the laundry and the dishes and the cooking and clean up after the kids. In a heartbeat.

foreverobama 3y ago

[dead]

khnov 3y ago

It is opensource but still costs nearly 25k dollar. why is it that expensive ?

RIMR 3y ago
The software is open source. The hardware is proprietary and protected by patents.
- dheera 3y ago
  I don't think patents are what is making this specific hardware expensive. Rather it's just a lack of market and supply chain scaling.
deleted 3y ago
[deleted]
kscottz 3y ago
Since when does open source mean cheap? Labor isn't free. Building custom PCBs and hardware in low quantity isn't cheap. Building, calibrating, and testing robots isn't cheap.
- yjftsjthsd-h 3y ago
  Now in all fairness, open source tends to mean cheaper because it does reduce how much has to be invented in-house, and also (sometimes) because it lets you crowd source free labor. In software, that can lead to stuff getting completely built for free (or close) because the base costs are low and mostly consist of labor that some people might be willing to do for free. In hardware, it's likely that open source still reduces the costs, but... you can make a thousand copies of a library for free; making a thousand copies of a part is never going to be free.
- tekknik 3y ago
  I’ve been getting back into robotics lately and one thing that’s rubbing me the wrong way is these days, with PCB way, everybody seems to be making their own boards. Why? Some boards are innovative, but how come everybody needs their own FOC controller? Can we get one project going and focus on that before adding yet another FOC controller, but this time with wireless!

dlivingston 3y ago

That's very cool. I have almost no experience with robotics, so excuse the silly questions: - How does it know what objects are? Does it use some sort of realtime object classifier neural net? What limitations are there here? - Does the robot know when it can't perform a request? I.e. if you ask it to move a large box or very heavy kettlebell? - How well does it do if the object is hidden or obscured? Does it go looking for it? What if it must move another object to get access to the requested one?

fishbotics 3y ago
Disclaimer: I'm not one of the authors, but I work in this area. You basically hit the nail on the head with these questions. This work is super cool, but you named a lot of the limitations with contemporary robot learning systems. 1. It's using an object classifier. It's described here (https://github.com/ok-robot/ok-robot/tree/main/ok-robot-navigation https://github.com/ok-robot/ok-robot/tree/main/ok-robot-navi...), but if I understanding it correctly basically they are using a ViT model (basically an image classification model) to do some labeling of images and projecting them onto a voxel grid. Then they are using language embeddings from CLIP to pair the language with the voxel grid. The limitations of this are that if they want this to run on the robot, they can't use the super huge versions of these models. While they could use a huge model on the cloud, that would introduce a lot of latency. 2. It almost certainly cannot identify invalid requests. There may be requests that are not covered by their language embeddings, in which case the robot would maybe do nothing. But it doesn't appear that this system has any knowledge of physics, other than the hardware limitations of the physical controller. 3. Hidden? Almost certainly wouldn't work. The voxel labeling relies on a module that labels the voxels and without visual info, it can't label them. Also, as far as I can tell, it doesn't appear to have very complex higher-order reasoning about, say, that a fork is in a drawer, which is in a kitchen, which is often in the back of a house. Partially obscured? That would be subject to the limitations of the visual classifier, so it might work. ViT is very good, but it probably depends on how obscured the object is.
- ativzzz 3y ago
  > While they could use a huge model on the cloud, that would introduce a lot of latency. Will all the recent work to make gen. AI faster (see groq for LLM & fal.ai for stable diffusion), I wonder if the latency will become low enough to make this a non-issue or at least good enough

deleted 3y ago

[deleted]

jamesdwilson 3y ago

I'd love to see this be usable as potentially a mower and or vacuum/mop with different swappable components.

bozhark 3y ago
Make it micro. I want mini robots cleaning dust and debris, silently and out of my way. I don’t want macro bots getting in my way
- observationist 3y ago
  I agree, micro bots would be best to handle the dirty jobs.
  - jamesdwilson 3y ago
    okay, so we all agree a Matryoshka doll like system similar to SD card and microSD card is appropriate then.
- mikeegg1 3y ago
  That's what Zorg though in _Fifth Element_.
happytiger 3y ago
Not to counterpoint, but just for the sake of discussion, i kind of want the opposite but for maybe similar reasons. I want flexible robots that can replace my human labor. I don’t want specialist robots that are obligate specialists. Laundry, cooking, dishes, sweeping, vacuuming, and other constantly recurring tasks are what I would love to see automated not just a “robot that sweeps” like the market has been trying to sell me. Ever since I read the second shift book about the unpaid extra 40 hour week women work doing domestic tasks I’ve dreamed of robots replacing that for humanity. It’s a massive cost to people individually and humanity overall, and kind of a silent epidemic. It’s crazy but freeing up half of humanity from the drudge work of daily chores is one of the most obvious disruptive technology plays. I rarely hear people put the robot revolution in this context, but I very much think we should start doing so. Here’s a good overview for the uninitiated: https://www.americanprogress.org/article/unequal-division-labor/ https://www.americanprogress.org/article/unequal-division-la...

bshah_ 3y ago

The failures analysis is super well done, nice work! Curious what qualifies as hardware failure, e.g. there's 5 trials where the "Realsense gave bad depth", and how that's determined.

MahiShafiullah 3y ago
Thanks! We collect all the data and analyze it post-facto to see what may have caused the failure. For example, on the 5 trials you mentioned, the Realsense gave wrong depth on transparent or semi-transparent objects, and so the pointcloud generated from the robot's head camera was simply wrong.

Geisterde 3y ago

I like the presentation of this, heres 10 different environments and multiple videos of each.

fnordpiglet 3y ago

This is remarkable and could be life changing for the disabled, elderly, gamers, or profoundly lazy and their caretakers.

marci 3y ago
I forgot where I saw that, but generally, improving things for people with disabilities improves things for everyone, like making sidewalks wheelchair friendly helps parents with a stroller, or people carrying heavy stuff, walking with a cane, young children on bicycles, people who can't see well...
- salviati 3y ago
  I heard this from Anna Martelli Ravenscroft in her presentation "Diversity as a Dependency" [0] [0] https://www.youtube.com/watch?v=wOpdDxJzNkw https://www.youtube.com/watch?v=wOpdDxJzNkw
- sandworm101 3y ago
  >> improving things for people with disabilities improves things for everyone Everything has its limits. Many years ago I was involved in building a series of staircases in a rock climbing area inside a park. There were about a hundred steps in a handful of orientations to get from the parking lot over a rocky hill to the small valleys behind. The project was primarily to prevent trail erosion and falls. These steps weren't going to even have handrails. (Think 2x6 framed boxes filled with dirt and bolted to the rock.) Then someone in government said if we wanted to use donated money inside a park we would have to somehow make the project wheelchair accessible. All stop. Project over. No stairs were built. Access trail remained a mess. We were going to replicate these stairs from another climbing area in BC. There is no way to make such a thing wheelchair accessible. https://sonnybou.ca/ssbou2001/skaha01.jpg https://sonnybou.ca/ssbou2001/skaha01.jpg
  - developerDan 3y ago
    It’s so frustrating that city leaders can’t even try to use common sense. Where I live a parking requirement blocked a restaurant from being built and our city council publicly acknowledged that there isn’t enough space for parking and a building, but “that’s the law” so they blocked it. Lazy idiots.

hansonpeter 3y ago

[dead]

deleted 3y ago

[deleted]

taco_emoji 3y ago

I know nothing about robotics, but can someone ELI5 why the robot makes so many extraneous movements? E.g. the video that shows it moving Takis from the desk to the nightstand, it approaches the desk, and then the arm mechanism moves all the way down (an unnecessary maneuver), then rises again before reaching the level needed to pick up the Takis.

throwup238 3y ago
A lot of those movements are there to zero out the axes so that each movement starts from a known good position and orients itself against the camera. Usually there's a switch that, for example, senses when the body goes all the way to the bottom, which is the origin for the whole positioning system. Several other movements are for safety since it doesn't have a bunch of cameras and really complex logic for collision avoidance so it resets to a smaller profile between moving around. Since motors are capable of very precise movements and errors accumulate, this is a best practice when starting new movements. Humans instead have a complex hand-eye coordination system that trained all of our lives (and some people are better at it than others).
- leptons 3y ago
  >A lot of those movements are there to zero out the axes so that each movement starts from a known good position and orients itself against the camera. Usually there's a switch that, for example, senses when the body goes all the way to the bottom, which is the origin for the whole positioning system. Ideally the "zeroing" should be done once when the robot "wakes up" or only once in a while, and there should be digital encoders on all motors, the position should always be known within a tiny margin of error, and not enough to cause a problem for positioning. At least that's how I'd do it, I'm not sure how they built this thing.
  - MahiShafiullah 3y ago
    It's always a trade-off! You could have more accurate sensors and motors that are more expensive, or you can have cheaper motors with no sensors and higher accumulated errors. Since this is more of a research project than a product, we went for a cheap robot with the slower-but-more-accurate approach.

chubs 3y ago

A friend is working on a slightly related project, I’m curious how they map out the room in voxels, anyone care to suggest how this is done?

MahiShafiullah 3y ago
The mapping process can be done with any RGB-D cameras, we use an iPhone pro but any apple devide with AR-Kit should work. Once we have a sequence of RGB-D images with associated camera poses, we can just backproject the pixels (and any associated information, like CLIP embeddings) using the depth into voxels.

btbuildem 3y ago

I've been watching this project for a while now, great progress! I envision an integration with a mobility aid (eg, a wheelchair) for someone with limited control over their limbs. Imagine a "smart" exoskeleton that can help with otherwise impossible tasks -- it could be a game-changer for so many people.

k4rli 3y ago

It's cool but what's the point for a normal person? Useful for warehouses and manufacturing but I don't see myself ever needing such things

polygamous_bat 3y ago
Are elderly or disabled people "normal" in your book? Do you see yourself or your loved ones growing old someday?
MahiShafiullah 3y ago
A large motivation behind this line of home-robot work for me is thinking about the elderly, people with disabilities, or busy parents who simply don't have enough time to do it all. I am personally hopeful that we can teach AI to take the jobs that no one wants rather than the jobs that everyone wants :)

rsync 3y ago

I very much want a stabilized platform vehicle that I can send point-to-point with a payload on it. So, a gyro-stabilized platform like a segway that I can send back and forth from point A to point B on a not-terrible-but-rough (walking path) route. I have tried to stay abreast of the options in the past and have never seen anything that matches this ... does anyone know if there is anything new that matches this use-case ? (the use-case is a tray of drinks and hors d'oeuvres that needs to go from one part of a property to another without spilling ... needs to be minimally all-terrain)

thebruce87m 3y ago
You’ve maybe seen these already in restaurants? https://www.pudurobotics.com/product/detail/bellabot https://www.pudurobotics.com/product/detail/bellabot Not sure I’ve seen them take drinks though, but definitely food.
- KomoD 3y ago
  Some of these robots are kind of "jittery" from what I've seen, I've seen soup delivered without spilling but it might not work well with drinks (not on "rough" terrain obviously)
fragmede 3y ago
https://www.youtube.com/watch?v=VGzRfvgnS_s https://www.youtube.com/watch?v=VGzRfvgnS_s The technology's out there, just need to glue it all together.
EnigmaFlare 3y ago
You can carry drinks on a plate suspended on strings. Keeps g pointing towards the bottom of the glasses. No electronic stabilizing needed.

owenpalmer 3y ago

Isn't this the same as dobb-e? https://dobb-e.com/ https://dobb-e.com/

ohnit 3y ago
The projects look related and have an author in common. Both are mentioned on the website for robot that they used: https://hello-robot.com/stretch-embodied-ai https://hello-robot.com/stretch-embodied-ai
MahiShafiullah 3y ago
No, although it has some of the same people on the team (aka I'm the first author there, and my advisor is advising both projects :) ) The primary difference is that this is zero-shot (meaning the robot needs 0 (zero!) new data in a new home) but has only two skills (pick and drop); where Dobb-E can have many skills but will need you to give some demonstrations in a new home.

westmeal 3y ago

Is the title a reference to OK computer or is it just something you all came up with?

MahiShafiullah 3y ago
The title has multiple meanings, some credits definitely should go to OK computer/radiohead, but also "OK Google" for controlling a home assistant, open-knowledge (OK) models, etc.

CodeWriter23 3y ago

Back in the day, my friend would lament not having “closetgrep” to find the needed thingy stored in an overfilled closet.

drivers99 3y ago
I bought 40 little cardboard boxes (VATTENTRÅG - they are pretty cheap, and shallow enough so you don't have to go digging too much) from IKEA and started putting what was in each one in text files so I could literally do that (grep for things). I still need to catalog 38 out of the 40 boxes though so I'm reconsidering my strategy.

ctoth 3y ago

For solving long term tasks like finding things that aren't there, you can turn the annotated scene into a templated description and feed it to a large-enough model trained on interactive fiction. You are standing in a kitchen. Ahead of you to your right there is a large refrigerator with the handle on the right side. There is a set of cabinets to your left with a plate sitting on the counter above them. > get beer You don't see any beer here. << COT: I know that beer is often found in the fridge. I should try opening the refrigerator > open fridge Opening the refrigerator reveals 4 cans of beer. > get beer taken Obviously we're still several years from this working, but it's very exciting to consider. Interactive Fiction narrative fed by real sensors plus chain-of-thought blocks as internal monologue.

garblegarble 3y ago
Great, now we can teach robots to wander around rooms looking for things, saying "keys, keys, keys... where would I put keys?"
- sillysaurusx 3y ago
  Get a Tile. I have one attached to my keys, and saying "hey Alexa, find my keys" has been really nice. We also have one taped to our remote, which turned out to be excellent since our couch constantly eats it. I just wish it lit up, but sound-only is fine. It would be really cool if the robot could just know where your keys are by attaching some kind of tile-type thing to it. If it already has a scan of your home, theoretically it could show a photo. But I have no idea if it’s possible to pinpoint an object via rfid.
- ponderings 3y ago
  I can pick up and place objects myself if only I could remember where I put them. You could take video data and have fuzzy identification of objects moving around, then throw away the video and keep track of the objects, the blue floppy thing (gloves) and the metal shiny deforming things (keys) then have a more constructive dialog about the keys. A voice responding, what do the keys look like? Is there a blue square thing on the key ring? The less identifiable the object the funnier the discussion. What shirt? You have many shirts! Oh, the blue one, you have 4 of those, one in the sink, one behind the bed, one in the laundry basket, one in the closet. Oh the one with stripes! Why didn't you say so, it's behind the bed bro. It could also ask you if they are suppose to be on the outside in the front door after you close it.

Whooping7116 3y ago

[dead]

nxobject 3y ago

Take that, SHRDLU!

ghostly_s 3y ago

why are these general purpose robots always so slow? Intuitively we expect machines to be able to do tasks faster than humans, but even the 5x speed video is much slower than a human could do the task.

jacobsenscott 3y ago

It appears slow, but tests show it completes most tasks more quickly, accurately, and with less complaining, than most members of the gen z cohort.

idiotsecant 3y ago
Weird energy.
DoktorDelta 3y ago
It must not have anything better to do with its time

ingend88 3y ago

For a long time, I wanted to use a robot with a gripper to make tea. Is there any 6DOF robot available within a reasonable price of <$1000 to do so ?

rapjr9 3y ago

Robots like this will have a small market until they can handle obstacles. The cat toy that the cat left in the middle of the floor, the papers that an open window blew off the table, the toys the kids left scattered about, the pencil that rolled off the desk while you were away, the dirty laundry you left laying on the floor, the ridge between carpet and hardwood floors, doors left open or closed, and more. That means there may be several tasks that intervene before a primary task can accomplished (move the toys, pick up the papers, pick up the laundry, open the door). Some obstacles will semi-permanently block a wheeled robot, such as cables, things stacked that you don't want moved, furniture, a sleeping pet, stacked unopened packages from the mail, etc. I believe this means general purpose home robots can not have wheels, they must have legs, perhaps more than two legs for stability. It may sound weird but I think the ideal design might be somewhere between a large friendly spider and a dog. It's odd how robotics has mostly fallen into this idea that the world is two dimensional and flat. They've idealized away the really difficult problems of dealing with mobility in a 3D world. Note that everything this robot does involves only planar horizontal surfaces. Basically it looks like a person had to go through the rooms and clean them up for the robot to function. Roomba's have the same problem.

iancmceachern 3y ago
This same problem exists in the hospital setting on anything with wheels. They solve it there with "cord pushers". Basically cattle guards for stuff on the floor. Not every problem needs a complex solution. I think the brilliance of this project is in its simplicity, particularly in the robot design.
- rapjr9 3y ago
  Take a look at these pictures: https://duckduckgo.com/?q=cluttered+old+persons+home&atb=v314-1&iax=images&ia=images https://duckduckgo.com/?q=cluttered+old+persons+home&atb=v31... I was thinking that people who live in an environment like this are most in need of a robot to help them.
  - iancmceachern 3y ago