9 ms·

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

Benjammer 10y ago
If a bug can kill millions then it's not "better than a human" though, right?
- darawk 10y ago
  Depends on how unlikely that bug is :) probability x value, etc.
- sriram_sun 10y ago
  Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars will always be able to drive manually or override the automated controls. At some point, technology will progress enough that the benefits will outweigh the risks and people will adopt.
  - xiphias 10y ago
    People won't be able to override a buggy software: they can't even do that now, just look at the remote Audi and BMW hacks that can brake the car on the highway.
    - seanp2k2 10y ago
      Or the jeep: https://blog.kaspersky.com/blackhat-jeep-cherokee-hack-explained/9493/ https://blog.kaspersky.com/blackhat-jeep-cherokee-hack-expla... Auto mfgs seem to be about 20-30 years behind when it comes to computers. Not really surprising that Tesla is whomping them on this front, given how SV people are scrambling to work there. You don't see that with the Big 3 or really any other car mfg.
  - seanp2k2 10y ago
    Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-everything-you-need-to-know-including-full-list-of-affected-vehicles/ http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they can get you scheduled for a software update? Remember that many cars can't update critical software OTA.
    - creshal 10y ago
      A friend of mine works for VW's engine computer division. Yes, those engine computers. After all I've heard of their development methods (or lack thereof), I'm surprised the engines even start more often than one time out of ten.
      - 72deluxe 10y ago
        My VW Golf has a bug where the driver's side door will be completely unresponsive after starting the ignition, with all the lights on the door being off too. After 5 to 10 seconds it will become responsive, which is a bit annoying if you're trying to open the windows to clear the damp mist on them, as you can't.... Also, if during normal routine you run through all four electric windows to close them (so passenger, driver, passenger rear, driver rear) in that order, you hear the solenoids click in a COMPLETELY different order. I am not sure if it is prioritising the messages in some way but the order that the windows "click" is not the order I press the buttons. Also, I can get the CD player to crash. Such minor noticeable issues make me think about the quality of the more important bits somewhat. The breakdown on Toyota's safety code was interesting; and frightening really.
      - qaq 10y ago
        Care to elaborate? Kernel developers are not practising SCRUM or TDD yet are shipping a fairly stable product.
essayist 10y ago
I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol units around the DC beltway notice three separate multi-Edsel phalanxes, drivers obviously trapped inside, each phalanx moving towards the Clara Barton Parkway, which enters DC from the west. Other units notice four more phalanxes, one comprising 20 Edsels, driving into DC from the east side, on Pennsylvania Avenue. At this point, traffic helicopters see similar car clusters, more than two dozen, all over DC, all converging on a spot that looks to be between the Washington Monument and the White House. We zoom in on the headquarters of the White House Secret Service. A woman is arguing vociferously that these cars have to be stopped before they get any closer to the White House. A colleague yells back that his wife is one of those commandeered cars and she, like the rest of the "hackjacked" drivers and passengers is innocent.
- Afforess 10y ago
  I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations. http://www.goodreads.com/book/show/6665847-daemon http://www.goodreads.com/book/show/6665847-daemon
  - noir_lord 10y ago
    One of my favourites, the sequels where similarly as good.
  - pavel_lishin 10y ago
    And a really fun read. If someone here decides to get this book, get the sequel as well - Daemon ends on kind of a cliff-hanger.
    - mitchellhislop 10y ago
      Freedom (TM) is the sequel, and the author (Daniel Suarez) has a few other near-term what-if-this-all-goes-skynet books which are equally good.
bduerst 10y ago
Sounds like current hardware defects and recalls for vehicles, with the difference being it's easier to fix.
Matt3o12_ 10y ago
There is something that's called fail safe(ly). In case of any error inside the car, it should sowly decelerate and pull over. Sure, it will cause a lot of traffic problems if say 10% of all cars did that at the same time but the damage would not be as severe as a ghost driver entering the freeway with 140mph. There are a few instances where some bad tesla batteries (the standard 12 volt batteries ironically) failed, and the cars handled it perfectly. It slowed down so that the driver could do safely pull over. Sure it did not happen to all cars at once, and autonomous cars migh not be able to do that by themselves but we have a log way to go to reach 100% autonomous driving (i.e. Without a steering wheel and a car that drives everywhere humans drive and not only on San Francisco's perfect sunny roads where it's been thoroughly tested on).
- hashkb 10y ago
  Google had fail safes which failed. What if a million cars pull over to the left instead of the right as a failed failsafe? Willing to risk your life?
  - thisisnotatest 10y ago
    Yes, I'm eager to risk my life on self-driving car failures; it would be a tremendous step up from risking my life on human drivers (including myself!) as I do on a daily basis. I'm also a biker. In 2013, 4,735 pedestrians and 743 bicyclists were killed in crashes with motor vehicles. http://www.pedbikeinfo.org/data/factsheet_crash.cfm http://www.pedbikeinfo.org/data/factsheet_crash.cfm In the future when self-driving or at least augmented driving is commonplace, I hope that number will be a lot lower.
distrill 10y ago
I think it still is that simple. Clearly killing millions is not better than human control, so would be unacceptable.
hinkley 10y ago
Imagine a leap second bug causing vehicles all over the US to crash into NYE revellers.

Benjammer 10y ago

If a bug can kill millions then it's not "better than a human" though, right?

darawk 10y ago
Depends on how unlikely that bug is :) probability x value, etc.
sriram_sun 10y ago
Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars will always be able to drive manually or override the automated controls. At some point, technology will progress enough that the benefits will outweigh the risks and people will adopt.
- xiphias 10y ago
  People won't be able to override a buggy software: they can't even do that now, just look at the remote Audi and BMW hacks that can brake the car on the highway.
  - seanp2k2 10y ago
    Or the jeep: https://blog.kaspersky.com/blackhat-jeep-cherokee-hack-explained/9493/ https://blog.kaspersky.com/blackhat-jeep-cherokee-hack-expla... Auto mfgs seem to be about 20-30 years behind when it comes to computers. Not really surprising that Tesla is whomping them on this front, given how SV people are scrambling to work there. You don't see that with the Big 3 or really any other car mfg.
- seanp2k2 10y ago
  Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-everything-you-need-to-know-including-full-list-of-affected-vehicles/ http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they can get you scheduled for a software update? Remember that many cars can't update critical software OTA.
  - creshal 10y ago
    A friend of mine works for VW's engine computer division. Yes, those engine computers. After all I've heard of their development methods (or lack thereof), I'm surprised the engines even start more often than one time out of ten.
    - 72deluxe 10y ago
      My VW Golf has a bug where the driver's side door will be completely unresponsive after starting the ignition, with all the lights on the door being off too. After 5 to 10 seconds it will become responsive, which is a bit annoying if you're trying to open the windows to clear the damp mist on them, as you can't.... Also, if during normal routine you run through all four electric windows to close them (so passenger, driver, passenger rear, driver rear) in that order, you hear the solenoids click in a COMPLETELY different order. I am not sure if it is prioritising the messages in some way but the order that the windows "click" is not the order I press the buttons. Also, I can get the CD player to crash. Such minor noticeable issues make me think about the quality of the more important bits somewhat. The breakdown on Toyota's safety code was interesting; and frightening really.
    - qaq 10y ago
      Care to elaborate? Kernel developers are not practising SCRUM or TDD yet are shipping a fairly stable product.

essayist 10y ago

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol units around the DC beltway notice three separate multi-Edsel phalanxes, drivers obviously trapped inside, each phalanx moving towards the Clara Barton Parkway, which enters DC from the west. Other units notice four more phalanxes, one comprising 20 Edsels, driving into DC from the east side, on Pennsylvania Avenue. At this point, traffic helicopters see similar car clusters, more than two dozen, all over DC, all converging on a spot that looks to be between the Washington Monument and the White House. We zoom in on the headquarters of the White House Secret Service. A woman is arguing vociferously that these cars have to be stopped before they get any closer to the White House. A colleague yells back that his wife is one of those commandeered cars and she, like the rest of the "hackjacked" drivers and passengers is innocent.

Afforess 10y ago
I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations. http://www.goodreads.com/book/show/6665847-daemon http://www.goodreads.com/book/show/6665847-daemon
- noir_lord 10y ago
  One of my favourites, the sequels where similarly as good.
- pavel_lishin 10y ago
  And a really fun read. If someone here decides to get this book, get the sequel as well - Daemon ends on kind of a cliff-hanger.
  - mitchellhislop 10y ago
    Freedom (TM) is the sequel, and the author (Daniel Suarez) has a few other near-term what-if-this-all-goes-skynet books which are equally good.

bduerst 10y ago

Sounds like current hardware defects and recalls for vehicles, with the difference being it's easier to fix.

Matt3o12_ 10y ago

There is something that's called fail safe(ly). In case of any error inside the car, it should sowly decelerate and pull over. Sure, it will cause a lot of traffic problems if say 10% of all cars did that at the same time but the damage would not be as severe as a ghost driver entering the freeway with 140mph. There are a few instances where some bad tesla batteries (the standard 12 volt batteries ironically) failed, and the cars handled it perfectly. It slowed down so that the driver could do safely pull over. Sure it did not happen to all cars at once, and autonomous cars migh not be able to do that by themselves but we have a log way to go to reach 100% autonomous driving (i.e. Without a steering wheel and a car that drives everywhere humans drive and not only on San Francisco's perfect sunny roads where it's been thoroughly tested on).

hashkb 10y ago
Google had fail safes which failed. What if a million cars pull over to the left instead of the right as a failed failsafe? Willing to risk your life?
- thisisnotatest 10y ago
  Yes, I'm eager to risk my life on self-driving car failures; it would be a tremendous step up from risking my life on human drivers (including myself!) as I do on a daily basis. I'm also a biker. In 2013, 4,735 pedestrians and 743 bicyclists were killed in crashes with motor vehicles. http://www.pedbikeinfo.org/data/factsheet_crash.cfm http://www.pedbikeinfo.org/data/factsheet_crash.cfm In the future when self-driving or at least augmented driving is commonplace, I hope that number will be a lot lower.

distrill 10y ago

I think it still is that simple. Clearly killing millions is not better than human control, so would be unacceptable.

hinkley 10y ago

Imagine a leap second bug causing vehicles all over the US to crash into NYE revellers.