6 ms·

> The FSD Beta system may cause crashes by allowing the affected vehicles to: “act unsafe around intersections, such as traveling straight through an intersecti

by apnew 4y ago

> The FSD Beta system may cause crashes by allowing the affected vehicles to: “act unsafe around intersections, such as traveling straight through an intersection while in a turn-only lane, entering a stop sign-controlled intersection without coming to a complete stop, or proceeding into an intersection during a steady yellow traffic signal without due caution,” according to the notice on the website of the National Highway Traffic Safety Administration. Does anyone have insights on what QA looks like at Tesla for FSD work? Because all of these seem table-stakes before even thinking about releasing the BETA FSD.

londons_explore 4y ago
I suspect they have thousands of tests, but ship code that passes only most of the tests... That's what makes it unfinished... It's never passed the 'drive from new York to LA with nobody touching the controls' test...
rvnx 4y ago
There are many of such tests in the open. There is even a former Tesla AI engineer that throws objects in front of the car on YouTube, as a demonstration. The results are not glorious at all :| (trying to find the channel back if someone knows). And random public tests too: https://www.youtube.com/watch?v=3mnG_Gbxf_w https://www.youtube.com/watch?v=3mnG_Gbxf_w This is a basic safety auto-braking. Just feels very wrong to even accept it goes into release.
- panick21_ 4y ago
  This is not a former Tesla engineer. This is competitor who wants to discredit Tesla and sell its own solution. The guy behind this is known to be untrustworthy, and many of the videos don't actually do what he claims. Notably he refused to release the videos that would prove his claims right. The reality is that Tesla scores high on all the automated breaking test done by government. The driver however can override this, and that is exactly what is being done in this video.
  - diggernet 4y ago
    So they scored high on whatever version of software was on the specific car tested by government at some point in time. Has any government done any testing at all on the version of software actually in use today?
    - panick21_ 4y ago
      The automated breaking is standard part of the software on every Tesla and is the software that was tested by multiple governments. This gives you certification. If Tesla would change or remove that software, it would be highly illegal. Tesla is actually notable that they are one of the only company that has implemented the highest standard of this safety feature on every car. These features are something Tesla is proud of have talked about quite a bit. They also mention that they have achieved the highest scores these evaluations. The test done above has been replicated and the car does break in automated driving. And it gives loud warning to the driver even under normal driving operation and does emergency breaking. Tesla assumes that if the driver hits the accelerator after the warning, the drive wants to accelerate based on the drivers judgment. This is what this video shows, notably the person that made this video has refused to release prove that the car actually was in Autopilot, refused to provide evidence that the driver didn't hit the accelerator and refused to provide audio from inside the car so it can be verified that there was no warning sound. In addition to that, the same person also claims things like 'millions of people will die if Autopilot isn't stopped' and that even under absurd assumptions is a legit insane thing to say. So what is more likely, that Tesla did something incredibly illegal removing and incredibly important safety critical software that is standard in every Tesla OR that a competitor is doing a PR campaign where they deliberately set up a situation to film a video where they can create maximum damage to Tesla and then advocate for their own solution. https://www.tesla.com/blog/model-y-earns-5-star-safety-rating-euro-ncap https://www.tesla.com/blog/model-y-earns-5-star-safety-ratin... So the question is who do you believe, Euro NCAP or a competitor who made a sensationalist viral video.
- Veserv 4y ago
  You are probably thinking of: https://www.youtube.com/@AIAddict https://www.youtube.com/@AIAddict
varjag 4y ago
That's the thing about neural networks: any QA is going to be superficial due to their statistical black box nature.
- dreamcompiler 4y ago
  Exactly. It's the same reason that no amount of unit tests can replace formal methods for safety-critical software, and we cannot apply formal methods to neural nets [yet].
  - IshKebab 4y ago
    That's not really true. Most safety critical software is tested without formal verification. They are just really really thorough and rigorous. Formal verification is obviously better if you can do it. But it's still really really difficult, and plenty of software simply can't be formally verified. Even in hardware where the problem is a lot easier we've only recently got the technology to formally verify a lot of things, and plenty of things are still out of reach. And even if you do formally verify some software it doesn't guarantee it is free of bugs.
- ChickenNugger 4y ago
  What? Black box testing has plenty of techniques: https://en.wikipedia.org/wiki/Black-box_testing https://en.wikipedia.org/wiki/Black-box_testing Whether it's a neural network inside or not is completely irrelevant. That's why it's called "black box".
  - erikerikson 4y ago
    This seems to ignore that if you look inside the box at code you could understand it whereas looking at the activation values is unlikely to illuminate.
  - varjag 4y ago
    Practical neural networks operate in enormous parameter spaces that are impossible to meaningfully test for all possible adversarial inputs and degraded outputs. Your FSD could well recognize stop signs in your battery of tests but not when someone drew a squirrel on it with green sharpie.
deleted 4y ago
[deleted]
yumraj 4y ago
> Does anyone have insights on what QA looks like at Tesla for FSD work? Yes. An army of Tesla owners perform the QA, in production.
- deleted 4y ago
  [deleted]
- dawnerd 4y ago
  Well first it goes to influencers that say it's perfect and good for stable release no matter what the car does! But in all seriousness they do have some small team that validates then it goes to employees.
jabagonuts 4y ago
As a human being and motor vehicle operator of many decades I have done all of the above, multiple times (very infrequently), both on purpose and on accident. I’m looking forward to the days when self-driving vehicles are normal, and human drivers are the exception. Until then, I’m glad companies and regulators are holding the robots to a higher standard than the meat computers.
- worik 4y ago
  > As a human being and motor vehicle operator of many decades I have done all of the above, multiple times Time to stop driving. That is not normal
deleted 4y ago
[deleted]
justapassenger 4y ago
> Does anyone have insights on what QA looks like at Tesla for FSD work? Because all of these seem table-stakes before even thinking about releasing the BETA FSD. Tesla is not exactly in love with QA. Especially for FSD. FSD is mainly 2 things: 1. (By far most important) shareholder value creating promise, that's been solved for 6 years according to their CEO. 2. Software engineering research project What FSD is not is a safety critical systems (which it should be). They focus on cool ML stuff and getting features, with any disregard for how to design, build and test safety critical systems. Validation and QA is basically non-existent.
- panick21_ 4y ago
  Do you have actual knowlage of Tesla internal QA processes, any kind of source at all? Based on there presentation, they for sure have a whole load of tests, many built directly from real world situation that the car has to handle. They simulate sensor input based on the simulation and check the car does the right thing. They very likely have some internal test drivers and before the software goes public it goes to the cars of the engineers. Those are just some of things we know about. I have no source on their approach to testing safety critical systems, but we do know that they have a lot of software that has based all test by all the major governments. They are one of the few (or only) car maker fully compliant to a number of standards on automated breaking in the US. We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. So they do clearly have some idea of how to do this stuff. So when making these claims I would like to know what they are based on. It might very well be true that their processes are insufficient but I would actual know some real data. Part of what a government could do, is forcing car maker to open their QA processes. Or the government could (should) have its own open test suit that a car needs to be able to handle, but clearly we are not there yet.
  - julianlam 4y ago
    > We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. I think you and I must've watched a different video.
TaylorAlexander 4y ago
Andrej Karpathy was the AI lead for most of the project and he has talked about the general system design. They have a set of regression tests they run on new code updates either by feeding in real world data and ensuring the code outputs the expected result, or running the code in simulation. It does seem worrying that they would miss things like this. Here’s a talk from Karpathy explaining the system in 2021: https://youtu.be/aNVbp0WKYzY https://youtu.be/aNVbp0WKYzY Though I don’t recall if he explains the regression testing in this talk, there’s a few good ones on YouTube.
- ertian 4y ago
  It's not even a bit surprising they'd miss things like this, IMHO. They do tests with a few (maybe even a lot of) intersections, but there are thousands upon thousands of intersections out there, including some where bushes are obscuring a stop sign, or the sign is at a funny angle, or sunlight is reflecting off the traffic lights, or heavy rain obscuring them, or plain old ambiguous signage...there's _bound_ to be mistakes. Human drivers make similar mistakes all the time. I used to think that fact was going to delay self-driving cars by a decade or more, because of the potential bad press involved in AI-caused accidents, but then along comes Tesla and enables the damn thing as a beta. I mean...good for them, but I've always wondered if it was going to last. I've been using it pretty consistently for a few months now (albeit with my foot near the brake at all times). I haven't experienced any of the above. Worst thing I've seen is the car slamming on the breaks on the freeway for...some reason? There was a pile-up in a tunnel caused by exactly that a month or so ago, so I've been careful not to use FSD when I'm being tailgated, or in dense traffic.
  - saalweachter 4y ago
    You know, there was an article on here last week about how there are only 4 billion floats, so just test them all. There are only like 16 million intersections in the US. Why not test them all?
    - TaylorAlexander 4y ago
      The thing is you already know everything you need to know about all 4 billion floats. Collecting data on every intersection in the US is quite difficult. Tesla does however collect data on edge cases and then train their system to respond correctly. They can for example trail a collection network to identify things that might be obscured stop signs, then have the fleet collect a whole bunch of examples, hand label those samples, and roll this new data in to the training system. This is explicitly how they handle edge cases. They can also create a new feature or network and roll it out in “shadow mode” where it is running but has no influence on the car, and then they can observe how these systems are behaving in the real world. The real issue I guess is when they release a new feature without trialing it in shadow mode, or if they have gaps in their testing and validation system.
Veserv 4y ago
It also does not know what one way street, do not enter, road closed, and speed limit signs are. Really, the only signs it appears to know about are stop signs. As for their QA process, in 2018 they had a braking distance problem on the Model 3. They learned of it, implemented a change that alters the safety critical operation of the brakes, then pushed it to production to all Model 3s without doing any rollout testing in less than a week [1]. So, their QA process is probably: compiles, run a few times on the nearby streets (I am pretty sure they do not own a test track as I have never seen a picture of tricked out Teslas doing testing runs at any of their facilities), ship it. [1] https://www.consumerreports.org/car-safety/tesla-model-3-gets-cr-recommendation-after-braking-update/ https://www.consumerreports.org/car-safety/tesla-model-3-get...
- ggreer 4y ago
  Teslas have understood speed limit signs since 2020.[1] 1. https://finance.yahoo.com/news/upcoming-tesla-software-2020-36-213057062.html https://finance.yahoo.com/news/upcoming-tesla-software-2020-...
  - Veserv 4y ago
    It uses maps for that. I have a winding road near me with a speed limit of 35 mph, but 15 mph on certain curves as indicated by a speed limit sign. It ignores those speed limit signs and will attempt to make the turns at 35 mph resulting in it wildly swerving into the other lane and around a blind turn with maybe 30 feet of visibility. It has also attempted to do it so poorly that it would have driven across the lane and then over the cliff without immediate intervention. Unsupported claims by a manufacturer that compulsively lies about the capabilities of their products except when directly called on it are the opposite of compelling evidence.
    - ggreer 4y ago
      I'm talking about standard speed limit signs. You're talking about the signs that warn about sharp turns and advise maximum speeds. Yes it would be good if the software understood those signs, but that's a different issue. Teslas definitely read speed limit signs. I've had mine correctly detect and follow speed limits in areas without connectivity or map data. It also follows speed limits on private drives (if there is a sign) and obeys temporary speed limit signs that have been put up in construction zones.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]

by apnew 4y ago

londons_explore 4y ago

I suspect they have thousands of tests, but ship code that passes only most of the tests... That's what makes it unfinished... It's never passed the 'drive from new York to LA with nobody touching the controls' test...

rvnx 4y ago

There are many of such tests in the open. There is even a former Tesla AI engineer that throws objects in front of the car on YouTube, as a demonstration. The results are not glorious at all :| (trying to find the channel back if someone knows). And random public tests too: https://www.youtube.com/watch?v=3mnG_Gbxf_w https://www.youtube.com/watch?v=3mnG_Gbxf_w This is a basic safety auto-braking. Just feels very wrong to even accept it goes into release.

panick21_ 4y ago
This is not a former Tesla engineer. This is competitor who wants to discredit Tesla and sell its own solution. The guy behind this is known to be untrustworthy, and many of the videos don't actually do what he claims. Notably he refused to release the videos that would prove his claims right. The reality is that Tesla scores high on all the automated breaking test done by government. The driver however can override this, and that is exactly what is being done in this video.
- diggernet 4y ago
  So they scored high on whatever version of software was on the specific car tested by government at some point in time. Has any government done any testing at all on the version of software actually in use today?
  - panick21_ 4y ago
    The automated breaking is standard part of the software on every Tesla and is the software that was tested by multiple governments. This gives you certification. If Tesla would change or remove that software, it would be highly illegal. Tesla is actually notable that they are one of the only company that has implemented the highest standard of this safety feature on every car. These features are something Tesla is proud of have talked about quite a bit. They also mention that they have achieved the highest scores these evaluations. The test done above has been replicated and the car does break in automated driving. And it gives loud warning to the driver even under normal driving operation and does emergency breaking. Tesla assumes that if the driver hits the accelerator after the warning, the drive wants to accelerate based on the drivers judgment. This is what this video shows, notably the person that made this video has refused to release prove that the car actually was in Autopilot, refused to provide evidence that the driver didn't hit the accelerator and refused to provide audio from inside the car so it can be verified that there was no warning sound. In addition to that, the same person also claims things like 'millions of people will die if Autopilot isn't stopped' and that even under absurd assumptions is a legit insane thing to say. So what is more likely, that Tesla did something incredibly illegal removing and incredibly important safety critical software that is standard in every Tesla OR that a competitor is doing a PR campaign where they deliberately set up a situation to film a video where they can create maximum damage to Tesla and then advocate for their own solution. https://www.tesla.com/blog/model-y-earns-5-star-safety-rating-euro-ncap https://www.tesla.com/blog/model-y-earns-5-star-safety-ratin... So the question is who do you believe, Euro NCAP or a competitor who made a sensationalist viral video.
Veserv 4y ago
You are probably thinking of: https://www.youtube.com/@AIAddict https://www.youtube.com/@AIAddict

varjag 4y ago

That's the thing about neural networks: any QA is going to be superficial due to their statistical black box nature.

dreamcompiler 4y ago
Exactly. It's the same reason that no amount of unit tests can replace formal methods for safety-critical software, and we cannot apply formal methods to neural nets [yet].
- IshKebab 4y ago
  That's not really true. Most safety critical software is tested without formal verification. They are just really really thorough and rigorous. Formal verification is obviously better if you can do it. But it's still really really difficult, and plenty of software simply can't be formally verified. Even in hardware where the problem is a lot easier we've only recently got the technology to formally verify a lot of things, and plenty of things are still out of reach. And even if you do formally verify some software it doesn't guarantee it is free of bugs.
ChickenNugger 4y ago
What? Black box testing has plenty of techniques: https://en.wikipedia.org/wiki/Black-box_testing https://en.wikipedia.org/wiki/Black-box_testing Whether it's a neural network inside or not is completely irrelevant. That's why it's called "black box".
- erikerikson 4y ago
  This seems to ignore that if you look inside the box at code you could understand it whereas looking at the activation values is unlikely to illuminate.
- varjag 4y ago
  Practical neural networks operate in enormous parameter spaces that are impossible to meaningfully test for all possible adversarial inputs and degraded outputs. Your FSD could well recognize stop signs in your battery of tests but not when someone drew a squirrel on it with green sharpie.

deleted 4y ago

[deleted]

yumraj 4y ago

> Does anyone have insights on what QA looks like at Tesla for FSD work? Yes. An army of Tesla owners perform the QA, in production.

deleted 4y ago
[deleted]
dawnerd 4y ago
Well first it goes to influencers that say it's perfect and good for stable release no matter what the car does! But in all seriousness they do have some small team that validates then it goes to employees.

jabagonuts 4y ago

As a human being and motor vehicle operator of many decades I have done all of the above, multiple times (very infrequently), both on purpose and on accident. I’m looking forward to the days when self-driving vehicles are normal, and human drivers are the exception. Until then, I’m glad companies and regulators are holding the robots to a higher standard than the meat computers.

worik 4y ago
> As a human being and motor vehicle operator of many decades I have done all of the above, multiple times Time to stop driving. That is not normal

deleted 4y ago

[deleted]

justapassenger 4y ago

> Does anyone have insights on what QA looks like at Tesla for FSD work? Because all of these seem table-stakes before even thinking about releasing the BETA FSD. Tesla is not exactly in love with QA. Especially for FSD. FSD is mainly 2 things: 1. (By far most important) shareholder value creating promise, that's been solved for 6 years according to their CEO. 2. Software engineering research project What FSD is not is a safety critical systems (which it should be). They focus on cool ML stuff and getting features, with any disregard for how to design, build and test safety critical systems. Validation and QA is basically non-existent.

panick21_ 4y ago
Do you have actual knowlage of Tesla internal QA processes, any kind of source at all? Based on there presentation, they for sure have a whole load of tests, many built directly from real world situation that the car has to handle. They simulate sensor input based on the simulation and check the car does the right thing. They very likely have some internal test drivers and before the software goes public it goes to the cars of the engineers. Those are just some of things we know about. I have no source on their approach to testing safety critical systems, but we do know that they have a lot of software that has based all test by all the major governments. They are one of the few (or only) car maker fully compliant to a number of standards on automated breaking in the US. We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. So they do clearly have some idea of how to do this stuff. So when making these claims I would like to know what they are based on. It might very well be true that their processes are insufficient but I would actual know some real data. Part of what a government could do, is forcing car maker to open their QA processes. Or the government could (should) have its own open test suit that a car needs to be able to handle, but clearly we are not there yet.
- julianlam 4y ago
  > We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. I think you and I must've watched a different video.

TaylorAlexander 4y ago

Andrej Karpathy was the AI lead for most of the project and he has talked about the general system design. They have a set of regression tests they run on new code updates either by feeding in real world data and ensuring the code outputs the expected result, or running the code in simulation. It does seem worrying that they would miss things like this. Here’s a talk from Karpathy explaining the system in 2021: https://youtu.be/aNVbp0WKYzY https://youtu.be/aNVbp0WKYzY Though I don’t recall if he explains the regression testing in this talk, there’s a few good ones on YouTube.

ertian 4y ago
It's not even a bit surprising they'd miss things like this, IMHO. They do tests with a few (maybe even a lot of) intersections, but there are thousands upon thousands of intersections out there, including some where bushes are obscuring a stop sign, or the sign is at a funny angle, or sunlight is reflecting off the traffic lights, or heavy rain obscuring them, or plain old ambiguous signage...there's _bound_ to be mistakes. Human drivers make similar mistakes all the time. I used to think that fact was going to delay self-driving cars by a decade or more, because of the potential bad press involved in AI-caused accidents, but then along comes Tesla and enables the damn thing as a beta. I mean...good for them, but I've always wondered if it was going to last. I've been using it pretty consistently for a few months now (albeit with my foot near the brake at all times). I haven't experienced any of the above. Worst thing I've seen is the car slamming on the breaks on the freeway for...some reason? There was a pile-up in a tunnel caused by exactly that a month or so ago, so I've been careful not to use FSD when I'm being tailgated, or in dense traffic.
- saalweachter 4y ago
  You know, there was an article on here last week about how there are only 4 billion floats, so just test them all. There are only like 16 million intersections in the US. Why not test them all?
  - TaylorAlexander 4y ago
    The thing is you already know everything you need to know about all 4 billion floats. Collecting data on every intersection in the US is quite difficult. Tesla does however collect data on edge cases and then train their system to respond correctly. They can for example trail a collection network to identify things that might be obscured stop signs, then have the fleet collect a whole bunch of examples, hand label those samples, and roll this new data in to the training system. This is explicitly how they handle edge cases. They can also create a new feature or network and roll it out in “shadow mode” where it is running but has no influence on the car, and then they can observe how these systems are behaving in the real world. The real issue I guess is when they release a new feature without trialing it in shadow mode, or if they have gaps in their testing and validation system.

Veserv 4y ago

It also does not know what one way street, do not enter, road closed, and speed limit signs are. Really, the only signs it appears to know about are stop signs. As for their QA process, in 2018 they had a braking distance problem on the Model 3. They learned of it, implemented a change that alters the safety critical operation of the brakes, then pushed it to production to all Model 3s without doing any rollout testing in less than a week [1]. So, their QA process is probably: compiles, run a few times on the nearby streets (I am pretty sure they do not own a test track as I have never seen a picture of tricked out Teslas doing testing runs at any of their facilities), ship it. [1] https://www.consumerreports.org/car-safety/tesla-model-3-gets-cr-recommendation-after-braking-update/ https://www.consumerreports.org/car-safety/tesla-model-3-get...

ggreer 4y ago
Teslas have understood speed limit signs since 2020.[1] 1. https://finance.yahoo.com/news/upcoming-tesla-software-2020-36-213057062.html https://finance.yahoo.com/news/upcoming-tesla-software-2020-...
- Veserv 4y ago
  It uses maps for that. I have a winding road near me with a speed limit of 35 mph, but 15 mph on certain curves as indicated by a speed limit sign. It ignores those speed limit signs and will attempt to make the turns at 35 mph resulting in it wildly swerving into the other lane and around a blind turn with maybe 30 feet of visibility. It has also attempted to do it so poorly that it would have driven across the lane and then over the cliff without immediate intervention. Unsupported claims by a manufacturer that compulsively lies about the capabilities of their products except when directly called on it are the opposite of compelling evidence.
  - ggreer 4y ago
    I'm talking about standard speed limit signs. You're talking about the signs that warn about sharp turns and advise maximum speeds. Yes it would be good if the software understood those signs, but that's a different issue. Teslas definitely read speed limit signs. I've had mine correctly detect and follow speed limits in areas without connectivity or map data. It also follows speed limits on private drives (if there is a sign) and obeys temporary speed limit signs that have been put up in construction zones.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]