5 ms·

I am a principal engineer for a major autonomous vehicle company. You can break this statement down into two components: Adding more sensors slows his team now

I am a principal engineer for a major autonomous vehicle company. You can break this statement down into two components: Adding more sensors slows his team now more than it improves system performance I'll take his word on this. It is a lot of work to incorporate multiple sensors. All necessary information is already in the pixel-space. I hate to disagree with someone as distinguished as Karpathy, but this is simply not what I have observed from all of that data that we have access to. Given my knowledge of the various stacks deployed today, I would never ever ever get into a vehicle using a vision only stack and expect it to perform in some of the challenging environments encountered during testing.

kfarr 4y ago
Full on agreement. There are literally videos of Teslas smashing into stationary vehicles on the highway at night using only vision camera for FSD. No way any rational actor could claim the visible pixel space is sufficient in that scenario compared to LIDAR, Radar, etc
- alsodumb 4y ago
  It's funny you use Radar as an example of 'good sensor' while it is well known that most or maybe almost all?) of the stationary vehicle accidents you're talking about happened because of Radars inability to detect a stationary obstacle. On the other hand, RGB data does have that information, we use it everyday to avoid obstacles, even under foggy and rainy conditions (I'm no LIDAR expert but I know it sucks in rainy conditions) I am not saying I support a vision only stack, but all I am saying is it is certainly possible to deploy a vision only stack in the future.
  - krapht 4y ago
    You mean some radars inability to detect stationary obstacles. Clutter rejection has a lot of more sophisticated algorithms to apply with greater compute power to throw at the problem.
    - Nomentatus 4y ago
      These were flat truck/ambulance surfaces encountered at an angle - exactly the conditions that the first stealth fighter, with its angular surfaces used to evade some of the best radar in the world, because from most angles, no radio waves would be reflected back to the radar sensing device. It's hard to get the job done with nothing.
  - threeseed 4y ago
    > Therefore, for this simple ADAS algorithm using roof mounted LIDAR, heavy rain does not prove to be a particularly important factor in the system performance. https://www.mdpi.com/2079-9292/8/1/89/htm https://www.mdpi.com/2079-9292/8/1/89/htm
alsodumb 4y ago
I think one should distinguish between 'all necessary information is already in the pixel-space' vs 'we already know how to extract all the information needed from pixel-space' The fact that (most) humans manage to drive around safely and successfully in current roads proves that the information needed exists in the pixel-space (not just current image, but say current + history). We don't yet have stacks that can successfully map everything needed from this information but I don't think Dr. Karpathy ever claimed that. (I am not a principal engineer but a mere PhD student who argues daily with people on how RGB information is underappreciated and under utilized)
- diskzero 4y ago
  I'll agree with you that there are still techniques to be discovered. I also agree that most humans manage to drive in challenging conditions, but their margins for error become slimmer and slimmer. I personally want my autonomous robot vehicle to be way more efficient and safer than the best human operator and also able to deal with conditions that any sane human would pull to the side of the road when encountering.
  - alsodumb 4y ago
    Definitely agree with your second point! In theory, the reaction time and complete environment awareness should itself make an autonomous system way safer than human drivers. In some way, I am against the philosophy of using HD maps + LIDAR data for highly accurate localization which most companies seem to be using these days. I believe that this approach is inherently brittle and is an 'easy way out' to the hard localization problem. I think more resources should be put into developing more natural, no HD map dependency techniques. PS: It is my understanding that most of the major players were using HD maps, not sure if it is still true.
  - bumby 4y ago
    >their margins for error become slimmer and slimmer. Can you elaborate on this? I've always felt like the margins of error are getting wider because the automotive tech (particularly safety features) are so vastly improved. I doubt people would be able to text and drive as much, for example, if they were driving a 1950s era Willys jeep just because it requires so much more attention to keep on the road by comparison to modern vehicles.
cma 4y ago
Compare their occupancy map with what you get out of the latest LIDAR Waymo is using and it is scary (occupancy is harder as it fills in what is occluded, but Tesla's looks like Minecraft-style 1x1x1m resolution).
jbverschoor 4y ago
It’s good enough for people, so all the info is there. Doesn’t mean it’s better or easier
Dunedan 4y ago
Out of curiosity: Could you please elaborate what such challenging environments can be?

kfarr 4y ago

Full on agreement. There are literally videos of Teslas smashing into stationary vehicles on the highway at night using only vision camera for FSD. No way any rational actor could claim the visible pixel space is sufficient in that scenario compared to LIDAR, Radar, etc

alsodumb 4y ago
It's funny you use Radar as an example of 'good sensor' while it is well known that most or maybe almost all?) of the stationary vehicle accidents you're talking about happened because of Radars inability to detect a stationary obstacle. On the other hand, RGB data does have that information, we use it everyday to avoid obstacles, even under foggy and rainy conditions (I'm no LIDAR expert but I know it sucks in rainy conditions) I am not saying I support a vision only stack, but all I am saying is it is certainly possible to deploy a vision only stack in the future.
- krapht 4y ago
  You mean some radars inability to detect stationary obstacles. Clutter rejection has a lot of more sophisticated algorithms to apply with greater compute power to throw at the problem.
  - Nomentatus 4y ago
    These were flat truck/ambulance surfaces encountered at an angle - exactly the conditions that the first stealth fighter, with its angular surfaces used to evade some of the best radar in the world, because from most angles, no radio waves would be reflected back to the radar sensing device. It's hard to get the job done with nothing.
- threeseed 4y ago
  > Therefore, for this simple ADAS algorithm using roof mounted LIDAR, heavy rain does not prove to be a particularly important factor in the system performance. https://www.mdpi.com/2079-9292/8/1/89/htm https://www.mdpi.com/2079-9292/8/1/89/htm

alsodumb 4y ago

I think one should distinguish between 'all necessary information is already in the pixel-space' vs 'we already know how to extract all the information needed from pixel-space' The fact that (most) humans manage to drive around safely and successfully in current roads proves that the information needed exists in the pixel-space (not just current image, but say current + history). We don't yet have stacks that can successfully map everything needed from this information but I don't think Dr. Karpathy ever claimed that. (I am not a principal engineer but a mere PhD student who argues daily with people on how RGB information is underappreciated and under utilized)

diskzero 4y ago
I'll agree with you that there are still techniques to be discovered. I also agree that most humans manage to drive in challenging conditions, but their margins for error become slimmer and slimmer. I personally want my autonomous robot vehicle to be way more efficient and safer than the best human operator and also able to deal with conditions that any sane human would pull to the side of the road when encountering.
- alsodumb 4y ago
  Definitely agree with your second point! In theory, the reaction time and complete environment awareness should itself make an autonomous system way safer than human drivers. In some way, I am against the philosophy of using HD maps + LIDAR data for highly accurate localization which most companies seem to be using these days. I believe that this approach is inherently brittle and is an 'easy way out' to the hard localization problem. I think more resources should be put into developing more natural, no HD map dependency techniques. PS: It is my understanding that most of the major players were using HD maps, not sure if it is still true.
- bumby 4y ago
  >their margins for error become slimmer and slimmer. Can you elaborate on this? I've always felt like the margins of error are getting wider because the automotive tech (particularly safety features) are so vastly improved. I doubt people would be able to text and drive as much, for example, if they were driving a 1950s era Willys jeep just because it requires so much more attention to keep on the road by comparison to modern vehicles.

cma 4y ago

Compare their occupancy map with what you get out of the latest LIDAR Waymo is using and it is scary (occupancy is harder as it fills in what is occluded, but Tesla's looks like Minecraft-style 1x1x1m resolution).

jbverschoor 4y ago

It’s good enough for people, so all the info is there. Doesn’t mean it’s better or easier

Dunedan 4y ago

Out of curiosity: Could you please elaborate what such challenging environments can be?