ai.bythebay.io: Danny Shapiro, AI and Self-Driving Cars
Recording: ai.bythebay.io: Danny Shapiro, AI and Self-Driving Cars
So what I wanted to do is spend a few minutes and just talk about what's happening in the in the self-driving car space. Um I I want to show you this this was the very first self-driving car we did. And this was probably about 20 years ago. And I think you guys know you know Nvidia kind of came out of video games. That's where we started. We were creating 3D graphics hardware and really helped the video game industry take off and it continues to grow today. But I think what we've seen is obviously video games have evolved um pretty dramatically and the amount of computation that's now part of video games is something like this. So you see not just crude graphics, but a huge amount of computation going on here
You have um very sophisticated rendering. You have lighting being calculated. You have different textures, different surfaces, but also vehicle dynamics. The physics of racing, of driving is all factored into the games. So this all requires a massive amount of computation and this is really why we've seen the GPU or the graphics processing unit with now its thousands of cores really have so many different applications. So graphics of course is just one of the key drivers of computing. But what we've seen really is is over decades these whole new trends or these inflection points where new computing models come into play. So first it was the PC, right? You have GPUs going into the PC and everyone connecting to the internet
And that's kind of this whole new revolution that happened um back in the in the 2000s. Then we shifted to mobile, right? Everyone now has smartphones and your phone in many cases can replace what you do on your laptop or on your PC. Apps, again, connected to the cloud. So, you have again a whole new computing shift that took place and really, you know, with the iPhone and other smartphones. And so now really within the last year or two, there's this whole new push, the Internet of Things, self-driving cars being part of that, but also artificial intelligence is changing the way people write software. It's changing the way applications are developed and how people interact with data. So, this is really what I want to focus on now is how deep learning, a form of artificial intelligence, is really changing how code is written, how apps are developed, and how industries are basically being transformed. So, this is a very simplistic model and I'm sure some of you, um, you know, are are familiar with this, but for those of you who aren't, I just wanted to take a few minutes to talk about some of the basics of how deep learning works
And it's this whole cycle of how we first have a lot of data that we need to train a neural net, a train a system on. And so this is going to happen offline, this is going to happen in a data center. I was talking to somebody earlier, it happens on your home PC if you're training your own neural nets, but again, you're taking lots of data, could be images, could be voice, could be medical data, could be financial data. And you basically have these inputs and outputs and you're training the system how to recognize patterns of data. So, once we build that neural net model, though, then we can deploy that in the device, on a vehicle, on your phone. Could also be in a data center doing a a cloud service app. But whatever it is, we basically have this cycle of training, deploying and inferencing, but then also being able to collect more data and feed that back into the loop to continue to train and evolve your models. So, I'll give you a couple of examples of of what's been going on
And these are just some headlines recently clipped. Um training a system to play poker, right? Basically, having tons and tons of data, all these hands and professional and amateur poker players, and what the cards were and what they did, and the system's now been able to be trained to beat some of the best poker players in the world. Of course, there was the game of Go last year, very um pivotal event in AI and deep learning where Google trained their AlphaGo system how to beat Lee Sedol, the grand champion in Go. But again, anytime there's there's games, there's data behind it, could be medical imaging data, whatever it is, we have all that set of inputs and outputs. What are the correct answers, and the system can then learn how to make decisions um better than you or I possibly could. And so, what we're seeing now is this explosion from game playing to cancer research. This was just announced today, and we're working with Google. They're using our GPUs in a data center to be able to analyze millions and millions of either X-rays or CAT scans or other medical data to be able to better diagnose cancer or virtually any other disease
It's really remarkable breakthrough. But also, I mean, it just extends everywhere. Wells Fargo starting a whole new division, salesforce.com, Oracle, eBay, all these companies are recognizing that artificial intelligence is a way to transform and gain new insights into their business, and really understand how to streamline and replace the fact that people were writing all kinds of hard-coded algorithms, and instead let the data write the software. What's interesting is this even goes into design. Autodesk is working on ways to have AI essentially augment engineers. Instead of somebody designing the chassis of a car, they input a lot of parameters and they basically have provided a lot of data inputs, all the constraints to the equation, and the system actually comes up with a lot of different options that designers hadn't even thought of. So, this is taking place today that AI is being used to generate new products. So, again, if we look at the kind of the whole spectrum, right? We have AI playing games, we have it learning how to paint and do art, we have it doing voice processing, right? Deep learning is at the basis of Google, okay Google and Siri and all these back ends are all running on GPUs running deep learning algorithms to process everything you say
So, again, we have this huge spectrum. This whole new computing model is taking place. But, I think one of the most challenging problems really is driving. You and I take it for granted, right? But, you've been driving for years and years and years and you've built up this whole memory and this understanding of what to do, how to react, what can happen. And again, this is a perfectly suited problem for artificial intelligence. So, if we break it down, essentially, we have the need to replace the human, right? The brain and the physical aspects are part of what a whole self-driving car system is. So, when you're driving, you're you're using your eyes and you're using your ears. You're figuring out what's around you
You're sensing, right? And then you take action based on what you sense and understand. So, if we break down this pipeline, first what we need to do is perceive what's going on. And so, all these self-driving cars, you start to see more cameras, radar, ultrasonics, maybe lidar, laser scanner. So, those are your eyes and your ears. We're collecting data. But, just getting that data isn't good enough. We have to figure out what is that data, right? An image only is a bunch of pixels and I'll I'll talk more about that in a second. But once we get all this information, we have to figure out what it is
Is it a person? Is it a car? Is it a tree? Is it a fire hydrant? Is it a dog? Whatever could be out there. So we have to teach the car this huge vocabulary essentially of what it can recognize and what's out there. From that we have to figure out then of course what it is and how is it going to behave. A car moves differently than a pedestrian which moves differently than a bike which is different than a motorcycle which is different than a truck. And a FedEx truck moves differently than an ambulance. And you have to do different things depending on whether it's a truck or an ambulance. So again there's a huge amount of information, a huge amount of understanding and reasoning that needs to be built into this neural net model. Then we also have to be able to predict what's going to happen
And so again that comes down to the behaviors. What is a car going to do? If it's moving at a speed it's going to continue to move approximately at the same speed and where is it going to be in the future? And so we have to basically track everything around the car and figure out where it's going to be in the future. The other aspect of self-driving cars where artificial intelligence comes in is the map. And so I'm not just talking about a regular map that says we're here at the dog patch and we want to go to downtown but rather the map has to be highly detailed, high definition. It's an HD map. And so it needs to know exactly how wide is the street, how many lanes are on the street, which is a straight lane, which is a right turn only lane, where are other particular landmarks, crosswalks, streetlights, stop signs. So again if we have this kind of ground truth, an understanding of the environment coupled with what we're scanning and understanding, then what we can do is localize ourselves, figure out where on that map we are, and then understand how the environment is moving around us. People walking, bikes riding, cars turning, and then we can predict where they are, and then we plan a path forward
So again, this is a massive amount of computation. And you guys, again, as you're driving, you sort of take this for granted, right? You just sort of look around and you naturally steer. But again, we as humans are distracted often. We make mistakes. We have bad judgment sometimes. And again, the result is over 40,000 people in the US killed, over 1.2 million people killed a year in automobile accidents. And so using AI computing, being able to have a system that's constantly scanning, understanding, interpreting, reacting, and then controlling the vehicle, um is going to make driving on our roads a lot, lot safer. So if we look at a a neural net, essentially one little frame of video, I wanted to kind of highlight what happens here
So, video, of course, is 30 pictures per second coming in, and we might have eight cameras around the car, as well as other radar sensors or lidars, as I mentioned. But let's just take a single frame of video. It's a bunch of pixels, right? A picture. Maybe it's called just a million pixels. Every pixel just a numeric value of the color. So when you see this picture, you instantly recognize that's a car. To the computer, all it is is a bunch of color values. And so what the computer needs to do is figure out, well, how do you go from that series of dots with different colors to an actual piece of information? What we have to do then is basically break down that image going through a bunch of layers of this neural net
And so it's a variety of algorithms with inputs and outputs. And so initially, what we might do is scan that image and look for edges. So we use an image processing technique to establish what's an edge in the image. The reason edges are important is you If you about it, as you look around this room, right, you're taking in essentially all these pixels. The wall that's white, right, you don't really pay much attention to these big white areas, but what you notice is the lines, the edges between the wall and maybe the window, or these steel struts, or the ceiling, right, wherever the sides of the screen, wherever there's edges, that denotes an object. So, again, we'll take these edges, straight edges, curved edges, and start to group them together to form elements. So, in the image, we start to see wheels, or windows, or headlights, or other elements. And so, these different layers of the neural net take inputs, edges, start to form objects, elements, and then assemble them together to form a car
So, ultimately, we can go through with enough data, train the system how to recognize cars, trucks, people. But even able to recognize what type of car it is. So, if you have enough data, and you show the system enough pictures of cars, it will learn how to recognize a car. And we show pictures of pedestrians, it'll recognize a pedestrian. If we show it a live huge enough library of cars, it can recognize that it's an Audi versus a BMW versus a Mercedes versus a Honda, whatever it is. And again, think of it like you think of it as a child learning. The more information you expose to that person, the more they remember and learn. So, again, this is just happening on a massive scale, basically feeding it millions and millions of images
One thing that what we did um recently with with Audi was they had a database of all the German street signs. And they basically needed to teach their car how to recognize these signs. They had spent over 2 years working with a smart camera company, actually writing code, right, writing algorithms to detect that's stop sign, that's a yield sign, and hard coding what color it is, what shape it is. What we was we came in with a deep learning based system and all they did was train it. So, they didn't have to write code, they just fed it all the data. And so, they said, "Here's all the pictures of stop signs taken at all different angles, different times of day, different weather conditions." So, the system formed an understanding what a stop sign looked like. It formed an understanding of speed limit sign, of a yield sign, merge, etc., etc. So, after 4 hours of feeding data into the deep learning system, it was able to achieve a better level of perception than 2 years of engineers developing hard-coded algorithms
So, again, the fact that you guys are here, right? You're talking about how is AI transforming industries, being able to have the data write the code is so much more efficient and so much more flexible than hard-coding algorithms. And so, the benefit here now is that Audi can take the system to Japan, totally different set of signs, different language, different character set. They don't have to go rewrite all the code, but rather feed it new data and now the system can recognize all the street signs in Germany as well as Japan. You know, driving up here, this was again, this was taken yesterday on 101. Never seen this sign before. I don't know if you guys have. I think it's Caltrans or the DOT has a sense of humor. But again, what we can do is we can have a system that can recognize signs, but if it's never seen this sign before, it's not going to know what to do
But again, through the power of deep learning, we can have it read this sign and interpret what it means. So, again, if there's variable signs, there's no way you can write code to account for things like this, but a deep learning system with natural language understanding would be able to interpret what's going on and then take appropriate action. So, one of the things that we've been focusing on is also bringing AI into the car, not just outside. And so, we've introduced this notion called the AI Co-Pilot. So, in addition to having sensors outside the car, have a cameras, again the lidar, radar, forming this whole 360° understanding, what we want to do is also understand what's happening inside the car. And so, we can use the combination of data outside and inside to assist the driver. So, I want to show you a couple of examples uh of how this would work. So, what you're seeing is the view outside the windshield and some of the cameras around the car
And so, in this case, when it detects something that's anomalous, maybe it's somebody um walking out in the street, a bicycle, uh another car turning, it can alert the driver. The other thing we can do is actually be watching the driver to understand what's their state of mind. So, another example, um in this case, a motorcycle coming up and splitting lanes. You might not notice So, again, we can bring information to the driver to make them more aware of things that they they might not be aware of. So, by watching the driver, we can we can learn a lot. We can do facial recognition. You could imagine you start your car without a key, but based on the person that's in the car, or the car can adjust all the settings, the seat, and the mirrors, and the radio presets, again, based on just recognizing who's in the car. But another key thing is again, being able to recognize is that driver paying attention or not
Are they distracted? So, again, using deep learning, we can train the system to monitor the driver, to understand their state of mind, to be able to track their head, um what are they looking at? We can track specifically their eyes. Are they falling asleep? Um are they paying attention? Where they looking? Suppose the driver is actually looking down at their phone, not good, but people do it. We want to be able to again provide appropriate alerts. Or maybe they're actually paying attention, but they're looking to the left to see if a car is coming and somebody stepped off the curb on the right. So again, we can provide alerts if the car senses a pedestrian crossing, but the driver's not looking. Another thing that's been a lot of work done um for the hearing impaired and being able to use deep neural nets to to read lips. So we train it on facial expressions and the position of the mouth. And let's suppose you're in a car and the radio's really loud, but you want to give a voice command, the system can read the lips of the driver and then take appropriate action
So again, there's a whole slew of different kinds of technologies that we can bring inside the vehicle to be able to understand again, who's in the car, how many people maybe are in the car, and change the personal experience depending on um the occupants as well as as what they're doing. So as we look at neural nets, again, it's it's a whole new computing model. There's so many things we can do with this. I want to show you a couple examples of different AIs, different neural nets running in the car. Um one taking the video from a front-facing camera and being able to just detect where's the free space. And again, we're analyzing pixels. We've trained the system and so we're able to understand where is it safe to drive. Um we can also detect other vehicles as I mentioned earlier and basically create a whole three-dimensional bounding box around these cars
So think of it as a video game, right? We kind of get back to our roots. We're creating this whole virtual environment, recognizing everything around the car in three-dimensional space, and then being able to plot our path forward. And so what we see is again, a whole variety of different neural nets running simultaneously in the car. And so we've developed a number of these. We make them available in a whole software package called DriveWorks. It's something that if those of you who are taking the Udacity course on self-driving cars, you're going to experience this. But we have a whole bunch of different layers of what's going on in the scene here. And essentially these are different neural nets that are processing the same data, detecting lanes, detecting other vehicles, detecting free space
And again, just like any kind of of code that gets written, we can continue to plug different inputs and outputs together to create new neural nets. And that's exactly what customers like Mercedes and Audi and Tesla and Volvo and other partners of ours are doing. They're taking our GPUs, our whole system that's basically the AI computing platform in the car, and building applications on top of it. What we did um most recently at CES, I want to share this, was kind of fun. We This is a shot of one of the the big parking lots um next to the Las Vegas Convention Center. And we wanted to showcase the power of what could be done in a vehicle um just using deep learning. And in this case um what we did is we sort of cordoned off a chunk of a parking lot, and we created a small course. And what we wanted to do is again show sort of dynamic conditions, different road surfaces
We had asphalt, we had dirt, uh we had AstroTurf, um and in some cases we had lane markings, in some cases there were no lane markings, um some cases there might have just been landscaping along the sides. We wanted to simulate kinds of real-world things that you experience all the time. You're driving along the road and there's good lanes, but some cases the lanes disappear. Maybe there's construction. The challenge, of course, is if you've hard-coded or if you specifically instructed the car to recognize specific things, and it doesn't find them, it doesn't know what to do. So there are systems out there that basically look for lane markings, but if the road's been resurfaced and there's no lane markings, the system will just give up, right? It won't be able to function. So, we wanted again use deep learning to figure out how do you drive in ambiguous situations? How do you drive if there's no lane markings? So, our research team had been working for about 18 months developing something called PilotNet. What PilotNet was again a different known that it's not detecting lanes, it's not detecting pedestrians, it's not detecting other cars, but what PilotNet did was take in the front-facing camera and map it to the steering wheel
And so, we went out, we drove many thousands of miles in different conditions, and it basically learned how to understand the environment. And if there were lanes, great, but if there no lanes, it didn't care. It used other cues to figure out where it was safe to drive. So, I want to show you a couple of examples of what we did here. So, this is in in Las Vegas on this course. And so, Janine got in the car here. Um There's nobody behind the wheel, right? We have a remote start and stop, but you can see in places where there's landscaping rocks, she drives through it. In this case, we moved a construction arrow in, and the car saw that there was an arrow there and took a detour
When we remove the arrow, the car drives straight. So, again, it was able to learn how to recognize all these different kinds of things and take the appropriate action. So, this demonstration actually was just a single camera running one neural net that we called PilotNet. Now, in reality, to to put these kinds of vehicles out on the road, we're going to run a lot of different neural nets. We want to have lane detection, we're going to have pedestrian detection, we're going to have car detection, we're going to bring in HD maps. And so, there's all these levels of redundancy and different types of sensors to ensure that the vehicle is safe. But the power of AI and the power of deep learning is incredible here. The fact that we're able to take information, transform it, and use it to make decisions better than you or I could do, is pretty remarkable
So, we wanted to showcase this in a couple of other places. We uh we brought the car up to Lombard Street. Again, a pretty challenging um venue. And there's really there's no there's no lanes here. And in some cases, it's really hard to even see around the corner. But again, with the power of deep learning, it's we're able to take other cues, again, that you as a human driver would do. Right? You see curves, you see bushes, you see other things, and you're going to use that to make your decision to steer. And so, in this case, again, we're able to use data to write the code, instead of using engineering hours and people sitting in front of a computer to write the code
Now, if we combine the AI Co-Pilot capabilities, natural language processing, monitoring the driver, with the sensor array outside, then we can put these things together to actually have a full self-driving car. Again, this was a combination of a lot of different neural nets running simultaneously in the car. I think that's what kind of is the the key to all this. It's not just a single AI running, but many different simultaneous neural nets that are all doing different things. They're all operating on the same data, but the key is bringing all this into a single centralized computer. In cars today, there may be a hundred different processors, and the complexity is just too great. People who are are writing code for these different processors have no idea what's going on elsewhere in the car, and the complexity and the cost has really skyrocketed. So, what we're doing is basically bringing a centralized supercomputer into the car that can run all these different artificial intelligence networks and perform different functions, and of course then be updated over the air
Tesla has really pioneered this approach, and it's been incredible. That car gets smarter and smarter and smarter over time, unlike any other car. And so, the ability to write new code, to have it learn from the data, and part of that cycle, train it in the data center, deploy it on the edge, in the field, in the device, is what's really differentiating that vehicle and is a model for others to follow. So moving forward, we have a lot of work still to do. We're developing solutions for level two driving, level three driving, as well as level four, some cases cars without steering wheels. But it's the path is going to be slow and we have to make sure we prove the technology. But essentially, it all comes back to video games again. Many of you that are doing some of this research recognize you need a lot of data to train
And in fact, people are using Grand Theft Auto to train neural nets how to recognize objects. So again, you have this whole virtual environment where we can tag data, pedestrians, cars, trucks. And the other thing though is simulation plays a very vital role because we want to be able to train cars how to recognize children running out in front of the car and being able to stop. But I don't think anyone wants to send their children out in front of a real car to test whether it's going to work or not. So again, we can use simulation both to train as well as validate these neural nets and really take potentially hazardous situations and do them in a very safe fashion. The other area that's pretty exciting where we're seeing this AI technology being used is in motorsport. So Roborace is a whole new circuit that's come out. And so this is Formula One class vehicle, right? It's probably 2,000 lb
It's a full-size car. But you see there's no place to put a driver. It's fully autonomous. So it's running using our Drive PX technology, it's our DriveWorks software. And all these different race teams are now going to be putting these cars on the track to compete against each other. Every car is the same, right? It has the same motors, the same battery, the same sensors, the same brain, in fact, the hardware. The only thing that's going to differentiate these cars are the paint and the software. So, this now becomes an AI challenge
It's not about the mechanics or the race dynamics that traditional motorsport is about, but instead it's an AI challenge. Who can develop the smartest neural nets to understand the environment and understand the behavior of the other race cars and the strategy of racing. So, this is a pretty exciting circuit that's just starting to take off. Um the car was just unveiled last week at Mobile World Congress. And I think we're going to see a lot of really exciting stuff happen in the future. So, again, whether we're talking about passenger cars, trucks, shuttles, race cars, I think the future of transportation is going to be based on artificial intelligence. Thank you very much. I I think we have a few minutes for questions, so if there's anyone, I guess there's a mic
Hi. Can you elaborate a little bit about your PilotNet product that you have? Uh I'm interested in when is it available? Is it in a chair person stuff like that? So, PilotNet is one of our neural nets and that was what basically drove BB-8 around uh the parking lot in Las Vegas. That's what drove it down Lombard Street. Um so, in the automotive space, we're not really a consumer company, right? We develop products that auto makers and the tier-one suppliers like Bosch or ZF or Continental or Denso or Delphi all use. So, PilotNet and our whole DriveWorks stack is available to those customers. For people that are wanting to experiment and develop on their own, we have something called Jetson, which is our small developer kit that again is software compatible and you can run a lot of that software. I don't think PilotNet is made publicly available yet. Um, but if you're the best route to I think access some of this would be through you the Udacity course
I don't know if you're taking that or not. We're working very closely with them and we're providing a lot of hardware and software support to them. Both people can actually develop on the hardware itself or they have some simulators there. Hello. Hey, thank you for the wonderful presentation. Um, my question is uh Nvidia has published a lot of papers like end-to-end driving and a lot of those network structures are really helpful for uh entry-level like like like like me. I'm wondering going forward is your team's focus more on the software development like algorithm development or uh the hardware like GPU even the car design the car doesn't have to look the way that it looked today. Uh what is the focus or what are the next steps that your team is mainly uh working So, automotive is a huge industry and in fact Nvidia has been working with automakers for more than 20 25 years
You think about all the cars are designed using Nvidia graphics. Uh they're engineered all the CAD is on Nvidia. Um simulation I didn't show here but we do all kinds of virtual crash tests and and virtual wind tunnel simulations is all on Nvidia. As we move the processors into the car um it's what we're talking about now processing data for self-driving. We also have a huge infotainment business. All the graphics in the car whether it's a Tesla screen or the Audi virtual cockpit that's all Nvidia. So, those are all areas we continue to invest in and develop and evolve. The hardware is always going to get faster, right? I mean that's just the nature of the computing business
So, we're investing in how do we develop new GPUs and in fact a platform for the car. So, our Drive PX system is designed for the car. It has inputs for radar and lidar and cameras and it's designed to be automotive grade. It's very different than a traditional graphics card. It has to work in sub-freezing temperatures. It's got to work in the heat of the desert. Whenever you can turn on the car, the computer has to run. Your phone won't do that
But if you leave the phone in the car on a hot sunny day, it'll shut off. So, we're always evolving the hardware, making it better and tuning it for artificial intelligence. Our GPUs today are designed to run deep neural networks in addition to graphics. But your point you're right, software is huge. We actually have probably twice the number of software engineers than we do hardware engineers. I don't think a lot of people recognize how much software Nvidia creates. So, it's the operating system, it's the drivers, but it's the libraries, it's all the CUDA NN, right? CUDA for parallel programming, all the neural nets. So, we're continuing to invest very heavily
And again, creating kind of the libraries and the tools for you to develop on, so you don't have to go back and start from scratch and re-engineer it. But we provide a lot of neural nets, and that becomes a framework for you to build applications on. Uh you showed us a demo with the uh cameras um neural net running on top of the camera data. Um is it the same Can you use the same kind of um uh algorithms even on lidar data? Absolutely. And how does that work in Yeah, so again, if you think about it, neural nets taking data. We don't care whether it's camera data, radar, lidar. I mean, a neural net can process financial data, stock market ticker information. It can bring in, you know, scans from medical imaging or voice, right? So, it's just information
Um in the case of the different sensors on a car, the benefit is that each sensor has its own strength and weakness, right? Camera gives us nice color, so we can tell is the street light red or green? Lidar gives us great distance and information in in bad weather, but the LIDAR can't really tell us is it a red or a green light. And so when we combine these different forms, you know, radar's really good at giving us depth and speed information. It's hard to extract that from camera. You can, but there's a lot of extra work required. So when we combine multiple different sensor types, we fuse them together, that gives us the most accurate picture. Um but again, LIDAR essentially gives us a three-dimensional point cloud versus video, which is two-dimensional, and we have to extract that 3D. So there's just different types of data, but again, from a neural net's perspective, if it's been trained, it doesn't matter. Now the thing is you can't take a neural net that's been trained on video image and feed it LIDAR
It's It's just not going to know what to do there. Can you hear me? Uh Um what do you do with the bad data? When you're driving and say you make a mistake, uh do you just throw that out or do you use negative reinforcement? What do you do with it? That's so That's a really good question. I think it's it's important to recognize that the car is not learning from you as you drive in real time. All the training happens back in the data center. So initially there's a lot of of work that that we do or our customers do where they're driving with professional drivers. What they're doing then is as the fleet is driving around, they're collecting data, but that data will go back to the data center, be analyzed and validated and tested before a software update gets pushed back out to the car. What do you do with the bad data? Do you throw it out? In In Yeah, sure, in some cases. I mean, so let me give you one example though of what what could happen
Suppose we first have trained the car in the city, and so we've driven around lots of miles, we've done simulation, and so the neural net knows how to recognize pedestrians and crosswalks and street lights, and school zones, and taxis pulling out, and all the kind of crazy stuff that happens in downtown San Francisco. And the first time you drive that car out of the country, you see some cows, you see deer, there's a moose or something. If the neural net has never been trained, it's not going to know what that is. Now, it's going to recognize there's something, and it's going to know not to run into it, but it's not going to understand what it is exactly or how it might behave. But so, what we could do is capture that information of that situation, and send that back to the data center. And so, there's data scientists that are involved in looking at, you know, key things, right? So, again, this is up to the software that the auto makers going to do what they want to classify, what they don't classify, what they want to detect as maybe anomalous. But in this case, they may say, "You know what? We need to now enhance our software." And so, they'll go out and they'll collect all kinds of images of deer, and they'll simulate it, and they'll train it again. So, now, when a new software update comes out, you have deer detection, cow detection, moose detection, whatever large animal detection in Volvo calls it
But the key thing there is, yeah, a cow is going to behave very differently than a deer, right? If it detects a cow over here on the side on a pasture, it knows the cow is not going to go over the fence and go bolting or leaping into the roadway, whereas a deer could do that. And so, again, this is part of just training and through experience. Um but again, as you're driving, if there's something it doesn't detect or it doesn't understand, it's going to flag it and avoid it, but not take any kind of um special action at that point in time necessarily. Okay, we got one more question coming here as soon as I can get to them. Thank you. Uh you mentioned end-to-end neural network systems that you're developing. Is it alternative approach for using many neural networks for different task, just to train one uh small one deep neural network for end-to-end driving. Uh But I'm not try this to the four question
I think we're doing both a combination of different networks running simultaneously. So, some are more traditionally trained to be able to detect lanes. We call it DriveNet. So, we're detecting cars and we use that for, you know, automatic emergency braking. We're detecting pedestrians, lanes. But in addition to that, we're using end-to-end. Think of it as a redundant system. So, we're able to understand the environment holistically so that if there aren't lanes, we can still drive
I mean, you you drive down 101 or 280, it's constantly having lanes moved or repaved. So, you can't just rely on one specific type of information. But again, having multiple neural nets provides that level of redundancy and an extra smarts, if you will, to be able to drive and handle any kind of situation. So, is it completely maintained system for They're complementary. That's right. Yeah, so we're not we're not advocating just end-to-end, but end-to-end is a way to augment and enhance the system. All right, I think we can do one more question. One last question
One last question. I'm just excited about self-driving cars cuz people use turn signals finally. Hi, um can you tell me how the system deals with conflicts between those neural nets? Right, if you've got the lane detection one saying here are the lanes and you've got the end-to-end one saying actually no, I think I know better in this situation. How does that deal and especially as those those nets change and evolve? Sure. So, again, think of it as a video game, right? What we're trying to do is build a three-dimensional model in the brain of the car based on what it senses all around. Um and so again, there's there's no I don't necessarily have a an exact answer because it's a very nebulous question. But but essentially of all these inputs feeding in that we're creating this environment. We We have a map
We really need to have a map. This that's the known environment. And then the sensors are going to try to figure out where we are on that map based on what it knows as fixed, like the curbs, the lanes if they're there, other objects, and then what it senses as dynamic objects in that scene. So, if it doesn't detect lanes and it knows from the map that there should be lanes there, and we essentially see our video image with the curbs or other milestones, landmarks, it's going to kind of create this model and overlay them. I guess I I'm not quite following what what you're saying if there's different thing Either you're going to see something or I think what's what's more likely is you have different sensors, and one sensor might say a radar may say, "Hey, it looks like there's something big in front of me." and the camera doesn't see it. I think that's probably more likely, and that's kind of what's what's going on today. Now, I think these cars are going to err on the side of caution, right? These cars are not going to speed, they're going to drive very safely, they're not going to take risks that you or I always take, they're not going to be in a hurry, they're not distracted. So, if one sensor is sensing something and the other sensor doesn't, it's going to then assume there's something there
Great. Great. Thank you, Danny. Thank you very much. Thanks, everyone. Have a great conference.