ai.bythebay.io: John Eggert, The Role of High Definition 3D LiDAR in Self Driving Cars
Recording: ai.bythebay.io: John Eggert, The Role of High Definition 3D LiDAR in Self Driving Cars
okay thank you very much for joining me today very excited to talk to you about lidar technology as mentioned it is the elephant in the room because I think it's one of the last great unexplored sensing modalities with autonomous driving sensors what I'm here to talk to you about is Bella dine is not a data science company by any means we just streamed the data but I think we have a compelling reason why we think lidar is here to stay and it's only going to become more ubiquitous over the next few years so right now I think mining lidar data is definitely a very much an open field in which many of you developers can explore first a little bit about us we started in 1983 so one of the oldest startup companies in the Silicon Valley we just went through our first round of funding late last year we started with Vella nine subwoofers some of you who may be familiar with that we were the first distortion-free subwoofers using digital signal processing to process information from the cones 16,000 times per second feeding back into servos to reduce distortion in the in the Senate in the speaker's our CEO is sort of an engineering polymath and so his interests varied from a wide variety of fields including mechatronics so he entered a contest called Robot Wars some of you might remember that these were gladiator robots we entered with a product called drill Zillah and you can see the drill in the front would drill into the competition and we did actually finish in second place this little 100 person subwoofer company out in Morgan Hill so you combine digital signal processing mechatronics so what is the next challenge for a company like that the ultimate robot the DARPA Grand Challenges robot cars and velvetine again did an entry into that against universities and some corporate interests and we finished third worldwide which is great the problem with that was we only had six miles that we went through the course before we failed but that was a very interesting competition for us because it informed our approach to self-driving and what we felt the necessary sensing was to make a car drive by itself and after that we came out with our high-definition 3d lidar which you see on the bottom photo there and we entered that for the second DARPA challenge and finally became just a supplier by the time of the DARPA urban challenge in 2007 so what lessons did we learn after that first DARPA challenge we entered with stereo cameras and the problem with stereo cameras are pretty much well known today at that time though we discovered that yeah ambient lighting conditions were very important and when you depend on light all the time you sometimes can result in failures in darkness low angle sunlight shadows the other thing we learned in self-driving is that we needed very fast update rate very high resolution sensing and maximum horizontal field of view ideally 360 degrees around the vehicle all of these things are not unique to robotics but they become particularly critical when you're talking about a high speed robotic system like a car another key point we wanted to be sure of as we couldn't have false positives we couldn't just be guessing about what was out there we need to measure what was out there false positives in the early DARPA challenges were the reasons why you saw cars moving a few feet and stopping moving a few feet and stopping so what we came up with in that second DARPA challenge was not particularly high tech it was a very simple and elegant approach but the implementation was the challenge there so essentially we took a typical 2d rangefinder you know not much different than this laser pointer here except we put a detector mated to it so you can measure the time of flight to see how long it how are away an object is based on the speed of light but develop ions IP covers tomb areas primarily the first is the multiple channels doing that all in a sequence very short time windows nanoseconds really that the detector is listening for the laser signal also rotating 360 degrees so combining those two things we were able to create an instantaneous 3d CAD model of the environment around the vehicle and when I say instantaneous if you think of a blink of an eye a few hundred milliseconds in that time we're measuring and firing sensors are firing lasers at 400,000 times in that blink of an eye so you can imagine how quickly we can create a 3d point cloud with our technology so and this is kind of an example of how quickly you can build up 3d information around the environment this is how a typical map is made all of these mapping companies as you see out there right now doing high-definition maps this is just pure lidar data stitched together in a technique called slam and you can see at the end you have a product there that's a 3d map so this can be done in real-time pretty much with a valentine sensor because the amount of data that we'd stream so what was the result of all of these technologies that we brought to the fore 2007 DARPA urban challenge which was the first demonstration of urban driving the valentine sensor was featured on the top two vehicles and it's mentioned here that it's Stanford and Carnegie Mellon pretty much the incubators for a lot of what you see today in level 4 and level 5 driving both of those programs fed directly into the Google self-driving program the you know recently uber purchased more or less the Carnegie Mellon robotics program as well so how does 3d lidar enable automated driving what are the key things that we do that other sensors can't well let's start with taking a look at what other sensors do what performance of other sensors are this is Car and Driver report from early last year monitoring how well cars could detect lanes lane markers so very simple part of the whole self-driving problem just Lane finding lane keeping and this is the best that cameras could do as of 2016 so in a 50 kilometer course the red numbers show you how many times the system disengaged because it couldn't find the lines and this was a course that was around in Arbor Michigan by far the best performers the Tesla vehicle still probably the best level to self-driving car out there by far and within 50 kilometers 26 times the system had to disengage because it couldn't read the lines the Infinity was as bad as 93 times within a 50 kilometer course so obviously not good enough for a full level for driving it can't even handle a simple task of lane keeping some other examples of the camera challenge you can see the ambient lighting here that's what we found in the first DARPA challenge very clearly you're not going to be able to pull much data out of something like this similarly very complicated scenes that are rapidly evolving and changing you get something like this way too much data for to be parsed and to figure out what's going on in that scene an example of a radar challenge is right here radar is a relatively low resolution sensing modality so if you look at that bridge that's going over the the road there it really can't tell where that is in the z-plane is it a wall right in front of the car is a bridge over the vehicle so these are some of the problems that you see with other sensing in fact this something like this is exactly what happened in the Tesla accident last year so how does lidar perform a little bit better in those situations this is an example of lane-keeping cameras as I mentioned haven't really proven their reliability and we feel that with lidar it becomes a much more robust system being able to detect lanes so our sensor gives you two pieces of information it gives you a position in space and it gives you a reflectivity so with lane markings they have a higher reflectivity than the asphalt around it so you very clearly can see those popping up and if you're looking at the the image on the right lower right-hand that purplish image you can see the lane markings they have a much greater reflectivity and they stand right out also for 3d non painted markers like Botts dots those kinds of things again you have additional spatial resolution that you can determine the location and space of things like that another situation poor ambient lighting this is raw data from a Valentine lidar it's not quite a video image but it's pretty high resolution stuff if you take a look at it the different colors represent different reflectivity is no objects that we're hitting and you can see this is kind of like the Tesla incident if you will you can see that truck drive right by and of course what the advantage of lidar data is it's 3d data so you can look at it from any perspective and you can see the very bright retro-reflective the red stuff that you see on this are retro reflective objects like the back of the truck that would be very reflective lane markings another use case for autonomous driving level 4 level 5 then I should point out this is one area that even the Volvo isn't doing whether gothenburg trials is merging onto a on ramp onto a freeway again you need a very high horizontal field of view sensor to be able to do something like this and you'd have very high resolution spatially to be able to determine where in space the other vehicles are and there no other sensors out there that are able to tell you exactly where other cars are when you're trying to merge with them it's a very dangerous maneuver for cameras or radar something that we have a lot of in the United States are these skewed intersections where you'll come up to a highway road and you'll need to make a left turn on to it the cross traffic might be moving you know 45 50 60 miles per hour and you need to be able to slowly pull out and make a left turn onto something like that without getting into an accident and so you need something again with a 360 degree view around the vehicle and you need to have a very long range high spatial resolution view of that so if you're looking at the right image that's raw lidar data and that's a light art that we've made that sees 200 meters and so you can see that if you're carefully look you can slowly track or you can carefully track the little cars that are moving in and out of that scene and you have an exact XYZ position of every car in that frame encroachment lane encroachment radar has a problem with these sorts of situations very noisy you need a lot of filtering on it and you can't really detect the signal of the noise in something like this the image on the right you'd be lucky if you do get a flag on the back of a load like that but again this is something that lidar can see pretty well in 3d space example of unmapped objects for self-driving road debris so anything standing above the ground plane that's something that lidar again will be measuring so a camera will kind of estimate what's out there whereas a lidar is actually measuring so something like this or depression below the ground let's say a pothole like we have a lot of these days in the Bay Area these are the kinds of things that a lidar might be able to help you detect a lot better and this is probably an area that's pretty ripe for a machine learning so that said I just kind of sold you on how great lidar performs in this space how does it actually perform we have a couple of data points we have the the state of California 2015 and 2016 results these here show the 2015 results of self-driving cars that are licensed in state of California the y-axis here shows the number of kilometres between dis engagements in other words how far the car drive before the human driver has to take over because of some error and it really is no contest if you looked at it last year Google was driving almost 2,000 kilometers between times the driver had to take control most of the other companies I think or at least an order of magnitude less than that 40 50 kilometers at best some of them were as low as less than one kilometer and I think one of the key differentiators between those two besides the experience of Google which is indeed vast is the fact that Google was the only one there using a high-definition 3d lidar point cloud so they had that advantage over the other competitors at the time at the time in 2015 none of those were using anything more than a 2d lidar but is it just googles experience that makes them drive so well so we take a look at the 2016 l4 performance I chose to take a look at this data a little bit differently than the previous years on the y-axis here I've put a measure of driving complexity so the least complex situation would be just kind of a highway autopilot sort of thing maybe no Lane changing even just a very simple adaptive cruise control lane keeping type of application that would be the least complex of driving situation most complex would be driving here in the city of San Francisco and what we found from the data this year if we kind of read through all of the reports is that there was one company that actually exceeded Google in terms of the complexity of situations it was trying to drive in and that was cruise automation and I'm sure a lot of you've heard of cruise automation it's a company that had a relatively short history in the self-driving space in fact if none of the folks there were from the DARPA challenges and so they kind of jumped right into the space and people were wondering why didn't why did GM pay a billion dollars for this company last year well this is one of the reasons why suddenly GM has catapulted themselves into driving in the most complex situations and if you've tracked the cruise data over the last year you could see early in the year they were like a few kilometers between dis engagements and towards the towards the end of the year there were well over a hundred maybe a couple hundred kilometers between dis engagements so crews made huge leaps in it without a long history in self-driving and I'd like to think that part of that was hardware related if you look at the internet pictures of crews they've got a couple of very obvious high-definition lidar sensors on top of the vehicles so you know this kind of begs the question is is it all AI or you know is it is it a matter of what kind of hardware sensing suite you have on the vehicle that ensures your success cruise automation perform quite well though by jumping into high-definition lidar we feel this is just some other data just again I'm trying to impress upon you the fact that the market is changing and lidar is here to stay and it's just going to grow when I started at Valentine a couple of years ago which makes me the old guy about what in 2015 there was only one company among the automotive OEMs traditional OEMs that considered using high-definition lidar for a love for driving that is from press releases everyone knows this is Ford and they were the only one everyone else said yeah we can use it with low definition lidar and the majority in fact felt that lidar wasn't even necessary to tackle the problem vision was plenty of data vision and radar would be plenty and at that time Ford was probably asking themselves you know are we crazy or why are the only ones using this expensive sensor right now you know are we not seeing something that the rest of the market is able to see but just the opposite happened the rest of the market kind of came around to Ford's way of thinking so in 2016 almost half of the companies out there a third of the companies out there we're using high-definition lidar decided that that was the way they needed to pursue level for driving level for and above driving and today there's only one company left among the OEMs that I'm aware of that feels that lidar is not necessary at all for full level for driving so the market has definitely come around to this way of thinking and I think it leaves a lot of opportunities for people who are looking to perform machine learning on lidar data if we take a look at also the acceptance of lidar data among the non-traditional automakers I think across the board everyone is using high-definition lidar I I don't I can't think of a company that is pursuing level 4 in level 5 without it and you've seen pictures of all of these companies in the news a lot of these are confidential now they've become confidential as we go so pretty soon the whole thing will be confidential but all of these photos are available on the Internet so where is light are going to go next to ensure that it indeed becomes a sensor on every single car made out there the keys that we need to accomplish at Villa dine are maintained the range and resolution so the very large sensor that you see on top of the old Google cars the HDL sixty-four valid ID we need to maintain that resolution in and if anything we need to improve upon that and so this year Vella nine has announced that we've developed a sensor that sees twice as far as our previous generation of sensors instead of a hundred meters now we're at 200 meters and so we're slowly climbing up that that bridge to be able to get to high speed driving at level four of course because they're going to be put into cars form-factor is a concern that first sensor I showed you at the beginning was about a 1 meter diameter sensor on top of the vehicle obviously not very practical and today our smallest sensor is about a hundred millimeters in diameter being automotive weight is very important as well we've gone from about a 30 kilogram sensor to an 800 gram sensor today with 32 channels and cost I've heard I think my friend Wolfgang mentioned that we would be below 250 dollars I'd see please hold off one moment give us a little more time but we have publicly announced that we think it's just a matter of time before we get to a 500-dollar sensor or below our CEO likes to say that this sensor is just a bag of electronics you make enough of them and the price will come down kind of give you an idea of the scale of what Vela Dyne has done we we are the biggest player in this space of self-driving but up until 2015 we probably sold a couple of hundred sensors into the automotive space per year so 200 sensors and so when we're talking about millions of sensors obviously there's a lot of opportunity to bring that $85,000 cost down by orders of magnitude and of course probably the key driver behind all of this to make sure that it gets introduced into cars is the automotive durability this will be quite a challenge but Vela 9 again 10 years working more or less the same space you know our technology even our solid-state approaches are going to be one laser one detector : early aligned multiples of them it's just a matter of how inexpensive we can make each of those channels that will get us the lower cost but the automotive durability we're pursuing along the same technology path that we have for ten years and so we've got a lot of experience in the field that picture of the truck there are those autonomous dump trucks that have been driving in Australia for about five years they have a very high duty cycle they don't they don't take many breaks they run more or less around the clock they get shot with fire hoses daily you can see it's a very dusty environment iron ore mines and so we do have some history in trying to ruggedized this sensor and so we think we're going along the right path as we proceed and so this year we've introduced to some of our select automotive OEM customers the VLT 32 sensor we like to think that that fundamental architecture will be the one that we're going to see in millions by the Bayeux 2020 2021 that timeframe so we think there is definitely a path to getting lidar sensors on every vehicle but it's just a space that not a lot of of machine learning has been focused on you know getting XYZ data plus reflectivity how does that feed into your algorithms I've put together just off top my head again we're not we're not the ones coming up with these these needs but these are kinds of things I've seen from some of our customers one of the advantages I have in Bella dines because we're kind of in the lead right now we get kind of a crowd sourced information from customers about what their needs are in the marketplace I kind of put into three general categories here but some of the more focused tasks that I think auto self-driving companies are pursuing our pothole detection that's more of an eight a SAP location but high-definition lidar so you know I would imagine we probably the best possible way to go about this finding a hole beneath the ground plane I mean that's exactly the kind of information that we send to you also we provide reflectivity so there may be some information in that that can be mined a road debris I mentioned that tire carcass or earlier on again that's another thing that I think is probably ripe for more machine learning work to be done three dimensional lane markings reflectors that pop up remember we have reflectivity x' as well so we see reflective items very very well in any kind of lighting and also being a three-dimensional marker we we see that as well so there are a lot of opportunities we think for focus tasks to identify 3d objects in the environment some of the more broader lidar based tasks localization finding your place where you are within a map object identification algorithms perhaps that's I think a lot of the the literature points to lidar plus camera in that segmentation of the environment that's one that's traditionally been done for lidar data from aerial aerial applications like forestry and things like that I just got kind of an example here of segmentation this is just segmentation with pure lidar data so what they've done with slam here is they've identified the ground plane they had mined it identified the road surface with our reflectivity measurements you can see the lines in the road the lane markers and the crosswalks and you see the stuff above the ground playing the trees and whatnot and all of this is done with pure lidar data not using any kind of machine learning or artificial intelligence of any kind so you can imagine a lot of this could probably be cleaned up as it is from aerial mapping with machine learning also I think another opportunity for your developers it would be the artificial intelligence infrastructure to support lidar for self-driving right now data data itself is something that doesn't exist out there there's not much lidar data available because again if you think about it there was Google and there was Ford for many years that were driving around collecting data and even they aren't collecting it on a massive scale they're collecting data you know a few cars at a time here and there over a few miles so whoever owns that data I'm not sure who those who those companies are going to be in the future but if you have access to lidar and a lot of pickles you're gonna have a huge advantage in the self-driving Wars also scalable and accurate David labeling seems to be a challenge a lot of our customers are in fact a whole and pretty much do it the same way they have very manpower intensive labeling methodologies that are very cumbersome not necessarily accurate and they they probably aren't scalable for the massive amounts of data they're going to want to process in the future so I think there's probably an opportunity for for companies looking into this space as well so I think that's about all I've got time for today but thank you very much any questions this might be an informal question but I wonder why there isn't enough lidar data does it have to be on a self-driving car and or could we just have those sensors on every car and collect Road data yeah I there's just not that many lidar sensors out there right now that are should say 3d I definitely our sensors you know we have a lot of those very short single line sensors for emergency braking that have been on the market for about ten years but really that's not light our data per se this level of data again I mentioned we we made four cars 200 just a couple of years ago you know even today we're only making them in the thousand so there's not a lot of people collecting this data there's a lot of aerial data collected that might be of use though thank you very much Oh Rebecca [Music] yeah I mean I think Elon Musk has made it very clear he doesn't want lidar on his vehicles and you can look at the California DMV this year is probably the best indication of what they're doing for level for driving and they're not trying the most challenging situations and they're there you know if you look at the data I think it was once every few few miles they're having to disengage I mean we saw the one very bad example of what happened last year when the white truck across their path and the white truck with a with a low angle sunlight background was not visible by the camera and to the radar it was could have been an over you know could have been a bridge over the road it could have been anything so yeah lidar gives it that three-dimensional aspect that you don't have [Music] [Applause]