ai.bythebay.io: Darren Nakhuda, Building Training Data for Autonomous Driving
Recording: ai.bythebay.io: Darren Nakhuda, Building Training Data for Autonomous Driving
hi everybody I'm Darren Auda I'm the founder and CTO of Mighty Ai and um thanks for being here we've had some really cool talks today about all aspects of self-driving cars and like Alexi said we focus on one particular angle of that which is building training data and on the AI side we've been building training data across a variety of different verticals but this is definitely one of the most exciting ones so for us when we talk about all the different ways you can collect data and all the different sensors that are out there the big question is what do you do with that because raw data doesn't do anybody any good so how do you get into a usable State and uh before I dive into exactly what we do in the driving space let me just tell you a little bit more about our company and give you a little context for the role that we play so we like to call our platform training data as a service and we started about two and a half years ago um myself and three co-founders started this company in Seattle we've raised about $27 million to date including most recently from Intel Capital Google Ventures and Accenture and really what we do is help companies usually take the data they already have but annotate it and get it in a state where they can trust it so they can use it to train and scale their computer vision and natural language models as you can see there's several components to what training data as a service means and that's starting with having the raw content sometimes we help uh collect that as well but the next step is Task creation which is where we've spent a lot of time building dedicated uis to perform different tasks so you've seen examples of drawing bounding boxes or categorization or free form text Cy all the way through some of the stuff I'll show you today around semantic segmentation of images and a lot more advanced tooling with other platforms often you have to build those those things yourself it's a lot of work and especially a lot of iteration to get it right based on how your community performs The Next Step of pre-qualifying users is really you know what you would consider the T typical crowdsourcing model we have a community of folks the big difference is we know a lot about them so starting out on day one we launched on iOS only on mobile devices we required Facebook connect and we allow them to connect their profile to Google to um to their uh PayPal and basic and they're LinkedIn and using that profile of who the person is we figure out you know what are they likely to be able to perform well on for example in a retail task you may want your Target customer to be the ones doing your annotation so they use the right words in the right language um Beyond how they identify and how they answer surveys there's C defin a level of qualification to make sure they're also aligned with what you want so we work with our customers to Define what you know what are the goals or what are some of the gold standards that you're looking for in the data you're collecting and make sure that the people are aligned that they can follow the instructions that they know how to use the tools so if if it's a requirement that they can draw bounding boxes we make sure that we test them and make sure they can do that from there the task distribution step is where we decide who of that Community are the right people to do a task and and we push that to them so whether they're on desktop or on their mobile device we'll send them a notification saying there's available task and the next step tasks get done it's the magic step so that you know this is where we pay individuals to do task on our platform and we obviously with any system where you're paying people there's there's fraud there's getting to know new people and see kind of calibrating them against others and that's what the loop with quality insurance is about so we have our own internal models on user reputation and data quality so that we can assess the likelihood that an answer is correct even if we don't have a correct answer and from there what what comes out of the whole system is label data so that is what would be ready to be um used as training data into a system there's an additional Loop that can happen after that which is obviously validation of the model so once we take that data and give it to a to a customer to buil to test their model they can come back with here's what our model is returning um is it correct and we can use humans to do that as well um so part of you know why we started this was that each of the founders had used crowdsourcing a lot in the past I worked at Amazon I also worked at a bunch of startups that were were using a variety of internal resources crowd sourcing external vendors trying to figure out how do we annotate data and collect data and you know what we learned is that it's really hard to scale an internal team so getting a few interns to do some number of labels that's pretty easy when when that scale explodes or you need you know more spiky traffic think of it as you know the uh AWS was for computing rather than having a fixed Workforce you can have a scaling Workforce um one of the startups I built was called Teach Street and we were structuring uh Learning Systems in uh in Seattle and then we launched another market so we were finding all the yoga teachers all the musicians all the tutors categorizing their content um uh parsing their schedules making a structured catalog and we did it with people in our office really me know manually looking at listings crawling the web and it it worked great for one market but there was no way we could go to eight markets in the nationally with without um a little bit more help so we went to crowdsourcing we used um the big platform from the big company in Seattle and they um you know it was a lot of learning we spent a lot of time figuring out what are the right instr instructions how do we do QA how do we understand fraud and and basically how much do we pay how many people do we ask and that was a full-time job and we and it's still a lot of problems and every person I talked to who was also using the same platform had the same experience so I thought rather than have everybody have to relearn the same lessons over and over again let's start a new company abstract some of those Concepts and kind of let them focus on what they do and take care of the data for them so this is just a list of some of the customer cers we work with they are mainly large Enterprises but they span a lot of industries from automotive and Retail Healthcare travel and we we do a variety of um AI we support a lot of variety of AIS from computer vision to natural language so now that you have the context let's talk about driving um there are several public data sets that we used as references when we first got started and they were a good place to start so a lot of you probably seen cityscapes Kitty and some of the research and data sets coming out of Coco um they are super valuable for evaluating kind of what what's possible but they're small you know their general purpose there's licensing issues when you come to wanting to use it commercially and this is my one joke of the day so everybody's ready um they don't really get you where you want to go okay I didn't think I was going to work um so um you know most of our customers have really specialized requirements specific locations definitions of the types of objects they want to classify that's actually proprietary to a lot of them is kind of what what they want to label and why and they need a lot more data than the open data sets you know some of the differences could be location based so the roads the road markings even the types of vehicles you see in Europe or Asia are going to be a lot different than what you see on the streets around here and um and same thing with weather and driving behaviors as far as school buses stopping traffic delivery vehicles and so really that's why most of our customers are investing in building their own data sets that they can use so you know here are some sample images that came from our own data collection of some of the different scenarios that we deal with um the first one is obviously this a street with Mark Road markings traffic signs traffic construction um snow which as we've heard multiple times is is a really hard challenge um parked construction vehicle and then obviously low light snow on the non-drivable surface but it's on the sidewalk so it's probably good to know it's there it's an emergency exit but it's not the common driving platform and these are the types of things that we we might label so as we came to think about this we you know we first naively approached the problem like any other computer vision problem that we've helped annotate and we used our current tools and we we built um we used our basically our object annotation tool to label images and the punch line is it failed spectacularly um found it was really hard to outline overlapping objects get 100% precision and it was just very timec consuming you can see the the laundry list of the types of things that were trying to be labeled it was just really hard to use so we went back to the drawing board and we built a new annotation tool this one taking some of the ideas around segmentation and doing true segmentation so sharing lines having common edges and this actually failed as well it was a lot better but uh annotating a single image would take a really long time for any one individual it was hard for make sure they got all this the details in the instructions because the rules about annotating the a vehicle versus snow or uh a tree were all different and how to handle groups of individuals like the the folks on the left in that photo versus two individuals are split up like if you can drive your car between them you know probably want to have them two as two individual images so we know that if if there are a cluster of people that's definitely you know important to know as well and that goes into occlusion of partially visible objects and knowing where the where the other side that you can't quite see in the still image might might be in which direction it might be moving so there's a lot of confusion around this so we went back to what we do best which is breaking task into microtask and rather than doing everything at once we broke it into logical groupings so in this case we had folks annotate every car in the image just vehicles um so and then we did a separate task for the same image where it was just the road road markings or just the road surfaces and we tried to make sure the community was educated around the definition and the requirements of each one so as you can imagine that was a lot of a lot of requirements a lot of reading a lot of cting of folks to to get you know the right rules for each object but in the end it actually helped a lot you know the biggest challenge we had with this approach was around recall so making sure every single car or every single pedestrian or every single road marking in the image was marked um because doing it as a whole if most of these cars were marked but two weren't and we said well then we have to go back and remark the entire image to make sure it's usable otherwise it's might as well throw it away so we went back one more time and we broke it up into even more microtask and so in this um we call this recursion we'd actually annotate each um each car one by one so we'd say don't worry about the whole image are there any cars in this image that have not yet been annotated and if there are um draw draw a mask around it and so here you can see an animation of kind of how the drawing process works over time so the different colors are the different classes of buildings and vehicles uh Riders and vehicles and Road markings and each each one of those shapes was drawn by a different individual and they were all aggregated together to end up with what I think is a pretty high quality result um and you know one of the biggest problems with it is it still took a lot of time every single drawing every single you know one pixel accurate marking around a vehicle was something that was being done by hand by tapping on on a on the mouse or by tapping on our touch interface on mobile so the next step of what we've been doing is we've been working on using our own computer vision models to pre- segment the images so in this case we looking we're using an algorithm that we developed to find shapes and if you can see the yellow lines those are all super pixels and we now Chang the task from having to draw These Fine Lines to just labeling them so it's kind of a paint by numbers style approach so somebody can just drag their finger or their Mouse over the different areas and say this everything in the in the lower area is a road there's Road markings there's vehicles and and the lines are are by definition crisp so using this along with our underlying models that I talked about earlier around user reputation and targeting an assessment it's made it really a like made us uniquely able to um to do this at scale and do do this at a reasonable cost so looking ahead mean you've seen these pictures throughout the day you've heard tons about different types of sensors right now in driving we've primarily talked about uh photos in in uh camera images but obviously there are other sensors out there there there's a big one we've talked about all day which is lar and um those are definitely on you know on our r on our radar another joke um and you know definitely um something we're looking at so you know it's it's all on our future road map I can't dive too deep into it but that's definitely top of mind and um yeah so so that was a lot and um try to rush through it for you guys so um as far as what's next my recommendation is as you think about what kinds of data you need obviously you're going to collect a lot of it you have to figure out how you're going to store it figure out what you're going to do with it so as you can see it's really difficult even with photographic images to get the right level of quality and the right level of detail that you want takes a lot of time it's probably not what you want to be doing um without being a sales pitch we're really good at doing it and uh and you know I think it can give you a lot of time back to doing the actual research building the models validating them and then and then um having a a great closed loop so if you're interested in talking more I'm I'm around we can talk more um I guess I wrapped up early so we have time for a couple questions hello hi hi um so that was a really interesting talk thank you I wanted to ask you we've been talking a little bit about trust um of AI specifically users trusting Ai and I thought there was a really interesting parallel between that and your user reputation algorithm would you mind giving a little bit more detail into how you go about calculating how much you trust a user um sure I so I have to stay pretty high level on some of the reputation stuff because that's kind of our secret sauce but the um the N is looking at both their ability to do particular types of tasks and how many you know how they've done them over time and and calibrating that with other people doing tasks known answers and and overall you know our own QA of the data and the customer QA so we have feedback loops and um ultimately when data comes into our system for validation it doesn't really matter whether it comes from another human or whether it comes from a system the the task is the same so your question of trust and whether something is good it's the source is independent of whether or not the data is good do maybe one more question anybody have question don't see so we're going to take a few minutes here get it set up for a panel so um last break of the day talking about our vendors upstairs and we'll be back in a little bit thank you thanks [Applause] ah