Scale By The Bay 2018: Sonam Kanungo, Connected Car Ecosystem: An Architectural Overview
Recording: Scale By The Bay 2018: Sonam Kanungo, Connected Car Ecosystem: An Architectural Overview
so today we will be talking about the connected car data platform that I and my team have been developing at mercedes-benz so let's go over the overview of the star so I'll be talking about the data sources and then we will go over the topic of data privacy we will look at the high-level architecture of our data platform and then I will go deep into one specific topic which is data unification module which is a smaller part of our overall architecture and then I'll let you know about our future plans and then towards the end of the presentation we will have some time for any questions that you guys have okay so starting from data sources so mercedes-benz vehicles are driven all over the world pretty much every single country so you can imagine that like we receive data from all of this different region and our data strategy has been divided into three main regions or force region is America and Asia Pacific second region is China and third region is Eastern and Central Europe and the reason for having three different the reason why we have three different region is because we have a different data privacy and data regulations rule in each of these region for example China has completely different rules and regulation as compared to what is being accepted by instance and eastern and central Europe by because of the gdpr which was introduced in May of this year so we see overall we receive data in different region now in each of this geography geographical region we receive data from different models of the car so let's talk about different models so our sedan has a different software in the head unit as compared to our our SUV version which is GL E and then same goes for like each of these each of these cars have a different version of software being introduced every year so for example the software version which is which was introduced in 2017 is going to be little different than what will be introduced in 2019 so what I'm trying to say is that like we even if it's the same car we have different beta version and then we receive data from different sources from within the car and then from outside of the car as well so you would think that like what's the source outside of the car so mercedes-benz has an app which is called Mercedes me app and what this app does is that it basically provides an interface between your car and your your mobile so the some of the feature that that has been introduced in the in the European region by this car is that you can simply say park and unpark from your app and then your car will be parked and a nun parked and this is yet to be introduced in US but I guess like it will be coming soon so overall we have like these different variation of our data sources so we have a lot of data sources okay that's a good thing but then like being a customer focused brand what we want to really take care of is customer privacy so we do not want to ingest or receive any of this data until unless we have received a constant from the customer so even our entire data ingestion starts only and only after a customer says that yes use my data for your product improvement and then even after we receive the consent from the customer what we want to still ensure is that the way we are handling our customers data is in a customer's benefit we do not want to like make any like anything out of it like until unless the customer is the end goal of our product so the privacy and the security is handled in a really secure manner in a really proper manner so what I mean when I say this so we do not persist any PII we do not expressed any information that can be traced back to any individual customer any information that can be tracked to the location of any individual customer so before we purchased any of the data we get rid of like all of this all of the information and we have like excellent anonymization strategies and place to to be able to do that and then that being said let's move on to the architecture of our data platform so what I'm showing you here is a very high level architecture because I mean like I thought that like for the purpose of this talk I'll go were just one module and then we will focus mainly on that so so as I said earlier in this in this architecture we first of all receive a concern from a customer while diversity's me app once we receive the constant we start receiving the data from the car so start a car start sending telemetry data to us what telemetry data is basically in the head unit when an API is being called from a function we log that function and that's what being sent to our back-end and then we receive some data from our series me as well and this data is forwarded into a data unification module the purpose of this unification module is that when we receive from the data from like all of this like different variety of sources for our data scientists to analyze this data in a way that like which is easier and efficient for them we want to homogenize it in a way that like where every row on on our final data store makes sense like is one data point and then is kind of a kind of repository for like all of the information that happened in like those few seconds or those few minutes as I said earlier we persist the new duplicated and only anonymized data in our persistent storage and then once we have this data in our persistent storage this data can be accessed by our data scientist and then they create solution for product improvement so now talking just about the data unification module so I've taken like a small example here so we receive telemetry data version 1 let's say that this is coming from like one of the model of Mercedes Benz which is a 300 and the telemetry telemetry data version 2 which is coming from another model the su G le so what we do is we combine all of this data together and then we create one datastore which we call telemetry data so here we have all of the telemetry data coming from any vehicle formatted you know you know in a way which is which is which is homogenous basically and then we combine this data with the data which is coming from the app once we have like all of this combination done and all of this data transformed and cleaned and passed we finally store it in our persistent storage when it is ready for our data scientists to be analyzed and then here I'm just showing you a small code snippet so for our data unification money will be used Park in Scala and I'm showing you an example where we are I'm using like a with watermark feature of spark and then like we are talking about like DNA unification and like I'm sure that like this is like pretty easy to be done like if you have to write a spark bad job you write it and like you can understand that like it's doable it's it's nothing rocket science but then the challenge that we are facing at mercedes-benz is that because of the GDP our and and our our commitment to privacy we do not have a lot of time to deal with this data we I hardly have a window to do all of this like to do all of this data ingestion data cleaning data parsing and then do the anonymization in the end and persisters so all of this unification is done in stream so from the time that a data point is generated in the vehicle or on the app the data is ingested unified transformed anonymized installed within few minutes and I think I think like that's something that I really take proud of ok yeah so so in the in the second block I'm doing the Union I'm doing a join here which is based on an ID so this ID depend on on the on the source of the data for example like every vehicle has a we call identification number so if you do this join on the basis of make an identification number and then like the timestamp so what we have analyzed so far is that for the amount of data that we are dealing with I mean you can imagine like as many cars that we have running globally and then so the amount of data that we are dealing with it kind of affect our our our our data storage that how how long do we want to hold this data in there in a spark in memory frame data frame or or in the cluster so these are the number that I mean these are just an example of like what usually work for us but then like we always play around these numbers and then like see that like what makes more sense in terms of like joining these data set together and what makes more sense for our data science to have to have a a unified data Scala is our preferred language of programming for our offer all of our data platform development so we have developed of course like spark in Scala and then we very recently started doing some things we were writing a lot of UDF's in Scala and then it happened that some time like UDF which is like needed for this kind of transformation it's the same which is needed by another kind of transformation so what we have done is that we created a scholar library and then what we do is we internally do releases of the Scala library and these UDF's are very specific to to vehicle functions because I mean all to vehicle domain and then we share it across like the other team members and then this has been proven useful for our data scientists as well because like when they analyze the data they also like find it useful so we have been like doing this internal eventually in future maybe we might as well do an open source of these libraries so that's something that we want to do in future now so I talked about the future plans of the data engineering team there is something that mercedes-benz started doing recently is that we are I mean I and my team we are currently working to implement an ISO standard which is called two zero zero seven eight and I'll go more in detail about about this protocol so what the standardization is that this is a this is a way of creating API specifically for the OEMs to share this data through I mean like to any other third party or any other entity across or like outside the organization and the reason why we feel that like it is important for like any om to like work on this kind of standardization is that in future these these API will be able to provide an interface between the third-party apps on your vehicle by somebody else other than the OEM and I feel that like that is an explorer which that is a space which is yet to be explored by by by a lot of like car makers and that's why like we are rigorously working on developing this and then one thing that I want to like address back is that while we are doing all of this still the main focus of the brand is to keep our customers satisfied and not just satisfied like give them the features give them the product which really makes their life easy and and I mean like build a community basically where our customer enjoy their car even more thank you any questions so whether great work they'd like you have better images it's all kind of data so it's data which is coming from from sensors data which is coming from the API internally but not necessarily image because this data platform that I'm talking here is not necessarily the autonomous driving platform but this is the this is the data layer of the autonomous auto start driven car I just order magnitude and enhancer so so we so the head unit sends the radar only and only when the car is in operation and not when the car is spawned so I mean at an average we we receive data about I think like we receive data every few minutes and then it also depends on how much of activities is being done in the in the vehicle like for example like how many times of feature is triggered how many times certain action is being called so it depends on the data size as well so but then overall like the data volume that we receive is pretty huge so when I'm talking about data volume so I'm talking about like the data coming from each car x number of car x globally i do not know the exact number [Music] with the regular is like polymathics or is it so it's it's telematics data and then this is I mean like so this is not for the cars which are like driven like until 2017 so this feature only like started from the cars in 2018 and then in that as well like the cars which can be connected to Mercedes me app okay thank you [Applause]