Scale By The Bay 2021 : Evan Chan, Location Based Data Engineering for Good with PySpark:Graphs
evan thank you very much and everyone welcome to scale by the way it's a it's a privilege to be invited back to speak here it's uh i love this conference i've made a lot of friends and you know learned a lot of things over the years and i'm here to talk to you about location-based data engineering for good with pi spark that's kind of a long title so we'll dissect that and what it means so uh just a quick word from my employer urban logic what we are is imagine that you could do uh data science with a cool a lot of cool data from maps location data all kinds of things and you could present cool visuals and help your community improve how it plans itself that's basically what we do is uh we are a software platform that presents analyzes and does machine learning data science on um on data to provide insights for cities counties states etc and we use rust python machine learning and stuff on all kinds of interesting use cases some of which we will go into uh today so location-based data what is it well everyone uses their smartphones and when you use your phone you use apps that often times report on um you know where you are for various reasons this could include maps but but other other apps will use it like apps that serve ads which are almost all of them does this so for better or worse you get there's this data that comes from all kinds of different mobile providers it comes from google apple et cetera and there's a large variety of this data with different coverages and densities and what you can it can be used for a lot of purposes but what we like to use it for is to process this and condense it into notions about where people are coming and going and that can help with all kinds of analysis to improve your communities so let me give a couple examples example one has to do with what we call amenity and accessibility planning so imagine that we're talking about for example parks in a city right so who visits or where do people visit these parks come from and where do they go afterwards how can planning for city parks their locations what recreational facilities they have for example like if they have if they should have a dog park or not how can they be planned better what kinds of people are most served or least served by the existing parks and if they were to if a city was to zone another park you know where it could be placed to serve underserved populations right so um so so what we can do here is that we can turn this location-based data into notion of origins and destinations for people that visit parks like where did they come from uh where do they go and this can help people plan uh this can help cities plan where the parks are and and plan what kind of amenities they should have like you know disability this kind of thing another example has to do with transit planning so here what i'm showing are origins and destinations for people traveling through transit in this case subway links here in vancouver and the different colors basically to represent origins versus destinations we were coming through transit so by looking at this data would after we aggregate it from the raw data we can visualize travel patterns and figure out for folks that are uh traveling on this transit route in like you know how effective is it if we were to extend if the transit route was extended you know where could it go and so so there's a lot of usefulness to do planning transportation a third example would be what are called play streets they are apparently popular in certain very popular in certain parts of the world but the idea is that we want to close off portions of city streets to provide play spaces and this is great especially for families you can have a lot of just you know you have uh just a lot of space and in a lot of neighborhoods there's a lack of play spaces so this is something that really helps and by combining location-based data traffic data demographic data we can help inform uh good potential places to implement this kind of policy so i hope these examples give you a flavor of a broad use of location-based data to help improve our communities and cities so now let's switch to data engineering so we want to build pipelines for processing [Music] this data and what are some of our goals that we would have for producing these pipelines well we want them to be scalable to be able to handle bigger loads of data we want this to be reproducible that we can run and run with the existing set of data again and produce consistent results right we want uh these pipelines to be flexible so that and easy to maintain so that we can go in and you know someone else can come in and understand it and tweak it for you know whatever purpose we would like to be productive right so these are all goals that would like to meet currently at our company we use a framework uh a python framework called kedro so what is kedro so cadro is a framework for producing structured data pipelines the way it works is that you set up a directed graph of nodes and the nodes transform data uh you can think of inputs and outputs like data frames so they can translate transform data frames to something else and then you can take the output and transform it and and kendra produces a really nice visualization here where you can see for each node which has the f uh function right this is basically a node it's just it's essentially just a function that takes input transforms it to some output underneath it could be doing something very very complex and you have data inputs and outputs and you have parameters and based on those that would power your pipeline hey evan we have a question here um absolutely how does kedra compare with airflow ah i'll just give a really quick uh answer because the uh i will talk about airflow later yeah airflow uh is a is another is an orchestration package that that can orchestrate a lot of data processing i would say there are very different focuses pedro actually works with airflow but kedro is not focused on orchestrating and scheduling and running the pipelines it is more how to organize pipelines into a dac so it's a bit of a simpler abstraction um i'll but yeah we'll talk more about this a little bit later and i will talk about spark spark you know everyone uses spark by now uh i use spark since it was in the zero point something uh days when very few people had heard of it but now everyone uses it and and now spark supports python very very well i guess it's a big deal because um just uh python is so uh you know everyone uses it in the data world so the way i compare these two like is that um so kedro being just an organizing framework for pipelines uh supports spark but typically when people start out using uh things like pedro the base version is uh people are using pandas with it which is you know a single node and all your data basically has to fit in memory but you can then plug in pi spark into it and now you end up with a pipeline that is scalable and it's distributed it's still written in python and still written at a very high level with high level transforms but the data does not have to fit in memory and you can work with all of the huge number of deploy options for productionizing options that you get with spark so with that out of the way let's get into what i call origin destination inference this is a this is a method of extracting trips or origin destination pairs out of location data so we'll look at how we can do this using uh pi spark so the overall passing steps are first we want to normalize and sessionize our data so that we get it by device then we want to coalesce these events so basically what happens is you can imagine one event is uh your phone like sending in a ping to say i am at this location at this time basically right so that's one event um now a lot of so in order to create patterns out of it the first step is that we want to coalesce these events that are in what we determined with the same location uh we want to collect that to a single duration event we use inference to determine and turn these events into trips or origin destination pairs and and and and finally we can uh you know sort them and get them together so the initial steps you know we read some data out of our favorite cloud data store and you know spark can just read this data directly out of the cloud you know and and select some data which is uh this is actually a really cool i know this will be like super simple to all of you that work with big data i was just going to point out that this is actually a really big deal to to folks from who don't have a big data background because packages like appendis for example actually cannot do this they just work with single files you know on your local you know machine but you have to plug you have to add other plugins to be able to do this so it is really cool that spark can actually read whole directory sports of stuff and then you can usually sort your data uh state by device id easily one line right but it's pretty cool we take this for granted but there's a lot going on under the hood when we do this right and so the initial output we get might be something like this where you see there's a whole bunch of this data that is sorted by device id and now we can analyze you know the patterns of events that we get so um what we want to work with are windowing functions for coalition data so um the um the windowing functions what the way this works is that we have a sliding window that goes over the data and the you can see that i highlighted a current row so what you imagine is that a window will start from uh like typically it starts when the current road goes backwards you know so it kind of goes goes like this excuse me for a minute i need to mute i need to build something uh really really quick okay um sorry about that um so so basically the window is um you have this window over rows of data so imagine the window goes you know kind of it was a little bit hard to highlight it goes kind of like this so typically there's uh two directions you can go so windows you can look backwards which is called lag so you can see the index goes minus one minus two minus three so you can look backwards or you can look forwards uh which is uh which is called a lead function which is kind of neat you can actually look ahead and we're going to use that so the first step is that we want to figure out well how far is my current event for my last point so you notice that we start by doing a window.partitionby in order by this will divide your data into events for each device and ordered by time which is really convenient which is the order we want to go through things then the way that we compute the the distance from the previous point is that we take the latitude and we subtract from the previous latitude which is which is using this lag function and one means that we're going one row back and we do this over the window and then we compute the hypotenuse distance by doing the delta of the latitude and longitude so pretty simple and we see the output here uh where we see the null is basically well this is the beginning of this device id so there's no previous row and so you see for the ones that are the same we got the zero as we would expect now once we have the distance from the previous what we do is that we want to group events we want to group a bunch of events that are at quote unquote the same location so to do this we add additional columns for first row and last row and what first row means is that we want first row to be true the first row in the group where the where it stays the same and the way we can tell that is by looking at this distance that we just computed if the distance from the previous one is above a certain threshold then we would count that as not staying in the same location so therefore it is now the first row of the new group the last row so this is really neat we we figure out the last group by seeing if the next row has first row equals true so this is an example of a look ahead and and this is really neat that you can do this it sets up a kind of this is all very high level but you know this creates like you know very interesting state after we have the first and last row markers now we can um now we can try to compute statistics so let's start with a really simple one we can count the number of events that happened between the first and last row of a group like how many events there was when someone stayed at one place and the way we do this is by starting another row you see there's a lot of these temp sorry another column there's a lot of these temporary columns and what we do is that we track the row number when first row is true and otherwise we output it now and then we have this new function called last where we can look up um the the value from the first row and so we would just take the current row number and we subtract that and now we got the count so um [Music] and and so we would repeat this across everything else that we want to aggregate and we can end up with this coalesce output where for each row basically you're looking at um you know you're looking you're looking at hey this row has 41 event you know someone was there 41 times and they were there you know like for 18 hours you know at this location so so this allows you to then do a lot more inferencing around trips there's a couple different methods the easiest method would be to use statistical methods uh statistical thresholds so you can say that an origin or destination is when the device has stayed at you know x location for uh y minutes you know or the number of count and or the number of counts about a certain amount so you can use kind of fixed thresholds it's a really simple start to to infer some it works okay um you can do smarter things like you can say on average a lot of people uh just go to and from work right so we can make an assumption that the location with the longest dwell time is likely to be the home location and the second longest on a weekday might be in office and you can confirm that based on their their travel patterns and so on um you can you can make you can combine several of these and make things a lot smarter using things like k-means clustering you know so you can use all these methods to try to infer you know trip information and then we need to check our work so all these inference algorithms have some bias such as using homework locations is biased towards um a standard uh work lifestyle right with kind of fixed home fixed work inference and so there's certain you know populations you are probably biasing against if you use a method like that and they all need a lot of experimentation with a lot of parameters and then you validate it and you come up with an idea of where the trips are so and then after that then analysts at the cities and other places can use this work and hopefully improve our transit and other and other things like that um another area you can go which is really neat is that you can use graphs for analysis so here what i've done is to aggregate my location-based data into a graph with counts so i've put strategic locations such as transit stops and neighborhoods as notes as a graph so you can see here i've been note called neighborhood one you know stop stop a stop b and so on and at stop a i can see where people are coming from you know that they're coming from maybe another stop or some neighborhood in another neighborhood and i can see the mode of transit that they're coming in from and i can see where they're going you know i can see that um you know people from starbay like to walk downtown but some of them might take a bus to another stop and so on so when i aggregate this data in a graph i can do this analysis and traverse some notes and figure out um travel patterns beyond just origins of the destinations it can can figure out uh you know really really interesting uh things uh see where bottlenecks are so this is uh a pretty powerful technique that is underexplored all right so we've talked about uh processing and doing inference on location-based data which is really neat and i think i think you'll see a lot of examples of of this like today and in the future but i'd like to now switch to the last half to talk about this data engineering and producing reliable pipelines so this is a really huge space um because there are i would say that as a data engineer you can spend pretty much all of your time just keeping up with all of the projects that are happening in the space and there's a a really huge amount but the common concerns everyone needs scheduling orchestration failure handling retries also really tools and you can there's such a large range right like you can start with a super simple setup with something like kedro or some other basic pipeline tool with just base kubernetes stuff you do get some resilience that way more likely folks would use something like apache airflow so um so really like um someone had asked about airflow before i look at airflow as like it's like so airflow provides you with a directed graph also but it also does orchestration also does scheduling and um so it's a much higher level thing than than cadre casual is more like a you know the the directed graph piece in airflow i think pedro actually does it slightly better than airflow but airflow is much more complete right so um airflow is also uh there's a lot of cloud environments that support airflow and there's even websites that have hosted airflow so there's astronomer that i o i don't know if folks have uh used them uh if actually if whatever folks have used it'd be great to get their you know feedback on how some of these there's obviously ammo flow and there's other similar frameworks and ml flow itself is really popular and hosted in various places you have cloud-based solutions like google dataflow and data bricks and other things that are more specific um and finally you have even higher level uh you have some specific like you have a lot of uh cloud or paid solutions that are really um they are more like complete like etl kind of pipelines and um like there's gaboola and other things like that um i will switch to uh talking about quality engineering uh this this is a really big deal nowadays uh because data bugs are not like service bugs that because it's not enough to fix and redeploy it and going back to fix all data is really expensive i find that so there's a lot of good data quality frameworks i just listed two that there's a really standard one in python world put great expectations it produces these pretty reports um and i found one that is specifically for pi spark where you can write a sql which is kind of neat because it allows uh more people to be able to write these tests on assertions on data i think whatever data quality framework you use to catch stuff i think the important thing is like how you incorporate it into your methodology there's a lot of you know basically this is the ops part of data engineering is you know things need to be set up you need to have a way for people to regularly get these reports you need to alert on them right and um i think data versioning will also be really key so using things like dvc you know dlt etc like so um being able to compare different versions of your data so that if you do fix it you can see what was fixed right and finally observability is really important right for data pipelines and i guess my comment on this space is that this is a space with a lot of movement um i'm interested in hearing your tics as someone that worked on obsid observability but more the traditional app possibility my my observation is that uh there's a lot of tools that aren't always a great fit for uh like if you just use standard metrics and monitoring frameworks um you know oftentimes it's like a circle and into a you know square you know like or sorry a square into a a thing uh well yeah you know what i mean right so they don't quite fit that well like for example metrics infrastructure is typically designed for long-running processes and not designed for transient jobs they don't make it easy for you to compare different uh by runs and stages uh logging tools are oftentimes not easy when to use or not helpful when i mean they are helpful but uh when you have more complex pipelines and complex stacks and distributed data systems then they become [Music] really difficult to trace what's going on i think that you'll see like the use of tracing tools and a lot more specialized data pipeline tools i think you're seeing this already but i think you'll see more and more very specialized uh data pipeline or data observability ml model observability kind of tools coming out and this is a space that is very exciting and this would be a great topic to chat about too and that is what i have today thank you very much for your time feel free to reach out to me on twitter github i have a webpage there or instagram if you like photos so um yeah thank you very much i think devin we can i don't know what time for a question or we should you