data.bythebay.io: Sandy Ryza, Analyzing Massive Time Series Data with Spark
Recording: data.bythebay.io: Sandy Ryza, Analyzing Massive Time Series Data with Spark
who here has written a spark job before cool um who here has done time series analysis cool um was this like time series statistical analysis raise your hand for that okay how about like processing massive time series data and maybe a streaming fashion oh that's great because we're not going to talk about that at all um but we are going to talk about the first two things um so I'm Sandy uh I work at Clover health I used to be at cladera for a while I sort of LED our initial spark team at cladera then moved over to a more data sciency kind of role helping you know various customers use their like these sort of awesome but very difficult to play with big data tools um and during my time there I wrote this this Library uh uh that would allow you to process enormous time series data with spark so I'm going to talk about sort of Time series data in general spark in general big data in general uh and then dive specifically into why you want to build a big data time series library and what it should do if you uh if you were to go about doing that um first of all what is time series data so we're going to answer like maybe four questions in this talk what is time series data oh this is really weird the way that it's doing that yeah okay um what makes time series data unique so why do we talk about time series data as opposed to you know in opposition to just like data in general um what's special about it that requires special uh you know special tools to deal with um what what do we do with it what are the sort of unique things that time series data sort of like demands us to us to do uh and last of all how do we deal with lots of it we have maybe more time series data then can fit on a single computer so time series data very this is a sort of the classic hockey STI graph from global warming you can see temperatures being you know relatively chill up until uh the Industrial Age and then they they rock it up over here this is a classic uh Classic univariate Time series we're taking basically samples at regular intervals um you know in this case these samples are floating Point numbers um and uh you know we we can visualize them over some some time sequence um so so univarate time series data we have samples at regular intervals then we have some you know particular scale or observation I'd like to distinguish time series data from what I like to call like time based event data so time series data has time stamps but it's not the only kind of data that has time stamps often if you're dealing with event data it'll have time stamps um but but but you know what distinguishes time series data is that there's some notion of a of a regular interval some notion of sampling from some sort of more possibly continuous process multivariate time series data similar time uh univarate time series data but instead of a scalar at each time point we have some Vector values so we might be measuring multiple things that each have a value at each time Point what makes time Ser data unique why is it different than other kinds of data so the the first maybe biggest thing is that there's this basic assumption in most of Statistics that data is IID it's independent identically distributed um and time series data basically breaks this Bas basic assumption in all sorts of uh fancy and unpleasant ways um so here are sort of the the many ways that time series data breaks this assumption so so so so first of all there's this basic structural concept that unless you're like in the plot of the movie primer time series data uh what happens now depends on the past and never on the future so you can never look at a a future data point and um use that to predict what's going to happen or uh in the past um what's also you know a little bit subtler but Salient as well is what happens now depends on recent past more than the distant past um so this is sort of a basic Assumption of lots of the Classic econometrics Time series models like ARA Etc um that uh that if you exper you know if yesterday was a pretty hot day today is going to be uh was more likely to be a hot day but you don't necessarily want to look at the weather like four years ago to predict um you know have the same weight on uh on whether today is a hot day um time series data has uh seasonality um so for example if you're trying to analyze uh traffic data you know let's say the how fast cars are going at some particular point on the freeway as we move through time um we can definitely improve our model by noticing that on week days we see more traffic or you know during certain Rush Hour periods we see more traffic and then last of all what happens now often depends on absolute moment in time so this is sort of a problem with lots of people who will be uh doing sort of Time series studies and they'll see some big spike um and if they're thinking about a few you know different covariates that might be might be affecting that uh you know it can be confounding to their model but some totally uh totally separate thing that's going on like a big weather event might be causing that Spike um the second challenge with time series data outside of this like statistical challenge is more of a software engineering and understanding kind of challenge um which is that human time and um sort of absolute time since the beginning of the universe uh can be very different um so time series software has to uh basically concern itself with smoothing out the differences between those right like um a uh you know one month does not have the same number of uh nanocs as another month depending on what month that is um and so you know this is like a little example from mat lab of how it's sort of Smooths over this you can specify um time time intervals in terms of uh you know years months and days not in terms of like nanc since the epoch any questions about this kind of stuff so far great what do we do with time series data um one of the biggest things we do with time series data is forecasting so you have a bunch of uh observations from the past you want to take some guess some educated guess at what is going to happen in the future um spping out time dependencies you know this this is a sort of sort of weirder one um but but but often if you want to be able to do your things with your data you the the fact that it's not IID is very unpleasant um so you can if you can use a Time series statistical model to sort of understand the time dependencies in your data you can often uh take the residuals from that model and understand what your data would look like without these time dependencies um this is important in uh for example Monte Carlo models for uh estimating Financial Risk um because once you remove these uh dependencies you can do all sorts of bootstrapping and and reordering of these points um and then apply the model back later and generate sort of more uh observations from uh this larger distribution over your outcomes um and then detecting anomaly this is one that sort of comes up less often in traditional econometrics and more often in the like iot kind of World um but uh you know let's let's say you're analyzing uh you know huge Factory you have tons of little sensor datas coming in all the time uh you want to be able to understand um a you know any particular weird event is it actually weird or is does it mix like if you see a really high value for some measurement that's coming off some machine is that actually weird or is it not that weird because the last you know 20 values were sort of uh were sort of high as well and that's just to be expected during this period um any questions about what we do with time series data uh yeah so the question was kind of give an example of how we could take out the time dependencies so uh often this is done by fitting a model to the data so fitting some sort of model that has time dependencies built into it so like an ARA model um which Which models what happens now as a sort of as like a a linear combination of things that happened in the recent past or a garch model which uh looks at the the volatility and accounts for the fact that sort of uh when there's High variance it happens in periods so so if you can sort of fit a model like that to your data then you can look at the residuals after that model you can say what is uh what are you know the recent observations think that should be happening at this point and what's the difference there and that difference will be sort of the uh dependency less um uh view at your data assuming that your model is like accurate in describing your data um all right we're going to keep going because we're not uh super abundant on time so there's a bunch of existing approaches that people use for dealing with with time series data mat lab has its econometrics toolbox uh python has pandas which has some really nice time series functionality that I personally really love R has the zoo and XTS packages SAS has the etss package and sort of what what's what's common between all of these um maybe in Python we should be including pandas Plus stats models what's common between all of these is that there there's sort of this unification with um software for manipulating time series for converting uh time series back and forth between human time and and machine time um you know let's say doing uh basic like rolling window operations cleaning data imputing data um and then these are married to a bunch of time serious statistical models like ARA models like gar models for example um on top of this let's not forget window functions and SQL um which are you know really useful in certain cases uh maybe in a more like business analytics World for for modeling time dependencies um this is a sort of the spark I believe this is the yeah this is the the Sparks python API um which has very nice window function uh functionality as well um and there are these window functions like lag which lets you um look at you know what happened recently and and and rank but these functions sort of often break down when you're dealing with more complex time series analysis um because like let's say how far you go back in the future it's it's it's hard to specify that in like a principled way so introducing the time series for spark project um the goal here is you know much like these other uh uh single node packages have uh like a data manipulation aspect and a statistics aspect um the goal for the time series for spark project is first of all to provide a natural API for manipulating large time series data um and then second of all providing us set statistical routines for modeling large time series data in a more statistical kind of way um it has apis in Scola Java and python the the Scola and Java apis are much more mature than the the python apis um so sort of the big question when you're thinking about distributed data and you know distributed time series data these kinds of manipulations is how do you lay out the data first of all within a machine um and then second of all across machines um so I have this sort of vocabulary I've come up with it's probably sort of similar to the vocabulary that someone else has come up with um about describing different ways of laying out time series so we'll sort of go through the the the three different ways that I've seen in various different libraries um the first of all one is this observational layout so this is a very like sort of tall and skinny layout the idea is that every record in this in this table um is a single observation so it's a time stamp it's a key which is what we're observing um so so so this can represent a multivariate Time series but in this sort of long um squeezed way um so you know let's say a is the weather that we see in Antarctica you know a particular time uh and B is the weather that we see in California um so so we we um each uh each row has some time stamp and some particular location that we're have an observation for and then the actual value for that observation two degrees in this case um there's a wider layout which I like to call the instance layout so so the observations layout is really good for um uh ingesting data you know as observations come in from different sources you can just append it to a file like this instead of actually having to mutate that file or uh do an aggregation or rewrite it but it's not really an ideal for anal uh layout for analysis in the uh in the instance layout each row corresponds to a single moment in time and all the data for that moment in time is in that row and the columns are just the you know particular locations that we have these measurements for um this is sort of a layout that's nice for classical machine learning kind of stuff um we have uh you know uh each one of these is a particular example we might have a label and then we'll you know we'll say let's try to predict you know based on the temperatures in these different places what the temperature is uh in this place so so it's nice for that kind of analysis but it's not really that great for time series analysis what we're really interested in um is you know for a particular location what's the sequence of temperatures that we've observed and you know how can we smooth those out or fit a model to that um so the last layout is this this time series layout and the idea is that um in this case each row is a is a key so so let's say in our example a location in the world and then we store in that Row the entire time series for that key you know just as an array of values um along with this uh with you know with this entire structure we store a datetime index and the datetime index is basically mapping positions in that array to uh you know human time or absolute time in the in the history of the world um this makes it easier to index into this into structure um using you know time instead of position um so you know let's talk about Distributing for a little bit in real world applications you know I've noticed that individual time seros are not that big right so if you want to let's say look at this the stock market um uh collect like hourly or even minutely data for you know uh a particular stock over the last hundred years that's still like you know maybe hundreds of megabytes easily enough to sit sit on a single machine but there might be lots of these time series so you might have payments for every Mortgage in America metrics for every robot in a factory or an Investment Portfolio with hundreds of thousands of derivatives um so this is the basic interface uh that's provided by the time series rdd which is you know a piece that comes inside the spark time series package first of all you have this rdd um sort of mimicking the time series layout that we have we have a a string which is the key you know it might be the location and then we have this Vector of doubles which is our actual time series some uh some some sequence of values over time then we have this datetime index object which is responsible for mapping uh indices into those arrays into um you know actual times um so a Time series Rd sort of distributed partitioning looks like this um for each of these different different keys for each of you know assuming we're storing time series for stocks um uh each of these colors is what's on a computer all by itself so you know for for Google we have on a single computer all the uh observations over all time um but for other stock uh you know stock tickers we might have those on on separate computers um this makes it really easy to do super efficient time series operations we don't have to do uh shuffles or aggregations across multiple machines if we want to do something like fit in AR Rema model to every um uh every piece of data in our uh in our data set so this is sort of I think we're rushing through I normally get this talk over a slightly longer period of time uh but to rush through uh this is what it looks like to use the library so we're loading up a time series R over here um we take a Subs slice of it over some period of time so we say we only care about the data between uh April 10th of 2015 to April 14th of 2015 um and Library can sort of convert that into operations that happen over all these time series um then we can do uh you know fill in missing values based on a linear interp interpolation so sort of um for each missing value look at the value that came before it and the value came after it uh and just linearly interpolate between those two and then last of all this last line um a uh like removes time and dependent effects we were talking about by uh fitting a LMA model to every time series in the data set um and then calling this uh remove time dependent effects on it so this is the basic like date time index API um we can specify a uniform index that is over some particular time period of time uh at some regular at some regular interval or there's also irregular datetime indices um that are just composed of like a list of of time stamps um we have time series models like ARA and garch we have uh a bunch of sort of Time series related statistical tests for example the Durban wson test is something that's often used for uh testing autocorrelation of data um what would we like to do in the future so so we there's this sort basic functionality that's locked down of um you know doing a bunch of manipulations over time series data um I'd really like to have more full windowing so right now we have sort of a basic window through Scala apis um but there's this whole like notion of tumbling Windows sliding Windows um and sessions that I think we could bake a little bit more natively into the into the library um full full resampling so sort of if you ever use the Panda's resampling functionality it's super nice I've actually started adding a bit of this but um we could iron it out a little bit more and then sort of the the big step that um some people think the library needs to take is able to actually distribute uh individual time series across both of the machines and so this would be the case where for a particular um entity you have um you know too many observations over time to to um uh fit on a single machine uh and then you know from a stats perspective um Vector Auto regression so right now we have mostly time series models that operate on sort of like single uh on on single univariate time series I'd like to add time series models that operate on multivariate time series um there's this whole other battery of statistical functionality um I think uh that is all I have so thanks a lot question any questions for Sandy okay so I was thinking about this uh uh fact that recent time the it was one of factors right uh recent time how do you have any strategy or meth method for deciding when is the best point for recent time is it depending on the area or domain or how do you think about setting the right yeah that's a good question I think it depends a lot on the model that you're trying to fit so if you're familiar with the sort like basic ARA modeling framework an ARA model has these uh two parameters or I guess three parameters but two that depend on how far back you need to care about uh with recent time um p and Q uh so you know in basic REM modeling you just give those and yeah there's the trouble that you're talking about um but uh Professor Rob Hindman if that's how you pronounce his name sort of came up with this uh algorithm for doing Auto ARA so basically figuring out an optimal op and Q um and so that that's basically done by looking at the data and saying um you know if we include time dependencies of this of this order or do they help us actually explain what's going on you know um uh or you know could a simpler model make more sense uh so that algorithm his algorithm is actually implemented in the library and you can use it to sort of figure out what the best fitting Arena model is for your data um so does Auto ARA also check for the order of integration in the case of ARA uh yes it does so yeah there there's that third parameter which is the differencing parameter so the way the auto Rema works is I think it actually just it does the differencing first and it's like a a stepwise thing and then once it's like figured out a differencing order that it thinks it makes sense then it tries to optimize pnq okay it looks like that's it thank thank you very much give a hand Sandy for a great talk the next talk