Scale By The Bay 2021 : Elad Rosenheim, Funnel Rocket: A Serverless Query Engine
Recording: Scale By The Bay 2021 : Elad Rosenheim, Funnel Rocket: A Serverless Query Engine
um thank you and hi uh usually i love seeing people's faces so i'm i imagine that you'll learn and i'll be glad to take any questions that you have so parties there any questions during the sessions we'll try to take them on and you know if we can't handle it during the session we'll do it later in in the special chats okay great okay so the topic of this conversation um is kind of an open source sunglass query engine uh that i've developed uh in my time as at my previous company uh the name is final rocket but i think maybe the the real question and maybe the real motivation for this uh for this presentation is like do we really need another silverless query engine and how does one oh let me frame like this you know uh quoting from paraphrasing from you futurama how did i get to this point in life and say okay i'll just write my own um you know if you're a veteran developer or you know anything with engineering and you come to this conclusion it needs some explanation so i'll try to show some of the background and i'll be happy to take questions and you know the whole underlying concept is yeah there's no silver bullet there's no just one right thing but at certain points you realize what your needs are and then you may want to go on a test it you know go your own test so that does happen and i'll try to share my experience okay so just a bit of self introduction without selling myself all my employers too much um so i've been working professionally since about 97 so i'm officially a boomer i guess uh this that's my what my daughters told me um and i've been really all around the stack so front and back end full stack all that is really interesting to me and i'm trying to maintain that same passion for all things uh uh software development i work in stockbit which is a small startup that does a website builder for developers um but i only joined there a few months ago and before that uh just uh when corona where we started hitting i was still working at dynamic fields where i worked for several years and that's why i developed a funnel orchid which is this query engine uh an interesting curiosity about dynamic gear that's been acquired by mcdonald's so that doesn't happen every day um but i think for context for this conversation it does um it does do a service to talk about just to understand what the problem is or what the problem domain is so dynamic yield if you look at the marketing pitch don't say okay we're assass provider of personalization i mean a b testing recommendations targeting audience segmentation and what it really means like in in terms of what we actually do is work was you know a lot of high volume websites we track the visitor journey uh we analyze it we segment it and i i put it in bold because it's it's really related to the problem that we're trying to solve here and once we know you know more about these distinct user groups we can apply any number of optimizations so it's really like a white hat operation i should say like we're not buying or selling user data or any of that sort we're just working for a specific customer at the time and trying to optimize the website better for whatever use segments the user segments are there so in terms of like let's say the large customers because there's always a mix um like each one of these large customers can have millions of monthly unique users um maybe like dozens of millions and then a hundred million or maybe much more monthly events and when i talk about events i uh most our customers are like e-commerce uh stores so it means like page views and then e-commerce relevant events like add to cart some product or purchase product and so on and the word users is important because we're not just looking at specific events we're trying to understand users so we unders we want to understand the real user journey per customer um and that's a big part of the problem and we'll get into the problem you know in stages um just a bit more about the context of you know so i call it like it's like like the king and i and of course the rise of open source big data and and kind of my perspective in it and these are the years like the eight years i spent the dynamic were really like following this this trend so when i started there like 2015 um hadoop was still a thing like yeah and and and i think it's a reminder that it all started with google every time they released like a white paper so there was mapreduce and google file system and then bigtable and everything kind of evolved into an apache project so yeah hadoop was still a thing and then people understood like hadoop is kind of low level it's slow it's inefficient it doesn't really give you any of the nice functional constructs that when we know when spark came that was as close as to to a silver bullet in the big data space um as as i recall so spark was a big thing and then of course uh the parquet file format came along for analytics and by the way it also comes uh so per can bigquery and all these things are all coming from this other google white paper which you may or may not know called google dremel so dremel is a power tool and if you're looking at the history of big data it all started right with mapreduce and batch and all these things and then people realized hey we want things that are more interactive we want to have these big data sets and without a lot of pre-baking and etl we want to start doing queries and you know potentially ideally if we can sql queries over big data where it rests so the first attempt i would say maybe uh at this was google dremel which then became also a service to the public called bigquery so this all started from this problem and i think the first stage was like if you look at bigquery of how it does things it's done things that are really really similar to perk but um it still has this etl and then amazon's athena came along and they said like hey the data is in f3 uh it's a trust you're using common file formats and we're just aiming a little army of query workers at your data at rest and this was a big improvement like all these etl steps they are a pain if you spend any time around them and of course the cloud so the cloud became cheaper and faster of course but also something that people don't sometimes appreciate as much as they should i guess is better networking and the reason is not just it's cheaper and faster but also some architectures that were not possible when the networking was mediocre as it was in aws at the time are now possible so a large part of what i'm doing would not have been possible if you couldn't decouple compute from storage because any kind of decoupled solution needs a very fast network otherwise it's not feasible so this is a very big step forward and i'm so glad that aws also picked up the ball and the networking now is so much better than it was before um so really if you look at the cloud now you have this wide range of options like i'm mostly proficient with aws so that's that's where i tested and developed things but i i always try to avoid locking so if i'm looking at aws today you have anything from almost bare metal to completely serverless so it's really a spectrum and you decide you know the more you need to manage yourself the lower the cost is so if you you know at one end you have virtual machines and then you have style gate and all these managed services and then at the far end you get to serverless like it's the fastest options to get compute and the least you should think about to get there but also the highest cpu per second cost i might say but there's another angle to this which is that the level of granularity of the billing program is always improving so it kind of tilts the scale and what i mean is if i can do something with serverless and i pay a lot per second but i'm just staying for the actual milliseconds when i when i need it up it can still be much much more cost effective than even if i have spot instances but they're lying around and most of the time they're not doing anything so if i can use serverless at a high cost per second but i'm just using it for the exact moments the exact milliseconds where i need to do something it might be much more cost effective than all these big clusters lying around whether they're rever reserved or on demand or even spot instances so that's another consideration um and i would say i was with all these silver bullets and this has been a great great advancement in less than a decade maybe but i would say there are a lot of things with big data that still suck i i have no other word for it like shuffles is the bane of big data and complexity of the clusters and so on is still bad and life in general in big data is pretty hard at least when you think about auto scaling and cost optimization it's a really hard problem and i can talk for hours and why but you know if you want to know more talk about we can do it later in the q a i think this is a problem that's that's that's been interesting for me for years um so let's talk about the actual problem at hand because we're talking about final rocket not just any specific gripes that i had with big data even though they are very interesting um so what's the problem and the problem is i was kind of thinking about it so there are many people in the world right and this is a problem i'll explain why but i think in general it's a problem and you know people know it's a problem that there are many of them and yet they persist but let's talk about why is this a problem so it's only a problem of scale of course kidding and and let's think about this the problem that i was tasked with solving so um a product manager at the time you know we started talking about this functionality of how do we explore user audiences so we have this all these data that's flowing in it's siloed per customer um and a business user or a marketer or whoever you whatever you call this world is going into a web ui and they're trying to understand the user audiences so they need to do these queries that that really can shine a light on specific the specific journeys and traits specific user groups so they can say something like hey i want to see users who did this uh less than five times in a week and then they did something else and they they didn't finish by doing something else that we would hope they would do so this is a kind of a funnel query or a sequence query and over this kind of like uh final query you also want to apply a lot of aggregated analytics which is another layer of complexity because um if you actually look at the details like doing sequences and funnels with classic databases is a big problem it's usually very awkward to do it if possible and then it crashes its scale and then applying um all these uh aggregations and all these breakdowns like by country and by region and by device and by whatever else you want and having a bunch of them this is really a performance killer um and and also you know when people talk about solutions and i'll mention click house at the end like click house for example like like any other technology it excels at certain things and it's it's harder to do other things so you can do sequences fast but when you start to do all these aggregations based on the results it gets slow so they're always like in real world you always get into these uh walls so sequences are hard and sometimes you see systems where they have funnels and they have sequences and it's all like baked in so you do it and tomorrow you come and you see like the results for that day or the result for the last week but here we're talking about diesels for exploring so it's ad hoc queries it's an interactive feature for end users meaning it's not like like i don't know some bi analyst in my own company and they can wait 5 or 10 or 15 or sometime 20 seconds and they know it's fine like if you use amazon latino it's basically an internal tool so if you have to wait and even if sometimes it fails or whatever okay it's a great tool but it's for internal purposes if i want to make something um available to my users i need to make sure the sla is really really clear so we wanted flexible with all kinds of conditions and ad hoc rules we wanted cheap of course and we wanted fast um go and build it and of course i'm over simplifying because the pm was great and she didn't she didn't actually leave it at that we worked a lot of the functionality but that's basically the mission statement as it was and i know i'm playing with the fonts i think it's uh quirky but uh um so so here's my thought process um so if i'm looking at this problem of sequences and funnels so we're talking about users right and each user they have their own history could be shown could be long of of events so like all the questions that i'm applying at the user level i'm like doing something i'm doing something akin to like group by user id or whatever it's called and then start asking questions and the problem is group by works really well even on a lot of data when you have 10 groups a thousand groups maybe 10 000 groups but why don't we you have 20 millions and within each of these 20 million groups now i say okay find the user who did x like find groups when x happened and then find uh groups where x happened and then y happened following that and with every group these timings are different so it needs to have this order in each and every group and this is a hard problem that usually databases and i'm talking about prosperous and and even past presto and other databases that i've looked at they're just not optimized they're great products and i'm not saying like i invented the wheel but there are some certain problems that you see and they are not these fringe problems but they're still not important enough for the mainstream databases to do them well so we have this very high cardinality group by problem and sequences are making it harder and then we have to apply a bunch of aggregations and something i should say is another thing which complicated life from for us at least was that if you look at click house and many of these solutions the assumption is you're just pushing more and more data in so data comes in and you maybe apply some transformations and you append it and you're not updating anything that came before because that's much much much less efficient but for us because we're tracking users sometimes every day we discover that user x and user y are actually the same person from multiple devices and they logged in from multiple devices so in terms of optimizing the experience for them we need to treat them as one so we basically i want to say like 1984 but we're going back and fixing the history so it's not even just an append only problem for us some problems are like that and and it means you can use more tooling but if you're using tooling that kind of has this basic assumption of pen only and then you have to start fixing things it's a big pain and of course um there's always the option to spend more money more money on the problem until it blows up but at some point it's just it's not reasonable and the eye of sauron is on you and i mean like the cfo or the ceo and they're like what's going on here and then you start making excuses and you don't want to make excuses right um so what was our initial approach which worked nice for for a while for a bunch of years so nobody wants to reinvent the wheel right it's it's long it's bad and so on um so we tried not to reinvent the wheel but i'm gonna make a claim that um when you're saying you're not a reinventing the wheel sometimes you have to customize it so much that you end up inventing a wheel or another wheel based on a common platform so it's not just about how do you re-invent you know or with are you inventing the wheel or not but really how much if you have a custom problem to solve so we started using elasticsearch which was pretty good already at the time and we modeled each user as a document so now all the data within a user is now in a single document which makes it much easier to search for a lot of things that need to happen within a specific user or specific document and we use sparks to take the whole data and index it to elasticsearch there's a nice connector for it meaning you use spark water workers to to make that this distributed indexing and they're always a cost like they're always the price for everything so elasticsearch could be really really fast when you search things if you do it right but the really really high cost is the indexing indexing is a really really expensive operation in terms of time resources money uh all these metrics so they're really you only shift the problems and create problems somewhere else um and it sounds really cool like like with a lot of solutions like you start out and you have this x amount of data and it works out great and it's cheap but then you realize oh but i can't really do the sequences the way they should be unless i write a custom plug-in to elasticsearch and i apply some weird data encoding to make the data more optimized more compact more friendly for my plugins and you start going into these rabbit holes over the years and it's a gradual process like you're just you're like this frog thing boiling water so at first it's all cool and then you know the heat goes up and up and up until it's like becomes unbearable and it gets hard and it gets expensive and even slow and the ceo is unhappy um and to quote from maybe austin powers when the ceo is unhappy people die or at least or at least it's about it's a big bummer and when you think about one one other problem that people don't often talk about is this coupling so of course there's an elasticsearch connector for spark but what it really means is that you're slowing down spark when two online systems talk to each other and one is much much slower than the other you shift the problem and you pay a lot for spark basically just sitting there and saying hey can i can i send the next batch over now is it okay and you pay for this time well spark is not doing because the scale of this system is literally different so any kind of of talk um at scale between two systems you have to think about the capital and then decoupling is a separate process right so it adds this complexity also so it is a problem and that's at some point and um of course i'm paraphrasing from from game of thrones here i realized that you know uh all clusters must serve but at some point they must die and it's the same kind of connotation as it is with game of thrones um must die it means that it inherently is gonna happen so this is something that we have to accept like these long hours these nights this complexity it's just in the nature of clusters and when i say clusters i mean real time and streaming and message queue and batch processing and all of them like i did it all i've done them all and they all have this complexity and they all die at the end and in the most unfortunate hours at times so can you do something else and one of the uh one of the things that i do and i learned over the years how to do it better is when we look at the problem forget about all the solutions and all the cool tech that the young developers want to try and try to really crystallize what are the traits that we want to have like what's really important for us and then the solutions they kind of they kind of present themselves from these basic ideas but you have to forget about you know everything that you know about solutions and just think about the problems and think about what what are the good things that you want to have in your solution and how critical are they for you because they're always going to be compromised so we don't want to reinvent the wheel but again we don't want to reinvent too much we want to rely on existing technology but it's going to be some work to bring it all together and we want to have low maintenance because all these long nights and it's it takes a toll on people and we wanted to be fast and wanted to be the couple we talked about like spock is doing its preparation job and then it doesn't have to do or wait for someone else okay and it doesn't have to be sparky and the other things and we want to have a good visibility on costs and if i can use the same solution for both ad hoc queries in batch that's great if i can't subvamil i want to have minimal predicting of data like as close to all data as i can and low overhead and when i say low overhead it also means like if i'm creating a spark cluster now there's so much time meaning money and time that is spent on just getting the cluster ready to do anything and for me this is this is pure overhead like and you might say okay it's a it's a big tool it has all this functionality but in the end i'm paying for a lot of setup that i'm probably not gonna use like i just wanna spend time and cost on the actual problem that they have at hand i want to get nearer to that point so here's my thought process um we want to go back to old data let's let's go with 4k we don't want any kind of etl and we want to do data crunching sequences and funnels and navigation so basically there's something which is already great and it's called pandas and then it connects to parquet really well with this project apache error and by the way pandas and apache are by the same great guy wes mckinney who really should be ought to think every week uh not even doing it for any kind of profit and then okay i i all these tools die like i know how to run things at a single task level how can i run like thousands of them concurrently and we said like no servers no clusters minimal orchestration um so maybe serverless can it work with serverless like um how i'm gonna orchestrate all these things will it be fast enough will be cost effective um will it be tied or locked into a specific vendor so that's pretty much where i got to think about aws lambda as a solution and when i was skeptical at the beginning but i'll tell you um honestly i was impressed at how good a solution i mean how much sure a solution it is today and maybe it's not surprising okay everybody's had years to work on this and it has all these features that are really really good for us so it can do um you can invoke lambda synchronously meaning you can you can just launch i don't know ten thousands of them and just it will cue them up but queuing them up doesn't mean it gets slow like it will um handle them as fast as it possibly can but then if you have like any kind of concurrency limits and so on you won't be just hitting the wall like it's like fight and forget but it runs really fast and things don't it works things don't break for no apparent reason so the default concurrency is pretty good like thousand concurrent and you can extend it and you can ask for a quota increase and the billing becomes better we talked about how billing affects things because when i have millisecond billing i think less about hey this takes a second point one so that sucks now i have to pay for two seconds no it's i'm just paying for the work done and extending the limits like in terms of ram in terms of cpu i'm not really really constraining what i can do but still sometimes if i'm doing something that should be really really distributed i should keep it at like once one worker is one cpu let's keep it simple so the limits are actually much higher what we probably need and it runs python great um and of course the whole this cold start and warm stuff but i realized like things stay warm so generally if you're just using them over time it stays worn really nicely and it just works um so there's a lot to it and that's the point like going back to the meme at the start of the conversation that's kind of like that was the point where like okay i'm just gonna write my own with lambdas and pandas and then i realized like then i had this down a moment okay but still like orchestration is hard like during clusters it's hard i mean it's it seems like a big task like maybe i'll like underestimate it completely and then i don't want to have lock-in and i want to make it agnostic how do i achieve all of this and the truth is and this is where i employed the kind of solution that i bring from time to time and it's not exactly what you'd expect um but this hero that i'm talking of is actually redis and it might be surprising because reddish is okay it's a database it's for caching it's whatever but how does it help us so one thing that i realized over time and then i use this pattern over and over is this is a great tool for actually doing an orchestration on the cheap because it has all these constructs that make it so easy to do this kind of like distributed orchestration so of course it has metadata and caching of course it does atomic contrast so if you have to count three tries and so on you have these guarantees of atomicity which are really really important if you have a distributed system so radix is really popular for q systems like when there's uh there's a redis list which acts like a queue and then workers come in and they take work in their own you know in in their own pace and it has sets meaning if unlike a list if i'm a worker and i have a slice or a kefir cache and i see that work for it i can try to grab that specific work and this is an optimization that i actually put inside funnel rocket um and it that is just make it so easier because it's this kind of removal is atomic and it's guaranteed so we don't have to work as hard on all these orchestration needs and it just works it really is simple to set up it's quite cheap there's a lot of manage options and it's just you know um i don't have a lot of bad things to say about it um it's like this monosodium glutamate over everything in your system uh so we have all the ingredients we talked about lambdas pandas parquet okay um redis what's what's the order of events and i'm i'm getting to you know how it actually works so we're letting spark still create data sets um of n 4k files and we're saving them in cloud storage so that's the end i mean in terms of what spark is doing it just puts the stuff in cloud storage which is really fast and tells the final rocket api i have this api service hey here is this data set that exists and it gets registered in redis so this is the preparation step and when i need to query any customer query done in our web ui becomes a json format query and an api server the same api server is called to now one display and what the the api server does is it invokes a whole bunch of lambda functions asynchronously and all the communication like the delete rights the results the status of things everything goes through lettuce and this process is really you know the project is open source i encourage you to look and see it's it's not that hard to create orchestration nowadays but it's not because i'm super smart it's because you have to be moderately smart but mostly you're standing on the shoulders of giants and i cannot overestimate how much work has been done um to make it easier for us so really i mean i was amazed by how well it worked and again not because my card is so awesome when i was doing things that are so complex or novel but just gluing things in the right way over these constructs that we have can be immensely powerful so when you think about reinventing the wheel i'm not reinventing the wheel i'm just gluing the parts in this in a different way and i'm getting a really powerful solution and this is just in terms of like to tie it up this is the diagram of on one end you have data preparations coming in from spark into per care format in cloud storage we register the data sets uh via the api server and the analysts so any kind of the end user when they run queries it becomes a json query it gets to hit the same api server and something that i didn't mention is serverless is just one option to run it i really try to make it agnostic so if you want to launch thousand containers on spot instances and do some kind of batch queries at scale during the night it's also an option because the idea is to make everything agnostic and and the biggest piece that makes it agnostic is that all the communication flows through ladies so you really don't care who you're talking to like back and forth um i i don't i think we're pretty much almost out of time here um i i just want to say um of course there's a lot of syntax which is interesting in its own i don't want to get into it you can define funnels you can define very complex conditions at the user level and just top it off it's an open source project there's a guide showing you how to download an open data set that is public from toggle and then teaches you like um in phases how to apply queries how to prepare the data how to do funnels how to do aggregations so i don't need to spend a lot of time on that and really the crux of my conversation i think wasn't hey this tool does things um in specifically x or y by but how do you get to this place where you develop your own query engine and how feasible is it and what tools do we have at our disposal this this is actually what i i wanted to convey above all um and i think with that i do have some numbers here i can maybe share the presentation later i'm not sure uh there are some interesting numbers i'll kind of skip them um and yeah that's the end of the presentation at least and i urge you to if you're interested i have two blog posts on medium you can check them out and read about the context why all clusters must die and what is funnel rocket and it goes more in depth so that's it and i guess i'll sp i'll see you all in the special chat if you have any questions i'll see you there in a few moments you