SBTB 2019: Bowen Li, Apache Flink 2.0: Unified Enterprise Data Processing System and Beyond
[Music] hi my name is Bonnie I'm a chief linka meter and the senior engineer at Alibaba and today I'm so glad to be here to talk about weather we have been working on our chief Lincoln stream processing so here's our agenda today we will go over some use cases seeing Alibaba and other companies we will do some recap on the core concept or philosophies and characteristics of links to see what makes flink so successful in stream processing world and we will then take a look at where we are being working on beyond stream processing okay Alibaba Group has the largest e-commerce company and the ecosystem in the world we are with regard to GM where we are about twice as Amazon annually we have like consumer-facing website like alex rezze tall Baltimore be too big website like alibaba.com we have the largest logistic network in the world called China we have the largest public cloud in East Asia Alibaba cloud and the largest private financial institution and final financial link has been really widely adopted within Alibaba it supports all the business units real-time computation as well as all the offline and online search and the recommendations we also provide flink as a public service on Alibaba cloud called real-time compute just to give you some sense of how big of a Baba is so among the 365 days of the year there is a single day that's matters the most called singles to global shopping festival on November 11 and as you can see from the charts from 2016 the GMB all Alibaba still ecosystems of the single day has surpassed Black Friday shopping season across the United States and has been suprising errors we just had a really successful single-stage shopping festival last week so here's some really hot and fresh data on the single day on the first 21 second we made 114 million dollar sales on the first 96 second we made 1.4 billion dollar sales and all the single day the total sales is 38 billion u.s. dollars and generates about like 1,000 petabytes of data in 24 hours and at the peak of that traffic we support over 500,000 transactions per second Flinx support all this real-time computations really smoothly without any downtime at the peak traffic fling handles about 2 billion events and about 3 terabytes of data per second which is the more than doubles from 2018 I will just go over a couple of linked use cases within our company the first day is online machine learning so when our user interact with our app or website their events generated from their behaviors cough it hits our search service or recommendation service and then the data flows into our online machine learning service got ETL sampled then we do real time training and model validation testing and deploy it to score and scoring service and then feed back to the search and recommendation so all that happens in real time the service can handle hundreds of millions events per second with over 100 billion features the end-to-end latency is within second and all the training and model update happens in real time the second use case is real-time GMB dashboard so on a single day shopping festival Alibaba will publish a globally available dashboard that anyone can access and you can query like you can see how much is the GM way from the beginning of that day to that very second so this is like a pretty simple architecture of how we do that so basically all the business units will submit to their transactions into some local some central hub and we use flink to do exactly once processing on those transactions within sub-second latency outputs the result the GM way into each place and have the dashboard query it lots of companies are leveraging flink to do a fantastic stuff for example Netflix and Xiaomi are also leveraging flink for real-time AI all those red sharing apps are using flink to do dynamic pricing for example on Weber F whenever you type a destination then you'll get to meet you immediately get back an estimated price rate that takes considers into the distance all the drivers available all the consumers that want to repair the cap all this complicated computation are backed by flink in real time financial institutions are leveraging flink for fraud detection risk management consumer facing companies are using flink for real-time analytics and as flink become the de facto standard for stream processing in the industry it got offered by others public cloud players as as public services for example in the United States Kinesis analytics backed by flink next I want to help us all together we have what's flink is actually is a belt and what flink is so unique wise has been so successful in stream processing world here's a typical the typical pipeline you would build whistling you basically have applications or devices generating events input it into some streams or the data has been sitting in the distributed file system you have link tourism process know me in real time and then emits the results to the dance room so what is fling fling is first is about stateful computations where does state or butene state matters in stream processing world first when you do come especially complex stream processing you always want to process a record according to a contacts rate you want to look at some historical behavior and then decide what to do where is this record rather than just processing it so in order to look up into the contacts you have to have this contact sitting somewhere for you to query that is where the state comes into play and of course you want this state to be as local as possible to have this data locality and you can still the expensive network i/o and asking a much lower latency and last you want full control of the state right you don't want to be it's like completely external with all the control because of a Polock failures you want to be able to roll back in order to support the exactly one semantics fortunately flink supports building state back hands out of shelf and our state back has support HR betray data structures for example marathon lists or nested data structures in a foley for tolerant way we provide you stay back on the first is in memory is billable backends so it's used for like a faster you memory queries but if the state just got really large it can be spilled to disk and we also provide off core rocks DB state back end for terabytes of state data nymph link is we flink applies the stateful computations to event streams and event streams actually means a few things here and I want to work you walk you step by step the first is according to Flinx philosophy all your data is data streams so when we talk about batch chemistry processing we are actually just talking about some mechanism the fundamental difference under the hood is actually your data so when you see batch processing you'll usually have the other data at your hand and you know how large the data is where it is ray so in this case it's actually just bonded through data streams on the other hand will do stream processing the data just keep flowing in you don't know when the data will end so in that case it would be a hub unbounded data streams and that's that we believe is the key difference between bash and stream processing and thus technically all data processing is stream processing second different from most of other open source technologies flink is built with this streaming first architecture with pipeline execution so in pipeline execution record just flowing in and through the system without any delays that gives us the benefits of extremely high throughput and ultra-low latency this is super different from batch first system will staged execution where the data flows into our system but execute here the impacts in stages and you cannot proceed to knock the stage until the first stage complete all our data processing that cost extra latency and the advantage of this pipeline execution cannot become gained with so-called mini batch mechanism last one Allah lists event coming with time loss Hall open source technologies the Ural a just did you early does support the processing time well we would argue that the event time actually matters more in most case because even the time is when you then actually happens and usually it has timestamp embedded into the event JSON array and processing time is just when the event comes into our system and got processed so early it doesn't really mean anything and when you want to do logic with times you look for some advanced the semantics for example window aggregation like sliding window tumbling window you want to do session ization like you want to do a session window on that and you want to be able to join streams based on some time the processor should also be able to handle a lot of other data and lay the data fortunately if link support all this comprehensive data similar time cymatics from the very beginning flink is expressive so we provide users with this layered api so on the top you can have sick hole or table API if you don't want to program java scala up high zone that gives you a fully self service environment you don't need to understand how flink handles it i just do it just did a magic for you right in the middle it has the native stream processing API where you can read java scala up high so and on the bottom you have the process functions which expose loss of the fundamental building blocks to you like event state times so you can do you can manipulate times and state all by yourself to do very precise control flink is scalable so as I mentioned before in our use cases and other companies use kids like Netflix is normal for flink to handle trillions of Records per day with terabytes of state and it can run thousands of course no problem we pay really lots of attention to make it operationally friendly you can deploy a fling anywhere on kubernetes yarn missiles or standalone I'll computer and no problem you can TV played however you want like a job mode session mode with high availability setups last but not the least flink guarantees the to recover from fault and they can do everything exactly one cymatics when some open source technology claim the exactly once actually they are trying to confuse users with this with some different key differences laying there so there are two kinds of exactly once semantics the first is if they just talk about the exactly once within their system they are actually talking about exactly one state consistency it means each reactor flowing into your system on impact and only impact the state of this logic once and only once so that's what normal weather was normally the exactly once means and if you want to have this external connectors to interact with let's say Kinesis or Kafka is external streams then you are talking about end-to-end exactly once so if links actually support both where well from the very beginning if you use like Kafka transactions you can achieve end to end exactly once you read all you read a record only once from a Kafka stream and you may write to a Kafka stream only once and flink has this on the flightcheck point and CEO point a mechanism that doesn't sacrifice much of this performance so blinking a nutshell let me let us recap this is is stateful computations over event streams in expressive scalable operation focused and exactly once and a fault-tolerant way so as the last part i want you to talk to you about what the community has been working on actively recently in this different field let's look at the unified data processing or batch processing first so why does unified processing matters to companies and teams let's recap this called classic lambda architecture where you usually have two pipelines ray on the bottom you have an offline pipeline that read from green some historical data from a distributed system file system emits some historical the results and then you have this online pipeline where the stream processor read from a master in coop hub subsystem emits this real-time results and then you combine them together to upload to downstream that is this architecture was invented by the author of a storm is because this old style like stream processing engine cannot guarantee correctness all online part thus you have to periodically have this offline parts to correct the results of the online part well that brings lots of headache to the team for example for the infrastructure team you have to operate more than one infrastructure and to monitor and beyond colleges developers how to maintain their code and learn more technologies and from the business perspective you have to you have to code the base and you have to keep them in sync right you change something in the online part then you have to change it on the offline part if you forgot oops you like screwed so wife link as I talked a little bit about Flinx perspective on the bash and the stream and unbounded and the bounded data stream perspective is natural for us to use flank to solve this problem because flink is a streaming first and we believe batch is just a special case of streaming well even though flink is born with this vision before flink 1.8 the stack doesn't really look so so on the left hand you can see before 1.8 we actually have a unified API on the top but when during the execution or the stream or the java scala api layer we actually have separate API is ready have the data stream API and data set api and they will actually generate different execution plan to be run on task managers fortunately in 1.9 we completely change that on the right side you can see we unified the planner and the danced down the stack layer to be a single stack so you have this see : table API then you have a single planner to optimize both stream and batch views kisses then he generates a single unified stream transformation will stream operators to be to running to be able to run on flink tasks here is TBC the benchmarks that we wrong with in Alibaba so this is wrong was flanked against spark so blink is an internal fork with in Alibaba off link and as of one flink one point I blink has officially become a part of link so beyond just batch processing we go like two steps ahead one is we provide this link and a high of integration to tackle the data warehousing use case in 1.9 we launched the u.s. in national integration with some limited support on some limited high versions but in 1.10 which gonna be released at in this December we provide the full integration on read/write and the UDF's are all major popular high versions we also natively integrated link into that link so if you download something right now you can run fling queries sick hose natively was on any problem next I want to spend a little bit more time talk about machine learning ai and deep learning let's go back to the week the lambda architecture for machine learning is similar to the one for data processing where you have this offline training pipeline and online training pipeline the somewhere you combine this on the dynamic model from the online pipeline and the static model from the offline pipeline together deploy it into production to do inferred inference normally people would do nowadays is using flink for this online pipelines pre-processing and then at the end they deploy the model to fling to do a production inference and how we want to solve this problem is as simple as running flink everywhere so you have a single stack and a single code base and the infratry interact with to gain all the benefits how can we do that if we separate this pipeline into stages then we can look at each stage has separate requirements and half link can support all the entree at first you have data acquisition in this part you will require some rich connector support where all the data you can access data however and wherever you want it and also you want to have this method the data management data set management in the pre-processing part you will require stream and batch unification some strong cycle and API support in model training you have enhanced the iteration you would want some out of shelf machinery and Alber arrays and also some deep learning integration techniques in model validation serving you want the framework to handle serving registry for you and also be able to roll out and roll back on demand at the inference phase you want to have the online evaluation and just as a fundamental thing for all this phase it will require some kind of machine learning pipeline mechanism and also strong Python API support and fortunately all the brain part has been supported in flink the brown part we are actively working on and actually most of the company right now like Netflix they just develop these things that myself is not that hard to do then we will look at some of this red part in detail first there is a machine learning library in 1.10 we completely rewrite those machine learning libraries to make it based on machine learning pipeline we shall talk about in a minute and we provide some battle-tested algorithms for users to leverage we introduced a brand new machine learning pipeline this is basically mostly based on a paper so there will be two kinds of operators in machine learning pipeline one is transformer where you ingest the data and output data the second is estimator will ingest the data and output a model so in this case you will say you have this training cell training phase where the day that the table one flows in car trained by an estimator into a model pipeline and then you just deploy the model pipeline to the inference production environment to do whatever you want deep learning pipelines how people really do this is basically the same as the machine learning ones is you use flink for data acquisition and the pre-processing but use tensor flow caster for like model training validation and serving and you have this parameter tuning loop in the middle the structure you really like this you have a fling cluster and when you approve process the data it to some distributed file system or Kafka message queue as external dependent infrastructure and then ingested in your tensor flow cluster to do all the training well it has lots of headaches is that it you have three structures to manage you have this intermediate external dependency and others like lambda headaches you already have in the in the in the previously mentioned slice right so the solution is simple as I mentioned use Flinx to do everything but hawk basically you would do wrong tensorflow we're seeing of link cluster so it will look like this it's less single fling cluster the data from flink got pre-processed with group quality we just flowed directly into the tensor flow operator that living aside from blink operators and it's actually fits perfectly into them machine learning pipeline we proposed right you will see if link as transformer as in the tensor flow operators as an estimator so it's free fits perfectly last but not least service lots of companies are using service and of course like AWS lambda this service idea is basically it's event-driven so some event happens trigger to a lambda and you do something and the emitted to the downstream it has lots of benefits is elastic you can secure up and down with our problem on demand it's pretty lightweight you just wrote the code doesn't need to worry about how to deploy it or anything and the resources consumes is pretty limited well it has lots of challenges to for examples all this on a shelf lambda solution doesn't provide a good state so you have to manage your own external state to worry about the consistencies the consistency the network hi old external storage capacity and it's a little bit hard to build complex logic let's see what if you want to chain two or more lambda e SS lambda functions together right you have to emit the data tools like nieces and then gestating another lambda function so let's summarize what we want for server list its event-driven we want a good management for state and we want to be able to compose some complex logic with server list does that doesn't that selects unfamiliar to everybody its stream processing right so I'm happy to tell you about this new project called state function which is announced at the October this year in fling forward Berlin is upon last week is officially become part of Apache flink project so please go and check it out so what this project is about building service functions on top of our chief link and half link to manage all this state and compensation for you so yeah that's all the content for today thank you very much you can get in touch with us on the mailing list of our Twitter do you have time for questions okay [Applause] [Music] [Applause] [Music] could you comment on comparing that Frank versus the spark and I mean comparing these two frameworks yeah I mean I'll just mention a few ideas but for like full comparison you can search online or I highly recommend to run some products to do some experiment with two frameworks because sometimes there can be like force claim or you know missed misleading content but basically first the spark is born with like batch batch first thing in mind and fling his first is streaming first so there's like fundamental difference on the execution like pipeline versus stage and also with regard to functionalities fling supports all the comprehensive like time semantics and operations where you can't do that in spark even though in spark structure screamingly create this ticketing like they are doing this a long time ago but it has been like sitting there about like two years so I highly recommend to do some experiment yourself try to run some production logic you'll see where it goes yeah me again so you mentioned extremely high throughput and that's the it's you showed some pretty impressive parity results there the the conventional wisdom for years was that you know because spark was a micro batch system okay sorry cuz it C Allah there sorry you mentioned sort of extremely high throughput from blink I guess which is really impressive the conventional wisdom for years was that because spark was a micro batch system for sort of mechanical sympathy reasons and also for kind of like stream fusion compilation reasons it could achieve higher throughput but I guess now that there is a unified like planner compiler in flank now it becomes possible for flink to do the same kind of optimizations right is that is that true or I'm trying to like what's the what's the reason for that for the massive increase in performance that we're seeing yes so the question is wife link like a performance better performance actually better on the backside array so yeah as I mentioned the execution model is fundamentally different and also we add lots of optimizations with round like batch on-screen processing for example vectorization all that so yeah it's just gives us this I guess it's just some advantage from the very beginning yeah it's just you need to like kind of chewing it and the haulage the potential representation my question is about how obviously you believe that flink is kind of a should be used as a universal computational one time across all these different computations but I guess my question is how protocol the system is in terms of what how what's the wide dynamic range of users who can successfully interact with it meaning like you put a data scientist in the Jupiter notebook they start to explore stuff like can that person successfully interact with flink within the alibaba organization or what whites white that swath the builders can successfully work on the system so the question is there any like successful use cases what data scientists to use fleeing notebooks that it's more that was like an example of a type of user who is responsible for building doing ETL and building models I'm not totally focused on the notebook example but just like what can a wide variety of people successfully build on this system or is it kind of a narrow set of data engineers who really get streaming in their core and this kind of advanced computational model I'll try to answer if I try to answer that first the stream processing really has some stretch food for you to intern for example you need to understand some complex logic like state time semantics are that this is a little bit different from batch processing array so you do need some other good understanding and spend some time learning about it but with regard to the use cases and the people whether they are successful I would say so yeah it has been like adopted a very widely across the industry and I recommend you probably look at like fling forward some videos to see how those peoples are leveraging that I would say that's they are pretty successful with link I think we could take one more question or zero that works right [Applause] [Music]