BDSBTB 2015: Martin Zapletal, Large Volume Data Analysis on the Typesafe Reactive Platform
thank you everyone for coming my name is Martin I work for cake solutions in New York where consultancy we do consultancy around Scala akkas Park Cassandra Kafka and many other technologies in the distributed systems Big Data machine learning world as you've probably noticed that the importance of data analytics is increasing in recent years it's also the reason why we are at this conference but current state in many companies is that they're using destructive updates so they don't keep history of the data they use analytics tools with poor scalability and integration it's off the manual process it's slow iterations so it's not really suitable for large amounts of data and fast data but we often need to process large volumes of data really quickly often streaming manner one of the options is to utilize parallel computing there are many different programming models for parallel computing another level is distributed computing where we have multiple machines working together trying to reach a common goal but distributed computing is often misused it introduces many new complexities that may not be necessary for your problem including having to worry about concurrency time order message delivery network latency and so on even large companies like Baidu or Google I reportedly use GPU optimized algorithm still to run their most computation expensive queries and as you can see single threaded algorithms are often better performing than distributed systems but on the other hand distributed systems have many positive impacts including availability resilient resilience geographical distribution ability to run on cheap hardware scalability or just the ability to scale to date a large data set just wouldn't fit on a single machine distributed systems are largely about trade-offs that must be carefully considered even with all those aforementioned complexity some distributed systems have amazing performance a good example is actually spark it won the Daytona greatsword 100 thera-band terabyte benchmark or distributed databases when developing distributed application we really need to optimize focus on two levels of optimization so we need to optimize on on single machine so things like algorithms register locality case locality in memory or disk storage layouts data formats garbage collection optimization and so on and at the same time and optimizations and distributed level so things like data locality communication pattern optimization failure handling network traffic and so on also data processing pipeline does not consist from a single algorithm that we can optimize for GPU it has to cover the whole lifecycle of data so from loading the data from from external sources running some some transformations on the data training machine learning models running those analytics and data mining machine learning algorithms and finally storing the data somewhere again in a database so we need many technologies working together I will talk mostly today about data processing but there are also data stores integration and messaging tools distributed computing primitives cluster manager stuff schedulers deployment configuration management and and DevOps are increasingly important in the distributed systems area on there and we also need data analytics and machine learning libraries that work well without this distributed systems and in each of those categories we have many different different technologies many different tools each with very different use cases very different features attributes and different level of maturity so we really need to pick the best tools that suit us for our machine learning pipeline traditionally the data pipeline used to include a monolithic application and often a relational database with full acid properties representing mutable state this is very nice for the programmer because because it hides many of the complexities such as concurrency or shared memory from the developer but they are still there they're in the database but they're just handled for us this does not scale very well with larger throughput or larger data amounts or unstructured data so sometimes we need to pull those complexities out of the database and handle them ourselves for those reasons there were many other architectural patterns developed during the years the first one is c QR s where we have the read and write side scalability and technology separated so the right side is usually right optimized database often an append only lock with a very fast right and the read side is a denormalized and pre computed database that is very fast to answer reads for some some expected queries another typical architecture is batch pipeline where we have a large-scale database such as HDFS and then many technologies that help us mind the data and query that unstructured large-scale database alarmed architecture is another option where we add the stream layer to fix some of the issues of batch pipeline which is handling or answering some queries in near real-time at least partially and the newest one is so-called cap architecture or streaming data architecture that stores data in something fast that supports streaming data the stream is done an immutable source of truth as well as source of data for other systems the data can be replied query transport of analyzed and reached it's all done in streaming manner and it's integrated with other data the databases each of them can have different technology different data model and and different use cases the types of technologies I will talk about today fit very well into these architecture types such as streaming data and event processing but there's there's no one-size-fits-all architecture or technology the common attributes are that all the data are stored there are multiple technologies in the pipeline for different use cases sometimes there's duplication the normalization but what we really need to do is to identify that the places in our pipeline we need to identify the features we're expecting the guarantees attributes and we need to pick the best technologies that fit into those places in the pipeline we built an internal demonstration project open source project called mover which also which processes streaming sensor data from devices such as accelerometer and award smartphone in a pocket or heartbeat monitor to classify exercises in a GM in near real time it uses those types of technologies I will talk about about this the pattern seems to be common for many use cases in practice as well so the architecture is very similar to what was mentioned previously the blue box processes streaming data so we have the mobile device and then acha acha cluster or shardee' our characters are processing the streaming data we have acha streams within that distributed domain and we store the data in Cassandra database and in Kafka we use cassandra also as an event source journal for actors persistence and then the pink box is apache spark that we use that for more complex analytic such as suggestions to improvements or to users exercise routine track tracking compliance to exercise plan on training our machine learning models we can also train machine learning models in the in the pink box and we can send it back to the stream stream liar serialized so the mobile device can immediately start using the new model the new improved machine learning model at the same time we need to fulfill a slice and attributes such as responsiveness elasticity durability and the resilience so we need all those technologies to be distributed and have those properties so I'll talk about acha acha stream spark and focus on some of the important features they have with regards to a large volume data processing the first option one option to implement parallel algorithm is using futures we have all the good things that Scala and the functional programming provides us such as monadic composition or type system but still we must ensure we were handling memory and synchronization properly properly accessing shared state from multiple threads obviously means that they can interleave in more than one possible order so the result will be incorrect acha now is an active framework for truly concurrent and distributed systems one of the biggest features it provides it has threadsafe mutable state which serves as a consistency boundary so everything that goes through that one actor is single threaded so we have we have consistency within just within that one actor it's perfect for domain modeling in distributed environment are perfect for distributed state just distributing work to workers to actors and for implementation of certain communication patterns the programming model is very simple the actor can essentially just send messages create new actors and change behavior but it's often using properly so people try to use it as object oriented programming but it doesn't work like that a typical example of an actor this actor is persistent so it's backed by a event source journal it starts in a note registered state receives account command then it becomes registered it stores that that command in or the event in the events or journal so it can retrieve that state and then it becomes registered and answers two different set of commands it also uses acha distributed data for application to replicate its state with other replicas in the cluster and share that that's dead knowledge another typical use case is this so we have this actor which serves as a source for acha streams so every event or every every message received by this actor is turned into a an event in streams and then we can define the flow on that owned on that stream of events okays very powerful is most powerful when it runs on a cluster of machines it provides very powerful features such as persistent so back in an actor by an event source journal sharding which means horizontal partitioning or application where we have the same entity replicated multiple times in the cluster and then obviously we need to synchronize them but distributed systems like this are affected by the very nature of this tributary environment and then the cap theorem so they are often trade-offs that are required to be we need to consider them such as consistency and versus availability persistence failures delivery guarantees data locality performance concurrent updates and conflicts during replication divergence and go global invariants in the system one example is time and order in distributed system so in sequential program the order of events there's one total order of there's always the same so every time we run the program the result is all is exactly the same but we don't have that guarantee generally in distributed system global clock in total order are not possible to have in distributed system so we're often aiming for partial order and just just causality or vector clock in distributed system the livery guarantees generally are different in distributed systems most distributed systems only provide at most once delivery guarantee which means messages may be lost so you need to ensure the message is delivered you can achieve at least once delivery guarantee but that means the message may be received multiple times it may be duplicated exactly once the delivery is theoretically not possible to achieve there are some some operation semantics that help us to achieve what we need such as idempotent which works very well with at least once delivery failure detection and the recovery is very interesting problem in distributed systems because it's very difficult to distinguish between a node failure network failure and just high latency or the node not responding because of high amount of work or just doing garbage collection and components in distributed system fail independently they can fail anytime knows Network everything even large companies like Microsoft or Google have failures quite commonly in their data centers so this is from report from Kyle Kingsbury and peter baelish the crucial is how to handle those failures in distributed systems and but as you can see this is again from Kyle Kingsbury this block even mature technology such as MongoDB alreadies ryuk elasticsearch MySQL even don't handle those failures properly very often so they often lose acknowledged rights during which they return right should have never occurred and so on um acha is a very low level technology so we often have to handle those things ourselves manually another technology in the types of ecosystem is called akka streams it represents a very different programming model we have a publisher subscriber and stream of events between them and we apply some transformations to that stream so for instance as you can see on this marble diagram I can transform the stream to square and then filter based on a color it provides higher level API that can be used to define graphs reusable components even with cycles it's very convenient nice way out to declaratively express graphs so the code on the top it's a representation of neural network computation and as you can see it very nicely corresponds to the picture in the bottom so it's a very nice way how to express graphs but we need to make sure the streaming logic is correct otherwise we can have incorrect results or even dead logs and other other other issues especially with cycles in stream processing usually components can either have bounded buffer which may result in drug messages when the buffer is full or unbounded buffer which may result in out of memory exceptions so I customs provide us a way how to handle back pressure innaka for instance we would have to do it ourselves we would have to manage the back pressure somehow ourselves you can define different back pressure strategies the graph on the Left shows the slowing down subscriber with a bounded buffer so it starts dropping messages at some point and the picture on the right is exactly the same scenario but this time we use back pressure so the publisher and subscriber they synchronize their speeds our typical use case on transformation on streaming data are time series so in this instance we re running for your transformation on a window to identify frequencies and distinguish between periods of exercise when the user is exercising and when the user is not exercising Oh another typical use case is transformation on time series so we're turning the time series into a different representation of the same time series so we're using dimensionality reduction and feature like feature extraction we're essentially trying to represent the time series with less features and with better probabilistic distribution so we better with the classifier in this case this is symbolic aggregate approximation but the ultimate goal really is to run time series through a train model and classify what what the user did what what exercise it was so in this case we have XYZ axis of accelerometer data and we want the SVM or on deep neural network in our case to tell us it was seven dumbbell art and alternating curls and the last technology I'll talk about this Apaches part it's an in-memory data flow distributed data processing framer it hands both dreaming and batch computation it distributes computation using your higher level a level API load it balances the load so it runs the the computation in the cluster most computation to data is fall to around has very rich programming model including so the the the main abstraction is called resilient distributed data set it's fall to around provides mean for means for caching serialization has two types of operations transformations which are lazy form the directed acyclic graph and actions that actually executed computational graph has some some distributed computing primitives such as accumulators or broadcast very variables has very good integration with other tools as very powerful streaming and has also machine learning and graph processing libraries but again there are many many of those tools have some subtle important impacts that we really need to understand properly for instance accumulators or side effects when when running apache spark computation when there's a failure the computation part of the computation needs to be rerun so it may it means that the debt side effects can be applied multiple times therefore is a resulting in something that you're not expecting if even if spark grants exactly wants delivery for the result some computations are very simple to paralyze so in this case this is trivially parallelizable so we just spoke just run some computations in different threads or even on different machines and then we just collect the result but it's not always the case some some some algorithms are very difficult to paralyse using any any programming model so in spark we can use this higher level API we define operation but it's important to understand the different operations have different different semantics and different performance characteristics so map for instance applies some some transformation some function to each element in each partition which is which can be very efficient on the other hand reduced by period requires so-called shuffle operation so it shuffles the data between nodes it requires a lot of network traffic which is usually very very inefficient spark optimizer is called catalyst it has multiple phases so it optimizes the query that you have written it parses the query to get logical plan that optimizes the logical plan creates multiple physical plans and uses cost-based optimizer to pick the best physical plan then generates code using scarce classic codes and only dead code is shipped and executed it uses techniques known from relational database optimizations such as removing sub queries evaluating static expressions simplifying filters pushing filters through projections removing redundant columns in the query merging projections and so on at the same time it often chooses the best physical execution strategy to utilize knowledge of the underlying data store so it tries to run the computation as close to the data as possible to avoid network traffic currently those also ongoing work on explicit memory management cache of our computations code generation optimization GP utilization and adaptive planning our data phrase the data frames ap is as much I mentioned this morning it provides more optimization than the strongly type art it is because user-defined functions are difficult to optimize but Apache fling for instance can do many of the optimization with type data because they're using bytecode inspection so there's there's really a lot going on behind the scenes when you're running apache spark queries but in the end it's still programmers job to implement the algorithm properly using the proper algorithm data structures just using those primitives and the API that spark provides the typical use case for spark in our mover project is to try in machine learning models so we split the data into training and testing data set we run some pre-processing so a normalization feature extraction dimensionality reduction we train the machine learning learning model we use the other data set to test it to optimize the parameters of the machine learning model we pick the best model and then we can start using it we can use ml labour or other existing compatible machine learning library spark also provides a nice way to express machine learning pipelines using the pipelining API so in this case we create the pipeline's it's ordered stages we said the normalizer feature extractor and then we use linear regression and then we use this pipeline to train the machine learning model and the same pipeline to process the data and actually use it also in this case you're trying to predict users our next exercise intensity using just linear regression another use case we have is to identify similarity between time series to find similar users or label data automatically so in in conclusion there are many technologies available for processing of large data volumes the type save technologies are very strong in this area but what we need to do is we need to identify features attributes guarantees and use cases in our pipeline and we need to pick the best tools that fit those places in the pipeline and second choosing the best tool for our use case is not enough there are many specific internals and many specific features that may not work as we expect so it's really important to learn those technologies very deeply and to understand them to be able to implement our programs with with sufficient quality and often also correctness and third the selection of environment is very important the environment can be synchronous asynchronous or even distributed if we have use cases it acquires it each requires very different way of thinking theoretical knowledge tools and n approaches thank you very much in any questions so the question is if there's any framework to run a cluster on mesas I don't think there's any framework but it's definitely possible to run our cluster to deploy a cluster using using missiles okay thank you