SFSpark.org: Eryn Muetzel, Lets Talk SMACK
Recording: SFSpark.org: Eryn Muetzel, Lets Talk SMACK
all right so I'll be sure to keep this pretty short I want to move on to the real interesting part but just to give an overview you know here at mesosphere what we see I'm going on with our customers and our open source users and a little bit of trends so everybody knows this quote right software is you know all companies today really need to be software businesses we're really facing a fundamental shift in how companies are engaging with their customers so you know if you look at you know the to thousands even still going on today a lot of companies were experimenting and trying to deploy things like Hadoop you know and really build these big big data warehouses and a lot of companies face a lot of challenges of doing this there's a recent Gartner survey that said something like 50 percent of Hadoop deployments actually failed and today you know really successful companies are going beyond big data to what we have been calling fast data so fast data is really using streaming data coming in real time from customers to then provide real-time services to those customers some examples things like autonomous cars connected devices connected home things like that so I always like visual examples this is a photo from Vatican City in 2005 during the announcement of the Pope you can see there's one guy there in the corner he's holding up his flip phone maybe it's making a call to his mom or something so what's not what's interesting is not that photo but how it compares to this photo so this is from 2013 same event and what we see here is not everybody just that they have smartphones but there have these connected devices they have endpoints and they have expectations for services being delivered to them on those endpoints so you know companies today really need to be able to meet the needs of this type of a customer in these types of users so just to give a few examples of some companies kind of leading the charge in terms of these data-driven modern applications one is GE so they're building an industrial IOT platform to optimize asset management and also operations so they're basically doing machine intelligence and data analytics from cloud data center and also to machines that are at the edge howdy is another example so outie is delivering a traffic light information system that actually helps drivers better navigate cities Yelp is is another one a lot of people know what yelps doing but Yelp is you know crowdsourcing reviews of local businesses and actually Yelp runs millions of tests every day and because of May so they're able to push code to production you know very very quickly and you know in hours and minutes instead of days in the last year is 2 Sigma 2 Sigma is a hedge fund that's based in New York and they're using machine learning to basically build new algorithms to predict the future of financial markets so modern enterprise apps like I showed on the previous slide can handle the the large amount of data and a large amount of users because of two things one is containerization and the second is Big Data services and these are made possible by platform services so platform services following to these two areas container orchestration which basically coordinates how containerized micro services get deployed and run on a cluster and then data services which basically capture store process the data that's created by those containerized maker services so in the last few years continuing every day we just see this huge explosion of the number of data stores and analytics engines so you know as a developer as a data engineer you're looking at all these options and it can be overwhelming but it's also exciting because there's so many options for how to build these fast data applications so how do you how do you choose right or how do you know which is most popular this is a visual basically you know a fast data application needs some basic functionality along each of these lines so the first is you know events come in through connected devices whether that's a car a wearable your phone and then you need to ensure that data is captured with no loss and you have to be able to you know take in millions of events per second analytics so you need to be able to you know analyze data to gain new insights whether that's streaming or batch machine learning and then you need to be able to store that data it needs to be highly available so you know data is coming in you're analyzing it quickly you can't lose any of that data it also needs to be scalable to meet the needs of you know having huge numbers of users and then finally you need to be able to present insight to a human or you needs either or trigger a reaction and a connected device or application so the snacked stack is really you know a key set of technologies that has emerged as the most common way to build these fast data applications so I'll just go through each of these quickly at a high level but you're going to learn way more than I could ever teach you in these 30 seconds so the first is Kafka so Kafka is a distributed highly available messaging system is created at LinkedIn and today it's used in something like over a hundred plus web scale companies very popular next is spark so spark is a large-scale data processing engine it does micro batching which makes it very fast and it supports everything's like sequel and machine learning and graph computation next is Cassandra so Cassandra is a distributed highly available database it's an Apache project that was born at Facebook and then akka is a toolkit and runtime to simplify the development of these data-driven apps and then finally meso so meso this is a cluster resource manager it's a platform essentially to run and deploy data services in containers maysa has this concept of a two-level scheduler so Mesa itself is relatively simple with basically our gates resources and then it allows basically each of these applications on top or data services to have specific scheduling logic baked in that actually takes into account you know the capabilities and factors of that application so this is just a summary of you know why you'd want to build and run your streaming and fast data applications with a snack stack so it's essentially an integrated set of data services it allows you to ingest analyze and store streaming data a simple deployment and operations to get your apps to market faster it's highly available because of the meso underpinning so you don't miss any customer interactions and then finally because of Mesa switch essentially pulls the resources in your data center or cloud it allows you to run at very high utilization and save money on cloud and data center costs okay since you're at mesosphere I'm going to give you the five-second overview of mesosphere so you know mesosphere is focus here is really to make running Mesa as simple as possible so we have a open source project called DCOs that essentially allows you to pool and run containers data services and basically allows you to operate or automate the operations of those data services so you can do things like one-click deployment of data services automated upgrades updates with rollback things like monitoring metrics manage persistent storage volumes so essentially we're trying to make Mae so and the data services that run on my toes as easy as possible I'm just a quick snapshot this is what we call the DC US universe it's essentially in a proposed Ettore for packages for things like spark Kafka Cassandra so we have both open source and then partner supported so you know confluent and data stack light Bend for instance all right and then I'll in there so that's it I'll hand it back over to Corbin and your speakers for tonight