SFSpark.org: Let's talk SMACK: Meet the SMACK Stack Experts Panel Discussion

Recording: SFSpark.org: Let's talk SMACK: Meet the SMACK Stack Experts Panel Discussion

rather than introduce these fine gentlemen I'm going to have them introduce themselves and tell us a little bit about what they're doing in the smack stack area and if you guys can just explain what role your product plays in the smack stack tell us a little bit about that that'd be great yeah whoo yes I'm Chris frankly I when the smack stack was created I think I was the S the spark I words yes I was working for data bricks the creators of spark and since then I actually created something called the pancake stack that was a mockery of this Mac stack and I was just in I think I was in London a couple weeks ago and someone recognized me as yeah the pancake guy so I wasn't sure how to take that yeah he invited me back to Amsterdam for a smoke and a pancake which I guess is a common phrase that I didn't realize which was kind of creepy but it sounded fun so yeah pancake stack is essentially a P as presto just something you guys should be looking at what else a is airflow I have like three days of back and like pancakes AK what's the other ones of course Cassandra T is tensorflow right which we're doing quite a bit of and I have to say I've been using kubernetes quite a bit so with that I'm going to hand it over to you Mike everyone my name is view and chef like cassava I work a confluent we're company packing Apache cockpit and Kafka is designed to handle it's a streaming platform designed to handle very very large-scale streaming data so it's able to handle actually a couple companies well over a trillion messages per day I think even nearing possibly 2 trillion per day at a couple and so handles sort of accepting all that data we handle integrating with other systems over instant in with other systems into stack and then also doing some transformation on that I'm Scott Livingston I work at Credit Karma I don't actually represent any of those pieces but I'm here on behalf of vodka I work on the recommender system at Credit Karma and we use akka in production and have been for somewhere around two years now where we run a cluster that is responsible for actually evaluating all of the data science models that we run when we're trying to give predictions about things for our users yeah hopefully this is pretty fun hi I'm Russell Spitzer I'm an engineer at data stacks and we're one of the Cassandra companies and I spend most of my time working on the integration between Cassandra and sparks so if you're using Cassandra and you're using spark you probably used my code at some point as the Cassandra representative I guess I should tell you Cassandra is a distributed database that does extremely large customer clusters for extremely large customers and is in use by pretty much all of the the big computer companies that are out there that don't have their own proprietary special magic database that they're using hi I'm Neil Conway I'm an engineer at mesosphere so I work mostly on mezzos so in that kind of platform diagram Mays is the lowest level so it's basically pulling together all your cluster resources and making it easy for application schedulers like the different high level parts stack to launch workloads and something else we do is we you know work with partners and and community people to take the software that is you know can run on mazes and package it up into repositories it makes it very easy to install and very easy to run those services and I think that's kind of interesting aspect of the way these services get run in practice in modern data centers is that there's kind of a mismatch between the way this offer is shipped and the way that you need to run it at scale like running Cassandra on 10,000 nodes he is kind of a fundamentally different undertaking than running Cassandra on a 5 or 7 or cluster and so that's one of the things we're looking at is trying to say okay what are the missing attractions that you need to add at the system level at the infrastructure level to make it easier to let's say you know upgrade a 10,000 or sandra cluster operator 25-minute to central cluster in a way that you know it's very hard to solve if you're just the person writing cassandra or just a person writing compliment so as part of that we're building a new sdk for building distributed service on top of DC us and up with mezzos which i think a lot of this stuff is actually built when you're running on top of DCs you're running it through an sdk we've been working on what do I do with this hello oh this works this is so strange okay so I've had some people ask me questions I don't know the answers to and it'd be interesting to compare the meso sand kubernetes answers to this I it's the docks I've seen tell me that if I have racks of physical servers that I could install kubernetes or mace O's and a portion and schedule the rest of the smack stack or whatever you have to run different loads and different machines and there's lots of strategies for deciding how that's done efficiently and what failover looks like but it's if you don't want to run your own hardware and your say on some giant site like AWS or whatever you have it would be nice if you could just get spot instances when you know you're going to get a heavy load let's say at the end of the month you've got 10x below you have in the middle of the month and this happens every month it's predictable so what's involved or is it simple is it easy or do you need some integration between same Essos or kubernetes and this vendor such that you can get some short-term pre-allocate some short-term resources for the impending crush and once that's over let it go because you want to pay for it is that already baked in is that hard to do where we at with that great so I guess I can start answering that question because I guess it's mostly Amazo specific so I think so it's a good question and I think the aspects of that are built in and aspects of that you're going to have to configure to your particular workload your particular environment so the basic notion of looking at your workload and looking at your computer resources and deciding which parts of your workload should run on which parts of your resources that's kind of core to make those handy kubernetes right then if your set of compute resources is going to be dynamic so you're going to bring these spotlights it's online then it doesn't fundamental like from kubernetes perspective mesas directive you have more resources your workload is the same or it's changed rip you're still mapping that workload to the current set of resources I think part of where something like DCOs comes in or some of the kind of more integrated stacks around kubernetes let's say is making that more plug-and-play so out of the box with mezzos it would kind of be up to you to you know do the tooling to you know hook into the cloud API to spin up more resources or to decide which parts of your workload to run on which cloud resources part of a return do with these us is try to give you a more and an experience where it can be very easy to you know point and click and say okay I have an on-premise data center or I have a you know cloud you know collection of cloud resources in one datacenter and I want to add some more resources here in a transient way and to make that kind of a more integrative and experience that's not like fundamentally changing Bezos that's still something there's those can do out of the box but to do it right now you know there's kind of some degree of scripting involved partly because mezzos tries to be very flexible right so like we you don't or philosophy behind meiosis hasn't been to build into mezzos like when the you know system looks like it's busy spin up extra resources mazes is just a very kind of low level of generic scheduler and then you'd want to build the scripting on top of that to respond to changes in your workload changes in your set of computer resources but yeah like I said with these 2's we're looking to make that more plug-and-play more seamless for kind of common use cases like spawn instances like transient workload spikes and so forth uh yeah I mean I got the thing again moving your workloads around automatically such that you have locality of data if you need five times I don't let's say the compute resource for 24 hours I mean these loads in the in this particular case a use case I'm thinking of its predictable you know exactly how much you're going to need and when you're going to need it so we don't it's much simpler than say unknown spiky stuff when some great orange buffoon tweets something and you have things going off right in this case you should be able to plan for that but I'm concerned about locality of data I want to be able to push processes aside such that the ones that need locality of data are co-located in the same rack there should be some awareness of rack and I guess my part of my question is let's say what's your API look like and how is it integrated let's let's take AWS for example or talk about any other vendor I'm happy to hear about other vendors in terms of integration because you you need awareness of what rack am i running on not give me a resource but where is the resource what kind of resource is it really is there anything in the API for that that supports so just to kind of go back to the first row the question I mean I think maybe I made a ton a little too difficult like it's not rocket science so if you're running your workloads in marathon you want to scale dynamically you know first you've hit some kind of provisioning API on abs just spin up more resources then you increase the number of M of Marathon instances that you want to be wrap those you know that's kind of like going to be to API goals exactly when you trigger those API calls what metrics you look at which instances of which apps you increase and so on like that's going to be applications of it but it so isn't rocket science in terms of data locality so most of the components in the stack have some notion of data locality some degree of record where no status and awareness something we're actually looking at doing right now is plumbing that it amazes itself so there's kind of a common vocabulary throughout the stack of talking about where resources are located there the zone and the region they're located in but yes certainly each of those pieces of the stack would have some notion of data locality but I you again after a bit of work to kind of get all the pieces on the same page in terms of you know getting everything running local like kubernetes stuff so the two questions I heard and I think they're separate questions are spot instant support and then auto scaling right like scaling up going down is that okay so and then thirdly federated clusters as well to sew across on-premise across like all the different cloud providers the ability to target specific right like clouds during specific times of the day things like that so like kubernetes and so are like fundamentally this is a docker container Orchestrator right that's like the most fundamental thing you can think of there's this tool called cops right like Kop s that has pretty much become the main way to spin up like kubernetes clusters it I think now supports not just Amazon but also really Google Sports Azure things like that so and then as of kubernetes 1:7 there's now full Federation support there's still one single master control plane so there's still single point of failure if that master control point goes down but at least now you can schedule across all the different early cloud providers but with the cops there's the notion so this is a separate community tool that works with kubernetes api that can not only auto scale the docker containers themselves across all the different federated clusters but can also do auto scaling at the actual load level so the ec2 instance the GCE instance the gk e node pools I think they're called and that's also so yeah so smart instances rightly that's an Amazon specific concept anyway right it's a generic concept but that's what you call them within Amazon basically give me some more compute tomorrow at 3 o'clock yeah okay so like down to that lovely like yeah so for the spot instances similar to right like data bricks I'll give a plug for my old company there you specify the spot price that you're interested in and if you get them you get them if you don't that's fine you could have partial spot instances partial on-demand specifically so actually scheduling more at a certain time actually I've never had a use case or I've never implemented that that use case for some reason typically it's any business that has yeah end of month stuff yeah as this problem it's a common problem I assure you although yeah I mean I haven't I guess I haven't done it I mean I guess with like data breaks for example you have these data bricks jobs I mean yeah you would just cron right like it's basically just a cron job within a notebook you know kind of thing but not only are you specifying the job to run but you're specifying the size of the cluster right so and which clusters to go to so I guess I like the short answer is all this stuff is either built kubernetes one seven is peak which is not out yet its its beta I think its beta 1 beta 2 but supports the Federation oh that's the big one but yeah the other stuff is all locality down to the rack level oh and so for for the locality yes all of kubernetes is very label friendly tag friendly so you can label a know GPU you can label note so you can also figure out which regions they're in which data centers they're in the rack level stuff I yes obviously data center you can label something to like figure out where which rack it is I think the best you can do in cloud would just be the same the same easier imagine that say 25 days out of the month you can get by with you know just some very small instances but five days of the month you need three racks and so now if you do is you say when the day comes up I want three racks I'm going to shut down I'm going to move over those other little fiddly instances into the racks until I'm done and then when I'm finished I'll release the racks and just go back to the no lesser but you need some integration between so may so super Nettie's with it the service provider in order for that to happen because the two api's need to have that level of granularity and I just haven't maybe haven't spent enough time digging in the api's but I sure hope that that exists ok so I think I can get us to a conclusion pretty pretty closely here so what you're talking about you absolutely can do it's you're right that there is a little bit of you know integration work to be done but you know as being pointed out it's basically a couple API calls so the what you're talking about basically in AWS what you do is you tag the instances with that if they're minimum the AV and less you're using one of the weird cluster types that you can like figure out whether they're actually on the same rack or not you actually absolutely have to do this and the reason I know about this is because when you went in when you integrate mezzos and Kafka so copy you're never going to do this you don't have scale up scale down like that generally almost nobody has that type of workload for their data but you do want to make sure that you have a rack aware setup where you make sure that replicas are on different racks and in AWS case it means different AG's and so there's definitely support and the integration that we have between D cos and Kafka absolutely can handle that you basically can tag the you type the instances when they're at being added to methods and then it can schedule appropriately to make sure that you get spread across multiple ATS and you can do the same thing for any other application does that pretty Kafka Thank You Nettie's also like you have those labels and when you go to schedule you have an anti affinity right like type of label where you say make sure this one's not on the same yes one right like one thing to point out too about the data locality because yes I write like see this all the time for the types of workloads that my company now pipeline IO does you know it's a lot of machine learning a lot of AI write like compute intensive right so and this was something I picked up at at like my old company Netflix before data bricks which is the first time I'd ever seen no HDFS and all s3 right so all of this data was being right like remotely pulled in and but it's because of the type of workload which was very compute intensive right that that was dominating not the actual you know data transfer so yeah we were able to kind of skirt around data locality it's not always the case any other questions so far back there hi I was wondering if you can go through an example from start to finish of one use case that uses the smack stack and how they used it for each part of this max back doing the dating site the spark a spark after dark yeah that was my idea I was using it for you know just for like my own little sandbox environment at like deta bricks for example I was not allowed to use any other tools except spark so I basically had make spark a database I had to make spark do everything and so of course that you know doesn't work so on the side I had to kind of hide from my my old bosses and do talks on other things beside spark but so it was spark streaming was taking in the ratings so we would show people you know pictures of like celebrities you know just what just like random things and they would rate those tuples you know the user ID and the rating or it was the user ID that who was doing the rating it was the item id which was the actor actress and it was the rating you know from 1 to 5 so that was going across kafka was being picked up by sparks streaming cough get integration it was then being written to Cassandra right away so pulling it off of Kafka's in spark streaming using the spark cassandra connector and then writing it out and so that was and then on the yet side the person when if you click refresh it would then show you recommendations based on oh and then yeah then that also kicked off a spark ml is just doing ALS behind the scenes right alternatingly squares are like collaborative filtering and then every you know like a minute or something it was generating your recommendations to into rebus and then pulling it and showing the user so and I think we forced in akka somewhere Khafre like you weren't there but we had to I think it was Ryan yeah that did it but yeah I showed up somewhere yeah I guess just to just add to that a little bit and it's basically using Kafka for event sourcing so you're recording raw events in Kafka and then you want to push it into other systems too for processing and for querying so you push it from from let's say it looked into Cassandra because then you get you know efficient query or certain types of querying you put it into spark for sort of analysis so you're going to get it loaded well I guess I guess into spark via either the copy integration I don't remember what we what it was back in a day but the you're getting into spark and then that's sort of for massive analysis and then actually akka is if you don't really have to for socket and what you're going to want to use akka for is probably something where you have you want to process the actual streaming data so rather than taking the raw events maybe you want to do something you want to get aggregates you want to get analytics for like online analytics for your site something like that you can handle that with akka can you comment on like what customers are using your technology for like some interesting use cases I know there's some people in the crowd using some of these technologies I don't know if someone wants to volunteer like what they're doing like today besides dating after darker so I'm not going to name specific names although there's some some quite a few out there especially the big Silicon Valley tech companies that obviously obviously talk about this a lot copy is sort of I mean you sort of use it for everything so it's a little hard to pin down to a specific application but I will cover a couple of different different options so you might be using it for analytics so this is actually probably one of the original use cases is just capturing raw data so then you can type it downstream to be to be analyzed possibly jumping into something like to dupe or into spark s3 whatever so that definition alone without being adopted across a large range of industries so you know retail retail they're using it to you know track all of their inventory where it's going where they need to put like send more send more products using it in car companies doing IO IOT solutions are basically delivering data from their car into cop death and passing it down door either trying to analyze stuff real-time so I would be again the naka use case or pass it downstream to Cassandra and things like Hadoop for the analysis and spark let's see what other industries and I'm missing I mean it's pretty much along the same lines as that is we see a lot of folks something I've seen more common these days is uh reward systems for you can imagine like coffee shops and things like that where you have a constant incoming stream of events that are you know need to be told for every customer you need to have a persistent record of that and that's why you need Cassandra somewhere in there and you have Kafka because you've got so many things coming in and you have to do kind of live updates to their state so that you have things that make sense for that customer because that they earn ten you know magic points they better see ten magic points when they check on their phone they're going to go back up and be very upset to whoever they just bought what they bought from so we'll see data coming in through Kafka cute going into sparks streaming and then I usually we see that's where we'll have a layer of analysis or something if you're familiar with Cassandra you know that not all query patterns are are possible the way you lay out your data so you'll use that step in spark streaming to move around your data or adjust it so that the data that you need to have available at a quick lookup is ready then you'll write that out to Cassandra and that's basically your constant streaming application that's running that's kind of the back end picture on the front you then have you know your your website which might be based in akka might be basically some other thing and that's where you're getting your live requests which access the data in Cassandra which is going to basically be your store of truth so your customers will never really interact with the backend portion they only interact with the database through your website and only those select pieces of data that you've analyzed score and then the whole setup of course is going to be managed by something like Nexus I don't know if you wanna talk about that you know I have a question for you so there was some questions about kubernetes earlier so I was curious on your take on you know what's different about running these technologies on you know the mesas platform versus just like container izing copter Cassandra or trying to run that like on kubernetes or something else right so I'm not a communities expert I won't claim to kind of you know talk about kubernetes in a detail I mean I do think though that the problem that these systems are the problem that we're trying to solve at mesosphere is a lot broader than just container orchestration that's a requirement for you know buildings applications and delivering the value but there's a lot more to it than just container appreciation so running these large staple services over persistent data integrating container based applications data data analytics applications making giving you experience like you know a lot of people who are running on premise data centers would love to get the experience they're able to get in the public louder they would be able to in the public cloud but operating those data centers is just really painful right now trying to solve that problem is a much bigger problem than just container restriction in the now so I think that's what we're trying to attack it another sphere is kind of that bigger problem of how do we develop distributed systems how do we deploy them how do we operate them in a way that it's portable class environment on premise on cloud and really unlocks the potential they have to you know very broad audience of customers because right now getting this value is really kind of rocket science and you know if you can't accept that a lot of more companies most companies all companies have to be software companies it does need to get fun infinitely easier to do stuff and to get value out of you know that opportunity questions from the audience anyone back here asked about use cases and you have a gentleman on the panel from Credit Karma can we hear what credit camera are doing absolutely so we don't use the entire smack stack I'll just get that other way now we do use spark we do use Kafka and we do use akka unfortunately we don't use Cassandra as much as I would like to and unfortunately we don't use Moses as much as I would like to but you know that's how business goes so what we do at Credit Karma I'll talk about akka first because that's what I'm supposed to do so we have a data science team that comes up with a bunch of predictive models for a whole bunch of different offers if you've ever been on the site before the way that we make money I just want to preface it by this is that there's no subscription or anything like that it actually is free but when you go to the site you'll see offers for things like credit cards personal loans a whole bunch of different financial services and next to those a lot of times you'll see things like approval odds those approval odds are calculated in an aqua cluster as well as the some some different models that try and determine what your likelihood to interact with that is given your likelihood to interact with it what's the likelihood that you actually go through the approval process and so on and so you actually get approved is the only way that we get money is if you get approved so it's in our best interest to kind of make sure that we're showing things users the right thing we use akka in production for the computation of our data science models we have somewhere around a thousand of them at this point and at any given request on the site you're likely to be evaluating your request is likely to trigger somewhere around 100 to 150 models that gets spread up across a 400 core cluster and then returned back to you and somewhere around 60 70 milliseconds at the 99th percentile doing that with raka was actually super super super easy because it has its clustered set up so that's from the akka side again if anybody is interested in me of this please come find me after from the spark side we actually use that to train all of our models and the important part when you're training is the data which a vast majority of it comes from Kaka so we have a whole bunch of data being adjusted as users are clicking on things taking offers browsing around the site so we have all kinds of tracking data that's coming through I don't actually remember what the rate is at this point but I want to say it's somewhere around 700 K events per second something along there which for some people is going to be a lot for some people it's not going to be that much and so we process all of that in other akka systems that are actually written in akka stream and from there we do all kinds of cool aggregations and spark and then use that data on the akka side very similar to actually what you were saying just without the Cassandra part unfortunately convincing out I know could camera you can derivation how do you manage your infrastructure you to know use the measures or open areas yeah so for for a long time the a lot of it was just done on bare metal nowadays as we're building out more and more services where all of them are docker eyes dat this point we manually maintain the docker servers that we have and I don't want to get too far into it because it's a pretty big rabbit hole and not necessarily relevant I don't recommend it to anyone but but again if you're interested come find me and I will explain more okay so another question to relate to the persistent storage so I'm just wondering another scale how to deploy the for example Kafka into a lot of boss of container but how to solve the persistent volume stuff I mean for example you need to position all of the stuff of the data into position volumes so how to solve the conflict I mean in one box we can only in a while volume to the container eg in large-scale how to solve the position bottom stuff I in terms of darker volume yeah I talked about him yeah oh okay I I can't say that that's something that we use at least not for actual real persistence things like config files or something like that maybe but Oh offer any of our databases are our Kafka instances now so am I understand the question correctly you're asking basically how do you deal with mounting volume 0 or after container so that then Kafka's storage can get restored into a new container and I just see I mean how to resolve I think the the short answer to this is you should just especially for running on mezzos you should just use the DCOs integration for Costco which takes care of this for you and you don't have to think about it this sort of slightly longer answer is that I mean meses basically has a way to get allow for the state and does it do the mounting for you so even if you were managing it sort of a little bit more manually in in mezzos I mean at some point you actually do run into a problem where like a node dies or something and you can't start it on that node in Kafka since things are already replicated that's actually ok if you are forced to you can bring up the container somewhere else without any state and it'll end up replicating the state from the other nodes that are still online so it's really just about preferring to be to be able to remount that data but it's actually not critical that you're able to and so actually before there was good integration and a good good framework good mezzos framework for doing this the way that people would do it is just run it effectively understand like marathon and you just sort of suffer with the fact that sometimes you end up having to replicate data but the better solution now is that there's a natural schedule that handles it for you okay thank you so I carry on in this question so sorry so the another question is that the in production and just wondering how many Mason's cluster do I need to spin out for example or so if I have a very big environment so probably I need to spin up more than one a Mason's class and how do I need how do I manage these two separate or several separate mesas cluster so the question is how about pointing destroyed running multiple measures clusters error right so missus itself is very scalable there are people running north of 30 or 40 thousand nodes and one mazes cluster that being said you don't necessarily want to do that because you know if you you know that does expose you the risk that if there is let's say some kind of bug in the mazes master that can harm the availability of that whole cluster which pretty meaningful slice of resources so a lot of people will end up dividing that larger pool of machines into a smaller collection of mazes clusters should depending on your workload and kind of your risk tolerance then once you have multiple mazes clusters then making those work together is kind of becomes a little more challenging than if they're one unified mazes cluster that's where Federation and those kinds of multi cluster multi data center features come in which is something probably mazes low behind kubernetes at the moment but it's something working on is Federation support and improving multi cluster operations okay thank you got a aqua question so we're looking at some high availability patterns you know to invoke caps theorem you know you can sort of choose between like the CH model you have to dislike some kind of master master type setup have like something hot and some nodes on standby you know you something like st-pierre zookeeper for that we're also looking at aqua cluster for more like a either CP ish or a pH type system where you know you can we can afford some sort of partition tolerance some partition in the network what are some patterns that you've seen using akka for building highly available systems on engine and do you see some folks using leaning more towards aqua clustered toolkit or and in case what you do see them like what are the use cases for that or n do you see people using more like the ch-ch model where people you know kind like marathon right is high availability to achieve using zookeeper with marathon and like an old Java ecosystem a lot of stuff is made H a through zookeeper what are some of the patterns and that you see fitting um that's a good question uh let's see so I'll talk a little bit about how our cluster works since we are using akka cluster the some of the things that it provides that are really really really nice are I was talking to somebody earlier how actually sitting right next to about how we went from running replicated machines that were not part of a cluster to running in a cluster in about an hour which is awesome that plus the ability to very easily auto scale that cluster up or just scale it regardless it doesn't have to be auto to a point where we actually didn't need any in any extra systems like consular zookeeper or something like that so that has worked really well for us we have other pieces of our infrastructure that are written more I guess the Twitter stack is one of the things that we use and and that is more on the using console for service discovery or zookeeper whatever you happen to use instead of letting the akka cluster handle that kind of stuff for you both of them have their advantages and disadvantages what I can say is that using the aqua cluster was extremely easy and it required only application level support no infrastructure level support whereas using something like zookeeper is going to also require infrastructure level support which is costly and might not actually be effective for what you're trying to work on yeah hopefully that answers your question I have a question around Kafka and like let's say I'm running active active copper cluster in two different regions do it like an easy way for me to maintain like consumer state if it has to failover to start consuming from the other region assuming like the data is like completely like replicated so when you say active active mode do you mean you actually have two separate clusters is not like stress trip yes so - so for active active in that case you actually have to have each topic basically owned by one cluster so I'm assuming that's the the setup feature sorry not like owned in defense that like there's you only produce to that topic in one cluster unless you're in the process of failing over right because active active wouldn't you can't you would just get replication cycles if you if you didn't do that so if that's the mode that you're talking about then you there isn't today and easy way to failover we did some work we actually built a confluent a product to help with replication we have some additional tools and for failing over we're sort of in the process of improving that the fundamental problems are give you sort of the nitty-gritty technical details so a fundamental problem is that the way that replication works across two clusters there's no mechanism in Kafka to say start replicating and I preserve the offset from the sorta cluster to the decimation cluster what your option today is basically when you do failover you can do timestamp based query and so you can we don't provide the tooling today but you can basically do that query in the new data center so like where do you want what timestamp you want a failover to and then just initialize all the consumer offsets by seeking to them so today you'd have to do that by hand but honestly it's like three or four lines of code it's not that complicated so a bit of a broad question here Scala or Java right I'm not going to touch that one I don't know if anybody else wants to things to say I'm a big Scala advocate if you're getting into SPARC right now Scala is going to be considerably easier until the Java eight support is a lot better right now you spend a lot of time if you're writing your application if you're writing it in Java passing around class types getting class types knowing what you know type parameters your map function adds in your output has things are a lot better if you're in the data frames datasets world but then you still have to managing coders and if I'm saying a lot of things right now that you're like I don't want to deal with that Scala's going to do it all for you well Java will not even Python is kind of better at that because you know Python doesn't really care what type anything is so it just handles it on the fly which is great but also not great so I'll say Scala but you know the JVM is great JVM is a great piece of machinery it's got a lot of great tooling around it you really can't go wrong doing Java or Scala based on what you feel comfortable with but if you're about to start your first spark application and you don't feel confident in Scala it might be worth taking a look at it just because it might save you a lot of pain down the line that's my my bet yeah okay so I sort of it depends on the type of application you're writing but one thing that I will know is that Kafka was originally written in Scala that actually creates a huge number of problems because we have client libraries and we needed people to be able to use them from Java as well which is terrible so now all the clients have been rewritten in Java and actually a substantial part of the of the broker is now getting shifted over to Java just sort of implicitly because we share code between the clients in the broker one of the reasons that was possible is because there are basically two ways that you can write Scala code you can write Scala code that basically looks like Java but takes advantage of a few of the niceties that Scala provides or you can write code that nobody else can read so or only crazy a functional programming language nerds so so I would say actually it probably doesn't matter that much if you jump to Scala but it's more about how you use use of Scala then necessarily making that decision that said I also can't stand developing itself as a compile time is way too slow so it really hurts your hydration iteration speed yes so similar to Kafka SPARC is actually right I mean it's still very much Scala heavy right like the full code base itself but parts of it and you know so things like project tungsten things like the dynamic code generation when it's going from a logical plan and it's dynamically generating source code that's Java source code and that gets compiled into bytecode and then cached that's that was the second part of project tungsten some of the earlier things that went from Scala to Java were I think was the nettie implementation the communication to be masters and workers and things like that funny enough akka was actually I think removed in spark to oh I don't know the specifics but the yeah and yeah there was something to I think about the random ports that would open and over firewalls that was yeah I was really yeah and I think the way that it was being used within the SPARC codebase I think just simple REST API is were but most yeah so like my point is most of like the really low level optimizations that are happening with these things and not sure this happen with Kafka as well to had to get written in Java right because of things like Nettie and you know sue the other libraries that they were relying on which were Java based I spend a lot of time going back and forth between Scala and Java which is why I cringe at this question you yeah Mike you're like one of the first people you came down to Netflix was one of the first times I it was back in the early days when like Netflix was looking at Scala yeah yeah right exactly Scala courses cow all right yes I always found you funny with the meetups cuz I like you I think you'd be ending a consulting gig and be like I'm available and it was like the perfect audience because they're all but and then oh yeah and so trying to go back and forth between Scala and Java is huge pain just because of the functional stuff and I do a lot of Python now too because of we're just like working with a lot of data scientists and so trying to do PI spark and figure all that stuff out but actually kind of like Python from a quick and dirty but I still get burned by the not typing are the new types yeah it drives me nuts so I mean I love Python and a lot of what we're talking about though the whole tungsten catalyst all of this stuff is available from Python you get all of those optimizations that were written at the low-level code all that happens automatically for me that the big thing is still just if I if I if I'm writing spark code and I think if you go online you can see Holden's done a ton of talks on this but you never really want to have a Python lambda in your big data ever because there's just a huge amount of penalty from serializing out of your JVM objects into a Python object and then of course when you're going back into spark again you have to serialize it back into a JVM object and you can imagine that that's very expensive so use data sets so Aaron at the beginning mentioned the two-tier scheduler with DCOs I was wondering if you can elaborate a little bit more about what that means and does in the relationships and marathon as well sure so the concept of like a teacher scheduler is trying to create a separation concerns so amazes itself does the low-level resource management so it manages actually launching the containers deciding which resources are available on which nodes and it implements a very kind of an initial notion of which cluster users should get which resources but it doesn't try to decide which workload should run where so the actual decision of given the resources in the cluster how to make use of those resources what workloads to run that's run on an application level scheduler so that's going to be you know a schedule for Kafka a schedule for Cassandra or what-have-you and the other thing so we initially called those these of software schedulers and certainly they're doing scheduling and then just to clarify so the reason that you want to have applications to the schedulers is that it would be very hard to write an optimal social or an intelligent scheduler for all possible workloads for your cluster and to bake that into mezzos so mazes tries to take this approach of being very scalable very simple very minimalistic kind of like you know the Linux kernel I mean it's not the most minimal system microkernel would be more minimal and maybe this kind of takes that kind of a microkernel approach where you try to avoid putting policy in mezzos you try to put the application Civic logic in the schedulers so that gives the opportunity then to write a really good concentric scheduler a really good caustic scheduler that really understands the properties of Cassandra Burke loads or the properties of Kafka workloads it's written by someone who is a copy expert Accenture expert would that happen to bake that all into into mezzos but I think the other thing those things are does that those schedules are doing and that's why I think the term scheduler is low misleading is that they part of what they're trying to do is trying to take these prepackaged software services like Cassandra and turn them into intelligent services so to make it so that you can deploy Kafka you know or Cassandra or each of these services in a very straightforward way scale them in a very straightforward way have you know fire to interface to understanding which applications using which resources they're kind of like we call them kind of service back planes where they're sitting between the server software like let's say Cassandra and then your infrastructure API is like AWS or GCP you guys they kind of fit the middle name mediate and they say okay if there's more workload if there's more work required you can move more more workload to a free server or if you want to upgrade one of these service schedulers that kind of back plane or the scheduler provides a convenient way to do that so to try to make running these services like fundamentally more easier and more self managing piggybacking on the scheduler question so Mesa site is a two-phase scheduler so there's also a shared state schedulers which sort of invoke a different sort of concurrency model well what are the trade-offs between to and when is it appropriate to use one versus the other now and I think you know there was a paper bites on the Google guys I don't know if Hinman was on it maybe some of us I think some of the data breaks got but so much we're doing like internships at Google you know who are writing about like Borg and Omega kubernetes and all those things you know some of the ideas they're eventually baked into may sauce but as far as like the different scheduler options out there like how do they kind of compare and like when you use one over the other night and I'm sure there's there might be some deep academic reasons for it for that there are any papers you could cite sure I mean so right so there's a bunch of different approaches of scheduling and I think you're also certain to see those approaches converge in some ways so just a kind of rough overview mazes is two level so you have a kind of a generic introductory level scheduler and then service level schedulers and then it's a pessimistic which means that when it offers resources to one framework those resources can't be used by anybody else until that framework has decided either use them or to decline them and there's a you know a bunch more you know so you could either be monolithic or two level you can be optimistic or pessimistic there are a bunch of dimensions and you know different schedules are playing in different parts of that space but I think they're also converging where some of the advantages of or their different trade-offs doing those models so one thing we're looking at it demeans us for example is adding optimistic capabilities to mazes so the idea there is that we would essentially kind of offer all the cluster resources to all the frameworks to all the consumers and then they could try to use those and then as they submit requests to using to use those resources some of the records would be turned down if there's a conflict or two thelonious the same thing so we can certainly go into one where'd you go maybe kind of the details we can talk about offline but I think that's kind of the high level is that there's it's a very rich space and there's a lot of options but in terms of what people need in practice I think it's kind of converging where you know you need certain kind of requirements in terms of being able to guarantee resources to certain clients to ensure that the rest of the reason you know try to balance culture utilization between maximizing utilization wealth and maximizing fairness so that people don't get starved and then if you want to meet those requirements you kind of do it with a couple different approaches and as I said the kind of popular which I think are kind of boring from each other so hi thanks for the great Smike overview I wonder how do you connect this in general to machine learning ai and of high which is happening right it looks like somatic is a good way to run machine learning pipelines and great karma I think is a case in point but I wonder other any patterns for m2m data pipelines in production where you run industrials great algorithms which are not Python yeah so the whole like company pipeline IO was built around this right so yes I appreciate the set up that was nice yeah thanks Lexi yeah the whole end and so basically getting data in doing you know if you think of your future transformations for the machine learning algorithms as just kind of straight-up ETL right they're just like different types of transformations so yeah for that we're using tesander using Kafka and then there's now ways to get tensor flow for example to talk to HDFS which is really handy and yeah so tensor flow just kind of like brainstorming here but yeah tensor flow kinda has their own file format it's called TF record which would I guess kind of be the equivalent of like a sequence file if you guys are you know familiar with some of those early on or like an avro or something like that so but yeah I mean smack is so it's the it's a platform level kind of thing so yeah with like AI and M L either just use cases on top so I believe people write like uber and you know yeah they're all using forms of smack and things like that right so yeah for all of their ml I guess to generic patterns I can think of off the top of my head is you have your your your time we're at the end of the day you run some kind of process where you take a huge amount of data you train your models you store those models and then load them into a streaming processing part of your your job which takes that last day's model and uses it to do new predictions based on when a user comes in another thing that actually I've got three options okay another one is where you have a batch job at night this is I wish there's someone here at Netflix that could correct me if I'm wrong on this but we're I think at night you take the entire set of data you make all of your predictions for the following day store those in Cassandra or some kind of other durable store because people don't really care on that kind of application whether they got an instant update to what their recommendation is and then you can also have machine learning algorithms that are constantly learning as new data is streaming in and that's where you need a specialized kind of algorithm we kind of baked into the streaming layer and then as the data comes in your model is both evaluating and training based on whatever outcomes you get and you just basically are constantly updating as you go through I don't think that the ecosystem is really mature enough to say any of these are easy out-of-the-box solutions they all take a lot of kind of researching what you're doing what your model is and my one big where I went with all things ml is you need to really know ahead of time what you're trying to do with with your machine learning before you start doing it very easy to convince yourself of things that are not true if you just start throwing data at math so while that it's true I would also say that the stack also the stock also makes it easy to - easy to experiment with this stuff so yeah there is some setup involved but the pattern that I see most in fact I don't see I haven't really seen anybody in practice doing online learning it's really hard you also don't really have any chance if you do stuff online you'll really have any chance to validate your models which makes it a little riskier to use so the pattern that I've seen most frequently is batch job on spark or something like that to do the to generate the model and then you apply it in line using whatever stream processing framework oftentimes based off of events directly in directly in Kafka is the one obviously I see a lot of and so you can apply it in line but one of the nice things is the stack also if you for example passage of data your raw data through top to first you have or you train multiple models in spark give an opportunity to basically just farm this out so what the pub/sub nature of your streaming data actually becomes really important there because it offers you the opportunity to try a lot of things in parallel right so almost from like one more thing too about tensorflow that i've been seeing more and more of them it's not tensorflow specific but at sort of larger scale you're like constantly training and you can spin up a bunch of like servers you have a huge training cluster that's pulling in batches off Kafka or off somewhere and then typically you're like checkpointing you know the the weights or the parameters the variables that you're learning and when you checkpoint you you then want to have some sort of sensor or some some sort of weights that then kick off a validation cluster you know like a smaller cluster so every you know hour something like that when you're checkpointing every half hour or whatever you then want to at the same time that you continue on the training side also spit up a validation on that current checkpoint and if the leg model value if the elect not all validation starts to show overfitting or other weird signs you want to propagate back and stop the training so that's a big pattern I've been seeing more and more out in the tensorflow world the online training stuff works better if you have linear models you know again I'm just kind of spewing things out here the was there a thing about online training if you've ever used Python scikit-learn so there's a method called partial fit right that a lot of people don't know about there's there's the traditional fit which expects all the data and you know on a single node because like it learns not very distributed but there's there's this sort of second method on specifically linear models in e regression logistic regression where you can continuously train and the name is partial fit and you're you know kind of partially passing in data and with these neural networks gets back to like tensorflow in neural networks like they're sort of always ready to keep being trains right because when you take checkpoints when you save off the current values they're meant to be restored and then training continues so you can sort of always think of neural weapon right yes these neural networks as having the ability for like continuous training but it's not always the best thing of course but yeah awesome any last questions as we're lowering down in time we're a little hotels are a service company that uses everything except Tomatoes can't afford it unfortunately but we're sort of in that middle territory we can't really use anything else because we need to process something like five to ten thousand messages second at times and we don't you know we can't really go full enterprise etcetera and and obviously it's very understandable that an immense amount of work goes into building each of these platforms and the companies that do that need to make money I was wondering if each of you or at least those who are are representing your respective companies can talk about whether it's your personal opinion or or any life on how the open source community is how it will be supported in the future and whether there are any kind of larger trends in what features will be supported in the next year to 3 years something like that that come to mind and kind of what what we as a company can do to sort of prepare for any changes it might be coming in the future support the needs that we have so we start saving money thank you definitely save your money all right so is that for all the yeah I don't question anyone who feels inclined to speak to that yeah the cost savings thing always comes up I we have this feature within pipeline IO for this auto traffic shifting as you know so there's auto scaling of course rescale up but think of you lose your spot instances within Amazon you're doing you know X number of predictions per second at some price and now you can shift over like maybe Google is cheaper at that moment so the thing with cost savings and you know this comes from I've been talking to a lot of VCS and and founders recently and yeah they all say the same thing right now don't talk about cost savings no one cares it's not so I'm personally a you know pretty frugal person so you know I'm a founder so I have to keep an eye on all the like Amazon bills and things like that so cost settings to me is important but I guess that you know for like people like Netflix you know they've got like millions of dollars per month I think the other big trend is continuous like this term continuous or obviously Kafka is huge you see SPARC using that term continuous you know continuous applications is the term coming out of data bricks a lot also one negative trend I think is a write like closed source closed source right you see data bricks the next few days you know very public about sort of bashing them forgoing the closed-source route I mean you have to do it you know again because if you cease you have to make money but a lot of the new things that you're going to see at the spark summit that are announced are going to be data break specific right we saw this with data stacks we see I think the burst verse first time I saw this with data stacks was when you were treating your Cassandra cluster as an analytics source right as a bulk read versus the traditional rent like transactional fast rights and the way Netflix did it you know we had separate clusters and and then we had gone away from separate clusters and we went straight to the SS tables and this was all Netflix created custom code you know you would create you go straight to the Cassandra SS tables you would do your ETL that way or your bulk reads that way so that you weren't going through the Cassandra front door and I think there was something similar for SPARC and then it was supposed to be open source then it wasn't open source it's closer so you know it's just natural business stuff but yeah yeah so I mean I guess one thing to point out is that infrastructure software basically today you can't go like close source for a bunch of stuff it's just you'll lose if you do that like regardless of your existing position of the market so in a lot of ways I think you should feel okay about the future because I think it's most of the important stuff is going to remain open source it's sort of interesting to hear about data bricks because I think it's a unusual choice I do work for a company that makes proprietary stuff around Kafka that said a very large fraction of our platform is also open source and generally you know the way that I'd say you know it's obviously on the product side products it's a product by product decision and but I do think of it as the stuff that we're going to charge for is the stuff that big enterprises are going to want to pay for anyway and so the stuff that's open source we're sort of like open source by default I would even say yeah but I think just by the nature of the trend in infrastructure software everything that you see up here I think you'll be in a a good position moving forward nobody's going to the sort of open source product because honestly a lot of we're customers well in from is from smaller smaller you know applications and then eventually grow and so you need to hook those people when they're only using the open source stuff the other thing that I'll point out is that think hard about where your costs actually are going if you end up using the open source version and operating it yourself you may be spending a lot more money than if you just go with a one of the hosted solutions and of course I have to done plug consequent cloud which is we just announced recently so yeah the other thing there is I'd say hosted versions it sort of like anti-lock in because you're still hosting out something you could eventually move to a little bit source so like open source version manage yourself if you wanted to but it you know oftentimes will be a very high cost savings over having to pay like another ops person I guess one one trend I'm seeing a lot is and then this is definitely from from the data stacks perspective it's one of those things where it's really important to have open API so that this locking doesn't happen at your application level but I would expect to see a lot more kind of enterprise differentiation coming in these products because I mean if you take a look at all the big cloud providers so definitely moving towards trying to provide their own hosted versions of all of these components and when that happens the questioners will they back these projects and the companies that are currently investing a lot of money I know want to make sure that they have a place in the market that they have a place to continue doing their work so I do think that you will definitely be able to interact with any of these components in their open source versions in the future but I think will probably be more differentiation between the the enterprise version and the non for a lot of component yeah well so the open-source side actually the the project got picked up by some folks at IBM who who took it and you know the question is where does it go from there will they continue putting money into it but at some level they wanted to make at data sets you know you want to you want to have a company that makes money that sells things that people use yeah that's kind of how that worked out I mean the the project's there for anyone to pick up I mean the big the big underlying question all this is who are you going to pay to write this software and if no one's paying anyone to write this software it's someone's hobby project and at that point they could always stop so just something to think about sure all of that makes perfect sense and I think we're a one company probably among many who have been extremely successful with with this patent and we're still in the process of growing until we're in that kind of awkward territory being ready to pay for the product as soon as possible as soon as we you know keep growing and and things are going very well so far but but obviously a little bit nervous about the foundation that we've built at this time so all that's very helpful thank you so maybe just add my two cents is not a question guys if we're a company that kind of landed on the other side of a I guess a large company kind of thing hey we're going to go enterprise and we're not going to really be too participated in the community anymore and I guess it really brings to mind a saying that you hear a lot of people talk about when it comes to open source it's you're going to pay for it in some way one is going to either come down to money or is it going to come down to time right and so once you've kind of got that cost equation you then look at what are the other benefits of of open sources you know yes you can get other people to run it and you leverage open api's you can potentially leverage you know many eyes looking at the source code as opposed to just a single set but then at the same time it means that you're not potentially focusing on your core competencies as well as a company right so for every company it's going to be a trade off for how important that particular technology is and we've actually seen a lot of companies where they've made a decision whereby you know if some kind of peripheral services they're like look we're just going to pay someone to do it because it's just it's far easier but for other things have said actually this is a core of what we do even though hey we're not here to be data based experts we're not here to be kind of you know infrastructure management experts it's so critical to what we do we need to build expertise in this that's why they've gone down the open-source path smack stack doesn't sort of emphasize which is unfortunate but is sort of monitoring and metrics collection all of these tools right like all of us up here get questions all the time how do we tune Kafka how do we tune spark how do we you know I like to like DCOs things like that and right like my first question is always what what kind of work but what sort of metrics are you even trying to optimize like what what do you have set up right like how do I know if I do you know give you five magical parameters that they're helping your specific workload and all that kind of stuff so yes we tend to use Prometheus just because it works nicely with kubernetes and things like that but spark and graphite work really well I think for like Kafka it's like JMX is weak well it gets exposed as Jen I Spy default but it's actually pluggable so you can pretty easily integrate it with whatever you want so I think to a couple Kafka talks recently and they were debugging talks you know which other the best ones it's yeah either you guys like your solution architects or people using Kafka and they just talk about these super obscure problems that there's no way that they could have debug down their own just from an outsider like you really really needed the JMX metrics and some central place to see the changes for Cassandra open sources they're not liquid there's like data stacks Enterprise obviously has a lot of the yeah yeah gotcha and then for like DCL yeah producers we kind of exposed when two metrics endpoints and I know DCOs so mazey suppose metros endpoints dcs integrates with is a data dog leader who with a bunch all right I'll take a question the sales guy in the room now yeah so we do have I mean I guess what I heard right is that most organizations that I go into have their own set of tools right they already do they already do monitoring they already have a way of doing it so all we're trying to do is make that simple right so a single way of getting logs out of the system a single way of getting metrics for all the things that we just talked about right so metrics at the infrastructure level at the you know Orchestrator level at the service level with some of these technologies and you at the application level like providing endpoints for you to publish metrics to and then pull into your tool of choice so Prometheus is one there's lots of open-source tools there's lots of proprietary tools that you can use that customers already have in the data center right so you [Music]