data.bythebay.io: Daria Mehra - Analytics as Code with Juttle
Recording: data.bythebay.io: Daria Mehra - Analytics as Code with Juttle
Um all right. So the the topic of my talk today is uh specifically operational analytics. Although most of the things I say will equally apply to BI you know business intelligence analytics and I've been around data for gosh 15 years now uh in data storage appliances and uh storage as service data search and more recently analytics and analytics specifically is kind of a young field and maybe I'm just saying that because I want to feel young but uh it has not matured to the point where uh the processes we follow when building analytics applications and dashboards and visualizations are at the same level of structure and control as is common for other areas of software development. And we've seen this before in pretty much every field within software development. When you start out, everybody is doing little quick one-off jobs. things are kind of done in different ways. Nothing standard. There's end solutions to pick from
You kind of cobble together what works for you. And then uh the maturation happens over time and certain solutions went out are considered uh the way to go. You don't have to reinvent the wheel anymore. Uh that has happened for example with infrastructure. We no longer configure machines one by one with you know little SSH scripts. there is chef and puppet and docker and that's kind of how you do it. It wouldn't occur to someone that to configure hosts, I should go to this UI and click some buttons and then when I want a new host, I'll just do that all over again and in some other custom way. But that seems to be what we do a lot with operational analytics with dashboarding UI solutions and with business analytics as well for what it's worth
Uh I think the future for analytics is that same movement toward handling code as as code because that's what it is. And this is what I'll talk about. There are some good things about having this ready UIs where you you click and charts appear. It's easy to get started. It's easy to get started if you're not a coder. Although that seems irrelevant to the field of operational analytics because people doing this are in fact coders. Um but sorry about that. It's uh it's easy to get uh pre-built charts from integrated monitoring solutions
So, for example, here is a data dog dashboard for elastic search. Well, they already figured out everything to monitor for elastic search. Here's a bunch of charts and um I actually like data dog, so I'll I'll speak on that a bit later. Or here is a customuilt Cubana dashboard. Um guess nothing much has happened for analysis for us lately. So there's not much data. Um it's it was easy to put together. That's that's the good stuff
However, you soon run into problems as you keep developing analytics logic and you end up with millions of dashboards. You know, this is just for one team kind of getting started trying stuff out. There's over a hundred charts. uh a newcomer to this would be like well what are the good charts well I don't know there are seven versions of this chart with essentially the same name which one is the right one maybe the latest one maybe not I don't know so solutions quickly grow to an unmaintainable u size if you need to make changes it's tricky for example let's say you discovered that well somebody went with say kubanas defaults and all the charts show averages then somebody comes in and goes, "Hey, you know, you shouldn't really be looking at averages. You should have peaks in your charts. You have 117 charts to update by hand by clicking through the UI." So, so that's unmaintainable. Uh the solutions are very customuilt. So, here is my dashboard
If somebody says, "Hey, I want a dashboard just like yours, but for this other metric I'm tracking," that's pretty tricky to do. Depending on what you're using for that, chances are you'll be literally looking at the screen, clicking through to see how it was done, which particular buttons were clicked, and then clicking it the same way. Uh there's no copying, there's no, you know, modifying and the resulting solutions are not transparent. There's no way to review that it was done right. Chances are people won't even ask. They're shown a dashboard. There's a pretty chart. chart goes to, you know, some threshold, you go, "Oh, well, you know, we're nearing the threshold
We should raise capacity." Uh, it's not usually a question of, "Well, how did you do this? How did you get at this?" You maybe get asked what was the data feeding into this. You probably will not be asked for what was the the specific logic, what statistic you used. It's implied that well, you know your job, you did this right. We do not make the same assumption for coding. We have code reviews. We find bugs. We find bugs at at review stage. That's a good thing
That's cheapest to fix. We find bugs at later stages. Uh pretty expensive to fix. These bugs, bugs in your analytics can affect a lot more than a bug that maybe turns away a few of your users. If you make a decision based on incorrect uh analytical report, you could be spending a whole lot of money in the wrong direction. So it would be good to get review for these things but since they're hidden behind UIs that would be hard to do. Um so there are different ways of doing this. We can in fact build analytical solutions based on code and make it transparent and you could do the same with code as you can with the UI
You can make a quick and dirty solution. You can in fact make a slow and dirty solution. Having code in the picture is not a panacea, but it gives you an option for building quick and clean if you do it the right way. So that's what we're getting to. Um, specifically what's operational analytics in the context of this talk? Well, we run a service at Quidd. We run a service that lets people analyze data, um, public news, company news, patent information. This is the result of my search for uh data analytics and code. And these are kind of the the clusters of news articles that that came out of this
And as I thought, there's no code in this. There's like um Spark stuff, big data, Google Analytics, predictive analytics. Only like a 4% cluster of this is actually about embedded analytics, which is where we're getting very close to code. this would be codebased solutions but by and large the space is not about that which is why I'm talking about it somebody has to um so when we run this service there's a complex stack behind it with you know spark and kafka and elastic search and a number of other things we want to know that it's up and working fine so in order to to get there we need anomaly detection we need alerting if there's a problem we want to be able to debug with data in hand with metrics and drilling down to logs. We want to know how is performance of various features, how is usage of features. So we want reports looking back at trends. We want to be able to do capacity planning kind of forwardlooking predictions and we also want to know which features make sense to build. What are the users actually doing with this? So that's the business analytics aspect
And we have a pipeline for this data. There's a monitoring stage. We've stored the data. We analyze it. We visualize or alert on it. This is what it looks like for us right now. Um, we have custom app logs and sort of metrics, not quite metrics, they're metrics, but incorporated in log lines as this JSON blobs. Uh, so that's going into log stash and then into elastic search and for now Kibbana for visualizing for operational needs
And then the same data is forked off into an aluma pipeline. They let you hang Python scripts uh that would enrich your data on the way then get it stored in commonly redshift Amazon's cheapest storage tier that you can still query and then the BI team is using Lucer which actually is code based. You write SQL and the dashboards are nicely SQL back to look as a variant on SQL and then we have host metrics and some service metrics say for elastic search and those are currently being collected into New Relic and so we end up with this completely disjoint views. So we would like to see that when a user action of say exporting to PowerPoint happens this is what happens to network traffic and CPU utilization. We can't actually overlay these things because they're entirely distinct dashboards. That's one of the problems we're addressing with a rework of this pipeline. And uh this is just a little background on where the data could possibly be coming from. So logs and metrics at host, service, and application level
If you're doing this yourself, which we pretty much are, then your logs will probably go to log stash and elastic search, at which point you may naturally be drawn toward visualizing them on Kibbana. And I'll show what that looks like. If you're collecting actual metrics, not like our JSON blob type metrics, but real time series data of time stamp, name, value, tags, you will probably be storing those in a time series database like Influx or OpenTSDB or a number of others and then also be kind of naturally propelled towards certain visualization solutions for those or you could not do it yourself and uh pay a service provider to do this for you. There are some choices in metrics and log space. There is not really still a good choice that would give you a unified handling of metrics and logs if if you're trying to, you know, pay somebody to take care of it. So when we've gotten this data collected, we want to visualize it. And this is what happens. So for time series databases, you'll probably use Graphana as your interface, although there are some other choices
For logs, you'll probably use Kibana. If you're using a third-party solution, they have a specific proprietary UI for that. And let's look at Kibbana. So, let me preface this with I don't hate Kibbana. There's nothing wrong with Kibbana when it's used as intended for what it was built for. It's a great UI for taking a look at your logs. It's great for log exploration. What it's not great for is using it as a production strength analytics solution for all your logs and metrics
wasn't meant for that. But a surprising number of people do that. I, you know, go to friends places and hey there is a Kibbana dashboard. I know what that is. And that's their solution for operational metrics. So that's an interesting choice which works for a little while while you're small and your requirements is small, but there's no access control for Kibana UI. So it's um everybody gets readwrite access to the same dashboard. you can very easily step all over somebody's work
Uh there is no math, you know, time series math calculator. There's just some built-in statistics and that's all you get. There's a number of other things that are missing from this. Graphana has some of those um but it doesn't handle logs. It's just metrics. So you you have to make your choices. Uh, Cubana works for some people who like us put metrics inside your logs because subd dog. Um, and this is what it looks like if you try to export to code from Kibana because for example, you say, "Hey, I at least want to back up my charts, put it in Git." Well, this this is just a little bit
It actually keeps going and going and going and going because it's this terrible escape JSON stuff. And the URLs are similarly the URL actually takes up a whole page if I pasted it in um because it has all the parameters in it. This is not the kind of code that I'm talking about as a UI though it's great you know here's my chart that's that's a nice chart easy but unmaintainable. So in my view, analytics is actually code whether you know it or not. It can be code hidden behind UIs, code written by someone else or it can be exposed to you. So you can do revision control on it. You can do code reviews. You can organize your visualizations, dashboards into directories that make sense, not a flat structure
You can uh dry it up and actually reuse code, which would be great. You can maybe even run tests on it. This is still a open question in my mind how your analytical logic should be tested. Kind of working on that one in my head. And there are people who do analytics as code but kind of take a completely do-it-yourselfer approach to it with say writing custom spark uh jobs uh that would then place the data in SQL and then you have your SQL queries that then feed into say a node app that's visualizing in high charts or you know pandas taking it into D3 and those are very complex pipelines to manage and your business logic of your analytics gets split across at least three stages with actually different languages and probably different people who know how to do it. So when you want to make a change in this, you go, oh, you know, we want to track this metric differently. We're going to reorganize the schema. Suddenly it's like, you know, how many engineers does it take to screw in this particular light bulb because nobody can do it by themselves
So what are the possible unified solutions to this? I've been kind of excited about Jupiter, you know, Apache's notebook solution, but it really natively integrates only with Spark and everything else is kind of mix and match stuff and it's a notebook. Uh they talk about dashboarding solutions that might be coming down the line but not there yet. And then there's Jettle which is something we did at at my prior project and it's in the open source. This is what Jettle looks like. It unifies the reading data from the source in this case from elastic search doing transformations on it your you know percentile sum averages whatever and custom reducers there's a lot of power and flexibility to the language so you can define your own computations on the data and the visualizations library is also integrated so it's as simple as this read reduce view on a time chart done um for what it's worth Here's a dashboard done in Jettle. So this is kind of a complicated recreation of another dashboard that we have in Kibbana. So the code for it looks equally complex and I don't have time in this talk to go through it. Plus this whole bit is not dried up at all because our metric schema is not perfect yet
So it should really be a 40line uh piece of code powering that entire dashboard. So, Jettle can read from many different sources, time series databases and elastic search and SQL and, you know, HTTP endpoints and files and whatever have you. It runs as a NodeJS process. You can join multiple data sources and process them together. Uh, it does handle alerting. Instead of sending the output to a visualization, you can send it to Slack or um Pager Duty. So I encourage you guys to check it out and see what you think of of the open source project. Um as a disclaimer the company behind it is no longer in business
So I work for quidd and we are using this for some specific cases. Uh Jettle as a language as a solution was in production for over a year. So this is pretty battle tested but the open source packaging is newer than that. So there are some rough edges around it and it would be great to get this community maintained because it's a great tool. Uh we won't be looking at that just in the interest of time. Uh what I want to cover is what if you don't want to do it yourself and learn a new language and kind of code up your analytics from the ground up. What are the better choices that still give you the option to treat your analytics as code? Data Dog is one that I found to be better than than the others because you get the UI, you get the visual builder, but you also export the definitions of charts and dashboards to JSON that's pretty readable with this little kind of DSL language here. And um unlike Cubanas export, this is something that you can put up for code review
You can actually make out the logic behind what what you're doing here. and you can pretty easily copy and modify and that whole code reuse part works. You do need to send your data to data dog. There's no on-prem solution unless you're huge and then maybe they'll talk to you about the custom build thing. Um, and there is no handling of logs, but they do handle events and overlay events on on metric uh time series charts. So, this is something that we're exploring and it's it's looking pretty good as an option. Other players that I've looked at do not provide anything similar to this. There's no export to code
It's just point-and-click UIs. So, you know, data dog right now is is a a contender. And so, this is what our pipeline for logs and metrics would look like if we went with data dog. it would replace New Relic for us and handle the custom metrics and events from our applications through through log stash um either through log stash or directly by posting to their API and cover the the host and service level metrics as well. We would still keep the the log stash bit that forks off to the business analytics line because that's working really well and makes sense. We may still keep Kibana for just having ro logs for debugging since data dog is not a logging solution and we'll get at least some uh ability to track uh code based analytics. It's not gentle, but it's it's a reasonable thing. And just in closing, what what the future holds for us? Well, this kind of blast from the past future
Over a year ago, the maker of Graphana made the statement that at some point in the coming one to three years, log analytics and metrics analytics and visualization will be solved by the same piece of software. Well, it's not. We're a year and a half into that prediction. I'm still waiting for that piece of software that will do everything the way I want to. Meanwhile, you can try jettle or you can tell me about what solutions you guys are using. I'm always on the lookout for what's out there. Thank you. Yes
Hi. Um, what is your thought on Zeppelin, Apache Drill and Tara? Because they also try to address this from a different uh perspective. For example, Apache drill can go against all kind of data sources, your uh Hadoop, NoSQL, relational SQL, JSON, RK file formats. So creating like a abstraction for all your SQL related queries and once you have it in a SQL format, there are tons of BI tools available. You can just plug in via GDBC or UBC and connect to any kind of data source. So that is one and zeppelin pretty much similar like Jupiter but has interpreters with n number of languages and any number of uh underlying data sources. So what are your thoughts on those? So SQL in my mind is not the optimal solution for handling time series data. It just gets really clunky when you try to express time series calculations through SQL
um just heard a talk uh from what was their name the the SQL variant with time series support guys and it's interesting but it it gets clunky the just the expression of it is not native to the problem. uh otherwise I've looked at Zeppelin and um the the part that seems attractive for a business is to get all the integrations where you get host metrics and service metrics coming in and you don't need to do anything. So that's what providers like data dog and you know Newurelic and Semiteex and bunch of others get you. So this is where my my interests lie right now just through the the needs of the business. We can't afford to build everything from scratch uh doing it ourselves even if we find kind of a a solution that expresses it well enough. Uh and once that monitoring part is solved by somebody else then building on top of it doesn't make sense. Then it's easier to go with the solution that's provided with your monitoring. So that's that's where I stand for the moment
And is this only more logs or is it pretty much all kind of data? Uh Jettle Jettle supports logs and metrics. Uh it it's got math and it's got search. So take a look. Sure. Oops. You didn't want me to unplug that, huh?