Detecting Anomalies in Streaming Data –...

data.bythebay.io: Subutai Ahmad, Detecting Anomalies in Streaming Data

Recording: data.bythebay.io: Subutai Ahmad, Detecting Anomalies in Streaming Data

um I'm just curious how many of you have heard of Nea or know what we're doing um like a few of you are any of you members of our open source Community or nck on I know that was a setup for you anyone else okay great um so you may know that uh Numa develops algorithms for machine intelligence that's inspired by Neuroscience and a lot of the new uh experimental data that's coming out of the field of Neuroscience but today I'm going to focus on one specific aspect of of how we deploy our algorithms which is analyzing streaming data and specifically I'm going to focus on detecting anomalies in streaming in streaming data okay so here's what I'm going to talk about um a little bit about kind of the generally about streaming analytics and anomaly detection and streaming data um streaming analytics imposes some constraints and requirements that are quite different from the what traditionally we expect in machine learning so I'm going to try to articulate uh some of that from kind of our standpoint I'm going to dive into how we use our uh technology hierarchical temporal memory for anomaly detection um and the technology has been designed for streaming data uh from the ground app um I'm then going to switch gears a little bit and talk a little bit generally about benchmarking so it turns out because the application area is has some constraints and requirements a little bit different you have to think about benchmarking in a little bit different way too and so we've created an open Benchmark for uh doing detecting anomalies in streaming data so I want to talk about that it's pretty unique in in what we've done and then I'll I'll summarize so it's a it's a somewhat algorithm Centric talk uh there's a there are a lot of um excellent sessions uh such as this week on kind of software infrastructure and overall INF infrastructure for streaming data but I feel like the focus on algorithms for streaming data has been really missing and I think because of the unique requirements of the field it's it's a bit different you have to think about it differently so this talk is really going to focus a little bit more on the on the algorithm side of of it okay okay so um real-time anomaly detection as we all know uh there's an exponential growth in Internet of Things sensors and real-time data collection systems and this um is driving an explosion of the availability of streaming data um I feel like the biggest application right now in this domain is is anomaly detection for machine learning uh techniques like classification and clustering and um so on it do have their use cases but U an anomaly detection gives you a very basic signal is there something wrong with my system and everyone can can use that uh the other stuff is a really a little bit harder to productized and harder to take action on um so that's one of the reasons we focus on anomaly detection to begin with um there's literally dozens to hundreds of applications in this area I've just listed a couple of them here you may want to monitor your it infrastructure um looking at your data center on your website whatever to see if there's something going on going wrong in your system uh you might want to uncover fraudulent transactions um you know monitor your energy consumption to see if there's something unusual there you may want to track vehicles uh see if there is some unusual movement in real time uh with your Fleet of uh vehicles um real-time Health monitoring is becoming extremely popular and certainly anomaly detection or detecting unusual things there is extremely important so uh anomaly detection has hundreds of applications in the general context of Internet of things and and real-time data but what's sometimes uh missed is that so detection is good detection is necessary but often the real goal is prevention you want to prevent something from happening so to take a look at sort of Health monitoring as an example suppose you have a a system that's monitoring your heart and the health of your heart in real time um it's great if it tells you if it detects a heart attack uh and the fact that you're having a heart attack right now but you know it would be far more valuable if it told me an hour ago that I'm might have an heart attack now that's something I can really take action on so detecting unusual behavior as early as possible not just whether you detect it or not is is really critical in in this area okay um let's take a example uh this is uh customer data that that we work with this is a large industrial machine and it's showing the temperature of of a of that machine over over many months as you can see there's three dots here labeled with anomalies um this particular one is a very obvious one this was a planned shutdown of the machine and that's good on the right hand side is the one they were really concerned about this was a catastrophic failure of the of the machine and this was uh unpredicted and it cost them hundreds of thousand of dollars with this very large system and what's interesting is that the middle anomaly actually um is the point at which the system started behaving uh abnormally so you can see that there's a funny fluctuation there it's a little hard for us to tell it's somewhat subtle but an expert in this area can tell uh told us that that was the point at which uh this thing started to go AR and so that that's if they could have detected anomaly at that point in time it could have saved them a lot of trouble now as you can see a simple threshold based system is not going to detect that there's a very subtle kind of temporal signature to it so uh this is the kind of thing that uh you see a lot of in in Internet of Things okay so let me set up the problem a little bit so you have uh thousands and thousands of sensors of metrics in your system um for any given system you have a stream of measurements uh at different points in time so it goes back sort of essentially infinitely into past and is going to keep going in the future that's kind of the streaming mentality and you're at the current time te and for anomaly detection what you want to do is given the past history of data uh decide right now whether the behavior of the system is in some anomalous State okay and then uh you have to report that decision do any bookkeeping if you need to retrain your system everything has to be done before the next input arrives okay so that's kind of the classic realtime setup um in particular for machine learning uh not not only do you have those constraints but you can't look ahead this is very different from the batch mentality you can't look ahead in time uh you have a you have to make a real-time decision there's no real training test set split um you know every system has a unique Behavior you start your system and you go um and you have to learn online as as you go along uh so things like cross validation and so on is has it's much harder to do in this kind of setup uh the system must be completely automated and customized to each stream every single machine I mean even if it's the identical model uh different deployments will start having very different Behavior very quickly so you need to really have customized models for each each thing but given that you don't really have a training set or test set um play it's it's it's challenging um so this is kind of the typical setup so um we've developed a technology called hierarchical temporal memory or HTM I'm not going to dive into the algorithm in too much detail but it's extremely powerful sequence memory algorithm uh that's derived from recent uh findings in experimental Neuroscience there's a lot that lot of documentation and papers uh that we published around this uh but just to kind of summarize the specific implementation it's a very high capacity memory based system uh it models temporal sequences in your data um it's an inherently a streaming algorithm it it has all of the properties that we want that I had in the previous slide it's a continuously learning and continuously predicting system system just like our brain is always predicting and always uh updating uh so is our algorithm um and uh it's it's uh there's really not much of a need to tune hyperparameters it's very insensitive to most of the parameters you you can work across a very wide uh variety of data without having to uh tune things custom for every data set uh it's a completely open- Source implementation and you can go to github.com nenta and there's tons of documentation about it we have a pretty thriving open source Community uh around this um so what I want to talk about is how we oh I'm sorry um one other thing about our HTM for it's going to be important for the anomaly detection is that um if you look at essentially what the HTM is doing it's always trying to predict future inputs okay so if you look at the current input to the system it's a stream of data X oft um internally the algorithm encodes this into a very sparse High dimensional Vector a of X of t and then it learns uh temporal sequences in the input stream and makes a prediction in the form of another very sparse Vector so Pi of X okay so Pi at X of T represents a prediction for the next input t+ one okay it's the exact same uh format of the vector it's just predicting what it's going to see next um so here's a diagram of that so when we apply it to anomaly detection um we do a couple of things with it we we have the input we compute what we call a raw anomaly score and then as the next step we compute an anomaly likelihood measure okay so if you look at the raw anomaly score what this is is it looks at the prediction from the previous time step this x of T minus one the current input and it computes a measure of how different the prediction is from the the current input okay so it's a it's a measure of prediction error at this point in time so this this is normalized so that it's zero if it was perfectly predicted and one if it was completely unpredicted okay so this is an instantaneous measure of the anomaly if you will um so you could actually threshold this and and use this to report anomalies but it turns out that in really noisy environments and really um you know unpredicted environments like we see in in typical deployments uh we can actually do a little bit better by Computing a likelihood score so the likelihood score is a second order measure um whereas the anomaly score tells you whether something is unpredicted right now uh the likelihood measure tell you uh is a measure of how predictable the overall system is and did that PR did that measure predictability change okay so it's a kind of it's a higher order uh mechanism and so what we do there is uh we first estimate the historical distribution of anomaly score so we're just using a gaussian uh distribution and then given a recent set of uh anomaly scores we do um we we look at the tail probabilities of this distribution and try to see How likely is the current set of anomaly scores given the historical distribution of anomaly scores okay and then we can detect an anomaly if the likelihood of the anomaly is really close to one okay so essentially we're modeling uh the predictability of the data as a gaussian and if you end up at the ends of that distribution then you're very likely to be an anomaly okay um so just to summarize um that part of it HTM uh the algorithm this is the heart of what's going on learns temporal sequences uh it continuously makes predictions and it's a continuously Learning System the anomaly score tries to ask the question was the current input predicted you know yes or no I measure from 0 to one um and then uh the anomaly likelihood is another measure on top of that says has the level of predictability in our system changed dramatically okay and that's what we end up using and later when we talk about bench marks I can quantify the difference why we might need a a likelihood score okay um okay so some uh results uh so this is um a few examples of detecting anomalies in an it uh setting this is from a a product uh deployment so this first uh example shows CPU usage on a server over time and as you can see there was a jump up in the middle and the system detected that as an anomaly um but notice that in this case I think it was a new software package that was deployed that had higher CPU usage so it detects a jump as an anomaly but now it's uh stabilized on a new normal and so the anomalies quickly die away and you're not continually reporting false positives it's automatically adjusted to that another thing to note here is that there is there are unpredictable events in here uh these little jumps but that's normal for this system so if you just look at the anomaly score by itself you'd actually get a false positives all the way along here but because of the way we use the likelihood score these are all this level of unpredictability is totally normal here uh here's a second example of a load balancer this is an extremely unpredictable stream um if you looked at load balancer l latencies in a website um they're often pretty low but quite often things Spike up and that's actually totally normal but there was a case where there was a configuration mistake in the load balancer and these high latency events happened much more frequently than normal and so the system quickly detects that um Can automatically detect that as as abnormal as well so this is an extremely unpredictable completely different from this uh kind of stream this is another example this actually happened on our servers um this was the case so we have our uh servers deployed on AWS and there was a point in time when the AWS API servers on the East Coast shut down and our system actually detected an anomaly an hour before the AP AWS API servers shut down so what happened is that uh their API server started to get slower and slower and our our system our server uh started to fluctuate the CPU usage started to fluctuate and this was a very unusual temporal behavior and our system automatically detected that as an anomaly even though any particular value in there is totally normal it's just that the behavior is one that hadn't been seen before so we actually detected that an hour before it was reported on the AWS uh stat board which is kind of cool so this um this product is uh called Gro it's actually available uh by a company called uh grock stream so you can go to grst stream.com uh they have they licens cfrs uh so there's a commercial server based product that detects an anomalies in it infrastructure it can run thousands of anomaly detection detector models in in real time um it takes approximately 10 milliseconds per input uh data to to make a decision and this includes the entire learning phase as well uh because it's continuously updating and um there's no parameter tuning required you don't really need to uh care about what type of machine or the characteristics you can just start streaming the data and work uh we've tried uh the system on many many different types of data um uh kind of a demo one that we use is uh looking at financial data so this particular example shows uh a stock kico Phillips and what we're doing is monitoring uh stock price stock volume and the Twitter mentions for the stock over time and you can see that um right here uh the Twitter volumes jumped and actually a single spike is totally normal in in Twitter but what's unusual is kind of a sustained period of activity here so it detects that as an anomaly and uh the next Once uh the market opened stock actually dropped and there was a higher than um normal you know stock volume uh my stock trading for that uh uh security and it turns out if you dive in and look at the Twitter it was because they had slash the quarterly dividend and in this case Twitter actually picked this up and started talking about it long before the market open and before the news was actually announced so this is uh kind of shows the power of real time uh anomal detection in in a streaming context so this um is a is an app called HTM for stocks it's a free demo application uh real-time demo application you can actually download it right now if you want on IOS app store or the Google Play Store um this continually monitors the top 200 stocks and then it sorts it by anomaly so you can see which Securities are in a most unusual State at any point in time HTM HTM is our acronym for hierarchical temporal memory so that's the algorithm that we yeah thanks um and this is also fully open source you can get um all the code for this here including all the model parameters and everything that drop thing you just mentioned is it also yours uh it's something that we've licensed to another company yeah so it's it's the I mean yeah it's it's theirs HTM that's also HTM yeah okay so let me switch gears a little bit now and talk about benchmarking um so I've been showing you you know isolated examples here and there of the system working but how do you know it's really working can you quantitatively measure it's it's working and in machine learning that that's important to do it's it's not enough just to look at a few examples you want to know uh you want to have a little more rigorous Benchmark of of whether it's working or not and it turns out for real-time anomaly detection this was not easy um so evaluating streaming anomaly detection has its own set of constraints because the underlying application domain has its own set of constraints um and most existing benchmarks and I'm talking specifically about anomaly detection now are designed for batch data and not streaming data um and it's really the other thing is we're interested in Practical applications and practical deployments not artificial data sets so it's really hard to find anomaly detection benchmarks that contain real world data that are labeled with actual anomalies uh this um basically doesn't exist uh there's one Benchmark out there that I know of uh other than ours which is the Yahoo Benchmark but unfortunately that's not an open Benchmark it's only open to universi so um but um but in theory that data should be uh useful as well have you Tak a at thear I haven't I'd love to talk to you more see if it if it meets this thing um so we saw there's a need for an open Benchmark that's designed to test realtime anomaly detection and uh we felt a standard Community Benchmark with real data could spur innovation in this field and and help everyone uh so that's what we created um and so we call it the Numa anomaly Benchmark um so this is uh NAB for short it's a rigorous Benchmark for anomaly detection in streaming applications um so there's basically two aspects to it one is a real world Benchmark data set so it has 365,000 data points from 58 streams 47 of them are actually real world data with real labeled anomalies and we do have 11 artificial streams so some of the examples I showed you earlier like this one comes from from NAB um the second aspect is a scoring mechanism we have a unique scoring mechanism that rewards early detection so again it's not sufficient to it's good if you detect an anomaly but it's even better if you detect an anomaly earlier um and then we also have different application profiles so you can trade off the cost of false positives versus false negatives here so this is a NAB is a completely open resource um the repository is an agpl licens repository it contains the data there source code for it uh documentation um and you can go to github.com uh SL nnab to find it we actually have an ongoing competition to expand NAB with more data because I feel we need an order of magnitude more data than this so I'd love to see if there's more data we can incorporate in here uh but let me talk a little bit about some of the the details of NAB um here's first a couple examples of data files this is um an hourly service demand and if you're used to working with this kind of data it has a typical daily up down kind of pattern uh in this case this is actually real time taxi usage data in New York City um they the New York City metropolitan Authority makes this available to everyone so you can see some examples of anomalies in here so there's a spike in an anomaly here this is actually the New York City marathon uh and if you look closely you can there there's actually funny patterns before that big spike so usage was anomalous before you actually got that spike in demand So in theory you should be able to detect you should be able to predict an anomaly before that Spike occurs c um then there's unusually low demand so it still goes up during the day but it's much lower than normal and that's that was Thanksgiving um you can see you can see that it would be hard for a threshold based system to detect uh that particular anomaly so that's one example of the data files we have this is a completely different one again in in the it domain uh this is uh CPU it's a server that's running these batch jobs um so it's sort of very spiky uh usage pattern um you can see here there's an anomaly where the spikiness suddenly becomes a lot more frequent so this is a change in the system Behavior at this point but then this is the new normal and so you don't want to keep getting false positives here and then you get another Spike anomaly at the end there okay so you can see the the characteristics of all these data files are completely different from one to the other um but they're all sort of typical things you see in in streaming data so the second uh as I mentioned one of the important things in NAB is how we score anomalies so what we did is we created the concept of a perfect detector for streaming uh for streaming analytics uh and we say the perfect detector would detect anomalies as soon as possible the earlier the better um you want to provide detections in real time um you want to have no false alarms if possible um you don't want to have parameter tuning um and uh you want to automatically adapt to changes in the statistics of the data these are all characteristics of real applications in streaming data so this is what we want to try to uh quantify in The Benchmark um and the scoring Benchmark scoring methods used in traditional benchmarks don't really capture this stuff so things like Precision recall They Don't Really incorporate the concept of early detection uh so you need to modify it's another dimension in in your uh in your scoring um there's typically because they're batched there's an artificial separation of training and test set sets well if you want to do continuous learning you don't want to stop learning when you're testing um you have to keep Le learning continuously um batch data files allow look ahead which is completely usually infeasible in in streaming data and they allow multi multiple passes through the data hundreds or thousands sometimes for uh many typical algorithms and usually that's infeasible in a in a real-time scenario um so how should we think about score um and if you here's an example and you can think about okay where is the anomaly here and where would you want to detect it so this thing there's a very obvious anomaly right at the bottom there where the system spikes down but the actual problem actually happened well before that when this the metric started going down so ideally you want to detect the anomaly here um so what but you don't really know where where it is uh so what NAB does is um so you have human labeled anomalies in this case let's say there and what NAB does is uh puts these windows around each labeled anomaly and what it says that you can detect an anomaly anywhere in the window that's a true positive but the way the scoring function works is that the you give higher score to earlier detections within that window okay so it's okay to detect it right at that red Point um but you will get a higher positive score if you detect it earlier in the window and then if you detect it outside the window it's considered a false positive and you get a negative score okay so I'm not going to go into the detailed equations for this we have a paper published on NAB that that goes into all all the detail for this um a couple of other points um there's application profiles which allow you to assign weights between false positives and uh false negatives and true positives um and you can imagine that different applications have different needs so if you have an EKG example let's say this heart attack uh example then your with a few false positives but you really don't want a false negative right that's that would be really bad um on the other hand if you look at it and devops um those professionals completely hate false positives um because it drives them nuts if their pager is always going off and it's actually usually okay to have a few false negatives because their systems are designed with some inherent robustness in there so uh you know exactly what the you know value of a false positive versus a false negative depends on the application so in NAB we you can actually test with different uh weightings of that and then NAB emulates uh some of the stuff I talked about earlier for real-time scenarios so there's no look ahead allowed for the algorithm you get one data point at a time and you have to make your detection before you see the next data point um there's no separation between training and test files every stream is its own um thing so you for every stream you invoke a Model start feeding it data and after after a short period you start counting the anomalies and then you can do your scoring on it and um there's no batch parameter tuning allowed uh you must uh it must be a fully automated system with a single set of parameters across all the data files if you're going to do any parameter tuning it must be done on the fly as you get the data okay so again this um emulates uh uh real well scenarios but this is a very different mentality from typical machine learning you can see but that's what's needed in in real time scenarios okay so um you want you can test algorithms with NAB we made it extremely easy to test uh to plug in and test different algorithms um I'm going to show results with uh several different algorithms now so we have the hierarchical temporal memory which is uh the algorithm I described earlier um we have results from Etsy Skyline which is a very popular open- Source anomaly detection uh technique uh it has a mixture of statistical experts and uh it's a continuously Learning System system uh Twitter released um an open source anomaly detection algorithm I think last year um and it does a form of robust outlier statistics plus uh some temporal modeling as well um we have B also Bas in online change Point detection which is kind of a classic uh technique that estimates the distribution of uh your metric values and tries to see how different you are from that um and it's a very principled way of of doing that and this is a real-time version of that uh technique uh so here are the results um on all the files so just look at this column for now um so the perfect detector we scale the score so the perfect detector gets 100 uh none of the detect algorithms are anywhere close to that right now so the HTM uh does does the best um and uh we I've split it out into two different types of runs if you include the likelihood computation you get a higher score than if you don't include the like likelihood comp computation so there's a significant difference uh between these two and this is typically for the really noisy environments that's where you see the the extra benefit um after that uh Twitter gets a score of 47 uh followed by Etsy and the Bas and change Point detection is essentially like as at random and this is because uh they make specific assumptions about the distribution of the data and some of the data files meet it but many of them don't where it does meet that assumption it works really well but you don't the point is you don't know in advance um so uh it's you know I think you should this algorithm could be expanded to do some mixture of distributions or or a broader class of distributions and that's something you can explore if you're uh developing these algorithms yeah why do you um try HTM with and without the likelihood um this is just um to see what the effect is of the likelihood um because we wanted to see if that we we felt the likelihood would help but we didn't know for sure until we run this so it's it's so in production in production we always run it with likelihood yeah there's no like undesirable computational burden no no it's it's very fast um so it's 10 milliseconds on my laptop uh for per data record for everything um and then the other columns there show some of the variations um if you look at different weightings of false positives and and false negatives I'm not going to go into that um so a couple of examples of uh of runs here so this is a uh CPU usage on a server um there's one anomaly here which all three algorithms get so the diamond is the HTM the square is uh Etsy Skyline and the plus is Twitter um in here so all three get that uh there's another uh smaller anomaly there and this is a shift in the usage pattern so uh Skyline and um Numa both get that uh the red things are false positives they're outside the window so um the HTM actually has some false positives outside of that window and then Twitter does not detect the shift but it then it gives a lot of false positives after that um so in this one uh the Etsy Skyline actually did best because it detected both anomalies and there was no false positives and it was right on um same as same time as everyone else um here's an example of the machine temperature one kind of zoomed in on that temporal anomaly um Numa is the only one that the the HM is the only one that really detects the purely temporal anomaly um everyone detects uh the catastrophic failure and then the HTM and uh Skyline actually have two false positives around here um here's another uh it um server usage data so there's a big spike up there um and Twitter and Etsy Skyline both detect that Spike up but the HTM actually detects it much earlier and if you look closely you can see there's actually a small shift in the usage um there's still fair amount of noise in the data and the shift is kind of within that noise but there is a noticeable shift there so if you can detect this temporal change you can detect uh anomalies earlier and this is extremely common in streaming data you often see subtle signals in the Data before the actual big catastrophic thing thank you um I mentioned the NAB competition so uh here's a a little bit more information on that so we see NAB as a resource for the streaming analytics Community it's a completely open Benchmark um we're at a 1.0 state but I feel for 2.0 we really need a lot more um data files uh with anomalies and and more algorithms tested to really make it a much more valuable to the community and so we're uh we have a NAB competition going on with cash prizes you can either uh test additional algorithms and while meeting all the constraints of NAB and uh report them or uh you can submit uh data files that have actual uh anomalies in this and if you have access to data I would highly urge you to submit uh this because it's extremely rare to get it's very hard to get real world data with actual anomalies and this would benefit the entire Community um in there so we have uh cash prices of $2,500 for algorithms and data it's really easy to enter not like we're going to get thousands of entries so it's a pretty high likelihood of winning this um so definitely consider ENT entering it um you can go to n.org naab for details there's a deadline I think it's uh by end of June or early July is the is the deadline for entering so I hope you uh some of you will consider that it's uh um it should be very easy easy to enter uh into this so just to summarize um anomaly detection for uh realtime streaming analytics poses some unique constraints uh so there's very stringent real-time constraints and automation requirements um and the typical bath meth methodologies do not really translate well into this domain you really need to shift your thinking a little bit um I talked about uh HTM learning algorithms which can be used to create a streaming andonly detection technique it uh system it works very well across a wide range of data sets at least the ones that we've tried uh it's open source uh and commercially Deployable um and then I discussed NAB which is a completely open source Benchmark for streaming anomaly detection uh it includes a label data set with real world data uh scoring methodology designed for practical real-time applications and then we have a competition to expand the data set for for future and then here's uh some of some resources uh that you can go to most of these links I've already um talked about HTM studio is a free desktop app that we're about to release uh that allows you to easily experiment with your data without sending it to the cloud or anything you don't have to be a programmer you can just download the app and try it so if you're interested in that please contact me and I can get you uh uh a link to that uh it's in a private beta state right now um the other thing I want to point out out uh is that our community has done um many different versions of uh our codebase and algorithms um in particular I want to point out Apache Flink which uh iron Wright has been spearheading in our community so he has uh done a complete port and a complete anomaly detection system um using htms on Apache Flink uh so I know from a deployment standpoint uh that could be really interesting to some of you um and that's it thank you any more questions I was curious how HCM handles trending data I'm not sure I saw any example in the talk so yeah so um trending data is typically fine um what will happen is um it's really if the the N the way a Trends uh really changes um so if it's sort of you know just a steady gradual um lineer it it should be fine um but um you know if it suddenly starts changes in in the way it's training it it'll cause a problem um we don't see that too much in streaming data but it's definitely a possibility if you look at like revenue for example that's going to be you know that one might have an exponential okay so it looks like the crucial point of your algorithm ofm is figuring out the model and then detecting the L of that of the anomaly as model so is the model specific model for hm or you have some generic kind of contr for kind of data or do they allow different models yeah it's a it's a generic uh model so oh sorry repeat the question yeah so the question is uh we have a the way our system works we have a HTM model followed by the anomaly score and the likelihood and the question is uh how specific is that model to the data so um you you have to define the number of input streams that's coming in and after that it's it's generic so in all of the examples a NAB actually is uh you get a time stamp and a and a scalar value um in the NAB data that I showed you so once you know those two things you can just insentient work across all of the different types of noise systems and patterns that I that I showed you so you don't have to tune the parameters at all HDM data model is yeah it it uh it's hard to say it's the best because for any specific thing you can always optimize and tune for that but it works very well across a very large uh category of data without fine-tuning and it's sort of interesting I mean I mentioned it comes we're very inspired by the Neuroscience data there's a lot of uh findings in neuroscience and how you get a very flexible generic system like that and so we kind of rely on that it doesn't mean you'll be the very best at any particular thing but you'll be very good at a wide range of things mention thatal distri but do you construct once you have the model do you really construct the specific noise distribution for each dat so the question is about the gaussian distribution and do we construct it specifically um for every stream reconstruct other distributions yeah so that that uh that distribution is on remember it's a second order thing so it's on the anomaly scores not on the underlying data so the underlying data can have any distribution uh so the Gin is purely modeling the anomaly scores and that by looking at a large set of data we've uh found that this gaussian works pretty well for that we don't have to tune it to specific things but it's it's possible that other distributions for in some cases could give better results in addition to that if you detect those fight meaning the extreme Focus so the question is can we detect extreme uh single Spike um and the answer is yes if it's a very clean data so you hardly ever see a spike then a single Spike will be very unusual but if you get pretty spiky data and spikiness is normal then a single spike is not going to cause an anomaly okay so a spike by itself does not mean anomaly it's a spike in the context of past behavior and it could or could not be an anomaly supp you are seeing under actually they have given frequency of SP yeah your likelihood major the K need to be adjust as well meaning you need to Define how the span of the interval that want yeah so so the question is if the frequency of the data changes do we need to change the way we're doing the distribution and the answer is no because it's the underlying HTM model is is um modeling all of the temporal behavior of the system so it's learning the common sequences so if it's a lower frequency system it'll have slower sequences if it's a higher frequency system it'll have higher sequences so the lohit is only on the output of that system it's a little tricky it's it's a second order thing it's it's uh uh it's not actually looking modeling the the data we keep certain window of the of the S only yeah so the question about memory usage um um the HTM system uh has a very high capacity memory for sequences so it can learn it can remember things uh very far back in time uh on the other hand it only it will overwrite things that are where in the same context you have different Behavior it will very uh specifically override those things and that's how it does its continuous learning any other questions uh do these do these benchmarks perform well in system as well do these uh benchmarks perform well in rule-based um so the for anomaly so the rule-based stuff that I'm familiar with is like from a simple setting a manual threshold or something like that those um those do not work well in practice I don't know if you've used those but you set up you can set a threshold um and then uh you know of course it's not just one system you have hundreds of systems you have to figure out what the right threshold is for everything and then whenever there's any change you suddenly start getting all sorts of false positives it's not automatically adjusting so you have to go back and retune it so what happens in practice is that everyone turns those off yeah so the question is about kind of robustness and reliability and fault tolerance and stuff um so that's kind of a system level thing so uh in some of the stuff we've showed we've uh the the systems do checkpoint um one the stuff that Iran has done with Apache Flink is really cool because it can do that in a distributed setup and take advantage of all the reliability guarantees that Flint provides uh using the calization so um so I think it's it's a kind of a system level question that we can you can mul signal not just single I have multiple yes so the question is have we tried multiple there seems we have um and actually our uh paper includes that but it's even harder to get real world data with multiple streams that are labeled with anomalies I would love to expand Nab in that way maybe Financial potentially potentially yeah yeah that could be one example that's right okay uh cut it short but hopefully be available afterward um just a super quick announcement from the organizers uh the sponsors upstairs fascinating really smart people really good to talk to and they're feeling a little bit lonely so if you feel like whatever where are the sponsors oh they are just in the top um uh sort of that corner the build so you go up up the stairs and then towards the back towards the street yeah where the coffee okay thank [Applause] you