data.bythebay.io: Finton Quill, Time series analytics for Big Data and IoT with Kx
Recording: data.bythebay.io: Finton Quill, Time series analytics for Big Data and IoT with Kx
miss Finn think well and so what I'm going to talk about today is just big data and IOT and so obviously when you're out conversations having conversations here in our conferences here you see a lot of different choices so I just thought I'd put this funny picture up and just for the sake of it and so as many choices out there but then like what what we want to do to make these choices what are the things we need to consider and I think when we step back and a lot from a business perspective I think there's three main things from your total cost of ownership you need to think of number one your hardware number two your software and number three or developers and more and more nowadays and the developer cost is completely ballooning compared to the other two costs so what can we do to actually make that a bit more efficient than the driver costs down and so as I say from a hardware perspective there's lots of things we need to learn I feel like many engineers nowadays many software engineers are actually like leaving hardware behind and they're not seeing some of the advances that are happening and i'm not writing their software accordingly to work on that hardware which is something that i feel that we've done at our company and but just in terms of like what do we need to think about with hardware and so number one is memory memory is exploding in terms of size and the cost is going down and now one terabytes of rama no machine has just come out of the utz that's pretty much standard and there's bigger machines coming along so I played with a 32 terabyte round machine a couple of weeks ago obviously you know that comes with a cost but you know the cost of roundly is coming down and more and more data can fit in RAM and we can also do in memory compression is coming down the line as well as coming directly in the operating system so you can take advantage of compression in memory and then there's also the idea of transactional memory which is very much upon us now as well and another consideration with her grade and a storage there's huge advantage advances in storage in the last couple of years and now we've got and non-volatile dims got nvme flash storage which are like orders of magnitude faster than a traditional you know standard spinning disk media that we're used to dealing with and then does this whole hyper converged approach as well and the idea of once again compressing data on disk and then the third point would be parallel ism obviously we've got them parallel ilysm inherent and machines now there's a multi-core machine so you want to be able to take advantage and write code to properly take you know and care of the multi-core machines and then even in the instruction set level now we also have vector instruction sets and directly in the machine instruction sets so we want to take advantage of those and we want to write our software appropriately and then we also want to run effectively in a distributed manner as well so to be able to you know find out over multiple machines and but do that efficiently which is the key here doing it efficiently because a lot of the tools that are out there nowadays are extremely inefficient for dealing with clusters of data and then there's the idea of file systems as well now obviously there's many file systems like be gfs btrfs etc that can take advantage of this distributed parallel file system nature and give us one view of the data and so that kind of covers a lot of you know the hardware side of things and as I say today I'm here to talk about like big data and the IOT side of things from a time series aspect and one big thing about IOT is everybody's talking about big data big data oh and structured this and structures becoming bigger and bigger and bigger but then at the same time they're telling us that IOT is becoming bigger and bigger and bigger and that's something that I still find you no trouble kind of like equating those two things because the most sensible thing is anything I've ever seen from an IOT device is inherently structured so if anything the size of the structured data is going to completely swamp the size of the unstructured data going forward and as well as that IOT it's inherently time series everything is timestamp ID value whether it's temperature whether it's you know pressure on any sort of measurement M so its all-time series nature and so it's structured and/or semi-structured data and then it's also they're doing analytics at the edge so not just having it in a you know a massive central data server but actually being able to do some of the analytics say directly on your Fitbit etc etc and so for that then you need a small footprint so you need to be able to run in these small devices and typically run on some sort of like ARM based architecture or like devices like the Intel quack that's coming out and and then you also be able to need to need to be able to deal with streams you need to be able to deal with data as it's coming in and not just when it's stored and you know and tabulated and then you also need to be able to deal with micro batch data especially if you're in areas where maybe there's poor connection in your sin data back over the wire so maybe it has to buffer it and then push it up in batches and so these are some of the considerations that we need to think of when we're thinking of IOT and big data going forward and so about our company KX and we've been around since 1993 and we're proud of a first derivative group which is publicly traded company and we have a relatively large user community not as much as other technologies and these are some of the industries that were involved in and so we've mainly been involved in the financial services industry so we've been dealing with banks for about 1993 onwards so quite a bit of time and finance in my opinion has been ahead of a lot of other industries and dealing with big data and especially dealing with streaming big data as well because it's been capturing data from the market since you know the 70s and even further and before that so you got the idea of like the lambda architecture and the copper architecture that's like that's not big news and finance you know these high frequency traders and big investment banks have been doing this for a long time so what can we learn about this and what you know we're seeing these volumes now coming out into other industry so what tools are available to deal with this and as you see here there's just some examples of graphs where you got like 18 billion quotes in a given day for just one exchange and there's multiple multiple exchanges out there so this is multiple terabytes for just one single data feed and so how do we catch it up and store that and so that's what I'm going to try and talk about a little bit more and so our product and what we do is we do streaming real-time and historical data so it's not like a lot of other solutions out there where they just do the streaming part or they just do the in memory part or they just do the on disk part we do all three of these facets which I think separates us from a lot of other technologies out there and this is just another you know funny diagram that I have here that small data big data etc etc and so our main product is called k DB plus and it's an integrated database and programming system and so there's two versions available there's a certitude bit free version that you can just download from our website KX calm and play around it and then we've got a 64-bit license version and there's no difference between the two versions other than the address space and limitations so you just got a four gigabyte address space limitation with a free version and then you're limited to basically your your rom on the 64-bit version and in terms of some of the features and we run on standard opera systems and hardware so we've got builds available for OS X for windows for a spark and and spark chips for solaris and also the intel on spark intel on solaris re and then also linux so that i'd say ninety ninety-five percent of our customers are using standard like red hat enterprise linux and yeah and then we're hardware agnostic as well to purely software solutions so it's about four hundred kilobytes in size so it's pretty small and so it fits typically inside and the cash and we support compression as well and so we can pour compression on disk and in flight and then when your am sending data over web sockets as well and so we can take advantage of all the parallel ISM available on multiple machines so multi-core machines and and many many machines in a cluster and i miss the in database analytics as well so that misleading it and traditionally obviously people separated the analytics from the database and so now we're actually bringing that together and that's that's you know not new and we can also support columnar relational I'm key value as well and in terms of the api's we support AP is for most major programming languages and so C C++ C sharp and these are all open source api's you can download them from our website and connect and then in terms of some of the interfaces as well we've got odbc version 3 compliance so you can hook it up to tableau server and then we've got JDBC HTTP and WebSocket the HTTP and WebSocket server actually built into the executable and so then on top of the database we've got a programming language queue and which I'm going to try and get into and show you a few examples and so what are the facets of the cube programming language well it's has sequel like semantics so it has crowed semantics so select update delete etc etc and then it also supports array or vector operations and so that's very very powerful than from a programming efficiency perspective because you basically get rid of a lot of their traditional control structures that you have in other languages like if Luke's do loops while loops because their operations run over the entire vector so it's very very easy and it so it really does reduce your code base and we also have native time series support which I'll get on to next and then it also has functional language features and that's built directly into the language as well so you've got the functional and the query in one so like sequel and PLC cool where you kind of had this to kind of separate things you can actually merge them both together with us and also create your own user-defined functions directly into the queries and so in terms of some of the time series analysis and what are the things that we support so we've got native temporal data types so we've got dictionaries tables key tables and 90 second time stamping millisecond time stamping are all first-class citizens into the database and awesome like afterthought or some why we that we load in after the fact and so as I said we do high-precision 90 second time stamping which is exceptionally important and a lot of the IOT industries nowadays where they're doing sensor analytics and reading off devices and then we can do temporal arithmetic so adding dividing you know subtracting dates and which is surprisingly uncommon in many many database platforms and we also have the ability to do by temporal joints which is not very common and so that's like if you've got two different time series data and you try and join them together and even if they're mismatched on time we'll do a lookup back in time and rather than say if you did up with a traditional SQL join you just would get a mismatch and we can also join the time series on the fly as well so as the data is coming in off the wire we can do a join even before the data is actually tabulated and so this is a sample architecture diagram here so you got your data coming in on the left and that could be from some like feed handler written in a compiled language like C C++ and that would be going into our events engine so all of these blue and boxes here are all just processes written in the queue language and so the data comes into the events engine first and the events engine is typically called a ticker plan to the financial world that's basically a data distribution engine so the first thing it does is it logs the data down to the log file on disk and that's for recovery purposes so then if any of these in memory systems go down you can replay the data from that log file and so the simplest example of a subscriber to disadvantage engine would be what we call the real-time database so that basically just opens a fire hose and gets all of the data so it gets a larger and larger to read today so it's quite memory intensive but it's not particularly CPU intensive because it's just doing simple obtained dependent and append and but then you can have specific streaming query engines or CEP engines and what they can do is they can maybe subscribe for a subset of the data say I want to subscribe for these four em stocks that I'm trading and I want to calculate the volume weighted average price or something like that as the date is coming in so it's going to be a little more CPU intensive because it's doing calculations on each message as it comes in and but then it's going to be a lot less memory intensive because maybe then you only have maybe one entry per subscription in your table so it's a lot less data so then if you want to get that volume weighted average price you just hit that engine and you get basically instantaneous results and then what happens a typically end up there running pre-configured interval is that data in the in-memory database gets and dumped down to a shard on disk so it creates a new day on disk so everything but say today's data is residing on disk and but the querying is exactly the same whether it's a streaming query engine the real-time database or the historical database it doesn't look any different and so i'll just run quickly onto a demo and so i'm running this on a server and which is based in new york and it has about half a terabyte of ram and it's got 22 terabytes of real new york stock exchange data for about three years and so if we just look at some of the sizes of the data set and so our quote data set is 538 billion records and our trade data set is 253 so there's over 1.2 trillion records between all the different tables and so basically the Q language in action this is it's an interpreter environment so you can just start typing so i can just great say a list of three numbers oh it's not coming up here sorry and i guess any questions yeah it typically comes in over IP see there's a few different ways you can do so you typically open it a simple and tcp/ip socket connection and you just push the data across over the water so as I said our hole and communications protocol is documented and it's available and that's what all those API is that i mentioned do as well i could just open a simple socket connection push data across another thing they can do is you can actually load shared objects into the actual execution space so you could actually push data in that way as well so it's actually just running in the same namespace and that's om say our Python API works for example is actually you can run say like pandas functions directly on the data in our database because it's running in the same memory space so just few different ways you can do or else you can load in from a flat file as well it could just be loading from a CSV file or something like that so you got a few different options there but for that particular kind of diagram that I've showing it typically they'll open a tcp/ip socket connection from the data source and push the data across yeah and but ya doing the shared object is typically faster because obviously it's working in the same memory space see you know you're not you don't have the tcp/ip overhead then what kind of analytics is possible with this language is beyond you know sequel yeah so it supports all of the standard sequel like analytics so as I said the crud semantics em and also has you saw some of the moving averages weighted average so it has a lot of statistical functions it has a lot of trigonometric functions that has a lot of matrix functions built in and we're working on a lot of liq machine learning libraries now we don't really have that kind of available and the dust stuff that we're working on we're probably just plugging down in as modules and then if you go we actually have a github repository as well where people have contributed a ton of like libraries that have written on top of the base product itself that the base product of stealth we try and keep it as kind of tight as possible and so it has of it i think it's 225 functions in total all together so a mix of algebraic trigonometric you know you name it no summer there and so make basic matrix functionality as well not like huge amount but you know some basic ones in it any other questions okay cool thank you very much