Building High Performance Microservices...

scala.bythebay.io: Dan Simon, Building High Performance Microservices with Colossus

Recording: scala.bythebay.io: Dan Simon, Building High Performance Microservices with Colossus

I you all right yeah Simon I am a staff engineer tumblr and today we talking to you a bit about building high-performance microservices with Colossus which is a framework that we've developed and used at tumblr so just a little bit about me before we get started like I said I'm staff engineer I've been at tumblr since about two thousand twelve so I've kind of been with the company as we've really grown significantly when I started we were maybe about 30-35 engineers now we're close to like 160 maybe more than that and in general I've been working with Scala since about 2010 and before that I was doing a PHP so that was a little bit of a change there but I think overalls for the better so anyway in this talk I'm going to be focusing on Colossus what it is how we use it at tumblr how it works and also with some code examples of what it looks like when you're using it and how we made it fast because one of the big things about Colossus is that it's really designed to be a high performance framework so just an overview about what Colossus is it's an i/o and service framework written in Scala it's a fully generalized framework at the core but the focus that we've had the whole time basically is geared on building microservices and it's bill it's built directly on top of Java niño but we also use acha for most of our thread management and concurrency and things like that I'll go into a lot more detail about how that works and of course it's open source and here's the URL and just as a brief history of where we've been with it I started working on Colossus towards the end of 2013 and it really started as a just a hackathon side project so the first version I built in 24 hours and it kind of just SAT around for a while then a few months later we decided to try using it in a project and it worked out really well so we started to like build it out more production eyes it and then finally towards the end of 2014 about a year later we open sourced it and then since that time it's grown significant significantly and it's come to a point now where pretty much powers our entire service architecture and a pretty large portion of tumblr overall and as I said the the two big focuses of Colossus have been performance and simplicity so what we're really trying to get is have a framework that makes it easy to write services and to really encourage a reactive and fault tolerant and scalable design all the kinds of things are really looking for out of micro services but at the same time you want to make sure that we introduced as little overhead as possible when doing that so I don't want to silly say that we're trying to make Colossus as fast as possible we want to try to make it as efficient as possible and making it fast is just kind of a corollary of that and then lastly we wanted to make sure that the framework itself was very small focused and easy to iterate on and I think that has been a pretty big factor in being able to make Colossus as fast and efficient as it is because as we've learned about building services and as we've learned you know what works and what doesn't were able to kind of iterate pretty quickly and change the design make it better so just kind of to rehash some definitions just so we're all on the same page just kind of want to talk a little bit about what I'm thinking of when we talk about services and microservices I'm going to kind of use those two terms interchangeably but basically what we're talking about is a stateless application server that is more or less geared towards taking in requests and processing them into responses and usually that's going to be a restful interface over HTTP but not necessarily that's the only way to do it all of the state is offloaded to databases and caches and it's designed to be highly concurrent you know fully utilizing our multi-core CPUs and it's generally considered to be part of a larger system so we're not building all of tumblr on one service we have a whole bunch of services each one handling one particular responsibility so in other words the basic takeaways our service owns its data it encapsulates state in terms of the fact that nothing is able to touch state that it is sitting in front of so we're talking about databases and caches and things like that and we're able to scale it horizontally and independently of the rest of the system and then basically when we think about what kinds of use cases we're looking at specifically with Colossus we can kind of take a step back and just kind of observe that most of services have more or less the same workflow and actually give a few examples of where we don't do this but in general the basic process is you're taking in requests your decoding them you're figuring out what these requests are asking for your then in turn going to be sending requests to these external databases or caches or other services you're going to be waiting for responses for them and then you're going to be building a response and sending back to the client who originally sent the request so in a nutshell that's basically the scenario that Colossus is looking at addressing so now I'm going to spend a little bit time and focus on how we use it and also base we'll talk a little bit about what tumblr looks like in general before I go into how we actually use Colossus and tumblr so we're by no means a full micro service architecture we're very much kind of in this PHP monolith system but more and more over the years schol services have been taking an increasingly large responsibility in terms of both front-end and back-end features so while we still have most of the core and legacy functionality in PHP as we're building out new features or we're slowly taking some features i needed to scale more independently and removing those into services and at this point i would say we have a few dozen services in production so we're not at the point where you have hundreds but we do have a decent number of services that are running in production for our data almost all of our primary data is in my sequel and memcache so we have hundreds and hundreds of my sequel machines many of them are running on SSDs and then at the same time we have also many many my sequel memcache machines I don't know how many terabytes of RAM are using but it's quite quite a few and then but we also have a lot of secondary data both in Redis HBase kafka we're really ramping up with Kafka more recently and also just in general HDFS has a lot of things like logs and things like that Collins is another open source project by tumblr which is what we use to manage all of our server infrastructure and management and getting our servers set up the way we want them and things like that and then lastly we use open TS TV to track all of our monitoring and metrics that we literally have i think almost 10,000 in visual metrics that are being fed into open TS to be looking at all kinds of various performance and user interactions and things like that across the entire site so kind of in one diagram this is sort of what a service at tumblr is going to look like and you can see that Colossus is kind of sits at the center of everything and the two big things that Colossus gives us in writing a service is going to be one an HTTP server and that's going to give us like the restful interface that we need to process requests and generate responses and it also gives us the ability to open these connections to other systems particularly the ones that we're really focused on with Colossus are memcache and Redis and also as well other services but then you can see that we have a whole bunch of other things that we have that make up a service in general that interact with Colossus and various ways probably the next most important one is how we interact with my sequel so we have an in-house library that handles all the logic for talking to our my sequel instances and in some case we'll have a single service talking to over a hundred databases so we have a pretty sophisticated library that handles all the sharding and all the routing that takes place to make that work then we also have libraries that deal with life cycle monitoring and this includes everything from making sure that a service starts up and shuts down correctly to monitoring it and also dealing with getting the metrics that our services generate out to open T SUV and lastly we have another library that handles lava all of our configuration this includes both deploying services getting them started up and also live configuration that can cause a service that changes behavior while it's still running and I guess the one other thing I'll point out is that we do i use akka actors pretty heavily within our services so even though a lot of like the core logic can sit entirely within Colossus we do often move functionality out actors especially when we have things like shared state or things we want to rap like a blocking API for example when we're talking to something like HBase we wrap that whole thing inside have an aqua actor so we can use that in multi-threaded environment so here's one specific example this is one of the last big projects that I worked on myself and this was a tumblr messaging which is something that we launched towards the end of last year and this is basically just private blog to blog instant messaging and it was an instant success we've had billions of messages sent in that time across hundreds of millions of blogs and but the one nice thing about this is that the the whole back end side is written in Colossus and the whole thing runs on roughly six machines and for the most part those six machines are only needed for redundancy we could actually run out on far fewer and we've had no major down time since launch the only really down time issues that we've had or mostly focused on a we have to do like my sequel maintenance or things like that but even that it's very very very little and our median latency for this service is about five milliseconds and our 99th percentile is about 15 milliseconds I think one of the important things to show is that with this service probably seventy to eighty percent of requests are only hitting memcache so this is one of like the real prime examples of the kinds of services that we're building where we're hitting memcache in my sequel but probably most of as much as we can try to stay entirely within memcache and that's sort of what causes our latency to go up towards the end is when we're falling back to my sequel but even with my sequel in some of these cases where some of the machines are an SSD so we're still able to get pretty high free low latency and a decent throughput so here's a couple other examples that'll touch on a bit so the first one is a project that was created by our search team it's called web scale afdb and it's basically the idea of taking memcache with my sequel on solid state and then sticking Colossus in front of that and treating the whole thing like a very scalable and durable key value store so a single node is able a single service machine is able to handle hundreds of thousands of operations per second even when those operations are going like a write operations to my sequel and actually this is a good example where I was saying before that most of our services use HTTP HTTP with a rest interface but this one actually is speaking the Redis protocol and so when a client is interacting with this service it's actually using a Redis client and it sort of treats it like it's a giant Redis store obviously the only commands that we really support out of that are like get and and like a multi guess we're not doing any of the other things that Redis would normally do like sets or lists but we are able to use a Redis client and just kind of it's a much lighter weight protocol than htp so it really helps facilitate those the low latency interactions that we're trying to get for and like I said this was originally built for search but now we're starting to look at wider integration and to rush this into the rest of the site man the last example I'll touch on is something called activity pages so this is a user facing feature it's a very basic real-time blog analytics that we provide for all users so if you have a blog on tumblr you can see basic things like how many likes you've been getting how many reblogs who are your biggest fans and things like that and this is one example we're not following that strict model of processing requests into responses we actually have sort of a secondary part of the service where in real time we're reading data out of Kafka and we're writing into both Redis and HBase so we're using red is kind of like a cache and then also writing to hbase for our long-term storage but we still provide an HTP interface to PHP which is the consumer of this service so as you hit the site it you're going to be hitting PHP and then PHP is going to be making a call out to the service to grab data to show in your activity page and so these are some of the big examples but like I said Colossus is now being used kind of all over the place in tumblr so like I said search so not just for web scale AfDB but a huge portion of the search infrastructure in general is built on Colossus and this includes both like the offline portions that are doing all of like the ml and recommendations generation and things like that but also the online portion that is doing things like type ahead and stuff like that and push notifications email logging analytics we have a ton of things where we have Colossus sitting in front of things like HDFS and other systems just kind of general event busts and message processing and also we're starting to now work focus a bit more on stream processing so we're not currently not using anything like spark but like I said we are really heavily ramping up our use of Kafka and we're starting to have a lot more integrations with Colossus in that regard so now I'm going to go into a bit about how it works and also what just a basic service looks like in terms of code so i guess as like a quick show of hands does anybody ever written anything either directly on java niño or using or writing something on Neddy yeah so I figure is pretty much nobody so basically the model that we follow is kind of the same model that most asynchronous frameworks have to follow nowadays if you're writing anything on top of niño which is where you have event loops and event loops are interacting with the underlying network and they're pulling events and you have for each connection that you that you're maintaining you have an event handler that is waiting to process those events as they come in so all of like the major frameworks are kind of working that way and in that regard we're sort of following suit so Colossus follows this reactive event-driven model where we have multiple independently running event loops generally one per CPU core but you're able to kind of configure that how you want and as I said an incoming or an outgoing connection is bound to a single event loop for its lifetime and then the user is going to supply a connection handler for each connection that is going to react to events that come in from those connections so mainly when new data is available to process or when the connection closes or when the connection first opens these are the kind of events that you're going to handle so this is a very rough overview what the core architecture looks like like I said Colossus is built on both niño Annika and the way this works is at the very lowest level we're doing all we're letting niño do all this event management & Event processing but all of these event loops are wrapped inside of acha actors so you can see here that we have two main actors at play we have a server actor which is the actor that actually opens your socket and listens for new connections and as those new connections come in it's going to forward them to one of n number of worker actors and these are the actors that maintain the event loops that process these events coming in on these connections and this is basically where all of like the real guts that Colossus live inside these workers so you can see that once the connection is opened it's going to stay bound to a single worker and we sort of use a router that's just in front of these to evenly distribute connections to each worker so the way these worker actors work is like I said they're maintaining an niño event loop but the way this actually works is we use actor messages to do the looping itself and what ends up happening is a worker is going to alternate between handling messages from niño these low-level connection messages and then handling actor messages coming in from other parts of the system and what this what this gives us is a very flexible system where we're able to very efficiently handle the niño events but then we're also able to handle events coming in from other places and that kind of allows us to integrate the event loops with things like akka actors and futures and things that kind of sit outside the workers and this is just kind of that I included here just kind of in a pseudocode of what this looks like actually inside the actor itself so you can see for example that we receive a new connection event and so then the worker is going to take that new connection from that server actor and bind it to its event loop so can start processing messages then it also has a select event and that's where it actually asks niño you know what are the connections that have events right now that are ready to process it will go through those and handle all the events 42 the H of those events to those connection handlers attached to the connections and then it will send itself a select message to do that again so the result of all of this is that most user code is going to be single threaded and in particular we're thinking about the context of a single connection everything is going to be single threaded and what this gives us is it allows us to do very very low latency on the kinds of small requests that are very typical of services so we're not doing any heavy processing in thread you know we're just kind of parsing HTTP requests for creating requests for mem cache or creating requests for other services and waiting for those responses so all this happens entirely in thread and then acha is what's going to be handling all the thread management and all the thread crosstalk so we're sending after messages from workers to the server and from servers to workers and then also from workers to your own actors that you can spin up and vice versa and what this ends up giving us is a very minimal overhead for concurrent operations and especially for concurrent operations when you're talking to a cash like memcached or to another service that's speaking HTTP and from what I've said here you know this might remind you of a framework that is also takes this approach would be something like nodejs where everything is single threaded and one of you know nodejs gives you sort of that promise where you can get very low latency and it's all asynchronous and reactive but also one of the biggest criticisms about something like that is you're really stuck in that single thread inside the event loop and if you want to do something that blocks or you have something that's CPU intensive you're kind of stuck or you have to kind of like hack around it and Colossus is integration with actors and futures really gets us out of that walled garden where when you have things that really need to be multi-threaded or things that need to block or things that are very CPU intensive where you don't want to block these event loops then you can just use actors and futures like you normally would so when it comes to actually building service there's two components that you have to provide the first is the request handler and this is what I've been saying where a request handler is attached to each connection as events come in on those connections Colossus is going to take care of all of like the decoding and giving you an HTTP request but then it's up to you to actually tell it you know what to do with this request and then the second component is what we call an initializer and this is something that has created once per worker and this is sort of where your setup logic your initialization goes and this is also where you're going to be able to open those connections to external systems so real quick here is a very simple request handler and as you can see it's really not much more than just having a partial function that map's HTTP requests to HTTP responses now request handlers can actually do quite a bit more but I have to kind of just keep it simple for now and as you can see the one thing I'll point out here which I'll get into shortly as you can see that the return type of this partial function is what we call a call back up HTTP response and just kind of in one sentence a call back of sort of like Colossus is version of a future but we'll get more into that shortly and then here's an example of an initializer so you can see this is really not much more than a you call a request handler factory but initializers actually do quite more than that as well but this again this is sort of like the bare minimum you need to start a service and here is basically all you need when it comes to the actual bootstrap code so you can see the first thing we do is we start an actor system and then what we do is we create a something call an i/o system and this is basically the wrapper around those event loops and everything like that whoops that's funny there we go thanks chrome and then last thing that we end up doing is actually starting the server where we give it a name we give it the port to listen on obviously there's a lot more configuration you can supply if you want to but then the last thing we give it it's just a function for creating those initializers what's going to happen is it's going to send this function to each of those worker actors that are running and the worker is going to create an initializer and then as new connections start to come in and we'll use that initializer to create connection handlers so everything I've talked about up until now is sort of like the bare minimum of just to create sort of like a hello world service but like I've been saying the one of the biggest features that we need in Colossus is be able to talk to external systems and in particular the low latency systems that we need to use like memcache and Redis and other services so in Colossus you do that by creating clients and clients are what give you these persistent connections to these external systems and clients support a codec based design I should have mentioned that servers do as well but right now we're just mostly focused on HTTP but with clients we want to be able to support things like the memcache and read as protocols as well as HTTP and other protocols that you might have so if you've worked with something like finagle you'll know that finagle is built around the idea of codex and this is one of the ideas that we borrowed and used in Colossus so here's kind of what this looks like when you're actually creating a client so as I said clients are created in the initializer of a service and what that means is because every worker has its own initializer when you open a client you're going to be actually opening one client connection per worker which more or less corresponds to one client connection per cpu core that you're running and then the request handlers that are being maintained by that worker are going to be using that client and that means that everything is still staying within the worker and as requests come in and you want to send requests out to memcache and Redis everything stays inside the worker and everything is in thread and all of the asynchronous communication is maintained by the worker itself so then what was really ends up giving us is what I call concurrency without parallel ism where we get all of the nice asynchronous non-blocking i/o that we want but ever thing is done in thread and this really cuts down on the overhead so now focus i'll just a little bit briefly on how that actually works and as i said before when you are writing your request handler i'd point out that the return type is what we call a call back and a callback is the way that we do all this in thread asynchronous operations and it's sort of a way of doing you know continuation passing and this is sort of another idea that I'll that I'll touch on with something like nodejs where when you want to do a sink io in nodejs you usually do it through continuation passing where you call a function with some parameters but then you give it a callback function that will be executed once that asynchronous i/o operation is complete and the pros about that is it's very efficient it's a nice very efficient way to do this asynchronous operations and it's good for low latency but then the cons is that you usually end up with code that looks like what I've shown here were as you do more and more operations it just starts getting more and more nested and you end up with what you know what we call callback hell so the callback type in Colossus basically takes the mechanics of continuation passing and presents it as a monad and what we end up with is basically something that has almost the exact same API as a scholar future so we have map flatmap recover we also have things like sequence and traverse and all those things and it gives us this nice very efficient single threaded asynchronous operations but now we can do all the things that we like to do in Scala where we as I shown here we put basically the same code and now is in a regular for comprehension which is the kind of things that we like to write and then lastly this is kind of just a quick example of what code actually looks like a bit so here's this a simple HTTP interface for Redis gets and sets and you can see here that we're sort of doing some very basic pattern matching on the HTP request we're sending a request to Redis and then we're mapping on that to produce our HTTP response so you can see that if you worked with any other scala framework like anything from like play to finagle to any of these others it looks very similar we're just kind of doing these operations and mapping and flat mapping on them and things like that and the last thing I'll point out is that clients actually do work with futures and this is very useful when you want to use like a register and memcache connection from outside of a worker for example if you have an actor that you want to have a connection directly you can do that and I guess what this would also say like it if you're looking at this and you're and you're thinking you know I don't really want to deal with callbacks I just want to work with futures like I'm used to then you actually we do support this where you can just do futures for everything it's still using callbacks under the hood but then this gives you the ability to just kind of stick with futures and not worry about that there is a performance it when you do that but it's in most cases it probably is not going to matter that much so those kind of the basics and okay I have a few minutes still so that's kind of like most of the use cases that we go for with Colossus where we're reading an HTP request making requests to write us from memcache or other services and things like that and then producing responses and then when we have to use things like my sequel or HBase or any of these other data stores that you know we we don't have a native Colossus client for then we can just use futures and just use like things like slick or other things or whatever API they provide and there's a few other things I'll touch on real quick that Colossus has probably the biggest one is our built-in support for metrics so as I said we have thousands of metrics going to TS DB and many of those are given to you out of the box in Colossus so things like latency throughput things like that are automatically generated for you you can obviously create your own so we have support for counters for rates and for histograms one of the things I guess I don't cant rly go into too much detail about but the connection handlers and the request handlers that I showed you before they can actually behave a lot like actors and this is actually important for when we do things that are not quite in like things that are not htp like for example for web socket which we're adding support for now you can actually treat your web socket connections kind of like actors and you get a proxy actor just a regular acha actor you can just pass that around send message messages to it and automatically gets routed into your low level Colossus connection handler and then like I said we are adding more support for things that are not typically like the straight request-response semantics of a service so like I said WebSocket is one of the big ones you're looking at we do have support for it now which is a bit experimental well and also things like streaming and also the ability to sort of add arbitrary code into workers that can use things like clients and stuff like that but aren't particularly actually spinning up a server so lastly I'm going to talk about a bit about how we made it fast and like I said one of the big focuses of Colossus has been tourist performance and I think in that regard we've actually really excelled at that and it's been a sort of an ongoing effort to constantly improve performance without sacrificing design and this graphic i'm showing you here is a kind of a rough gauge of how Colossus has performed in terms of benchmarks since we've open sourced it so when we open source it we were already kind of going under this pretense that it was a fairly high performance framework for low-latency things but over time we've actually significantly improved that since then and I guess before I go into the actual looking at a few numbers I'll just kind of give a quick disclaimer that benchmarks aren't everything and so obviously when it comes to benchmarking usually we're looking out something just like a hello world service that isn't doing anything significant it's or just taking requests and turning them into responses and that does capture a lot of what if we work has to do but it's not everything and it's also very easy to fall into this trap of just optimizing something just for a benchmark or not for the real world and I think looking at benchmarks and and doing these kinds of optimizations actually though has really helped us a lot in in improving our performance in the real world so now look at actually some numbers and so this is a these are some benchmarks provided by a company tech and power and this is actually a preview run these aren't actually the final results but I think the final results which should be published pretty soon these this preview is run just a week ago but I think the final results will pretty much be the same so this is basically showing for a hello world service you can kind of get a good idea of where Colossus sits in terms of performance and really kind of taking things to the next level in terms of doing these low latency operations and then even when we factor in all frameworks on the JVM we're still pretty much up there I don't know if this will act if this will stay you know we might get bumped down or something like that I'm not too concerned about trying to say that we're the fastest framework on the JVM but just the fact that we're up there is sort of like showing that we're kind of meeting this goal in terms of writing high performance code so I'm not going to focus too much on the numbers but I think it's safe to say that we're doing pretty well in terms of performance so i wanted to talk a bit about what goes into making code like this because it's been a lot of work to get Colossus to where it is and what I have here is sort of just a rough breakdown of what I feel has contributed to making Colossus as fast as it is so by and large the largest factor of this is going to be just the concurrency model that we've chosen the fact that keeping as much as we can single threaded really cuts down on the overhead needed for doing asynchronous operations so if we think about using things like actors and futures the overhead is pretty small but you know when you start to try doing millions of operations per second it does add up quite a bit and the the model that we've chosen where we try to reuse callbacks and especially when we're talking to our these low latency systems like Bredesen memcache it really helps us keep the latency down and drive through foot up and then again keeping things single threaded makes it a lot easier when we actually do decide to go in and really optimize something it makes a lot easier to optimize these performance critical paths and there's a bunch of other things that gives us like for example we really don't have a concept of buffer pool and this is getting a bit more lower level but when it comes to just kind of like the really low level mechanics of how we decode requests and encode messages and things like that rail will keep things pretty simple and I guess the last thing is we have a very so one thing I didn't mention was that back pressure is a feature that's built into Colossus across the board so as I've been talking about doing all these like high performance interactions back pressure is a very important part of all that to make sure that we're not you know knocking over systems and we're gracefully handling situations for example when we overload memcache or if we overload another service making sure that we handle the back pressure correctly but in many cases when it comes to the actual internal back pressure itself we do have situations where we allow things so sort of a temporarily exceed their limit to sort of keep things running smoothly but still have like actual back pressure when things go a little too overboard and so the next big thing that I feel like factors a lot into our performance gains is basically the idea of optimizing for pipelining and this is sort of like the one the one major feature that we actually kind of looked at specifically optimized for and this is the idea where if you have a client or if you have a server you can send multiple requests at the same time before you wait for the first response and this is something that HTTP that the memcache protocol and the registro to call all support and it's something that we utilize very very heavily in our systems so for example when we have like our tumblr messaging system when we're sending all these requests to memcache we're pipelining them at a very high rate sometimes we're sending hundreds of requests all the same time before we wait for a response so by not just allowing Colossus to do those kinds of things but optimizing for those situations gave us a very large performance improvement and actually if I just go back real quick when I showed this graph when we sort of had this big upswing and performance that was when we were kind of doing that kind of stuff optimizing for pipelining and things like that and then I guess the last thing I'll touch on real quick is just sort of like the last bit of like micro optimizations this is when we really kind of go all out we kind of take a lot of you know the nice things that you can get in Scala and we throw all that out we kind of just basically write Java in Scala and this is something that we've done very very selectively only in like the hottest code paths but I feel like it actually has given us a nice improvement and the one thing I'll say is that when we have whenever we do this we're always optimizing with metrics so we're never just looking at something and thinking oh this might be faster if we did it this way we're always benchmarking and sampling and things like that to make sure that we actually have a really noticeable gain and whenever we do these kinds of things row is also making sure that we hide these optimizations from the end user so we're not actually you know decreasing the usability of the framework just for performance one of the big areas is just you know parsing HTTP requests it's htp is not the most efficient protocol when it comes to parsing so making sure that you're doing it as fast as possible is actually a larger part of our performance gain and I initially thought it would be and then a lot of the other things though using just regular ol arrays and using Java linked lists and place of Scala collections you always hear about Scala collections are not that fast and I would say most the time we don't really worry about that especially one especially like for example in our HTTP request parser we had to make sure to use just like a plain ol arrays plain old linked lists because it actually gave us a very noticeable improvement in performance and then I guess another thing to point on is there are actually a lot of really small little things that you wouldn't necessarily think it would lead to a performance improvement and I guess in most cases they wouldn't but again when you're in these really hot code paths that actually makes a significant difference so the one that I always think about is just just the idea of calling dot size sources dot length on an array and if you think about that dot length is built into the array itself but when you do a call to dot size you're going to trigger that implicit conversion into like a wrapped array and just that one little bit of overhead if you're doing that millions of times per second it really adds up and then a few other things I point out here just like composing partial functions and just using plain while loops versus doing higher-order functions like map and foreach so again these are the things that I would never recommend anyone do at first and I would never recommend you do these things and like user-facing code but when it comes like the very low level stuff that we've done it does help quite a bit and I'm just about out of time which is good cuz I'm basically finished so as I said Colossus is actually still under very heavy development we're not at at one point o released yet i would say we're we're still kind of iterating and going forth from there but as i've shown even at the state we're in we're using it for very large features at tumblr and it's working out for us very well so the next big release is going to be 0.9 which is currently like in the release candidate phase and has a bunch of big improvements probably the one i'd like to talk about just a little bit as the fact that internally we're now sort of changing everything to look like a stream so the way things have worked internally now is sort of just kind of building things up as we have worked on them but now we've had taken all these ideas that we've had about doing things efficiently with asynchronous operations now we're turning internally into streams and I guess that's about it we're always looking for feedback I think everything that we've built into Colossus at this point is really been to solve our own problems but now i'm very interested in hearing about other people trying to use it and seeing you know what are some things that other people are running into that we don't handle yet or maybe things that we could handle better and things like that so that's about it so i guess we have a couple minutes for questions if anyone has anything they were here sorry okay so you're asking the performance improvement we got from using just a plain while loop versus math and free oh I mean I I think I put what I put I said twenty percent I think that's a rough a rough estimate of how much of improvement we got from doing some of the stuff that it matters more in some situations versus others be I wouldn't I wouldn't say it was like the thing that made us fast but it certainly helped yeah face right so the question was and we go back to that slide the question was in the example where we are writing data to both reticent hbase how do we handle the situations where data is not in Redis yeah pretty much what we do is is will fall back to hbase and I will do right through caching whereas we're sending back the response will write data to Redis I should have mentioned that sort of the the plan that we do for most of our stuff we do like look aside and right through caching where if something is not in cash will pull from the database write it back to the cash as we're sending back the response [Applause] you