scala.bythebay.io: Amos Schallich, Scaling the Internet of Things with Scala
Recording: scala.bythebay.io: Amos Schallich, Scaling the Internet of Things with Scala
you so i'm a mess i'm one of the co-founders in the head of the software team at euro and who here over the upcoming Thanksgiving holidays is going to have to run tech support for family or friends show of hands ok few people and how about on their Wi-Fi fixing their Wi-Fi for them a couple this is something that I've had to do myself many times and euro we're trying to fix home Wi-Fi so make it easier for people to use so this isn't an issue and today what I'm excited to talk about is how we've leveraged Scala and akka as part of our build system or part of our overall system to help with this goal and I'm excited to kind of share how you might be able to use this for your cloud connected IOT devices if you're building something similar or for your web services so what are the practical examples of how we've done this and another reason I'm excited is because i think these technologies are really really cool there it's why we're all here today so to give a background on what our product does is the the current one router model that people use and are used to covers really well one room within the house it covers your living room and maybe the room next to it and this is becoming more of an issues there's a periphery of Internet of Things devices that might be around the house or at the very edges of the house as well as people are streaming more video and they're streaming it not just in their living room but everywhere in the house you want fast internet everywhere so with that we work to solve that with having multiple devices that mesh wirelessly together and provide internet everywhere and so to make it so the setup for these is not even worse than setting up an existing traditional router we needed to have a cloud to really support and make it simpler to use and so using a cloud to back this system so i think our system is made up of three different components there's the hardware and software running on the the hardware and then the software running on it the euros that you would set up in your home in the match together there's the mobile application that people you use to set up their networks they use it to do advanced network configuration they use it to monitor and debug their networks and also to use newer features that we've been able to enable because we have a cloud such as parental controls and there's more that we're always working on the device is independently talk out to the cloud they are constantly talking to it it's providing them with their configuration and so the the cloud component is the most relevant to what I'm talking about today and that in part is because when I suggested using skull on the embedded devices my co-founder that was working on them flat out ignored me pretend probably wisely and so the cloud is what the devices talk to and send up their statuses their temperature sensor data or their speed test results and then it in turn sends them decisions that it's made it sends down run a speedtest download a firmware and these are all things that we've been able to do so why use the cloud and something that traditionally not had a cloud component why did we choose to go this way well the first two up their configuration firmware updates are something that we knew we wanted and we knew we wanted the cloud to manage especially as you have the distributed if you have a distributed system of devices within the home the cloud is actually really nice thing to be able to coordinate that and to be able to elect a gateway and own the configuration rather than one of those devices having to have it it's a way to offload decision-making so if anyone is working on something that has maybe less compute power or doesn't have the full context of what's going on any of our individual devices doesn't have much knowledge about the network as a whole the cloud can make decisions for it so decisions taking more than half a second well those should go to the cloud and the cloud can decide and it has the context of that network and all of our networks so hopefully we can eventually leverage machine learning across all of our networks to make everyone's network better and when deciding knowing that we needed to build a cloud and knowing that I was going to be the first one building this is there are three of us up front and building out the team we had to decide what tools would we be using and to decide on the tool and specifically the language we knew we'd be built building in AWS there's a set of constraints that I had to work against and these constraints led to requirements that the language needed to fulfill and so the first constraint is scalability these these devices and if you have automated kind of computers that are talking to you constantly they're much different than users and that they're always on and always internet connected by default and by design that's we're by definition even so we would have a huge number of requests and we wanted to be able to handle a large number of concurrent requests for every ichiro that we have makes at least one request for every 10 seconds so every 10,000 years you have you have a thousand requests per second and on day one we shipped over 20,000 so we knew that we had to handle two thousand requests per second day one and we've grown since then we we launched nine months ago so think of where where we might be we also knew that we had to be highly available so while your network is going to run without our cloud you're not going to be able to set it up you're not going to be able to make changes and that was a conscious decision we made because it would enable us to do a lot more but it was also something that we had to make sure that we handled so Scala works really well for these requirements it's highly performant it's super fast on the JVM it's compiled runs really fast it has easy-to-use threading models and ways to reason about what you're doing and so that's worked really well for the development team and its type safe and has good air handling primitives as all of you know but this is really nice when you have a lot of information coming in from these devices meta data speed test results diagnostic logs that you know at the very edge as you're receiving it that it's going to be well-formed all the way into your database I love Scala I really enjoyed working on it on projects prior to starting this company and knew that I would be writing a lot of it and so that did go into the decision-making process and while when we first started writing it we didn't know that we'd be using akka we had kind of thoughts that we might akka is now something that would prevent us from moving away from the language it's really enabled a lot of our technologies and a lot of the things that we do and so that's what I want to talk talk more on is how are we leveraging this to do things that I think are potentially different from how we would have thought about doing them five years ago or 10 years ago so why acha is I listen to parts of the two previous talk so this is a recurring theme I believe if people have been in the last two they know but it for acha we're using it to solve not the scalability of this availability piece but these actually very specific problems that are hard to solve or difficult to solve in other ways so the first is that we need to coordinate firmware updates across all of the Arizona network so a typical Network we sell three packs so it's going to be three devices in SF that might be two devices do the homes being a little small here and it could be up to ten and so coordinating those getting them from an unknown state into an own good stay on the same version this is the the coordination and the process management is something that a cur provides really well and I'll get into how in a minute we also want to track the status of all of the era's are they online are the internet connected and to do that that's these we use heartbeats that are coming into the cloud once every 10 seconds so we can notify the user in the application for example hey your device's online or no your internet is down and do that quickly and we want to be able to track the status of them all without spending a huge amount of money to scale or with without having it take down our cloud so we want it to be to be fast and then we want to detect win an arrow or a network loses connectivity and so this is the detection of an absence of something which is a problem that is hopefully something that people here are faced before and this is kind of our way of dealing with that we don't want to do a scan of the table to find of our full scan table scan to see oh these are the older ones that haven't been updated recently we want to have something that can find something is missing so how do we do that what is akka I know the two previous talks talked on it I am guessing a lot of people know about actors and akka but I'm happy to do a quick run does anyone here not know what actors are akka don't be shy raise your hand okay I'll do it I'll do a brief intro to them so and and akka is is a tool in Scala or a toolkit and it provides actors and similar to erling and so actors are a concurrent computational model that is based on message passing so you have an individual apke or actor in our case every euro that we have so every device out in the wild has its own in-memory representation in the JVM and it's it's actor and it's a single threaded model and it has the state of the device and so that and it's also because it's a message processing system it's also a queuing system so you have the messages come in it handles them one at a time which handles the synchronization because it's single threaded and it also it can have this state that's being updated along the way you can actually also do in a functional way which we do do but you can also use bars or mutable state so how would a heartbeat for us which are these ways of us knowing the Vice telling us hey we're okay it's hard to reach out to advice because they might be matted how does that look so you have a hop through your network every euro as I mentioned it's talking individually out to AWS I had a cloud outline here did not show up on the screen so you hit H a proxy which is our load balancer it directs it then okay it's a heartbeat I'm going to send you to this is more of a hypothetical how you might do it with if you weren't using actors sends it into the controllers they talk and write the heartbeat to the database maybe read the database to look for information then goes back out what our model ends up looking like is the same number of hops approximately and but without that database call you're not going to have to go to the database every time you're coming in it does add additional services which potentially makes your system slightly more complex but it's really made our developer lives much easier they really appreciate having this model for how they can think about these these devices in the state of the devices and how to manipulate them so digging and actually let's go back so now we're here we have our actor clusters digging in and going in and let's say we zoom in there so what does a heartbeat look like so it's coming off the device it's hitting in our system so it's hitting each a proxy it's being directed to one of five nodes interact or cluster these are actually crossed two to availability zones and they're fairly underutilized so we could out probably get away with fewer but we're adding in the redundancy so we don't go down so you hit your your actor cluster node each of which for us has five shards with 450 total shards across the five but you hit one actor cluster API from the the API that's where you take this message in this request you package it up into a message that you're going to then be sending to the actor and figure out where the actor that you want is and so you're using the the ID of the device to then go down go down to the actor and then the actor might write to the database of might not these were actually totally fine with there being this extra hot potentially between machines part of the time because it's much faster to do a network request a quest even across availability zones than having to go to the database for every request and here is really bad pseudocode but hopefully to give up you people an idea of how what does the actor look like for us how are we using this understand the state to do the the detection of absence so we start our actual actors are going to have a lot a lot more state but the relevant state here is that you have the last heartbeat and you know when you'll a threat to the database you have the receive function which is defined for akka and it's where you send your message and then it has pattern matching to figure out what is the type of message so here we have two different types of messages we can have we can have you there a heartbeat or we can have a check alive and in this case the ero sends it in its ends in a heartbeat it's ending up in this actor so the first thing that the heartbeat does is it updates the last heartbeat time and it checks if it's written in the last 30 minutes and 30 minutes here is somewhat arbitrary but it's much better than writing at every 10 seconds and so if it isn't there then it writes into the database it updates the last right time and it goes on this is saved as well so there's 6 180 fewer rights to the database because we're writing when things change if a Nero comes online which isn't shown here and it's been offline that'll get written as well but it's so it's writing the sads it's alive and here's the heartbeat we also as part of this when you want to know if the device is on and when its last heartbeat was because that's not in the database because it's not a memcache to this other layer are any users using the app it'll actually reach into the actor because the actors are going to have anything blocking and grab out the heartbeat there so this next piece this cancel scheduled and schedule here your we canceling a job that has been scheduled and would otherwise be coming back in because we know that we had a heartbeat we don't need to to wait and then we're restarting the schedule check alive and it's a 45 second loop so after 45 seconds if you haven't had it reek an saled it's going to come in and it's going to hit this last case called check alive do is double check to make sure there hasn't been a heartbeat and then it's going to write that hard p of the day base and potentially send a push notification to a user or other inform our users or us that this device has come offline so this is even having the scheduler running for every single actor we have and having it get cancelled and we started all the time is actually very performant we haven't had any issues with this and it's it's been a great way to to solve this somewhat interesting problem the next piece that I mentioned well and also through through these hops to the database because it's writing to a much less frequently we launched when we first launched we were writing everyone to the database and that was killing us and so being able to remove these has really both saved us money and heartache and potential downtime but also just made our lives easier so for updates the other part so how do you coordinate from an actor so in this case we have two different actors and the fact that actors can talk to each other well and coordinate amongst each other is key here so we have our network actor and then we have our euro actors so the network actor would have no which which devices belong to it and which ones it needs to update and then you have the eros out to the side so the network actor might get an get an update or an upgrade action which has the firmware version needs to go to and then it's going to send download actions to each your actor which then have a reference back to it so they can always say okay I've succeeded they send them down to the eros the ears might succeed one at a time it's not necessarily all going to happen same time downloads can also take up to 45 minutes on slow networks so you're wanting something that is going to be long-lived and not take a memory when it's not in use and then we fail which sucks but it actually happens moderately frequently maybe it's corrupted maybe something happened when it was downloading it so the network actor doesn't want to have the network reboot to apply this update it wants to retry so it maintains the state it knows that these other ones are ready to go it updates the actor and then it succeeds so from that success then we're able to stay reboot and get the reboots coordinated down to all of the devices an automatic update might happen in the middle of the night and take your network down for 30 seconds as its rebooting and you want those happen at the same time you'd want to be rebooting in at different times so from this I the key kind of learnings that we had is we were building the system which are some cautionary tales they want to make sure that your if you're using an actor cluster that you're always maintaining well depending on how you deploy but you're always maintaining backwards and forwards compatibility with with your actors and you're in essentially your persistence and and how they're talking to one another and so for us we're deploying because we don't want downtime we're taking down an actor one in the taaraq I'm an active cluster node one at a time it'll be running two different versions you need this backwards and forwards compatibility if you take them all down if you have to you can take them all down bring them back up and break that you want to avoid blocking i/o in an actor's execution context my pseudo code did not follow that because I was right into the database there but there's ways around that so you can put the database call on us in the future that you don't actually care about the result of that which we do do which we do we we don't we don't wait for the result there you don't want to allow connected clients or devices to synchronize if you think of a request taking maybe even 500 milliseconds so half a second and in a 10 second period which is our heartbeats intervals there's 20 of those if your devices end of synchronizing and all of them are hitting at one of those 500 millisecond periods you're in trouble you have a much higher load than you should if you have it spread so that actually did happen we tried to avoid it and it still happened don't allow our site and measure everything you don't you don't know where the the bottlenecks are going to be I thought up front that maybe play was going to add an overhead and it adds none so how do you make sure that you don't have overheads and so the cake away from this is that hopefully these are things that you can apply to if you are doing IOT devices are connected to cloud may be very relevant otherwise hopefully still relevant to your web services so this might be how do you do absent to detection how do you find that something doesn't exist as we do with our heartbeats and our statuses how do you do process coordination and and make that work really well for you and how actors talking to each other and maybe it can be different types of actors and how do you make sure that you're really performing our our 75th percentile time for these heartbeat requests to our cloud right now is four milliseconds and that is with multiple hops across different availability zones in ec2 so I I think that we hopefully you can apply some of these concepts to your work I've enjoyed sharing them and they have really enjoyed building this out and I think we've defined a bit of a paradigm of how you can do cloud-connected IOT devices and well so thank you [Applause] you