scale.bythebay.io: David Massart, Managing Nation-Wide Traffic Cameras and Sensors
Recording: scale.bythebay.io: David Massart, Managing Nation-Wide Traffic Cameras and Sensors
Hello everybody. So, I'm David Massart. Well, you probably don't know my company because we are really two people there and what we offer is consistency for uh for whoever wants us to help them with their data or big data problem. So, yes, and one of these customer is a company called Mac. And what they do among other things, well, they are hardware uh maker and one of their project is those traffic cameras. So, just to understand which way it was, so it's just a camera that will uh take a picture of a car and then there's some kind of OCR embedded in the camera that will uh build photo well, OCR the the plate number. The camera has also a GPS embedded, so they they know uh the GPS coordinate of of the camera and the timestamp and they they put this information in a in a message that they that they send. And well, usually all these cameras are really put at different places on the road and where the network activity can be can be poor
So, the pictures the picture itself is usually not sent with a message. Um the idea is that each time there's a message it's sent by the camera, but when when the camera can send this message and if the message cannot be sent or well, it can be kept in the camera for up to 7 days. So, that's the constraints we had to deal with. And uh if you look at the example of applications that you can well, what you can do with this information. So, basically a timestamp, a plate number and um and and and the and the location. So, we have things that's called blacklist which basically you have a list of wanted cars. And well, when the car wanted car is recognized, well, you know that the car was at some place and you can send this information. Um so, I've been surprised because you expect people being in part of this blacklist of wanted car to be relatively limited
So, it's when I look look at the the list for the first time, I mean, you have Belgium which which is a relatively small country with 10 million people and in this blacklist you have 1.5 million cars that are registered. So, it goes really from from I really well, from from terrorists with high priority to people who are driving without a without a driver's license or without insurance and all those things. So, that's relatively easy. I mean, if you have one camera, you know, you have this blacklist and you see the car to you have something. You can do other things like speeding. If you have two cameras, it's also relatively easy. Um it's really very simple math. I mean, you know the distance between the two uh the two cameras, you know uh um well, you you know the time that the car well, that I between two well, a car was seen at one camera and at the other and you know the speed limit
So, you know what the final the average speed is higher to the from higher than the speed limit or not. And so, all this information like um the blacklist or the distance between cameras and and the speed limit, etc. That's part of a configuration that you have to to add to a system. And then you have also things like overtaking where you have another segment with two cameras and basically it's a no passing zone, so you want to make sure that well, if people follow the rules, the first car in must be the first car out of the of the zone. So, that's that's three examples of things you can do with uh with with this. Um and okay, initially the system the this company had this was so, they were hardware company, they create the hardware, they create also well, the software that's runs this hardware that's embedded in the camera and some applications for their customer. And so, they started with a very uh standalone applications which was a MySQL database that was receiving all the messages and they were just operating on uh on this database. And with a solution like that, they were able to manage up to 60 60 cameras
And so, they came basically there was the the federal police in Belgium came last year and say, "Okay, yes, we we would like to be able to manage all the all the cameras in the country." And so, that's where they ask us to to help them to uh well, to to scale up to 20,000 cameras which is the estimate they want to have. And so, um well, so here's kind of solution we we came. So, the idea was to have some kind of something a lambda-like architecture and the idea of uh so, between a well, the idea of having a speed layer and a batch layer and that's mainly that was the the main reason for that was just to deal with these uh well, these network problem. So, the idea was that when uh when a camera is disconnected from the network and it happens all the time, well, we wanted to be able to well, first of all, sorry. First of all, when the when a camera is okay and just sending messages, you want to process these these messages as as fast as possible and that's the role of the speed layer. And so, when you have a camera that is disconnected, when the camera comes back um and send all these old messages, you don't want these messages to uh to interfere with the speed layer. So, they're just sent to this batch layer where they can uh well, they're registered and then you you can batch process them. So, it's just a um a security net where you you make sure you don't miss any event, but also you you yes, you use the speed layer to to very quickly generate the events
And to generate these events, we put in place the an event generator. So, as if you look at the diagram, so, the idea is to have this um I car connector which is just a way to manage the cameras and send the messages from the cameras to to Kafka and then Kafka is used to feed this event generator. And now I'm going to show you what what this event generator looks like. Okay, so the idea is of the event generator is really to have something that turns the camera messages into a potential traffic violation events. Um well, you just say potential because the machine doesn't uh well, say there is there's potentially a traffic violation, but it's the role of a human operator or the police officer to to just validate if it's really uh uh an yeah, violation or not. And the idea is to have a solution solution that's was extendable to support additional types of events. Not just well, the one I mentioned, but potentially new ones. So, we wanted to be able to easily plug um new new types of events and actually um to to be a resilient to generate events without downtime and to be elastic to to adapt to the level of traffic
And to do that, we we have implemented this this event generator in Scala with basically a mix of Akka cluster and Akka streams. Just a there's a nice library called Akka streams Kafka and we have a heavily relied on on that. Okay. So, I don't know if you've heard of this uh ma- master worker design pattern. So, that was a source of inspira- of inspiration, but yeah, it was already modified. So, in the original um model, you have a master that receive messages that consists of a list of tasks and distribute the task between the different different different workers. Here it's a little different because we don't have a list of of tasks really. Well, what we have is yes, list of functionalities more than tasks
And when a worker start a a task when it's a continuous task, it doesn't have a beginning or end basically. Um for the event generator, I mean if you have look at this worker one that does that process speeding, well, it's just taking events from the from Kafka, processing, generating the event, and sending them um to well, to another Kafka topic basically. So, this when this task starts, it's it never ends. So, the idea is what's more to have a some kind of configuration, the task list to give that to a master, and the master looks at okay, knows that he has to find workers to uh well, deal with a blacklist speeding, overtaking. And so, that's we use then workers. Each worker has a list of abilities. And well, you have some kind of a negotiation between the worker and the master to decide what what to do. So, the master, the workers are well, master is definitely a a character
Um Workers as a at the high level can be seen also as a as a characters, and uh the idea is we wanted really to be able to add a worker possibly with new functionalities, and uh add that to the network without having to uh well, to to change the configuration of the master. So, the way it works that the worker will just say, "Okay, yes, hello." Will find a master, and um contact it. Master will ask sir, the worker will say, "Okay, give me give me some work to do, and here are the list of my abilities." And then the the the master will say, "Okay, yes, I have a task that you are able to do." Or sorry, I have nothing to do for you to do at the moment. And if there is a task for the master, well, the master will uh ask the worker to perform the task. The worker will then start working on it, and notify the master. And so, starting to work on it it's really connect to Kafka, start reading messages from the cameras, process them, and generate events. So, that's that's the idea. Okay
So, since we were using Kafka, we we had to use Zookeeper. So, the idea was really for workers to find a um to to to find their master is was to to rely on Zookeeper. Actually, um well, we had plenty the the the architecture was one of the constraint on the architecture is it had to be very flexible because um well, it was sold at to two different customers. They are very well, they are mostly cities or municip- municipalities or regions that have very specific constraint, and the architecture was uh well, had to be flexible enough to adapt to whatever constraint was requested by the by the customer. So, for example, when we um deploy things in in Brussels, they wanted to um well, basically, they wanted to have two data centers, and one of their big acceptance test was, "Okay, we switch off one of the uh data centers, and we want to make sure that the system resumes and well, on the on the second data center." So, the idea was for the master to have some kind of um um well, um some kind of leader election using Zookeeper. So, I I don't know if you know it works, but each um candidate master actually will register in Zookeeper, get a turn, and Zookeeper will well, will well, it will Looking at Zookeeper, you can see which which one of the master was the first to register, and it becomes the master. And so, you can look at the list, and the idea was simply to put including the information that was sent to to Zookeeper, the address of of the master itself. So, that the worker in turns were able to by just looking at Zook- Zookeeper to see what well, what master was was the leader, the elect- elected leader, and um well, find the address, and then it was possible to to contact the this master, um and start the little negotiation that uh that I I described
Okay. Um so, now it's just looking at the the way we have implemented the the worker themselves. And um here also it's uh So, if you take the well, actually, it's better to to start looking at these diagrams from here. So, you have the worker here is that list of abilities, and uh the worker itself is a can be seen I mean, it's really an actor cluster. So, I don't know if you're familiar with Kafka, but Kafka you have a a topic so, a queue, and this queue can have multiple partition. And you need to have one uh one client per per partition in order to read all the data. So, the idea was for the worker to have a um for each uh for each task to start um um well, to have a consumer group for Kafka, and to instantiate in a in this cluster as many as as many instances um as you have a Kafka topic in on Kafka. So, so that that's that's relatively that was the idea
So, um the worker was just well, looking at his task for each task, looking at the actors able to do to perform this task. And they were instantiated so that they could read the data. So, one of the problem we had is okay, so um you one of the way you sort the data on Kafka um to to know on what topic the Kafka is sent is to use some kind of key, and the key we we had was um to use the plate number. It seems relatively natural. I mean, for a it's relatively easy. So, each time you have a a plate number, well, you the same the idea is that the same plate number will always go on the same um on on the same partition. And so, all the well, the same car, the information about the same car will always be processed by the same node, by the same uh by the same actor, right? So, that was that was kind of easy for the speeding because you have all the information about the same car. It's fine
And for the blacklist, that was fine also. You can divide the the blacklist between actors, so you have a always the same subset of the blacklist that is used by the the same actor. So, that's was easy. Where it didn't work so well was actually for things like the overtaking. Cuz in the overtaking, obviously, you deal with different cars. So, you have different cars, so you had no guarantee that a given overtaking for same segment will be uh processed by the same actor. So, Akka provides something like this called distributed data where you can have shared data between actors. We've tried to do that
Uh it doesn't work very well. And so, we had to uh to kind of cheat, and the solution we adopted was actually to create just for that uh that use case a a specific uh a a new Kafka topic where the key was actually um the segment well, the the the the no passing no passing zone. So, we were that by using a separate topic, we are sure to send actually all the data necessary to to to check for for overtaking to the same actor, and to avoid to share to share data between uh um yeah, between between actors that were potentially on on on different nodes of the of the cluster. And uh basically, that's uh that's it. That's the way uh That That's the way the um this this event generator works. Um so I'm the end of my talk. Um yes, that's this image. I don't know if just if you want if you wonder and then cool stuff when uh when you work on a in an hardware company, we needed a cluster to test things
So, we started the development on the on Amazon web services, but at some point people there say, "Oh, actually, we have all these motherboards from these cameras. We can just create a little cluster." So, they put six uh Well, they they create two towers like this with on each one six processors and you had a you had you had your mini cluster. You could you could experiment with. Thank you very much, dude. Um Do you have any any questions? How much time it takes from the moment that an average to the moment that you send the event from the camera or the camera that sent from the camera to like the end I mean, I guess you explained there like more or less what you really but what is what happens after that? You send to the database or Yeah. No, well, the thing is it depends of really the the type of of offense, but for things that are urgent, it takes it's less less than 1 second. Um Well, it depends. I mean, the slow part really is the part between the camera and the the Icar connector, the entrance of the systems
That's that's really depends on the network and sometimes I mean, sometimes we have fast network, sometimes it's very slow. In some cases, the the only way they use to connect the camera is to use some you know, the G4 phone network. It's really depends. But once the message is is in in the system uh uh the event is generated in yes, some some some 50 100 milliseconds. That was that was super fast. And that was one one of the requirements. Yes. What are you using to monitor the performance of Yeah, that that was we we um That that that that was I think the problem was never really resolved
So, so we we never had a really good solution for that. I mean, like a tool able to to do that. So, um usually when Basically, as long as things go well, that's that's fine. And when things become complicated, then we had to to start looking at logs on the different application and and track things. So, what we we've tried to put in place actually a way to a separate system that was look well, collecting the logs from the different from the different system, but it was totally ad hoc and made and uh provide well, some some extra metadata to to document that and put all the logs at one place so that we we could just uh look at those things, but we we really never had we I wish we We had and related Did you experience any stability problems while you were developing or scaling? No, it seems seems went pretty smooth I mean. So, we we had a few bugs. Oh, we what one thing actually Yeah, that's um the um Well, it was not really a technical problem. So, when we started we to deploy the the speeding module in a in tunnels, so it's a tunnel well, okay
Just a long tunnel. You have cameras at each entrance and every entrance and every exit. And you just measure the speed. And so, we we put the thing in place and well, it's run for the night and the day after the day after we look at the data and we had some extravagant I mean the speed limit was about 30 mph or something something like that and we had car uh well, especially one going at 150 mph or something. So, that was really the first deployment. Say, "Okay, there's really impossible to drive that fast." So, we had to send a technician look at the cameras, check everything, etc. Check also the system. Okay, we made a mistake somewhere
And no. Cool. Thank you. So, yeah. Okay, we've got to stop there, perhaps. Okay. Yeah, thanks.