SBTB 2019: Vishakha Gupta-Cledat, Machine Learning's Missed Opportunity in Visual Data Management
So I guess there's a lot of pressure since it's before drinks. I will try to keep this interesting. What we want to talk about today is how the data part being missed is actually a missed opportunity for machine learning. And I'll show you some examples and why we believe so and what we are doing about it. So back in March, a biotech AI company reached out to us. They wanted to store the images that they were dealing with, the metadata surrounding those images. They also wanted to store machine learning artifacts, and by that I mean feature vectors. They wanted to perform search using all this information
They had constraints on scale, performance. They wanted to pre-process the images as they were being accessed. And they wanted all of this stuff to be done by one system. Now, they did not find something off the shelf. In a world without Aperture data, what you are looking at is most of the ML engineers and data scientists are forced to put together, you know, Frankenstein collection of components where they have to put their metadata in one among the platoid of database choices that they have. It could be relational, graph, key value store, which may or may not evolve easily with their use case. Then they have to store the data somewhere. Could be one of the cloud storage, could be one of your distributed file systems
Uh, they have to establish connection between that data and the metadata. When they look for how do, how could they store feature vectors and have a persistence index to do similarity searches, they come up with an empty box of options. Not only that, uh, because they don't find all these solutions that meet all their needs, they have to put together point scripts that often lead to suboptimal solutions. And so the problem that our customer and a lot of ML engineers across different companies and different application verticals they face is that the intersection of machine learning and big visual data is a big unsolved problem. And before we, I go into how we solve or what we do about it, let me give you an idea of how big it is. If you think about the fact that 80 percent of the data on internet is visual in some form, so images or videos, and the fact that machine learning is already a 29 billion dollar market and it's growing at 36 percent annually, we are looking at, uh, millions of dollars wasted for a small business because of, you know, the time and productivity loss and delay in go to market. You, uh, it, the loss is bigger for bigger companies and if you add across the different application verticals and over a decade, we are talking about billions of dollars lost because of the lack of a good data management solution which was designed for machine learning and visual data in mind. So, Aperture Data, at Aperture Data what we want to do is solve the storage problem keeping VisualML in mind
We recognize the unique characteristics of, you know, images, videos, the importance of metadata surrounding it, and the characteristics of visual machine learning. So, what we offer is a way to store your, not just images, but your videos, other blobs, then the metadata that surrounds them, machine learning artifacts like feature vectors, into one platform, and we enable efficient and scalable search, and we also offer a way to pre-process this data. And I have demos to show how we do that so I can familiarize you with the API. And the one thing that we have focused and tried to build is a machine learning aware, unified, and simple API because our goal always has been to simplify the lives of ML engineers who are trying to put these, you know, complex platform engineering solutions together. Our platform, you can picture it fitting between, uh, the, the data input layers of like, you know, we can get a data from streaming things like Kafka, or we can, uh, read data from, uh, the cloud storage buckets or your file system. And we can the output of, uh, acquired from our system can be fed to the machine learning frameworks like Spark, uh, TensorFlow. We could also interact with Beam. Um, so, this is where we fit in, and after actually having been deployed now at the customer, we have been able to save them over a year of full stack engineering time
They're already running our platform with over 800 million metadata entities loaded in it, uh, with over 100 million data entities loaded in it, uh, with over 100 million data entities. So, we have been able to save them over a year of full stack engineering time. Uh, and in the performance experiments that we have done, by putting together state of the art solutions to approximate the portion of the functionality we offer, we have, depending on the use case, seen a variation about 2 to 10x performance performance, and the performance experiments that we have done by putting together state of the art solutions to approximate the portion of the functionality we offer, we have, depending on the use case, seen a variation about 2 to 10x performance performance, and we have seen a variation about 2 to 10x performance improvement by our system, uh, compared to when doing complex queries and, uh, doing data pre-processing. And I'll show you some sample data as we go along. So, uh, what I have done is built a demo in the order of, like, you know, how we handle more and more complex data types, and we've used different application examples to show you how it is, uh, our platform is can be applied across different application verticals. Uh, the other thing I would like to point out here is that the project has, the project that we have built on top of, Visual Data Management System or VDMS, has existed for some time, uh, we have been working on the project, uh, and we have been working on the project, uh, and we have been working on the project, uh, and we have been working on the last year. So, Aperture Data Platform is actually built on top of that open source core, uh, and the project is on GitHub, and, um, towards the end of the presentation, I'll also say what we plan to do as part of the company itself. So, um, let's see
So, you know, knowing Murphy's Law, I actually embedded the videos of us running the code here. What we did was we used, um, cancer image archive. So, first example is a medical imaging example because that was like our first application we used. Uh, we preloaded data for patients and their medical scans. So, we were, we had access to, like, um, uh, brain scans from this cancer image archive. And what you see here is that, um, you can import our, so VDMS is the open source package and you can import it in the Python, uh, Python notebook. All you need to do is connect to the server and the simple query that we want to start with is finding a patient. The first thing we want to do is, you know, just a simple query like you would do with any database of, you know, get a patient with a constraint on what was the age at which they were admitted
So here, like, let's say 85 years and, you know, the properties we want to access are like some patient barcode and their, uh, date at which they were admitted. Now, when we execute this query, it returns, and, you know, this is my question, we're going to try to play with him. Uh, so the, uh, return, it shows, well, there were two patients that met the criteria. You could just go ahead and change that criteria, make it, you know, over 80 years, and you'll, of course, see more patients that match, uh, that information. So far, so good. We haven't done anything special. It's just regular metadata queries. And you get the results that you need, and it's a good thing
So, uh, you can see that, uh, you can see that, uh, you can see that, uh, you can see that, uh, you can see that match, uh, that information. So far, so good. We haven't done anything special. It's just regular metadata queries. And you get the results that you need, and it's in a simple JSON format. Now we want to do something more complex. We want to use the patient information. We want to use the fact that we recorded their medical scans, and then we want to access the actual images themselves
So that's what we'll build here. And while we are accessing those brain scans, we want to pre-process them as they come to us. So we want to get them at, like, some particular resolution, even if they were originally stored in, like, a much higher resolution. So in this case, the first query, the first command that you put is the same. You access a patient. And now you actually do some connected query to it. You find the scan belonging to the patients that met your criteria before. And the next thing you use is our find image command
So we've defined these commands. And we can apply constraints on those, you know, you can choose to put whatever number on the images. And the interesting part is these operations. You can specify, you know, you want it resized to that size. So when you run this particular query, it'll get you images belonging to the scans of the patients that met the criteria you were looking for. So, you know, this is what we had in our, in the data that we were managing. You could go ahead and change the resolution again because, you know, different displays require different resolutions and you can change that on the fly. You don't have to store copies of that
Um, so this is like a classic example of what people have to do. But to do this, they end up having to combine together, you can already count something for metadata, something for data, something for pre-processing the images. At least three different solutions that you have to put together if you don't use something like Aperture Data Platform. Now, we also used this data set and like a, and we did a test with, you know, what I'm calling here the Frankenstein system which is basically the combination of different, uh, different state of the art solutions. Um, which was like an in-memory relational database, the Apache server to serve the images and OpenCV to run the pre-processing. And then on the right side is our open source code from back in December last year. So the new numbers are more customer internal so I couldn't show them here. But even with this, if you look at what the queries are doing, they are getting more and more complex as we go around
Uh, this particular query is accessing 1600 images and remember we were trying to resize them to 100 by 100 as we were accessing the query. So we have the query that we have to do with the query that we have to do with the query. So we have to see the query that we have to do with the query that we have to do with the query. So we have to see the query that we have to do with the query that we have to do with the query that we have to do with the query that we have to do with them to 100 by 100 as we were accessing. So the more processing you go put closer to the data the bigger the benefits you see. And we have seen like up to 10x performance depending on how complex the query was and how much data it was accessing. And not only that, there is something that gets ignored very often when you store data in a certain like you know full resolution and you take that data along and pass it to something like something like PyTorch. The first step typically is of a pre-processing where you resize it down to whatever your neural network can accept which is often lower resolution than what you stored the data at
So you transferred the full res to a different system and converted it down to lower. In here we measured what the difference was in terms of just the pure data transfer with that Frankenstein system and our system. And we actually you know we kind of suspected this would happen and this kind of proves our point that there was a 63% reduction in just the amount of data transfer. And we actually you know we kind of proves our point that there was a 63% reduction in just the amount of data transfer. Uh when you did all this processing near the data. Now imagine that for a much larger scale system when you are transferring millions of images around or you are in fact dealing with videos. So these benefits add up and these are the kind of things that have gotten kind of sidelined or hidden or they haven't come up to the front because of you know the way we have been dealing with data so far. Uh now I want to build like you know something more complicated
And this is something that's novel to our system where we provide a persistent index to uh index feature vectors. Uh using you know uh we have our internal implementation and we also provide implementation with the Facebook face library. Uh just adding the persistence to the index. So in this demo we have actually used a hiking example where um some some friends from my co-founder they went on a hike. There are pictures of that hike. And we have feature vectors extracted externally that are stored in the system already. What we are going to do in this case is we are going to take a new image that we have not seen before uh extract feature vectors from that image uh to recognize you know to basically associate the feature vectors with the different faces in that picture. And then we are going to find someone similar to one of the people who um okay except I need to play this
Um I think I already spoke to it assuming it was playing. All right. So we use this library on the client side uh that can help us extract um that can help us get the faces and extract feature vectors from them. So when you execute that this is the new picture we have not seen. These are the four people in the picture. And uh these are the faces and we have extracted feature vectors. And what we are going to do is from the tall guy whose name is Tom, we are going to use the feature vectors uh feature vector of that person to find uh to do similarity search that our platform actually supports. So as usual you know the first step just import our packet import the VDMS library and connect to it
And after one second. Yeah. So we connect to the library and then we use this descriptor. And now we use the find descriptor command where you really want to find the k-near neighbors. So what we can let you do is find k neighbors of the given descriptor. And here we are asking for four descriptors and we are asking for the distance between them, the ID that we stored them as, and what label was associated with that descriptor. Because you know you classify the descriptors and you like label the cluster so that you can actually return labels and find who this feature vector belongs to. So when you execute something like that and you pass along the feature vector as um as the blob array, what you get is there were four other feature vectors stored in the system that were close to this person uh and the label or that all of them had was Tom
So in this case now we want to make it more complex. Now we want to go find images and we want to see what images had those feature vectors. So we'll again go ahead, do the find descriptor just like we did before, and we can actually again request the distance uh for the same and we use the same person and we give the five- five neighbors this time. And we want to go connect that to uh the- the image that this feature vectors were extracted from. And we can again specify uh the operations to perform. So here we are again saying okay let's just resize it to uh to 200 by 200 whatever the original image might be. When you execute something like that uh which is building on top of the first query we did, you get actually four images where those feature vectors existed and that uh and the tall guy Tom is there in all of those pictures. And you know you can decide on the fly you want to see uh a bigger resolution of it or a smaller one or a smaller one or a resolution and there is nothing- not much required other than just changing those parameter
And to- to- to you know our metadata is stored like a graph so we can actually do a lot of neighbor hops. So in this case we make it more complex and do some more neighbor traversals where we first find the descriptor, we find the entity associated with that descriptor or the person in this case, and then we can see the and then we can actually go and find out im- all the images of that person. So even if you did not have descriptors extracted, you can get all the images that belong to that person. And we build our calls the same way. You know you find- you use the descriptor, you find the person, and you go ahead and find images and you get some of the properties for that person. When you execute something like that and you again do some- you know and- and you can specify operations at any point in time and you see the pictures of this person from even like you know from like the baby time for example. Because we had labeled that person there. So this- this is a pretty powerful functionality that is not supported in a lot of cases and this is something that we have actually even been able to use in let's say like a retail example when you want to find who this person is or have you seen this person before in another zone
And now to build up on something that's not- you don't really see videos being handled as well everywhere and you know the machine learning on videos has been such a challenging topic because it's so hard. Every time you have to transfer the entire video even though there are few key frames. Uh- so what our platform lets you do is actually extract and manipulate frames. You can get intervals from the videos and we demonstrate that with the media or like a small video example. It's a video clip from the movie Megamind where- this time I remember to play- where essentially it's- it's a ten second clip and we just want to show how you can actually change parameters of the video or access some interesting frames. So this is the original video that you can just read and play. So the way you would deal with our system is basically you'll read that video in a buffer. You'll connect- you've already connected to the server and now we have a command that lets you add a video
We recognize that part. You can associate metadata with that video so you can put in properties. It could be the name, whatever type that video has, any other information you want to store around the video. Um- and then you can also specify different codecs and containers. So you can also specify the codecs and containers. So you can also specify the codecs and containers. So if the video was in a different format, our server takes care of transcoding them into the right one before it stores those videos. So you pass along that blob and this- this query string which is written as a JSON string to the server
When you execute something like- when you pass along the blob and you execute that, it essentially will like you know return a status to you so you know we have- we- all of this stuff is documented on the docs page that I have linked in- in the different locations here. So um- now what we want to do, now that the video is stored, we actually want to find let's say frames 20, 50, 100 because you know that those frames are interesting. So we provide a find frames command where you can list all the frames you want. Um- and you can also provide constraints to find you know so which video do you want the frames out of which is where you provide the content. So you can also provide constraints to find you know so which video do you want the frames out of which is where you provide the content. So you can also provide the constraints. And you can also specify when the frames are returned do you want them as JPEG, you want them as PNG and the server takes care of transcoding those things. When you execute a query like that, essentially see now you get three frames because those are the ones you requested
You did not have to transfer the entire video to the client and the server took care of doing all that and if you have a query like that you can also provide constraints to find you know so which video do you want to see PNGs. You can do that. The size of the return frames has changed because PNGs are larger. Now there are more interesting things that you can apply on the overall video not just the frames on the overall video. You can actually apply the same sort of operations like you know the resizing and stuff on the video where you can say well I want to put constraints. I use the find video command. I want to put constraints to find my video. And when I do that I want to like resize it while I'm fetching it
So you know here this is the way we provide the constraint on it. This is what we save for the whole video. And in addition to that operation, in addition to the resize of the video, the frames in the video, uh to this particular size, you also want to say that I want to do this resizing on a specific interval. So for going from starting of 100 frame to 200 frame, that's where I want, that's how much of the video I want in this particular format. And the other thing you can do is you can drop frames in between. So that step lets you say return every 8th frame because sometimes with the video, you can drop frames in between. So that step lets you say return every 8th frame because sometimes with the video, you can drop frames in between. So that step lets you say return every 8th frame because sometimes with training, that's what you need
You drop certain frames in between. And um, you can fetch that. So when you execute a command like that, what you get is a video of, you know, it's shorter length. So you see it's larger than, the frames are bigger because you asked for a resize, and it's shorter length because it was a fixed duration. And when we play it, you can see that you can see that you can see that you can see that you can see that you can see that the frames are bigger because you asked for a resize. And it's shorter length because it was a fixed duration. And when we play it, you can see the sharpness of it because you drop certain frames. So it's not like the, you know, uh, as the original one
But, you know, all of these features combine, they make for a really powerful system that can let you manage all your data. And this is like, I just showed demonstration of visuals, things like images, videos, and feature vectors, and the metadata surrounding them. So you can see that the frames are bigger because you asked for a resize, and it's shorter length because it was a fixed duration. And when we play it, you can see the sharpness of it because you can see that the image is shorter length. So you can see that the image is shorter length. So you can see that the image is shorter length. So you can see that the image is shorter length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length length So, I'm wondering whether the, um, what is it there? I'm wondering if it's there? um, what is it? I'm wondering if it's there? um, what is it? I'll be here at the happy hour time so you can feel free to ask me then. Okay
Thank you. Okay. I'm wondering whether the, um, what is it there? Okay. I'm wondering whether the, um, what is it there? Okay. Hi. I'm wondering whether there are any algorithmic issues that are specific to images or visual that is holding this area back? And, uh, do you see any kind of hurdles there or new developments that might provide more breakthrough? So some, um, so one of the reasons why a lot of this stuff was not being handled was that there was so much to do on the compute side to even make deep learning fast and plausible on a real time. So a lot of the focus tended to be on the compute and scaling that. Um, and we just happened to like be working on data at the time
So we now are at a point where people are starting to notice a data problem and we have a solution. Um, the other thing is there are certain tricks that you can play with the format because now if you think about it, you're no longer doing image and videos just for human viewing. It's becoming more of a machine viewing so the constraints are different, the compression techniques can be different and we can play with all those things behind the scenes now. Um,