scale.bythebay.io: Manoj Mahalingam, Democratizing data with an internal data pipeline platform
Sure, how is everyone today? It's a weekend, so thank you all for coming. So I'm going to be talking about democratization of data. So just like how Askar on Thursday said that this talk is about Elmification of ML. This is one of the fancy titles that I could come up with for the proposal, but the title is actually going to be why and how we built an internal data pipeline platform at Indyx. So I'll start off by quickly introducing myself. I'm Manoj. I'm a principal engineer at Indyx. I've been working with Indyx for close to three years now
Before that I used to work for a company called ThoughtWorks. So all this has been based out of Chennai, India. So I'm born and brought up and did all my schooling, everything at Chennai. So before I go ahead, I'll introduce Indyx a bit because what we do is also gives you context around why we did some of the things that we are doing. So at Indyx we see that there are six business critical indexes. The first is around people, Google's data. Obviously there is Facebook, Twitter, LinkedIn for that. There is places data for which we have Google Maps, your maps
And there is index of documents which is Google and Microsoft with Bing. And there are businesses with Yelp and also LinkedIn. And there is also going to be an index for IoT devices. And the last missing piece that we see is the index of products. So we all know about Amazon, eBay, back home at India. We have Flipkart. So all these stores have built their own product data. So we want to be the one single source for all the data that is present in the world
If anyone thinks about product data, can we be the source for that data? So we also like to think of us as being the Google Maps of products. So Google Maps enabled someone like Uber to build their platform on top of. So can we be the Google Maps for some companies which are trying to build their products based on the product data that we could provide. So towards that we have like catalog data of around 2.1 billion product data with around 120 billion price points. So quickly look at the data pipeline that we have so far at Indyx. So this has been evolving over the last six years or so. And I've been part of it for the last three years. So obviously most of the data that we do have comes from crawling websites
So we crawl brand and retailers stores. We crawl around 30 million URLs a day. And we also have our partners and customers who provide us with their product data in form of feeds. So we combine these data into our ML data pipeline platform where we do things like deduping duplicate products and categorizing products into a single taxonomy that we have. Add more data like extract brand information and standardize that. Make sure all the attributes and the keys that are associated with the attributes are standardized. And also match product across stores. And then we make this data that comes out of this platform
This data pipeline as data that customers could use. And we also make this available as product catalog that people could get a feed out of. And we also index this data for our API. So this is like a full picture of the data pipeline at Indyx. So this middle part is the core meat which I'll be coming back to in with reference to the internal data pipeline platform that we have built. So this is our production system. And this has been sort of similar for the last few years. So let me start off by defining democratization of data
This is my understanding of it. And yeah, I did come up with a fancy title. I have to define it before going at it. So what I see as democratization of data is can we enable everyone in the organization to know where the data is and then enable them to access the data and then use that data for any business decision. Use it for serving the customer needs. Start using it for any research and experimentation. So that's what I think democratization of data is. At Indyx, we do have a lot of data
So this is a look of the scale of data at Indyx. It can be anywhere in the range of different companies. We are not at Twitter or Google scale, but we do have a lot of data. We have 2.1 billion product URLs. And we crawl about 8 TB of HTML data every day. Every day. And at the bottom, you see that we have 120 billion price points from the data that we have collected in the last five, six years. It's something about the file itself that when it goes into full screen, the resolution doesn't come out
Okay. It was fine for some time. I have a question. Is there any animation here? Yes. Okay. So there's animation. You can turn it off though. We can try changing the laptop, but it might be an issue where we can't go full screen
Okay. In which case, maybe we can- Just don't go full screen. Yeah. If we can do the presentation without going into full screen. Okay. Fine. It should be okay. Sorry, guys
Seems to have some technical place. So I'll continue like this. So we have data in different shapes and sizes. So data could be like, since we could do crawl a lot of data, data is in form of HTML pages. And then we convert them into different form. I'm still audible, right? Yeah. Okay. Sorry
So it could be in thrift and parquet, HABRO and all that. So and we also have data in form of CSCs and Google sheets and so on. And data sets can be in TBs. So we crawl about 80B per day, like I said. And it could also be in a few hundreds of KBs because we do get in data in different forms. We work on data, filter them out. We have ML folks who get data out, filter them, sample data and so on. So it's going to be in different shapes and sizes, right? So what we were feeling was the data's full potential could not be realized
So we have all this data and we all know that data is the potential power, right? But data that's not being used might as well not have been there. So how do we make sure that while it's still serving our production system, how can we make the data to be put to more innovative uses? How can we enable anyone in the organization to come up with a new idea? Okay. I thought about this. Can I look at the data to see if my idea is worth trying out? Does the data validate my thinking? And so on. So can we enable such a workflow? So I'll go into detail of what were the problems that we face. So that drives into what the goals were for our Internet Data Pipeline platform. So the data wasn't discoverable. So for example, when I joined the company, I knew that one particular data which was like full data that we had for our products data, what we call as the mass data, was located on HVS
And then after a few weeks of getting to know how to use it, because I was very new to all the systems, and then I realized that it's not in HVS anymore. It's been moved to HDFS. And then some point down the line, it was moved to S3. Since I was not really working with the team that was taking care of it, the production systems are all fine because a single engineering team is taking care of all that. But me working outside that team didn't really know where the things were. So if an engineer like me was facing such problems, I'm sure that our product management, customer success, and even our CEO, they want to look at the data, it's going to be near impossible. And the next problem was knowing the schema, right? Like the serialization frameworks like Thrift and Parquet solve it to an extent, but you need to know exactly what the schema is to be able to read the data and also know what the different fields are in it. So while some of the like, while Thrift and Parquet Avro can solve the problem to an extent, but especially for non-engineers, it's not going to be trivial to know what's within the data
Okay, I know the data is there, but do I really know what is within the data? And also, we don't want everyone to be writing code, like writing code, especially scalding code for our Hadoop jobs or for even Spark. To a large extent, it's not very easy to write code and then ship jars, put it into a cluster, start running the jobs, and so on, right? And even if you have engineers who can write ready-made jobs and make them available to others, they cannot handle all the different use cases and it will definitely limit the time, like limit your productivity to get looking at the data and start using it to what you want to use it for. And definitely, as a startup and for even for many companies, cost is not something that you can be easily saying, like yeah, this cluster is there, you want to run your job, you have your data, just start running your jobs. You can't easily say, just go and do that because we don't know what the consumption of resources is going to be, you'll have to take care of, auto-skilling, you'll have to take care of how much resources are being used for each of the nodes, what the node types are, what are the cores that are going to be assigned, the memory and so on. And if someone is, and if we as engineers say that to others that please be aware of all these things, they'll probably say, man, I'll just not use it, right? So, how can we like abstract this behind some system which will take care of the cost for us, which will make sure that we are running it in some same defaults while also giving power users the enough knobs so that they can do changes and start running the jobs that might need more power, more resources for that. So, with the problems specified, so these are going to be the goals of our internal data pipeline platform. Can we enable discovery of data? Can we allow the schema of the data to be introspective so that people can know what's exactly in the data and also what's the format of it and then and they can know like which is the fields that they're interested in, what the type is and so on. And also, can they do it without having to do a lot of code
So, that mean even if you can enable a SQL based workflow for certain more coding like activities and if you can have other built in transformations that people can just drag and drop and start using, that's the kind of workflow that we wanted. And we wanted it to be, like I said, it should be a UI and a wizard based one so that people are driven into making their choices rather than having to know exactly what is to be done. And can it also utilize the resources that are being used in a more cost friendly manner. And yes, we are building an internal platform, we do have a separate data pipeline platform, data pipeline ready which is serving our production needs. Can this also be a production platform where anyone can configure bits and pieces of our data pipeline and then start use consuming only those. So, if I go back to my data pipeline slide. So, we have different components here, right. So, can someone just choose to do the classification for some data and then provide that data to a customer because some customer might just want that data instead of having to wait for it to go through the entire pipeline
So, can we start even addressing those use cases. So, towards that we built a platform called what we internally called as MDA. It's like marketplace of data sets and algorithms. So, we started off with a very big and ambitious name for it. Like we wanted to be the marketplace of all product data and algorithms around it. And we even wanted other people to start coming and using like submitting their own data, submitting their algorithms to the platform and start running their pipelines. So, that's been the biggest big vision that we have for it. So, we are not close to the marketplace aspect of it yet
But what we have seen it is internally it's become like a great platform for internal data pipelines and yeah, it's really democratized our data. So, I'll quickly look at the tech stack and then I'll jump over to a demo of the platform. So, we at Index are a big Scala company. We have been using Scala ever since we started. So, we decided to use Spark for so we have been using Aadoop and Scalding for some time. But we see that Spark enables lot more things that we could do with our jobs. So, it was an obvious choice to start using Spark. And we built the backend with the microservice based on Akka HTTP talking to a Postgres database
And so, all the infrastructure is deployed on Kubernetes which is spun up using Ansible and Terraform. And recently we even started using GraphQL APIs from the Akka HTTP using Sangria. So, which enabled us to evolve the UI in a much faster way. And we use React for the front end code. So, I'll jump over to a demo of the product. Any questions so far? So, this is the main view of MDA. So, we have come up with the concept of pipelines, data sets, schema, and namespace. It's these are all terms that everyone uses in this area
So, I'll start off with the easiest one namespaces. Namespaces we introduced so that we can logically group every entity in the platform. So, that we can separate the entities across teams, across customer use cases and so on. So, that's the easy one. So, let's talk about schema. So, one of the problems that we said was the schema wasn't known. So, schema of the data is what will define what is within the data. What are the different fields? What is the actual format? It is in is it a parquet? Is it a parquet schema or Avro schema and so on? So, I'll go into schema first
Start off by creating a new schema just to show how the schema creation looks like. So, just going to create a test CSV schema and give a version to it. So, in the drop down, you can see that we provide options for different schema types. These are all the schema types that we support, which is what needed for our platform. It's going to be thrift based, CSV based, parquet Avro, JSON, TSV and so on. So, I'm going to create a simple CSV schema. So, any entity that we create in the platform is essentially a JSON config, but since we wanted to be it to be a platform where anyone can create things from the UI and don't really have to know how to write JSON properly, what is actual format of things are going to be. We also have a form-based UI where you can say what the CSV schema in this case is going to look like
So, I'm just adding a new field called URL. You can also add a store ID which could be an integer and then if I want, I can just go ahead and save the schema. I also have other options like does the CSV file have an header, does it have what is the quotation character if comma is encountered and what is the escape character and all the different options that we might have and we can just create a schema. So, this is one of the bigger schemas that we do have, it's for one of our product data. So, we can see that it has a lot of fields. People can come to this page and introspect what the different fields are, what are the types. They can know if it's going to be a much more complicated one like an array or a struct and really know what the different fields that are available in the data. So, there is this another schema that I will be showing
So, this is a Abro schema. So, you can see that the JSON is there, but people can go ahead and look at the fields and see what's in the data. So, it's a consistent interface that we across all the schemas. So, it can be a parquet Avro, it can be Thrift, it can be CSV, but finally, if you come to the fields page, you will know what the fields are in the schema. And we also provide ability to see what is the difference between different schema versions or schema has been evolving. So, the evolution is why is one of the reasons why we use Avro and Thrift, right? So, people can see what the difference. So, in this case, a particular field, a more complicated struct field was added. So, people can know what's the change across schema versions
So, schemas tackle what is going to be in the data. So, next, let's move on to the data sets, which will define where the data is and what's the storage for it like is it in S3, is it in Google Sheets, is it in FTP and so on. So, I can create a test CSV data set and use the test CSV schema that I just created and choose the version. And currently, we support data sets source to be in either in S3. So, we could have files that are present in S3 in terms of CSVs and the Parquet and Thrift files and so on or we also support Google Sheets as input. We do support FTP in the output side of things. So, we can give any path here and then this is how we publish your data set. So, any questions so far, folks? I would like this to be like a two-way conversation because something that I forgot to mention is the goal of this talk, we are coming all the way from Chennai
So, we want to share some of our learnings with the community but also learn from the community here to know like are we on the right track, how are the things that we are creating and also to learn and learn what others could provide us. So, I would like this to be like a two-way conversation. So, please ask any questions at any point in time. Okay. So, I have a question. So, you mentioned GraphQL in this. Yeah. These look like tabular, these look like square, you know, rectangular data sets
Where does the graph language come in? So, one of the misconceptions is GraphQL means graph data. So, GraphQL, you can think of it as REST 2.0. That's at least that's how I see it as. So, I can quickly show our GraphQL interface. GraphQL is what everyone uses. So, by GraphQL, I mean that the API that is driving this UI is written on top of GraphQL. It's not the actual data that is a graph data or anything. It's the so we have abstractions of pipelines, namespaces, schemas and data set, right? That data is represented in a GraphQL schema and the UI that we are seeing which is what I am showing is driven by that GraphQL schema
So, for example, so within this namespace, I will just quickly show our pipelines query will look like. So, this is listing out all the pipelines that are within this namespace, which is essentially what's driving this UI. So, the power of GraphQL is the client side, the UI or any other consumer of it like a mobile app can ask for the exact fees that they want. And then you so you have the entire schema available and then you just you pick and choose the parts that you need. So, that's the power of GraphQL. So, name and if I want the data set that is being used. So, just by adding this without having to it some other endpoint by just changing the schema that I'm requesting, now I'm able to get the pipelines along with the data set that they are using. So, that's GraphQL
It's essentially what I think of it is as REST 2.0 or the next iteration of REST. So, do you think it would be more as the, this is even more as the hierarchy or tree-based graph? Yeah. Yeah. It's more about APIs and instead of having to construct different endpoints in REST like you would do for different use cases or for different queries. Can we just post to one single GraphQL endpoint, give the schema that your client requires and you get back the data that the client requested? Yeah. Yeah. The reason I ask is because we're doing a lot of work with more complex graphs. Yeah
Yeah. So, that's not this was about yeah. Yeah. I think when when a lot of folks hear about GraphQL they misunderstand it in a different way. It's not about purely about dealing with graph data, but you can it can be a serving layer on top of graph data, but it's more like REST rethought. Yeah. Yeah. I've talked about like the users of this and I was thinking you're going to jump in more into the data sources and how people find data
Could you talk a little bit about who is the customer of this portal and how do they actually find the data they're looking for and what have you done to help that? Okay. So, so the customers of this is anyone within the organization. So, we have internal customers and we do in the future want to start exposing this to customers so that they can also start using this in a sales or more, but the initial customers are going to be the internal folks. So, it's going to be engineering, it could be product management, it could be customer success, anyone within the team. So, non technical people as well. so that they can come here, use the UI and then start looking at the data that they want. So, we have provided features to search, find the data and then go and look at the schema for the data set, what the version is, when was it last updated and even get notifications for when new data sets are getting published and so on. So, I'll probably quickly show that part
So, for example, people can get notification via email or via web books if you want to talk to some other system and even Slack obviously if they want to know, keep track of what's happening with a particular data set. Is it, is there a new version being published and we also use the same mechanism to trigger pipelines. So, if a new version of a data set is published, we can also configure pipelines which will start automatically as soon as that new version is available. Does that answer your question to an extent? Yes. Okay. So, I'll move on to the meat of the system which is pipelines. So, you can see a bunch of pipelines here. So, I'll start by creating a new pipeline
So, I'll choose one of the common data sets that we have. So, here we can. So, while creating a new pipeline obviously so far I've given a name for it and I've started by giving choosing the data set. So, here I can even if I want to see what the data set, how has it been evolving, I can see that. So, this one is like published very frequently and I can also see the schema from, I can click on this link and see what the schema was. So, it will take you to the similar UI that I was showing previously. So, once I've decided what the data set is, I can choose a particular version to use. I can say just start by using the latest version that is available and this is the part that I was talking about a bit back
And so, we can whenever we are creating a pipeline, we can say let it be a manual trigger pipeline which means I've created the pipeline, I will trigger it manually whenever I wanted to or we can configure it to trigger on every data set published. So, as soon as a new version of the data set is available, the pipeline will be triggered and it can be like a cron timer. So, that will run periodically if you want to start collecting statistics or metrics and so on. So, next you come to the place where you want to start defining the different transformations that you want to apply. So, we have a bunch of built-in transformations. So, there are transformations to do sampling, stratified sampling, there are transformations to run our classification algorithm, brand extraction algorithm and so on. So, there is even this override transform which so this part of our platform has been open source. So, I will talk about it at the end
And the most powerful transform at least in my mind is the SQL transform which gives you like a whole text editor where you can start typing any queries that you want. So, it can be like just type in a query. So, hash table name is this visible folks should I zoom in a bit. So, this is also like very interactive. It shows in the bottom. Sorry, it might not be visible but it keeps showing in the bottom where I am making any errors while I am doing the query. So, it is showing that title one is not really something that is present in the schema. So, since I have chosen a data set and the data set is associated with the schema
Here this transform while I am typing it knows what is that underlying data that I am going to work on and will want me if I am doing something wrong. So, I will have to fix the query to make sure it is doing fine. So, we have enabled full live SQL here. So, and anyway we are using the power of Spark SQL. So, it is not like we have done a lot to enable this functionality. So, but entire full I've SQL functionality is available and we have also written a lot of UDFs to make the task easier especially ones around very specific to our domain like to see if a UPC is valid to assign an ID to a new URL and so on. Like there will be ones for ISBN. So, we have written a bunch of UDFs and also even UDAFs which people could start using
So, all those UDFs are available in the SQL query. Male Speaker 2 So, I understand that you have a lot of preset transformations. Yeah. Male Speaker 2 Yeah. Male Speaker 2 Do you have anything that is like, you know, freeformers will allow you to write any sort of smart job that you want. Like say if I am a Scala developer, I want to be running a smart job at Scala. Male Speaker 2 Yeah. So, currently we don't have that functionality yet
So, we are still thinking of so that's something that's been coming up. So, can someone run a job with new custom set of UDFs? Can someone run a direct Spark job using this platform by defining a pipeline? So, that's something that we have thought of in some ways but since if someone is having code that they can ship into our existing Spark clusters and run it and that part is solved to a certain extent we didn't want to. We haven't done it yet but that's something that we are still evaluating especially from submitting custom UDFs. So, we have all these UDFs defined already. So, can someone run a job by also defining a jar from which we can pick up more additional custom UDFs? Something that they have just written. Can we start using this? That's also something that we are looking at. Yeah. Thank you for the question
Male Speaker 2 I can ask a follow-up question to that. Male Speaker 2 Yes. Yeah. Yeah. I'll come to that. Yeah. I'm going to come to that. Yeah
Thank you for the question. So, obviously, it can it do not be just one transform. I can start writing more and it could be also another SQL transform. So, I just selected a field called title here. So, if I do title one It's going to error out saying title is not valid. So, it's a much less much smaller schema right now. So, it's just saying title one is not one of the fields and the only available field right now is title. So, people can start writing multiple SQL queries
So, when we started this project two years ago, we provided this SQL transform thinking that mostly developers will use it but what we have seen is our customer success folks are using it to a large extent like they are writing like 10 different SQL transforms with code or even with SQL that I thought was not possible. So, I as engineer, I am shocked to see the kind of code that people have been writing on this. So, if you enable people, I think people will start using something. So, that's been the biggest lesson that I've seen, especially from this transform. So, I'll come to the output part of things. So, yeah, we are reading data. We are doing some transformations and then we'll have to start writing out. So, like I said, for output, we have support for S3, SFTP and Google Sheets
So, if I do Google Sheets, I'll be able to specify ID and the worksheet name that I want to use and I can write to the Google Sheets. We saw that a lot of our product management folks especially want stats to be pushed directly to Google Sheets so that then they can start looking at the data quickly and start making decisions on it. So, we started supporting Google Sheets. So, this is the part that you were asking about. So, from the output of a pipeline, I can just write an output to S3, but I can also choose to publish this output as a data set back into the system. So, here I can choose what the data set is and also specify the data set version. We also provide some placeholder variables like dollar date or dollar pipeline label and so on so that those get expanded as the pipeline runs. So, for example, if this run of the data set uses a particular data set version, I can say publish this output of this pipeline to another data set and use the input data sets version as the version for this guy
So, we also provide this capability. So, that's that answer your question. Yeah, a little bit. I was asking also for just like external spark jobs. Yes. I have that as a slide. I'll come to that. Yeah
Okay. I thought this is what we're asking, but we do have that also. That's something that we have as a use case as well. And then we can choose what the format is. It's going to be the same as what we are defining as a data set. And we can do even custom partitioning based on keys and so on with here and choose comparison options. Okay. Okay
Okay. So, one of the features that we have also provided is as we are writing the data, can we also do some aggregations on the data so that people can immediately know what is there in the data. So, we people can do select the field and do aggregations on how many distinct fields of certain things is. For example, we are working on product data. So, if someone is working on the data doing some transformations with the pipeline, they are writing out the output, can they also immediately see statistics like how many stores are present within the data that they are writing out, how many categories are present within the output they are writing out and so on. So, we provided that the stats features so that there is no need to run a separate job after writing out data. So, this is how the pipelines interface would look like. You can come and trigger new runs, pass the pipeline from running if we don't want it to be scheduled
And these have been the different runs on it. So, the runs view will show us what the data set was used, what the data set's version was, how many output records came from that run and how long it ran and so on. So, all the necessary information that people running pipelines, looking at what the pipelines are doing would want to know. And we also integrate as much as possible. So, we pull even pull in error logs from Spark and show that year so that people can know what the issue was. So, that we want people to be in the product and not move out and provide all the data as much as possible available here. So, in this case, the data that we were trying to read was not available. A simple error message
Any questions on this? I was wondering, how are you doing the classification? Can you speak maybe a little more about that? With this platform or classification for? No, on your platform. Okay. So, what we do is when we are running that particular transformation, which is available, probably quickly show that. So, what we do is when the Spark cluster comes up, we pull in the Docker image that is serving our model. And from this platform, we can specify where the service is running, what the port is, and which field represents the piece that the model is looking to get. So, for classification, our model looks for the URL of the product and the title of the product. And so, in this transformation, I can specify which field that the from the data set really as the URL and which field as the title. And this gets passed to the local service that's running on the Spark node
So, the way that we solve the algorithms part of things is we bring up the cluster, we pull in the necessary Docker images in each of the nodes and then serve the service from that. And these transformations can hit the local host and start using it. But potentially, sorry, one second. So, but potentially, it could also be a separate form where, for example, we could be hosting the service on Mesos or Kubernetes and also add this transformation point to that. So, which could be we specify what the URL is here and so on. Yeah. Yeah. So, on the input data set, can you only use one data set or can you compose multiple ? Yeah
Yeah. So, currently, in the input data set, we can only specify one. So, we are working on a way to make convert this into a much more complex DAG format. But what we can also do is in transformations like, say, the joint transform, in which you'll obviously want to join with another data set, we can select the data set that we want to join with and then choose the join type, which could be a column based join or a condition type based join and do any type of join that is available to Spark. So, essentially, what you would write as df.join, another df and give the columns, we have made it into a UI that anyone can start using. So, most of the things are available like that. So, currently, in the input data set, we can't specify the multiple data sets, because we started off with use cases that were more aligned to that. But now, we are seeing that lots of folks want to do multiple steps to different data sets and then combine them instead of doing steps to only one data set and then combine with another data set as is
So, we are seeing that and so, actually, we are working on that thing right now. Yeah. Okay. So, I will also show an example of how we have also built apps on top of this. Sorry. So, you can output the structure types, length of types. Yes. Is there any sort of validation to make sure that your transform is going to the correct type and then all the fields are there? And if you know the output type, is there anything to make the transform easier to write? Yeah
For the output side, we are not doing like creation time validation. We are only doing it at runtime. So, we will have to run the pipeline to know that you have made some mistakes. But yeah, it's a good thing that we'll have to fix. But we don't do validation. So, we do a lot of validations while you are reading the data and then applying transformations. But once the output is ready and you want to write it to a particular schema, we don't do any validation at that point in time. It will be a runtime failure, not a pipeline creation time failure
Yeah. Makes sense. Yeah. But yeah, nice suggestion. We did face that problems a few times. But it's not been a big problem that we have not fixed it yet. But we'll be fixing that also. Yeah
So, this is an example of another app that is built on top of this platform. So, this part of it has been open sourced in what we are called as how am I doing on time? So, just to give an example like we have also built other systems on top of this platform. So, we do a lot of work with product data right. So, the inner end data itself could be of bad quality and we do have a lot of models that turn on top of the data to do predictions like around classification and so on. So, we have come across a lot of cases where things aren't really right. So, we want to provide an ability for like everyone within the organization to okay I've seen these problems in the data. For example, a product which has an ISBN and we have found that ISBN to be valid. Then it's probably a book right then but we are classifying it as a video game
So, that's obviously wrong right. So, we provided a system where we can look at the data, take some sample based on some conditions and then write overrides which are JSON rules but using SQL conditions and then run those overrides against the platform. See how the rule will affect the data and then use that as overrides in our production system. So, that okay if I see an ISBN in our data and if that ISBN is valid please do mark it as books. So, this is that system which we called as Trinity. So, it does so it allows people to sample data based on some condition and then write some overrides which are essentially SQL conditions and so on. Yes, since I'm running out of time I'm not going to a lot of details but the underlying core system that we built on top of Spark and Spark SQL we have open sourced it as Spark plug. So, it's one of the things that we have it's a small enough conference that we were able to pull it out
So, if possible I'll talk about this in the on conference. So, going back to my slides. So, one of the ways that we integrate with the platform is in from the jobs we have provided an library by which people can submit datasets. It's just an API call to the system. So, as soon as a job in our production system is done with writing the output can just do an API call to our platform and say this dataset is now ready and new version is available. So, that's that answer your question there. So, that's how we write. So, we also have ability to read which I'll show
So, we also allow so this is one use case where we are getting feed data from our customer. So, we can mix it with our existing data in the MDA platform and enrich the data that our customer is providing and then do very specific stuff that the customer might need. So, they have their own data. We fill in some holes in their data with the data that we are getting from our platform and then just do a minimal classification and say brand extraction very simple example and then give that out as an output to our customer. So, without any engineering involvement any of anyone from our customer success team or anyone else can just configure this pipeline and get it and even have it on cron or with every dataset published and get started publishing the output for the customer. So, it really solves customer use cases as well. So, a lot of data scientists are using this platform to filter and pre-processed data. So, we have a lot of UDFs which help us to do very domain specific transformations for us
So, we are seeing that we have started using this for collecting our training data. Oops. Sorry, I wanted to do an animation here but it's not available to me. So, this is the place where jobs which want to read the data from the platform can use it via code. So, they can instantiate a simple client which will look at our API server and then just point use the client to read a DF from the namespace and specify the data set name and we will get a data frame out of it. So, I wanted to make a joke saying that based on the last evening's panel discussion this is ready to push your notebooks to production.io but unfortunately I didn't have animations for me. Okay. So, that's mostly what I add
I won't go through the timelines because I don't have time. But from future point of view we want to start working on multiple datasets. We want to start using schema inference so that not for everything we want to define a schema. So, if I just give a dataset path can it just infer the data. Spark already does it in a lot of cases, right. So, can we start integrating that into the platform that's something that we want to support. So, while we have hundreds of datasets there are only few datasets that people use often. So, can we start caching them
Can we start reusing the cache version in memory across different jobs like can we evaluate something like Apache Ignite to do that for us. And you also want to make this like a full self-serve solution for our customers which is just the initial reason why we call this as the grand name of marketplace of datasets and algorithms. And something that I learned also there is Airbnb open source tool called superset which also does some of the SQL stuff that we are doing. So, can we integrate with it or just to evaluate it to see how it is. And like I mentioned we have open source to very point one percent of what you have done. We try to open source much of what we do. It's the only thing that's stopping us is time that we have to do this. So, that's one of the reasons why we wanted to come here and have a talk
So, few folks have said that hey is this already available open source that we can start using it. Few folks have said that we have built something very similar. Can we share our learnings with each other. So, in a lot of ways our goals are being satisfied with the talks that we have been having here. Yeah, that's it. I'm on Twitter at Manoj LDS. So, do ping me for anything. Thank you folks.