data.bythebay.io: Karissa McKelvey - Data governance and distribution with Dat
Recording: data.bythebay.io: Karissa McKelvey - Data governance and distribution with Dat
hey everybody um my name is carissa and I work with Max Ogden hey y'all in the back hey hey in the back I'm about to give a talk okay okay um my name is Chris and mckelvey and I've been working on this project for about a year and a half we are a non-profit and we're funded by the Alfred P sloan Foundation we're a three-person team although we're about to hire four more people so be seven people and it's all open source we have over 800 modules on NPM and everything's written in JavaScript RC and we've been tasked with this question to answer this question how do we reproduce research today and trying to make that better and one of the things that I've been thinking a lot about is how reproducing a research is actually more like collaborating on research so when you create research whether that's with someone in your company or your organization or in your lab or whether with that someone from across the world because you publish that research you're actually collaborating with that person because you have to get their data and you have to know what they did and you have to actually talk to them and then what do you think about expanding research to data analysis more generally and that's one of the things start to get interesting is when you talk about how we can collaborate on data analysis throughout all realms of life every organization in the world and that's the kind of thing that interests me and what this project I think can bring so what happens when you collaborate on data analysis today what ends up happening is you start with like huge amounts of data that come from all over the place could be pictures of cats the start like pictures of the stars sales records or whatever I guess this is like an industry conference so I have to throw that in there but so you start with this and then people end up at least in universities and what the kind of research we did we saw that people are actually putting this on a shared drive and they're like hoping that nobody edits it at the same time or you know pulls the wrong data or accidentally bleats everything that's kind of like collaborating on big data today and then you take this you know gzip and you put it into a Python script which exports into csv s which you might put on github or something try and track it hopefully everything's fine and then you have someone who comes along and are and like messes with the CSV s and outputs their own CSVs and then you know you try to replicate that or reproduce that it's really difficult actually and this is just a very simple pipeline from science but you know we just saw in the last talked okay these pipelines can get really complicated and involved proprietary services and all sorts of things and so in the small use cases that I've been working with right now we've just been using really focusing on the use case of github where people are using github to publish and open data and we're trying to see how that like kind of fails or succeeds and where we can come in to fix that and a lot of the biggest problem we see is that get does not do very well with very large data sets as soon as you put anything over like 100 megabytes in there and start editing a lot it just starts blowing up and it takes forever to download and you have to do it with get you have to download everything in order to get any like even if you just want to see like one row in the data set you have to still download everything so pain in the ass another problem is the good old git pull rebase origin master anytime you put someone who's like familiar with our or sequel or someone who's just learning to be a data science scientist in charge of like interacting with a git repo good luck and have fun even if you put like a brand new software developer a junior developer and have them have a merge conflict they're going to get in trouble I've been using it for four or five years now and I still sometimes have to just you know I have problems with it so let's let's be clear about how domain experts aren't developers and we shouldn't have them learn developer tools like it someone who's really good at statistics they probably didn't learn that much get foo and they're not going to be that good at it they might be really really good at data analysis though so we still want them around there studying the Stars and the planets and like math and and we don't want to have them deal with it what they end up doing is Dropbox so Dropbox ends up being place for us to collaborate on data and I've been in the situation multiple times where I have to just put the CSVs into dropbox and the our user comes and starts analyzing it and all of a sudden we end up with a hundred different data files you know working model data june fourth 2012 or whatever and then you have like version to version 5 version 7 starts to get unwieldy and not only is dropbox not really good at doing historical data analysis dropbox is actually not hackable it has an API but it's not open for and actually like integrate it into a workflow as easily as you want to you can't like modify it it's not redundant like get is redundant you can like really easily put a bunch of get servers on a bunch of different places and if one fails it's not a huge deal with Dropbox it's still a centralized service free is it free no it's clearly expensive auditable this is another problem with it not having this historical feature where you can't actually see where the data has come from and it's not open source so we need something like get that it still has these properties of being hackable and auditable and historical but we need that for collaborating with data data sets and we also need something that's useful for building data pipelines so get itself is not actually that hackable it's written in bash try to put get in the browser for example or something like that and you start to have problems start to build something on top of get and you'll quickly notice that it's really hard so we wanted to also make something that was developer friendly so let's take this example where I have a bunch of different files that I'm an RA user and I'm like editing a bunch of different files that someone has given me what i can do is i can create a link to this like dropbox folder say and this data link is actually just a hash it's a unique cash that is based on the data that was actually stored in those files so if you actually go through each of these we actually go through each of these files individually and produce a hash the representative of those files so if I change like a single line in one of these files I'm going to get a brand new hash so then you can take up here or like a computer we call them peers because there are no servers and clients in this model this is a peer-to-peer model so we take two peers that want to share data from one to another the first thing they do is they talk to a peer introduction server or any sort of way to introduce them to each other then they finally find each other the pier introduction server says hey you have data that you want to give to this other person and the and then those computers talk to each other and share the data directly from each computer to each other so no data goes through a centralized server which reduces the amount of round trips makes it faster reduces your bandwidth costs etc etc it's actually really similar to bittorrent it's actually almost exactly like BitTorrent but we don't use the protocol of BitTorrent we actually just built a new more efficient and for variety of reasons which I'll go into in a minute so the way that we do peer introduction BitTorrent just does the DHT but we add DNS em dns tcp and we're going to add HTTP as well soon basically we'll actually have a dns implementation and no GS that you can run as a server that will do dns lookup and try and connect people together which is really cool because it's a lot faster than try and use the BitTorrent DHD and you can look use this right now it's a module on NPM called discovery swarm and then so the second thing that we do is really cool is we guarantee data integrity with merkel trees so it's just like get or block chains I said the word blockchains guarantee data integrity with like a decentralized model so if we're looking at all the different solutions that kind of are similar with data sharing and collaboration the first here is a bit torrent on the left the torrent is not open so BitTorrent sync is a proprietary protocol and so I'm actually even though the BitTorrent its cell is like an open thing it's not open source and it's not very hackable in that way Dropbox is obviously not open source and making go down the list so sign obel this means that you can authorize all of these allow you to like authorized and sign a certain feed of data which is really helpful for people if they want to make sure that nobody you don't want some random person to edit your data historical get and dad are both historical decentralize Dropbox is clearly not decentralized but the other three are content addressable I'll get into that a little bit later but basically is it does anyone know content addressable is okay it's the only a couple people the great thing about content addressability is that you don't have to download like the same data over and over and over and over again even if the data changes a little bit streaming so something that is streaming means that you can get the data at the same time that someone is uploading it and you can like listen for changes on that data so get for example you have to you have to wait for the entire commit to finish and be pushed to get before you can like download anything and you have to download everything at once it's like an HTTP request you can't just like get like the first 20 bytes and then the next 20 bytes and then the next 20 bytes as they're uploaded for example indexable so that is the only one that you can actually we can actually give you a URL to a particular byte range inside of a file inside of your repository of your data set which is really cool thing that we think is it's really important for specific specifically data scientists that will work with like huge huge huge data sets and they just want to see like a row inside of one of the CSVs but they don't want to download three terabytes you can go read more about this on our blog I just pushed a blog post about this so that date is different a few other ways it works in the browser which I'm going to be demoing in a minute it's entirely written a node um which is awesome because we can do really cool stuff like decentralized applications where the person just has to drag and drop a file onto their web browser and then all of a sudden it's into the swarm and someone can go get it on a server or some people get it somewhere else so you can build really cool applications where people just dragging and dropping files and then they're in the certain air in the system the diffs are built for data not for text files so get has their diff specifically designed for text files which creates a lot of problems which is a lot of the reason why it's slow streaming and indexable deck by trade news which already talked about so that's a core theory i'm going to show you this browser stuff because it's really fun so this is something you can find on github this is called hyper drive and this is the database that we're using in debt you can take a look at it macintosh / hyperdrive so what i'm going to do is drag and drop the csv so what if what it just did is it like hash the file and it gave me this hash so let me open a new window so it gave me this hash I don't know if you can see that so I can go to this link and what it's going to do is it's going to connect to each other and starts downloading the data so what I've actually done is served a data set from the browser to another browser without using any server whatsoever so now I can go and click on this data and I can see the first five rows of the data what it actually did is fetch the first five rows it didn't fetch all the data it just fetched the first five rows and if i use the bigger csv i could have like hit you know load me the first five rows before it finished downloading but i was talking too much something really cool that we also did was you can do this with movies so movie is a really good example of being able to stream something in real time so i'm just going to show you that i'm not lying so before this is finished downloading before this is finished lout downloading ah you can see it's us it's playing this movie before finished downloading cool so that's proof of concept now I can show you on so what I did here was do a dot link this is um our command line tool I did that link me to it again so what it does is it goes and it hashes the files it took like you know not that long to do 100 Meg's and then I get this hash which is actually dat colon slash slash we're kind of experimenting with this because we're kind of thinking about like what if you know we had browsers that supported this what would that actually mean and it's kind of interesting to think about that as like an upgrade on HTTP or something like that where the data is always going to be there and you can kind of reference it forever and you can guarantee that it's the same data because it's the same hash if any of the data changes at all it'll be a different hash so it's kind of exciting because HTTP if you changed one letter or if the server went down you'd never be able to get it again but with dad it's just like a new way to reference data that will always be available so let me open up a new browser here so that I can download this data by just saying dat and it says it's connecting and then they're uploading downloading to each other and so now this one on the right becomes a server again so if you wanted to put all the data on a bunch of different clusters you could just say dat space the hash and then it would distribute all the data to all the clusters and them or the computers and the more computers that are in the network the faster it's going to download because it's based on the same principles of BitTorrent are there any questions before I finish up and talk a little more about how it works okay so how do we how do we do this part of like the streaming and the indexing so what we do is we divide the file into the files into small chunks that are unlikely to change when the file is updated so for example with get I'm going to use a get example the get also does a similar thing but they do it really stupidly so first you get one line per chunk with get so say I edit one line from worldthe moaned the other three chunks are remain unchanged so what happens is get will just say okay so I'm going to send this one data want one line of data but that's but what we want what we really want is difficult for data not text so what we use is rabid fingerprinting which is you can go look it up content to find chunking there are some papers on it which is you scan through the file and create chunks based on the actual file content so you can like tweak the parameters of like what kind of chunks you want so say a new part is inserted into the middle of the file we create a new chunk the neighboring chunks are merged and so then only those so then you can only you can like kind of ignore the other chunks and you never have to go fetch those because they're the same so it only fetches the new data so you can get this on NPM as well we didn't ojs implementation but it has it has a/c dependency and actually if it's something you're excited about we would love someone to reimplement the see stuff in javascript but right now it's just in C with a with a JavaScript wrapper so this reduces bandwidth use and increases the speed for the entire swarm which is really exciting when you're starting to think about sharing terabytes and terabytes of data across lots of computers so what's gonna happen in the future so we're gonna have modules for doing backups on servers so right now if I'm sharing data from my computer and then I turn off my laptop the data is gone right it's not available on the network even if I give that hash to someone else if my computer is off they won't get that data so what we want to do is make it really easy for you to set up a server node and then you can just say publish to that or push to that and it'll automatically it'll like listen for changes and automatically pull pull your pull your links and rehost them the second thing we want to do is HTTP fallback so that'll make it really nice if you want to use google cloud or storage or s3 or internet archive or something like that so we're going to have like a archive format that you can put on an HTTP server then we're going to have a desktop application at least to make it really easy for your friends who don't use the command line and then we're gonna have a background Damon manager so you can have multiple deaths running on a single server without running like 20 different nodejs processes because that's one of the problems right now is nodejs each each time you run a debt it'll be 50 megabytes per server so having a daemon manager it'll be 50 megabytes for all of debts you could run like 20,000 dots on one server if you really wanted to and then the last thing we want to do is open data scrapers and this is what we just got four hundred twenty thousand dollars from the knight foundation to do so what we're we're hiring for more people and we're going to be turning all of the open data portals into dats so what we're going to do is every day we're going to scrape an open data portal and then we're going to get their hash of that data from that day and then we're going to put that into a DAT which is going to be visible on a website like that's going to actually act like a torrent tracker it's going to be like a torrent tracker for open data and so then you can go and see like oh the zip file was put up yesterday what did they change from the zip file from a week ago because an open government and an open data in general sometimes they're publishing like a zip file on an HTTP server and you have no idea what they changed one day to the next and so this will be really helpful for that particular use case but also a lot of other really awesome use cases so if you want to talk more about that I would love to talk to you about that after this because that's going to be manager on that project that's happening this year and thank you very much you can find us on NPM install dash gchat debt or look at hyperdrive which is the database we're using and any questions are very welcome yeah it only touches the neighboring chunks so that was that um this uh yes oh yeah so we don't do we don't we don't care what kind of dad you put in it'd be really cool if someone built something that was smart for CSVs but we don't have something at smart for CSVs no it's like a parameter that you tweak do you want to do 128 bytes or whatever yeah yeah it works pretty well for like on the terabyte range um yeah it actually hashes things really fast um the only problems i see right now are like doing that in the browser like browsers are pretty bad at that good question though yeah the chunk yeah the chunking with the original like hashing and chunking happens on one computer although it'd be really cool it would speed it up a lot if there was like a clustered version of that but there's not yeah no because it just goes through and like literally looks at the bytes okay yeah so that's one of the things that really differentiates us from get is that we built it with no Jas so it's really easy to get like streaming by a nature and it's also indexable by nature so you can say I want a bite range of this file and you can go get that so even if the repository is like a terabyte you can still say I want one I want the first row of that CSV and you can get it yeah yeah so each so when you actually do the chunk chunking it's actually doing a mini hash likes to have a hash of so in order to index it you would say like a hash of a hash basically or you would have a subhash I guess something like that I haven't tried it yet but you could probably put one of them into this you would think you think of that more like a filesystem layer or something like layer on top of file system so you could put if you're putting anything into a regular file system you could just put it into that just a list of files so if hdf is just like a zip files like looks like a file on the file system I don't know but if it's like a server then it won't work right yeah it'll fetch outlays Lee sorry it will it will fetch the stuff just from top to bottom like in the way that it was first hashed but unless you specify oh I want this by range it's just like a file system so there's actually you it riginal e is just light it looks like a folder directory with directories in it like that but there is an underlying module underneath hyperdrive called hyper core which is more flexible so hyper core just stores arbitrary binary data it doesn't assume that there's a drive so hyper drive is like the thing that kind of assumes that there's a file system so if you really wanted to just append for example here's the example if you really want to just append so you create a feed and you can just append arbitrary blocks and then on the other side you can like download these blocks and it'll just download hello and then world yeah that's what that's what I'm working on right now yeah I'm actually calling it hyper commit one of the things that we really really like and appreciate about node is that we really want to separate out everything as much as possible so it's just like turtles all the way down so that way people can be like oh I really like the idea of having a dag of hyperdrives but I don't want to use that so you can go ahead and do that so dad itself is just like a command-line wrapper around a database it's really simple but yeah so this thing hypercomm it is what I'm building right now which is something called a hyper log which is the dag part around a hyperdrive which is the storage part like we have some users but it's really all very super technical and super researchy right now on that end you are not envision people actually can push computer to your note in there and then that you handle the profession did I pushed Oh accomplished today no that would worry i would love I would love to get to that point actually there's a so we're a non-profit right now but I think there's definitely like there's definitely avenues I think for doing something like compute to data sort of services so one of the things that we've been working on is called hyper OS hyper OS I think it's the i/o so hyper OS so so we actually want to do is put containers so not just files so I'm talking about data files right now right just like a file system but if it's a file system you could actually put in like a container why not a container is just a list of files so then think about that if I'm have a container a DAT a DAT container and I put it to a server the server says okay I want to run Python what it will first do is say okay I need bin bash and so it'll go to this swarm and get the files and needs for bin bash and then it'll go get the files it needs for Python but then it won't download all the other stuff so it'll be really really really fast to like mountain boot and run a container so that's one of the things that we're really excited about but we don't have funding for right now so is kind of just like an experimental thing yeah we we aren't really i mean we appreciate docker and like docker but we're more interested in like the peer-to-peer model and so how we can like expand in the notion because docker containers it's all based on like this you know you download it and then you run it and we're really interested in like what if I want to stream it or right I think I think retire my different things so like you might have a docker container that first sets it up but then to send it to you I might put it in a DAT and give it to you and that would be faster then yeah so it might do docker container once but instead of giving you a docker script so then you run docker on your machine I only have to run docker once and then I've saved the state of my machine and I can actually send that state to you as a list of files so make sense ng some of you right to query the chance of a location so i can divide i really have the opportunity to compute in your own together coexist with you i will love to know where should I push my computer to where we are the charm oh jeez okay so i can minimize traffic your bounden will be on the network if I have so many concurrent sessions friend other you know my clients are yes I don't have it done anyways all your side your pet doesn't system right right uh oh why me it's a big question yeah are there any other questions i think i have another minute or two cool well come talk to me after if you want i'll be hanging out for another 20 minutes or so