scala.bythebay.io: Pathikrit Bhowmick, better-files: Towards safe and sane I/O in Scala
Recording: scala.bythebay.io: Pathikrit Bhowmick, better-files: Towards safe and sane I/O in Scala
you so you guys my name is fatty could vomit I'm here to talk about better files it's a scholar library that I wrote I was frustrated by doing and doing Iowans kala I'm going to talk a little bit about the library but more about like the philosophical underpinnings of doing Iowans color the decision choices that I had and the state of Io so when I mean I oh I specifically talking about file system I oh so these are things like reading a string from a file moving files around doing like file system operations like cha march on zipping guns up unzipping deleting stuff so I don't really mean network I or database I oh just pure file system I oh and unfortunately all of these things are quite hard to do in Scala our Java I know shell commands on the top of my head which I can do most of the things with but if you ask me to write Scala code which will do these things correctly I cannot write them which is surprising and unfortunate because most of the time I'm a more productive programmer when I write Scala then when I right shell but except when it comes to i/o so with so what do i do I you I just google and the how to read a string from a file and that that's what the state of art is unfortunately yeah okay so even if you land up on Stack Overflow like there are these questions read a file in Scala write a file in skull and you can see at the number of people who like avoided this question it's like the two top questions in Scala and this creature was taking like a year ago and it's more that number is way more now and even then the top answers are incorrect in baltic ones for different reasons well this guy mentions like hey i can read a file in python ruby and some other language go or something in one line how do i do it in scala the answer it's out of the screenshot is actually slightly incorrect because it leaves a file hand we're open and the top accepted answer right here it doesn't work in Scala 211 that library was never released for that and the last comment was like 34 years ago whatever this library stirs recommending so even if you copy paste on Stack Overflow doesn't always work ok so let's look at the state of so I want to append some string to a file in Scala and okay this code is looks normal I mean if you are or if you're like assistance program where this magmic might make sense to you you need to open an output stream you pass that stream into a writer you append the text and then you have to close all the things that you opened and this is okay java code not too bad and what the problem is this this is a lot of code to do a very simple thing which is to just append a text to a file and which is okay fine that six lines of code I'll stick it right at once and never look at it but a lot of the things are not obvious from this piece of code like what does it what happens the file does not exist what happens if you're trying to write your directory what if you don't have write permissions what if there's a write lock on it what if the file is the symlink are you going to follow the simple or are you going to try to write to that I node in particular and all of this is completely not obvious from this code or even from the JDK file right utils well that was fine and all that was i was showing java 6 code that was before java and IO came out in Java 7 so let's look at some java 7 java nayak think so what in Java 7 and 7 they introduced the niño package so java dot io dot file is old use the Java dot ni o dot file dot files utils or path utils that's the recommended of way of doing things and you want to copy from a source into a destination and this is what you do I mean this is kind of fast issues you wouldn't even write this wrapper you till you can just use file start copy and you'll be asking why am I talking about I oh this is great weather like if niño solves all your problems but guess what if you actually read the documentation of this thing it says that this only works for files or directories that do not have any other files in it and it that's completely surprising and you wouldn't even find it in the Java dog it's you have to go to this copied HTML oracle dachshund you'll find it or if you're running it you will get java nir file dot they're actually not empty exception when you try to copy a directory which has files in it okay so let's try to copy directories correctly so this is so Oracle in all their wisdom actually published a correct way to copy directories in this link in this copied or Java and IO examples package which is not available to the JDK and this is how you do it I was a little bit surprised to see this because I thought oh copy would be I list recursively and just do a BFS rdfs and copy like that but you want to use the file that walk file tree you to that the files are walk for three the reason being the operating system can give you better access to directories which has billions and millions of files you don't want to list that and loaded into memory but anyway this is the most idiomatic scholar code or scholar translation of copied Java to copy directories correctly and there's absolutely no way you could have come up with this and again it doesn't solve the problems i mentioned it doesn't tell you all these things what if the destination doesn't exist symlink handling how if you're copying between different file system does it actually work it doesn't okay so before we jump in to the scholar solutions let's look at the state of the art in the Java world so first as these two great libraries Apache Commons I oh and google guava and and they work great I have nothing against them they're used in production in many big companies including Google so you can safely use them there are two things being Java libraries it leaves a little bit of bad taste in my mouth one is one is you always have to do something util dot something and and files in Scala I would have done implicit decorations and put them on the file class okay fine that's just aesthetics another thing is Scala is a much more powerful type system and you can use type classes to do a lot of things for example the Java has a little you to call scanner which lets you slurp inputs like strings and so on and they defined like 10 or 15 different next to next boolean next blast so on and so forth it's called I would assume there would be type classes where you pass in some kind of readable type class and you can it's extensible you have one method you need to remember that's it but these are minor inconveniences or you can use plain Java and I oh but as we saw that's really dangerous like even this is again copied from Stack Overflow someone asks how do i account number of lines in a file this code is buggy it leaves a file handler open and it's not obvious at all and even the verbosity of just using plain j / and iOS ridiculous like if you want to charm the file like change the owner of a file you have to do this ridiculous thing like files start set owner you pass in the file and get the filesystem get the user principal lookup service and look a principal by name and pass the one or in and that's how you Sean a file and there's no way you could have found that unless you know like the javadocs quite hard so let's look at scholar solutions the first is scholar diode or source it doesn't do most of the things that you want to do it can read and write files but if you want to zip unzip change ownership deletes directories so on and so forth you can't do any of those so it's not that useful then there's color I oh this is an interesting thing I want to talk about it so well besides the fact the last commit was in 2012 four years ago five years ago and even that was updating the readme or something it it was the wrong approach in my opinion it tried to do too much it tried to bring it tried to create scholar counterparts of buffered readers and print writers and output stream readers and so on and so forth so now you have two problems you have to not only keep track of the java mental model you have to know what the scholar counterparts are which how to go between those two one which one to use when so it was needlessly complex that's a man adopts by Lee Howie he's at the conference this is an awesome library this was originally written for the ammonite shell and that's where it's used it's great go feel free to use it my I have some minor gripes with it one is it has the syntax this is how you remove a directory or a file this is not Scala code that we usually used to writing that's because this was used as was originally part of a man I'd show where you can write scala as shell commands and that's where this dsl is very useful and another thing is if you want to actually do something complex like if you want to write to a file but use a different codec or if you want to traverse a directory but you want to find do different link options so like follow not follow the same link for example and those were not possible in the those are it's not it doesn't take one those arguments at least last when I looked this summer or you can roll your own and if you actually google i/o util dot Scala you will find so many results and I think the Scala compiler has won the play has one inside svt has one inside acha has one inside and they pretty much do the same thing all over again and we as developers highly respect dried do not repeat yourself and I think I'm more important takeaways to not repeat other people and then this better files which I'm going to talk about it's that my grabber is a pretty terrible name for a library that it's a historical reason it's called better files because I started adding implicit stew the Java file and I call that implicit class better file a scholar and that's how the name stuck so before I talk too much about the design of scholar the i/o and scholar i want to talk about the philosophy and the different philosophies that exists so Scala is a platform as a language attracts a lot of different ideas and a lot of different views of doing things which is great as someone said Scala is a melting pot of ideas and so that's all combinatorial explosion involved if you take any two different weird features and you put them together they don't go well together in Scala but that's fine so to give you overview like doing the some special things about IO IO is inherently slow io is inherently error-prone like files cannot exist they may not have permissions so on and so forth and to tackle these problems there are very different schools of thought there is the reactive streams acha philosophy where I oh should be just streams and you should buy things around move things and source that kind of philosophy then there's also like Scala jess is becoming popular and it's like the node.js callback philosophy where you passing callbacks which has an error handler and a success handler also has scholar as a first class language of the JVM we have to deal with the Java baggage and you know the Java baggage is like all the input stream writers bufferedreader sprint writers even now I don't on the top of my head which one to use when and you have to deal with that many times when you write code in Scala and there would be there are also like the functional school of thought where I or should be a monadic and that should not be side effects errors should be in the type system and I should be done using I amo nerds that's the purely functional school of thought and the new a new new ideas are coming in like the dotty there was talk by Martin about effects and encoding impacts into the type system and how that can be implemented in dotty which is the future of skull in some respect and that's again brings a new way of thinking about IO as effects and at the end of the day as many of you in the room we're just engineers we want to build something and you just want to write one line of code and if it's stupid and blocking and it works you don't care if you want unblocking can wrap it in the future and move on and you can see all these different schools of thought collide and manifest in this particular scala slip so this was opened by someone to gather ideas about doing a better I library in Scala and everyone had a completely different opinion about the best way to go about and nothing really happened in the end so this is my very highly opinionated implementation of Island scholar so before I started to just crank out code I made some points of what I think should be design goals I should aspire to one is you should not be able you shouldn't need to include any other library to do iron scholar so I looked at the superset of all the utils or file you tools available in guava bashi Commons jawed it's a cute java library and the Java and I apart and file utils and better files is a superset of all of those no external dependencies this is just an aesthetic issue I didn't want to depend on any library just depend on Java I didn't want to do anything clever I just wanted to wrap niño and remove surprises removing consistencies from the API give a giver idiomatic scholar expose an idiomatic scholar interface there's another thing that I wanted to do I didn't want to introduce a new mental model I i justjust want file class and every new capabilities are added to the file class using implicit and so there's no like 20 different i/o classes you have to remember this is an obvious one API should be obvious like if you're copying things it should not work only for empty directories Scala has gives you a very powerful type system there are some things you can prevent at compile time like if you're trying to list a file that should be an error you can only Lister actor ease on the converse if you're trying to write to a file but it's a directory you should not be able to do that and we can leverage the scholar type system to do all those things if you want another thing is automatic resource management in Scott and Java you always have to do try finally closed in I tried to make the library use auto closing iterators as much as possible which basically means if there has a next is false it closed the resource underneath and give you capabilities so that you don't have to always try close it's a pain even with the Java 7 try with resource syntax again I wanted full configurability like for example the Codex thing you can pass in your own codec but you can pass in your own link options you can pass on your own open options which is you want to open the file and read more read more read write mode but you don't have to specify seven or eight parameters to do something simple all of them have default parameters which are saying okay tests good docs performance I didn't want to incur any performance penalty this is as fast as Java there's a little asterisk there because it's actually faster than Java in some places specifically scanner so that Java scanner uses reg access which are extremely slow this one uses parser Combinator's and which is much faster than Raj axis this is again an external article by Lee Howie about principle of least power I think this is the by far the most important design goal of this library or anywhere you should strive to chair so this is a clickable link you can google it or read about it the idea being is that you want to do something with the least possible abstraction or the least powerful extraction available to you so the core of this library is not reactive not monadic not effects based not pure no buzzwords that you hear in the scholar conference it's very simple wrapper in our niño and the reason being it's not that those things are bad you just want to use the least powerful abstraction to do something and you can build higher level abstractions once you have a solid ground so there are libraries built on top of better files for example archive file watcher which uses actors for watching files instead of threads but you need a really solid API underneath for you to build that and there's like a shapeless bay scanner which lets you scan H lists instead of tuples and those are getting built on top of better files so I'll give a quick tour of better file so this was a motivation for me to update the github dots so I can present it in a conference okay let me can you guys see so this is what butterflies looks like there's many ways of is 10 shooting a file you can use string interpolation that some nights dsl involving the forward slash you can use a classic constructor over here and one caveat is the bet that file class in better fast is also called file so if you don't like that you can do import Reno mean to call it scholar file and call the Java one the file I usually go the other way I call the Java 1j file and keep the one that I like so again the API is that simple this is how you do right appends get a string it's just one lines of code I'll come back to that yeah there are these some quarters in Scala like symbols so there's also symbolic ones available this one a parent's this one or this sorry the single left less than overrides in the double left arrow appends it same as the unix shell cad commands and that's a fluent interface so you can chain all of them they return the file object there are also slurp API switch don't completely read everything into memory they return it triggers these are self closing iterators so if you manage to iterate over it completely it will when they has next is false it will close the underlying string for you but for if some reason you crashed out in the middle or you took the first 50 elements and just left it at that it will not close the stream for you so that's another alternative for that I'll come back to that later and yeah most of this API is so as you can see you can just if you want to read the content you can just do this but all of this API still takes in about three or four default parameters so codec is one link option is one another one and you can specify pass those in but by default you don't have to pass any customizations to these methods and these are bridges to the Java I OPAC Java I or classes so like you can convert between any pairs of like how do I convert an input stream into a bufferedreader or like how do I convert a file into an output writer all those things are fairly easy your ID you can auto-suggest it or you can just guess they follow a pretty decent convention again most of them takes three or four parameters and all of them are defaulted and you can convert between any two of the any pair of java io classes fairly easily i add implicit stew all the java classes so you can just call an output stream buffered and you would get this one instead of creating new buffered output stream and passing the output stream in this is kind of alluding to the fact how you can use the pattern matching and scholar type system to do things so you can pattern match and on the three most common types of files which is a symbolic link the extractor is the destination or a directory the extractor is a lazy iterator of the children in the file or a regular file again the extractor is a contents of the file there could be other types of files which I neither directories regular files a symbolic links for example on unix you have process files our pipes there's our everything's a file in UNIX so those things are not captured in the time system just because they're so rare you can also use Scala has this you can use the extractor syntax on the left hand side to do this I don't know how many of you familiar with this so this would give you all the documents in this directory because kyle has an apply on the LHS other nice utils for like globbing this actually uses the operating system primitives instead of just listing cursive lee and doing a match and all the little utils touch clear rename move copy link symbolic link all the hash md5 chancho mod some pipe operators queering the file for logs creating new temporary files are all available and if you actually import the commands package you will get a DSL which very reminiscent of the command line you can do CPR em and mcdeere chon all that it just calls the regular file API I'm some more methods I'll skip past this like you can get extension content type modified time hidden are so on and so forth put the POSIX attributes so this is another example where you can pass in some specific link option to an aquarium fits that directory which means you want to see if the file is a directory or the symbolic link pointing to the destination is a directory or not to Marge there's various types of file comparison if the spot is same if the content is same the path is same just looks at the name of the path and the content actually does I or to figure out if the content is same as various files ordering provided for you like by name by size by depth by modification type again zip you don't have to google how to zip a file in java something it comes with better files it uses java oh this is the part where you can use auto close resource management so you can sew all the java classes that are closable have a order closed implicit method defined on them so you get a little monad here i would say since i don't depend depend on cat starts call Izzy this is actually just a traversable but you can pretend it's a monad and you can get the resource do something and this would close the resource for you and this forex syntax is just queued you can obviously write a map or a flat map like you can do this that also works and you have you have utilities to convert any iterators to an order close to trader and this is the scholar scanner so if you're familiar with the Java util dot scanner is a very useful class but it's really slow it uses reg axis and this scanner is extensible you can just define a scannable type class and it will scan for you and it's much faster it uses partial combinators underneath so this is an example where I say how do i scan an animal you just define an implicit tie plus for animals saying it's a scannable because given a scanner this is how i scan it and boom you can do scanner dot next animal and you can't do that in java and same way you can define an H list scanner so this is a good example we're building higher level abstractions become simple once you have a solid core so this is literally what the shapeless scanner looks like it's two lines of code anything whenever you work with H lists you just have to define two things how do i scan the base class H no well you can always do that and how do i scan something which has a head and a tail which is another H list and it returns scannable of th column quality and as long as both the head and the tails are scannable and you can define it now you can scan H lists instead of tuples and this is an example where we do that we're scanning the CSV and you get an H list back so file monitoring is another interesting oops while monitoring is another thing that Java I can do a whole other talk on how terrible the java file monitor is it has so many bugs it can only it cannot watch single files it cannot recursively watch directories it cannot if a new directory is created in a directory does not automatically start watching that it has a lot of inconsistencies the sky this one wraps those inconsistent and gives you a nice little interface so if you want to watch create a file monitor you just create a thread back file monitor and you override three methods on create on modifying and on delete and that's it that's pretty much a three lines of code and that's a acha based one which is again based on top of the thread based one where you can remove and add actors on the fly and that's a powerful extraction built on top of the simple file monitor so that was the kind of the flavor of the API have better files so I I don't want you are in a scholar conference so I want you to take back something from this talk not just like me advertising better fast to you so these are some things that I learned that would be useful for you when designing a general-purpose library so fluent a PS i talked about it our api is where you return the object back and so this is what it looks like so you have been appended returns the file back so you can change the commands but the problem here is like okay four appendices you need to return the file back but when you're copying something it's not obvious should you return this file back or should you return the destination and that's sort of obvious from the signature you have to document it I have to read the code so you can use colors literal types to encode that into the compiler and document it at the same time for free so for a parent you say like okay when I append I get the softer type back but when I move a file to a destination the return type is not softer type its destination card type and as a programmer this is really useful because you know when you are doing fluent api calls when i called move to here I don't have this regional file back I have this new file back when I want to work on things another takeaway is since I didn't want to depend on any external library traversable czar poor man's monads they have a4h in there which you can use to do things so here i define a structural type called closable so anything is closable as long as they have a close method on it even though java added auto closable interface but not but not to everything so I'd rely on the structural type and I say something in as a managed resource that on a as long as a is at least closeable as just a traversable and that's because I want to use the for each feature which takes in a function and I apply it and I finally close the resource and this adds an auto closed method to anything that's closeable which is the structure type so now you can use this you can get the input stream auto close this is added implicitly you can get an output stream out of clothes you use the pipe and that's it this will close both the input stream and the output stream when this for each block execute and as you may know the for each block is just a syntactic sugar for a flat map so so future work this is alluding to your question in a fluent api each of them are opening their own file handlers maybe there's a way like kind of like a database session or a transaction where they could reuse the file analyst and push them forward so currently it doesn't it's an interesting work that I'm exploring it's definitely possible but challenging other things are the distinguishing between files and fats because sometimes you want to do I oh and sometimes you don't like a good example would be like to check if a file has an extension well if you call it on a directory you don't want to return that say that the file has an extension even if it has a dot in it but no if a file it has a is a directory or not you have to do I oh but maybe you don't want to do I oh because you're running some MapReduce on another cluster and that file actually does not exist there so the library better fast currently does not distinguish too much between what is a file and what's I oh and what is a path versus files things that exist versus things that don't exist on the file system and wouldn't be doing I oh now that this is a great idea from ammonite to type the paths there's a lot of bugs can happen when you try to go from a relative to an absolute path in Java in Java all of them are strongly typed so you can screw yourself up in many different ways and there's a PR out for the scholar 2.1 to release i'm looking through it it's going to be merged and released shortly i'll talk about this later about how I this library can integrate into the scholar platform so for future discussion I'm running out of time so this is it that's the github length as an act of Gator chalon scholar ducks are online it's only 500 lines of code so feel free to jump into the code base look at it there's five times as many lines of tests versus lines of code or requests are welcome so that's it one last thing recruiting I work at a hedge fund called Puerto capital we do algorithmic trading quantitative modeling if you don't know anything about finance that's great because none of us know anything about finance where engineers we are data scientists with machine learning where NLP specialists so we work a lot on scholar spark these are our tech stack if you're interested one good thing about working in a hedge fund is you can immediately get feedback from your algorithms like if you work in a regular company no matter how smart algorithms you implement you're going to get the same paycheck here if you make a m'allister dollars on a trade you will see a six seven figure bonus on your paycheck because this is our money we trade so even if you don't you're not interested feel free to refer me anyone who might be interested so that's the email address and we have a 25k dollar bonus for reference all right thank you [Applause] have any questions yeah it's just activity of people discussing things nothing really so the question was slipped 19 does it there's a lot of activity what came out of it a lot of activity has a lot of people chimed in what they thought was the best way of doing things but there was no conclusion as far as I know any more question that's a good question yeah I think traversable once would encode the fact that you can only go through it once much more it's in that documentation but that's a very good point it should be reversible once thank you okay all right so if our five is actually it's not evil it's not yeah yeah more if you if you traverse if you it's only for reading and writing if you use the right method or the read and for reading for writing yes it will close it for you for reading as long as you slurp the full input and it will close it for you but if you crashed halfway through the input it won't but there you can use the dot auto closed version which will close it for you even if you crash out all right okay oh one more no okay all right listen [Applause] you