Scale By The Bay 2020: Li Haoyi, Getting Things Done in the Scala REPL
Recording: Scale By The Bay 2020: Li Haoyi, Getting Things Done in the Scala REPL
[Music] so this talks title getting work done the scala rappel a bit about myself so i'm on the developer tool team at databricks i used to be at dropbox also doing developer tools among other things so beta breaks a heavy uses of scala bazel and a bunch of other technologies that we use to be productive here python json javascript docker kubernetes so on um and the author of the book hands on scala which is what alexi are showing just now um it's a introduction to writing scala in a kind of productive and practical way and i do i maintain an author a lot of open source projects like ammonite mill fastpass and it's long laundry list that i'm not going to name read or read them all out so what's this presentation about this presentation is an introduction to doing things that mine scholar apple um and it's not just doing things like learning one plus one or learning how to create a linked list in memory it's about doing kind of more practical things that certainly actually pay your salary to do i want to go scrape a competitor's website i want to go automate some third-party service i want to go paralyze some web crawling and go explore the graph the graph expanding multiple uh expanding the internet how do i do that in the ammonite wrapper um without needing to go and set this whole new heavyweight project with like a build file and a file system hierarchy and all that stuff um so we'll be walking through several real world use cases that you can do it in ammonite rappel that you may find useful in your day-to-day work so the three topics i'll be covering uh one we'll do some web scraping two we'll execute like a small github project migration and three we'll do some parallel web crawling all within the comfort of our ammonite wrapper so first let's talk about web scraping um over here i have empty terminal empty browser i can open up ammonite um and here i have ammonite ready to use so the let's imagine that we wanted to scrape the wikipedia homepage so wikipedia if you go there and wikipedia or you can see there's like a homepage there's some titles there's a bunch of links let's say i'm doing some kind of i don't know sentiment analysis for some trading algorithm i want to figure out what's what's happening in the news today um so wikipedia has this information in this in the new segment um and has a bunch of links to go to like various people places events that happened today that they considered newsworthy so how do i scrape this information down onto my computer um so m9 doesn't come built in with a web scraping library because mlx is a scholar apple what mlight lets you do is it lets you do this import dollar iv and then you can kind of import some library from open source library that you can use in the repo so the most popular open source library for web scraping is called jsoop in it on the jvm it's a java library and so if you look at how this java library is like org.jsoop j soup um see j soup one dot one dot let's say 1.13.1 just following the version they have on the website and then we can um you you can import org.jsoop.jsoop um so this imports jsoop in the m night raffle and we can immediately start um just following the tutorial of this jsoop library in order to do useful things so here i am um someone asked a question or can i increase the font size a bit uh how about this does this work um if no one says anything i hope this i assume this works um so i can call let's say like dot dot title and get the title of the doc i can um get the headlines so this odd syntax the css select sector syntax and basically says that i inspect this element on screen you can see that it is within this div with idmp itn i want to find the bold elements within it and within the bold elements i want to find the anchor elements which have a which are titled a so i can do that and i can loop over these in order to get out the title and the href link so because it's a java library i need to convert it to scala um scala collections have java collections and then i can do for headline in use headlines dot and scala let's say yield let's say headline dot after title headline dot uh absolute url hit draft yeah and then see we now have all the links pulled down from wikipedia so we have this martin vascara with his wikipedia page we have this nagona karabakh ceasefire agreement with its wikipedia page with the indian premier league final with its wikipedia page link and so on so just like that i saw some information on internet i pulled down the library from the maven central open source repository and used the script information down into my terminal for me to use and i can loop over it save it to a file and so on so for example if i wanted to write this to file i could just do all right let's say let's say i want wikipedia news.json and i can use the built-in json library to write um to write this disk let's say like invent equals 4 to make it a bit prettier um trying to overwrite it now if i read it uh you can see all the data has been saved to disk not just so it's not just for me to play around with in the terminal but i can save it store it pass it elsewhere hand it over to other processes and general this uh m night rapport becomes not just a place for you to learn scala but also place for you to manipulate and wrangle data that comes from and may go to some other third-party sources let's consider a second example of web scraping let's say i want to scrape the mozilla development network api apis so mdn the mozilla development network has a big list of documentation for all the apis that are available in the browser so for example the abstract worker that you can use in the javascript runtime environment the browser has this documentation event handlers methods and so on and let's say that for whatever reason i want to grab every one of these pages and scrape down the first paragraph like summary of the documentation maybe i'm writing like a ide and want to provide in id help maybe i'm writing like a web editor for someone to make websites under provide in-editor help for when they try to access these apis um so how so now how can we get this data down from the using the mlx scalar repo um so similarly i can just use jsoop uh to connect to this uh website um so now i have this doc this uh html document downloaded in the past i can look for some elements in the html on the right of that i can use to access it so from document i can select the header whose id is interfaces interfaces i can then uh let's say do next.next which will give me this div which is contains all my links which i can then select let's say i want to select the links and um so now i have like uh links equals that and similar to what i did earlier for link in links and scala let's have to get the title of the link and i'll take the hit draft um and let's take one more let's do like link.text so now the text within the link like angle instance arrays uh and then i have the title html the title means this pop-up tooltip and the destination that that link goes to so now i have all these links uh let's call this like link data i want the first paragraph of each page pages documentation so to do that i can take for example the first the first url do connect jsoop.connect uh let's see val doc equals jsoop.connect and then um i can pull up the html and find that uh it's a first article whose id equals wikiarticle so it's article hash wiki article dot uh and then within that i want the first child which is a paragraph and then scala dot head and that gives me the first paragraph of this article so combining this with the original index page that i found earlier um and i get the link data um sorry let's run this um grabbing all the separate paragraphs is as straightforward as looping over um the text title and href from each of these links let's let me just grab the first 10 links just so we don't take too long and connecting to that the url and then selecting the um what did i want i wanted article whose id calls wiki article a child paragraph and getting the first element and let's say i want to store it together the text in order to make sense of it later let's convert this to a string uh so i need mr.get here so this will make 10 web requests will take a moment and now i have the first 10 um the first 10 pages from the mdm documentation page with the title of the page as well as the first paragraph documentation and as i showed earlier it's simple to write this to disk somehow so i can do like os dot right let's say [Music] docs dot json and yes i write res 21 with indent equals four and i have an osr read the current working directory says docs.json uh you can see all the data's been saved um so the so that's more or less a demonstration for our web scraping uh the purpose of this demonstration isn't to build like a fully featured web scraper that can handle every edge case but rather the purpose of this demonstration is just to show off the kinds of things you can do in the scala wrapper without going too deeply so i can just open ammonite grab the library i need grab some third-party uh third-party websites script the information down and save it to disk all in the comfort of the wrapper without needing to go anywhere else the next the next thing i'm going to cover is a small github project migration so this will teach us how to use um the mx color apple to automate third-party services over the network in order to accomplish something that we want to do so let's consider a use case so let's say i have github.com obviously howie test repository which is currently empty and let's say i have github.com lee hari let's say request scala which is not empty and i want to take all the issues from request scala and i want to move them over to this lee howie test repository um why i may want to do that maybe i want to maybe the original repo owner has gone awol and i want to pick up pick up development without losing any of the historical context maybe i want to maintain the old repo for some reason and you want to make a new repo that's the same issues another continue pick off the work pick up the work from there so the many reasons why i may want to do is migration and let's see how to do this using uh the ammonite scalar apple um so starting from here the easiest way to access github is to it programmatically is through its api so if you google github api list issues uh we'll see that there's this uh api endpoint to list issues which is i want to make a get request to just report repo repos owner repo issues url um so this requires a token which i have already so i'm just going to load the token into memory this way and you can use ammonite's built-in request library to make a get to the url of api https api.github.com repos uh lee howie request scala um and then um i need to pass in some actually i think any passing headers equals authorization header uh token token and here i have the data back in the http response i can parse into json and render it nicely indent equals four um you can see this is um so this is the metadata for the repo if i want to metadata for the issues i need to put that in so now if i make this a bit bigger and browse it in the terminal i can see that this is a bunch of json that i got back from github as part of this http request all formatted ic ready to use so this more or less maps to what we can see in this example here and notably what what we may want to migrate from the old issue to a new issue is let's say we want to map the title uh when to map the issue body let's you want to map the ish to login user login just so you know who created the old issue and maybe the old issue number so you can easily figure out which of the old issues this new issue corresponded to um so to do that um i simply can take this json let's call it like issues json and for each element in issues json as an array i want the elements title as a string i want the elements body as a string i want the elements user login as a string and i want the elements number as a number which is an integer let's call it issues issues data so now we have 12 issues to migrate the last thing we need to do is we need to figure out how to create a new issues on the new repository using these old issues so if you look for in on the github documentation page there's a section how to create the issue you're simply making a http post request so i can so i need to make a post request so it's like request dot post to api.github.com slash repos test lee howie says test issues and i need to pass in some data um as a dictionary let's say i'm for testing it's called title hello world uh let's call body um hello um and then i need to pass in the same authorization token uh header that i had earlier missing a comma uh https so now we've made the post to github and if i look at my issue tracker you'll see that my one new test issue that was created so now doing the migration is as simple as taking this post and putting it in a loop so earlier i had kind of extracted the issues data with the title for the user login number of each old issue i can take this just loop over it so for title body user number in issues data um i can make this request.post and i'll go instead of using hello world for title i use the old title the body i will use the old body with the old user id with the old issue number um and the token behave will behave the same way so if i loop over this as a the page you can see the new issue is being created on the right hand side while i while my code is running on the left hand side in the terminal um and now that i'm done you'll see that all i've all 12 of my issues migrated over each of which has the same title as the old issue let's put them side by side just to validate client-side certificates same title same body and because the new issue is going to have a different number as well as a different user since it's using my api token to create it i put the user and number as a part of the bot as a string as part of the body so people can easily cross-reference this um so this uh this isn't limited to only migrating issues github also provides a api for doing issue comments so uh github create issue comment let's say you see i can create issue comment by just posting this other endpoint so let's do that so i need to post the test issues let's say i want to post the issues number um number 13 and i want to post to slash comments and comments don't have a don't have a title it's just a body um hello hello you'll see that the post goes through and immediately i can see the effect of this action on the right hand side um so the point of this demonstration isn't to do a fully featured github project migrator you may want to migrate close issues we want to migrate milestones you may want to migrate comments you may want to migrate labels assignees reviewers and so on the point of this github project migration is to kind of show that you can use the mlx scalar apple not just as something to wrangle data on your own laptop but also as a kind of command and control console in order to go all the way out to your third-party services your web your web services your back-end your websites and control all of these from the sake comfort of your scala rappel so on the right hand side here i have um the github web page on the left hand side i have my scalar apple and they can talk to each other pretty seamlessly where i can post data to the github web page and i can get data back from the github web page let's say listing the comments out and there's no real boundary between my console and the open internet everything i can do on the internet i can accomplish through using the mlx scalar apple um uh salt milk can i check how much time i have left is it five minutes left or is it uh 15 minutes left oh um yeah i guess we have uh like six minutes but you can okay you can take you know here okay um uh if it's six minutes left i guess i'll see if i can speed through this last section which is parallel web crawling so um let's say i wanted to look at the um the mdn migrator we had earlier sorry where was it um connect so earlier we had run this small like uh web web scraper that will scrape documentation off the mdn developer pages so it makes it lists the interfaces here grab the links and then for each page it will go and grab the first paragraph of each of those links i limited it to 10 in order to make sure it doesn't take too long but you can see that as i increase the size of this list i'm scraping the time goes up significantly because it needs to go and scrape each of these pages sequentially one thing that scala scale language lets you do very easily is it lets you parallelize this kind of data fetching or data processing code and the built-in features in the scala library lets you do that very easily so let's say i import like scala dot concurrent dot all concurrent.all duration duration.duration to infinite weight on it and like detail.concurrent.executors um let's say create an execution context to use uh execution context start from executor service uh executors new fixed thread pool so what i'm doing here is i'm kind of setting a global i'm i'm setting a global um thread pool with eight threads in order to use for all my parallelism needs um i can then go to my uh this data fetching code earlier and instead of yielding the original results i can yield futures that will do this data fetching in parallel sorry here's my data fetching code so you can see that i have spawned off 20 futures and each future can run in parallel and this can function a lot faster than if we had done it all sequentially so let's say i call this like fail futures is this and val results equals f stop map await dot result for infinity um uh yeah okay this demo does not want to cooperate for some reason um well and i think we are running a bit out of time so i'll skip this last demo since um we don't have quite enough time to go through it in detail um but basically uh what this point of this presentation is to show how you can do a bunch of useful work in the m night scalar apple that you may not have traditionally considered doing in the repo so things like web scraping where i can go to a third-party service whether some competitors website or some open open website and grab data down as i need it or doing a github project migration where i can automate and send commands to some network service all from the comfort of my scala report i can pull down data i can create i can tell it to create issues i can create comments and i can just write code in my mlx calendar app on the left while sending commands to the um third-party service on the right running in the browser um and i i skip parallel web crawling due to time constraints but hopefully this kind of shows that the m9 scalar apple is um a much broader tool than just something you may use to learn scala it's not just something you use to do oneplus one or create a linked list or to learn how to discover collections it's actually a tool that you can use for your production use cases where i want to go and automate some workflow i want to go and query some data i want to go and analyze some um something that lives on the cloud you can't do it you can do it using python you make it you using bash but often using the mlx style wrapper may be the best way of doing it uh so we covered using another style rapper and we also covered the style of lightweight scala programming that doesn't have many of the heavyweight frameworks that you may be familiar with right using scale on the back end so back-end services in scala may use like heavyweight frameworks like uh finagle like cats uh like akka these are these big things that require a big investment in order to use on the other hand using mlx scalar apple is kind of like writing python but without the dependency management problems and with running much better performance and static typing and it allows you to do things like deep download third party dependencies into your apple much more easily than any other any other such like rapid development system that you do um yeah so that's the presentation getting work done the scholar rappel um these three topics last week i didn't quite have time to cover are covered in my book hands-on scala the book goes into much more detail so like web scraping will cover how to scrape the comments and migrate them over uh then we'll talk about parallel web crawling how to use both blocking as well as asynchronous non-blocking web crawling parallel web crawling in order to script wikipedia pages or mdn development network pages much faster than before um and i put a little discount code here so if anyone's watching and wants a 20 discount can go ahead and take it um yeah so that's the presentation uh thanks hope you all enjoyed it and found it interesting you