SBTB 2015: Shadaj Laddad, Scala: Power and Versatility
Recording: SBTB 2015: Shadaj Laddad, Scala: Power and Versatility
Shadhaj Reviewer Thanks. We should take this out. Hi, everyone. I'm Shadhaj, and today I'm going to be talking about Scala, Power, and Versatility. So in this talk, I'm going to be going through a series of projects that I've been working on in the past year that show off this power and versatility of Scala. So let's get started with the first project, which is Google Code Jam. So I participate in the Google Code Jam programming contest. It's run by Google, and in it, there are a series of rounds in which you have to solve multiple algorithmic problems
So with Google Code Jam, you're allowed to pick whichever language you would like to use to solve the problems. So I chose Scala. And to make this possible, Google Code Jam lets you run your algorithm on your local machine with a downloaded dataset, and then submit your source files, as well as the solution to the dataset to Google servers. But this process can be quite tedious. So I wrote an SBT plugin to automate submission of solutions to Google Code Jam. Okay. That should work. Okay
Cool. So first, let's go through the manual submission process for Google Code Jam. First, you download your input file. This is randomly generated on Google servers, and that's what you have to run your program with. Then you modify your code to match up with the name of the input file. Since you are allowed to resubmit a solution multiple times if you don't get it right the first time, Google generates a different file name for every input that you download. So you have to always make sure that you submit the correct input file. Then you run your code with this input file name changed
You upload your output file. Now just like your input file is going to change, your output file might change as well. So you have to make sure to upload the correct output file. Once you've done that, you have to zip up all of your source files, and finally you upload your source files. So for every single submission you have to do in Google Code Jam, you have to go through this entire six-step process. And it would get tedious, and sometimes I would screw up the zipping, and it just got quite tedious. So I looked for ways around this. The first thing I found was a Python command line tool for Google Code Jam, written by Google
But there are a few things that I didn't like about the tool. First of all, you had to store your password in a text file, and I just don't like the idea of doing that. And even with two-factor authentication, you had to bypass that by generating an application-specific password. So I looked for a way around this, and I couldn't find one, so I decided to write my own SBT plugin. So with the SBT plugin I wrote, all you do to submit is run GCJDRS, which stands for Download Run Submit, the ID of your problem, which is A, and small, which is the type of data set you want to run with. Is this a microphone? Okay, yeah. So once you run this, it'll download the input for the problem, run with the solution you specified in your config in your build.sbt, and submit it to Google. So let's try it out
So I'm going to go ahead and load up the project for qualification 2015. So this is the first Google Code Jam round in which I used this plugin during the actual round. So I can go ahead and boot up SBT. Now, the first thing I'm going to do is run GCJ status. What GCJ status lets me do is check how well I did in this competition. So during the competition itself, it will show you if you got the problem correct and other things like that. But now that the competition is over, it'll show me how well I did during the competition. So you can go ahead and run GCJ status
Now, the way I bypassed the text password method used in the Python command line tool is booting up a Java X web view to take in your account credentials. And here I actually don't even store your username or password. All you need to access the Code Jam APIs is your Google cookie. So I extract that once you sign in and use that to make all API requests. So I can go ahead and open up authenticator, which I'll need. Okay. And now I need my two-factor authentication code. Okay
So I can verify. Okay. So it just did a lot things. I'm going to scroll back up to see what it did. So first, this is what happened before it opened up the web view. It showed that you need to sign into your Google account. And it gave a warning not to quit the Java process because as soon as Java X opens up the web view, an SBT process will pop up in your task bar. So you don't want to quit that as that will quit SBT as well
Then after you sign in, it extracts the cookie needed to use Google Code Jam APIs. It gets the middleware tokens, which are the cross-site resource forgery tokens needed to use the Google Code Jam API. It gets data that it needs to show you your status, such as content status and problems. And then it shows me. So here I can see the first problem. I got both the small and large correct. And all other problems, I only got the small dataset correct. So I wasn't able to get an algorithm fast enough to do the other solutions
So now that we've done that, let's try something a little bit different. Let's actually submit a solution. So here the contest is in practice mode. So they let you submit test solutions to practice. So I'm going to submit problem A, which is standing ovation with a small dataset. So I can go ahead and run that. It's downloading the input and ran my program. Now I say yes, I want to submit my solution
It submits the solution and I got it correct. So that's the flow you go through when you're using this Google Code Jam plugin. So let's go back to Keynote and see how this works. So first the login flow. When I first ran the commit GCJ status, it went through a series of steps to log me into the Code Jam servers so that I could access all the data I needed. So let's go through the flow of how that happens. I modeled this login steps as a series of stages and methods of how to go between stages. So here I'm not logged in
I haven't logged into Google. I haven't logged into Code Jam servers. I've done nothing. I've just booted up SPT. Now to get to the next stage, which is signing into Google servers, I have to sign into my Google account. And it does this by opening up the Java X web view. Now that we did that, I'm in the logged in Google state. So now I've logged into Google servers
I have the cookie I need to sign into Code Jam servers, but I haven't yet done that. So to get to the next step, I have to get my middleware tokens, which are the cross-site resource forgery tokens necessary to access the Code Jam APIs. Now, next, now we're logged into Code Jam servers, but we still can't use the download, run, and submit tasks that you would use to submit a solution because you have to tell the Code Jam server that I'm going to be using your APIs now. And you have to do that by doing two things. One, you have to get the contest status, which tells you if the contest has yet started, if it's currently in progress, or if it's in practice mode. And the second, which is get contest problems, which allows us to get the ID of a problem given the letter ID, which is ABC. And then we can get the ID we need to use when we're talking to the Code Jam APIs. Now that we've done that, we're contest initialized
So this is the last state. Now we can run all the commands that we would like to. So here I have gcj status, which is the command I ran at the beginning. And then I have gcj drs, download, run, and submit. And I've also split up this into subtasks so that you can only run a few if you would like. So this plugin is open source. It's at this URL. And next, let's go on to the next project, which is Janalgo
So a few years ago, I wrote a bioinformatics library for Scala, which allows you to represent biological sequences as Scala collections. So then you can do all of your mat, flat, mat, flat mapping, and any other collection methods that you would like on a biological sequence. So Janalgo started out as a JVM-only project. But over the last year, I've made it compatible with Scala.js. So now you can run bioinformatics algorithms in the browser through Janalgo. Let's try it out. So I'm going to go ahead and switch over to the browser now because we're going to run that here. Now I'm going to go to Janalgo.sh.me, which is the website I've set up for Janalgo documentation
So here we can see a few of the things you can do. You can create a sequence such as DNA, RNA, or a protein with a string. You can also do it just given basis. So I've created types for each type of element you can have in a biological sequence. And here we have some transformations. You can do an alignment. But what we care about is at the bottom, which is the demo that lets us align sequences from Uniproct. So what this demo is going to do is it's going to grab two similar proteins, one from a mouse, one from a human, get those from Uniproct, which is a universal protein database, and then align them to compare the areas of those sequences in the browser
So first I'm going to pre-fill with the protein IDs I'm going to get, and I can align. So now it's making those network requests. It got those, it aligned them in the browser, and now we can see here that we have these two sequences. So at the bottom is the comparison. So a star means that they're identical. A colon means that there is a mismatch. And a space means that there is an insertion or deletion of a amino acid in one of the two proteins. So now we can go ahead and switch over to Keynote
Okay. So when I switched over to Danalgo to also work through Scala.js, I had a few issues that I had to tackle, and I had to do a lot of work through the code. So I had to do a lot of work through the code. It was really ridiculously automated. He had a button that would put the room into instant party mode by spinning a bunch of lights around, closing the blinds, turning on a strobe light, and it would do some really cool things. So I wanted to make one of these in my room. And so Ash was born. With Ash, everything that you can automate is modeled around a series of services
So you write services that run on your server as well as a component to that service that lets you control it from a web browser. So in this example, I have two different services. One is a Spotify service to control the Spotify music player. And the second one is a weather service, which lets me view current weather data. So one of the interesting things I did when I was rewriting Ash from scratch over the summer with Akka was creating a single pipe to communicate between all of these services. So you can have multiple services here. In this case, I have Spotify and just a weather service. But all of these services will communicate between the server and a browser with a single pipe
Here I'm using a web socket. So let's walk through what the control flow would be like. So let's say you clicked a button in Chrome and in the Spotify service. And let's say this is a play button. So you want to play the current song. The Spotify service, and here all services are represented as actors, will send a message to the service messenger. Service messengers act as the communication boundary between these two sides of Ash. So once this service messenger receives the message, it needs to serialize it to send over the web socket
So in this case, I use Boo Pickle, which lets me easily do cross-platform binary serialization. So here I take that object sent from the Spotify service, serialize it into binary data with Pickle, send it over the binary web socket, unpickle it on the other end. So this is where one of the advantages of Boo Pickle comes in because it is cross-platform. So you can use the same serializers and deserializers on both sides. So here I deserialize the data with unpickle and send it to the Spotify server side. So this is where you can use the server actor. Now if we're going in the reverse direction, if Spotify wants to talk to the Spotify client on the browser. First the server side component will send a message to service messenger
The service messenger will pickle it on the server and it will get sent over the web socket. The service messenger in Chrome will unpickle this and send it to the Spotify actor running in Chrome. So let's do a demo. So in this demo I'm going to be showing two services. One is Spotify from this demo and the other one is remotely controlling a motor which I have up here. So I'm going to be remotely rotating this motor. It's not just for a demo. There's actually a project behind this
I'm trying to automate the window blinds in my room. So we 3D printed this wheel and the idea is we mount it on a wall and the wheel will rotate the strings coming down from my window blinds so we can open and close them. But for here I just have the wheel on the motor and let's do this. So I can go ahead and switch over to IntelliJ. I'm going to boot up this. So in this case actually the way that Ash works with both this JVM and JS components are it has a cross built Scala JS project and all of the serializers are shared files that will get both compiled to both sides and then I have different implementations of things like service messengers to be handled differently on both ends. So here I'm going to boot up the JVM side of this project which is Ash JVM and I'm going to boot this up. And so this is going to boot up my play server
Okay, now I can go ahead and go to the website and I'm going to go to slash fast for the fast optimized version. So here I can go ahead and load it up. Now Ash is going to connect to Spotify. So here. Okay, cool. So now I can see the current song that I was playing and it even does some fancy things by extracting out prominent colors from the album arc to set colors here. So here now I can play the music. And I can go to the next song to play another song
And I can do some other things with this. So that's the first service. Now the second service is controlling the motor. So the setup I have here is I have a Raspberry Pi here with a motor controller on top. This is connected to my stepper motor and a power supply to run the motor. And I'm going to connect it to my laptop through this Ethernet connection. And one of the really cool things about Raspberry Pis is its LAN chip will automatically detect if you're just doing a direct connection between two laptops so you don't have to use a special cable to do that. So I should probably plug this into here
And plug this because I'm running out of ports on my laptop. So plug this in. Now my Raspberry Pi should be booting up. I need to connect my wire for that too. So let me connect my power supply to my Raspberry Pi. So many wires. So now I'm going to be connecting my power supply to my Raspberry Pi, which is right over here. So now the Raspberry Pi is going to boot up
And the way I've set this up is I have a jar deployed to my Raspberry Pi. When I boot that up, it's going to boot up an actor on my Raspberry Pi that's going to continuously broadcast UDP packets so that the server can identify this remote motor because this does not have a fixed IP address. So my laptop doesn't know where this Raspberry Pi is going to end up at. So I think it booted up. So now I can go ahead and SSH into it. Hopefully this will work. So we cannot find my Raspberry Pi. Let me try searching for devices on my network here
This is connected, right? Yep. Connected to my Raspberry Pi. Let's try that SSH again. Come on. Work. with me, Raspberry Pi. Sometimes the Raspberry Pi takes some time for it to boot up and connect to the network. Hopefully this will work
Let me just start pinging it first. Okay. Looks like we're not finding it. Let me try reconnecting this Ethernet cable. Come on. Work with me. Okay. So Raspberry Pi doesn't seem to be responding to me
It is on, though. How much time do I have? Okay. Let me check first that my laptop has detected it. Okay. So it has found it. Okay. Let's try discovering some devices on my network again. Okay
So it looks like we can't find the Raspberry Pi. So I'm just going to leave it connected for now. I'll go through how Ash works and maybe we can try at the end to see if we can connect to the Raspberry Pi. But basically the idea is we would have been able to connect to the Raspberry Pi with this and then toggle the motor rotation from the web interface. So in the meantime, let's go over how Ash works. So Ash is built upon services. have two components, a JavaScript end and a JVM end. And they have a shared component which is the serializers
So on the JavaScript end, you have an actor which handles taking an input from the browser. You have a voice component so you can voice control. So I can show that. So here if I reload, okay Ash. My laptop doesn't seem to be working either. Let's try that again. Okay Ash. Play next Spon- Yeah
Okay Ash. Play next song. Now playing Animals by Maroon 5. So now I'm able to voice control Spotify. It even responds with some voice cues that what song is playing now. So now I can control it again. Okay Ash. I think that's not a command
So let's go into the code and see what the command is. So I'm going to actually go into the code for the Spotify service here. So all services in Ash are implemented as actors. So here I actually have an actor which represents the Spotify running in my browser. So let's go to where I'm registering. So this should work. Pause the song. So let's try this
Okay Ash. Pause the song. Okay Ash doesn't seem to be wanting to work with me today. But yeah. When it's quite enough Chrome will tell Ash that now the voice recognition is complete and it usually finishes off. So that's Ash. Now we still have a few more things in Ash. One of the interesting things about Ash is that it's built for modularity
So you can use the Ash core and add services to it. So the way Ash detects services is you have to put a package identifier which in the case of Spotify is me.shetage.ash.spotify. And underneath this package you'll have serializers as well as a service actor. And in the case of the JavaScript end, since we can't use reflection with scholar.js, I have you export it so that I can find it using the global JavaScript namespace. So service detection on the JVM, this is just reflection. I find the package and I find the serializers object there as well as the service actor and I boot up that actor. So that's what I'm doing here. And in the JavaScript end, I have my service store which looks through all the services
So here I'm actually using the same resource scholar object generation from Janalgo. And here I just use scholar.js dynamic to go ahead and get all of those objects that I need to use. Now with serializers, I want all serializers to automatically be exported. So in the trait serializers, I use the annotation js export descendant object, which basically makes anything that extends a serializer be automatically exported. So here I require you to define a pickler of any ref and unpickler of any ref. And this is usually built with a composite pickler from boot pickle, which lets you specify multiple types that you can handle in this pickling. Now, one of the things that I had to do when I was writing the portion of Ash for JavaScript was how do you write an actor in scholar.js? Because Akka is not built for scholar.js. So that I could get that feel of writing an actor and having those messages sent, I wrote Appa, which is the fake Akka for scholar.js
So this is majority of Appa. Here I'm just implementing an actor. I just have fake implementations of everything, but it's functional. So I don't do any fancy web worker or asynchronous calls or anything. But you can still get the feel and have those state machines that you would expect from Akka. So here I have implementations of everything. And then you have an actor ref. And here I'm not doing anything fancy with actor paths
An actor ref just stores a reference to the actor it's referencing. And it has a method send, which takes a message, as well as an implicit sender so that you know who the sender was if you want to send a message back. So these two libraries are both open source. And so maybe now we can try and see if my Raspberry Pi woke up. Come on. Ah. Okay. So it looks like it's not going to work with me today
But when it does work, it works really nicely. It starts broadcasting out UDP packets. My server parses out those packets and figures out the IP address of the actor here. Then it connects to that actor. A remote link is established. We also set things like death watches to automatically reconnect to the actor if either end drops out. And with that, you're able to even voice control the motor. So that's another voice control I wrote
So that's it for all the projects I'm going to be showing today. So we've seen a lot of different projects that you can do at Scala today. We saw a SPT plugin for communicating with Google Code Jam servers and submitting solutions. We were able to run bioinformatics algorithms in the browser. And we were even able to see a primitive home automation system. So here are a few links to if you want to follow the stuff I'm working on. I put all my code on GitHub. I have a YouTube channel where I publish videos about Scala programming
And I also have a website. And I'm ready to answer any questions if you have them. I'm ready to answer any questions. Yeah. So I listen to the server whenever I find a packet that is for that service because you might have multiple Raspberry Pis connected to the same network. So I look for one that's for that service and connect to that after. Any other questions? Okay. Thank you.