sfscala.org: Alexy Khrabrov interviews Li Haoyi

sfscala.org: Alexy Khrabrov interviews Li Haoyi

Recording: sfscala.org: Alexy Khrabrov interviews Li Haoyi

[Music] hello everybody uh I'm Alexi kraber the organizer of SF Scala and here on location at lithium with two talks today fast pars by and uh we actually have H us uh it's great to have you here it's the first time we actually have you at us Scola Meetup we had at some uh Scala byway conferences about this the first meet up talk so this is the first Q&A we get to do with you and you know that you know your work is very popular in the community your videos have huge following uh so folks have always the question how you can do all this right and so basically say you know you work at Dropbox by day and you play with scar by night and by the output you do at least you know work of five or 10 people you know at least that's visible side of the scholar so so how how you able to do all this uh like is it is it your like are you super productive are you kind of super efficient uh you know because you create multiple projects uh kind of one falling logically from the other so can you share you know do you know how this how this works um sure so um a lot of the projects are actually very old ideas that I have been thinking of for at least like four to five years long before I started using Scala um the library I'm going to talk be talking about today which is fast pars arises from me learning scalar par combinators back in 2012 mhm which arises from me trying to pass things back in 2011 through 2010 Okay and like you try doing something there are some tools for example at the time in 2012 I found like oh scal PR combinators are very nice and they solve a lot of problems and no one uses them for real and you try to fight why no one uses them and it's because when you try to use them there are certain problems that get in your way and it turns out in many of these cases the problems are not insurmountable the problems are relatively straightforward things that no one has had the kit has had the time to go and fix and make better MH and so fast par although itself as a library only took me something like two days to implement it's something that I've thought about for at least several years now I've implemented I've implemented one pass combinated library before in Python I've used two of them in Scala M the inbuilt ones and parboil 2 and then fast pass is the result of this experience okay yeah and so parboil based combinators used in your Scola compiler your Scola parser yes uh which in turn emanated from uh scal Tech right so did you write Scala compiler scal parser because you had to do it in scal Tech um I do not remember why I wrote the Scala parsa but I I think that was the reason you know here's my theory correct me if I'm wrong right like again again so so we'll know that you you know popularize scholar JS you a lot of work on scholar JS so kind of you're kind of mostly known for like really great work on scholar JS which brought into the mainstream so so here's my theory so you have a book heads on scholar GS which you know you wrote for the web which and then you wrote scol Tech in order to write scolar hands on scol scholar jazz book right in order to write scol Tech you had to a scolar parser because scol Tech is fully you know scholar compatible right so you can you can have any scholar there and then parser commentators you need to have to ride the scull parser right but the first pars is the next Generation right than the other one yes okay so I guess the answer is to how I I pick these projects is I build something I find something that's missing and I go build that and often it's not that hard to build you just need to not make the same mistakes as the past right right yeah um and so fast pass is a logical conclusion and fast pass has its own like bit set and other stuff inside that I had to build because of standard Library versions were not good um and like I don't I don't want to build as much as I do but given given the tools that are available out there and the tools I think I can make myself um it is the in my opinion the correct answer yes how to get these things out um and then the reason I packaged them up as individual libraries rather than having just one huge project is because I don't have time to maintain a huge project and if I want these things to stay alive I need to get other people to to use it at the very least to give me bug reports and send in bug bug fixes as you find them and that's why every project I make I package it up very nicely put it on the internet with SW of documentation saying this is how you use it Go and use it don't need to ask me any questions it's all here right and it's the same for scjs like I have something like six different projects I work with scy JS all of them can be can work completely independently and it's for the same reason I can't maintain a Scala JS web framework I don't have the time to do that right but I can maintain a bunch of libraries and it's much easier easier to get people to buy into one Library than is get people to buy into a huge framework yes and that's why I've done it in this way okay that's that's good to know because you know a lot of people are wondering you know why don't we have a Scola GS based web framework which will be like play right or like Ruben ra before and now especially with AKA HTTP it seems like a very reasonable back end for that and somebody should go ahead and package that right but we don't see this quite yet and uh you know that's like an interesting question and I understand it's probably a lot of work like more than one person right so do you think this is uh because colleages itself is more comp like complex than kind of typical front ends like angular or so or should it just stay the same way like everybody should figure out their own back end and figure out their own way to connect you know scol GS to back end I think that um currently scholar is largely early adopters that's one issue another issue is that Scala JS is a whole new world and whole new way of doing things like people have used Scala for a long time on the back end using Scala for interactive interfaces is a completely different experience yes and so the best practices are have not yet been discovered like people have tried many different things many of them have failed some have succeeded some of mine have succeeded some of mine have failed as well and it's clear what will win out in the end and it seems a bit to to me at least it's premature to say that we know what we're doing we're going to build an awesome framework because honestly we don't know what we're doing and like every week every month someone tries a new thing and maybe it'll work maybe it won't and I think that over time as a community grows and matures um some best practices will win out naturally and those will become the web Frameworks of of the future MH um and for me I don't have the manpower to make a framework myself maybe if I was working full-time on this I could but I'm not I'm doing this maybe 10 15 hours a week mhm and so my goal is to get as many other people excited as possible in order for them to get to do work for me without me having to pay the money right um and that's why I do it in this way and we definitely want to help you in all possible ways I think like a lot of folks who you know see work they really motivated to to try it and so as you mentioned like you know last uh SC by the by conference there were four talks about schol GS and you basically said like full talks in about this in Scala by the way I'm not doing any of them and so my job is done right uh so which which and you did the talk amonite which is the new kind of rapple so I I must ask here because you know like you mentioned that you know some something was not done properly and you decided just to do it properly so I think this is a fundamental difference majority of people will be afraid to tackle something like scull parser or jvm so you have like a model of gvm right in javascripts uh so so in Scola right and so so this is uh this is interesting right because for most people when they need to do something like you know like not like the Gen will pause and say okay like you know skull parer Martin DKI spend years riding scull parer I'm not going to ride skull parser you know or you know jvm obviously like like you know or sound Oracle spend you know men men centuries on on this I'm not going to try this so but you basically uh you you try this and you show that you know you have so your scholar pars about 1500 lines and it Parts 90 9% of scholar so so so you actually demonstrate that these things which are en nominally complex can be easy or you make them look easy so I wonder so how do you know because a lot of people don't do this because they don't think it's doable and so you kind of you know make this kind kind of bold claim and you just kind of share a results so how do you know like what's doable what's not how do you know before you approach you know like jvm and Scala that it will be doable and uh like why are you not afraid to just kind of try this stuff um every project has its own judgment and own motivation the the Scala parsa arose from my experience that using scalar par combinators makes it very easy to par complex syntax very quickly in like dozens of lines of code rather than thousands mhm the jvm on the other hand arose from the fact that one of my friends had written a jvm in coffee script and I looked at his jvm and it works it could you can run the Java compiler in new browser it can run Rhino it can run Ka scheme and it was like three 3,000 NES of coffee script and I thought well if this guy who I know can write a jvm in 3,000 n of coffee script I'm sure I could write a jvm in about the same amount of Scala and so I did that you know this is a case of you thought something was impossible someone else demonstrates that this is hey this is not impossible this is actually doable and so you go and try and do it yourself and see how far you get um so that's for that project so the other project for example my functional reactive programming Library scal RX um arose pretty naturally out of refactoring a bunch of front end UI code I just refactored it first question is okay I don't know where to update the UI so you sorry the first question is when I first render the UI changing the bits that you care about is annoying okay so you make it such when render the UI you can just say render and you'll reender the whole chunk the next problem is well which chunks do I want to render and then okay so you try to keep track of which chunks depend on which other another chunks to render before when you change something um and after that you refactor it a bit and you get this nice library and then I took this Library which was made for some freelance project and I looked at this project called Scala react in that's done by Martin's lab in E epfl and it's like okay this project Scala react seems like exactly what my uh freelance project was it needs in fact I almost have the same thing so I tried to download that project like most academic projects it doesn't work doesn't compile you make it compile it blows up at run time you make it work at run time and after a while you hack it together it kind of works and you realize that well what's the point I already basically implemented the parts that I care about they actually work I don't need to depend on someone's PhD thesis MH and so that's how that came about um the templating library I made SC uh Scala tags is actually something like the seventh templating language I have written among them like one in PHP two of them in Python one in C one in in F and then this one is in Scala mhm so maybe it's a bit obsessive to have done so many of them ever since I started doing web programming in 2010 but as a result of all this experience writing this one was relatively straightforward because I knew what I wanted it to look like I knew how I wanted to use it um so every Project's different I guess and that's actually you've been validated because I think scol Tex is now been used outside scholar GS it's been used as generally you know HTML generation in in scholar right so I think so all this experience kind of uh converged and you know people are using it uh so again it's kind of interesting so I think you're kind of again you following kind of A New Path your own path because a lot of people who in technology World get so excited about technology that basically once they find the technology they like they usually like start doing it 100% right so so you know and eventually find a way you know if they create new software they found a company around this and they basically you know spend all their time so and obviously you have so many different things you can pick one of them and basically spend all your time doing this but you basically maintain this balance right and you basically say you know you you write Python and JavaScript by day are you excited about this python JavaScript like what what keeps you doing this you know like outside of all kind of kind of business and kind of jobs and uh other aspects I'm just wondering technically right like is uh this python we are working on is this as intellectually satisfying as you know the schola and schola GS or kind of how you like why wouldn't you you know not like go all the way and do everything with scholar GS or kind of you know uh focus on one of these Technologies which you are developing because there are very many ways that can be you know lead to kind of new businesses and and startups and or whatnot right like uh I'm wondering how do you maintain this balance between kind of you know boring Python and JavaScript on the one side and kind of exciting scholar and Scholar on the other side so I had offer to go write Scala fulltime in the winter of 2014 and I've had several people contact me but that was the only like time I follow through and got a concrete offer MH um and I decide not to take it because I've decided that a lot of the work I do in at work at in Python and coffee script at Dropbox and open source in the scal community is actually very complimentary um as an example I would not have written fast Paras the library I'm going to talk about today if I had not written a different par combinated library in Python before not that Dropbox C before in school I would not have worked on scholar GS in my open source time if I had not spent a bunch of time working on the web in Dropbox in at Dropbox in coffee script and python at Dropbox there's a bunch of tools that we have and one of them for example is a static analyzer that crawls your coffee script crawls your python make sure the names line up and enforces basic invariance it can't enforce types but in forces basic things like names have to be defined somewhere right um I would not have written that at Dropbox if I had did not have my experience working open source in various Scala projects and there's a lot of this kind of cross-pollination that my opinion has been very valuable I wouldn't have written the amonite Scala repple if I had not used the IPython reppel every day at Dropbox and if I had not used uh needed a good repple to work on my static analyzer at Dropbox M and because I wrote this nice am rapper I now have a set analyze at drobbox which can query like find me all the dependencies of this file Union with all dependencies of that file and filter those that start with something so that's the sort of thing you need a good rapple for and the work open source has benefited my work at Dropbox work at Dropbox has benefited my open source work yep and it's I'm not going to probably not going to give it give it up so soon got it okay now that that explains so kind of so drbox provides kind of industrial motivation you know software used by millions of people so you see like what really work at scale and then you kind of uh take the learnings and apply them to different domains well that makes sense uh so again like a lot of people are very curious how you can do all this so maybe you can share a little bit of your background so like at what age did you start programming how you know quickly did you learn programming and kind of what very like first experience first programm languages what like most interesting experiences kind of growing up as a developer sure so my first exp my first programming language was XML and I was modifying user interface files for my EverQuest user interface can move can move around where a health bar is shown or what color your health bar is and that is all done by modifying these XML files which represent the user interface um maybe that doesn't count after that after that my next experience programming was in the Warcraft 3 trigger editor so this is something which you Ed to make the warcraft like interactive campaigns for the for the for the game you can say that oh if you if a player walks into a area when you define area by by dragging and dropping area on the ground perform action what is the action they click on the list of possible actions maybe select the if statement if and then there's a condition you can click on there's an action you can click on and then there's an else you can click on and you can have for Loops you can have variables you can have objects you have arrays and you have all sorts of things you can do with this trigger trigger editor it's under underlying this trigger editor you have this language called Jaz which is only used by blizzard for their games but I never actually went that deep um so I spent several years fiddling with that probably from well so I wrote my first XML probably when I was 11 or 12 years old I spent time fiddling with Warcraft world edit trigger World editor and Trigger editor from maybe like 13 to 14 15 years old MH and when I was 15 years old a bunch of people in my school which had at that point just signed up to do the international Borat program this international program other schools do got together and said that hey this program which everyone else seems to do optionally has this computer science like computer science um course this addition to like math and physics and history and geography and like and biology can do computer science so a bunch of us in I think in here would be called grade nine and grade 10 got together and say hey can you provide us a computer science course cuz like we're not yet at this level we're going to do the real Borat but maybe we getting ready for it yes um so there may be like 20 of us a bunch of us got together in the school computer lab every day after school then we learn programming and during geography class which we didn't take we would go to library and work on our programming homework mhm um so that's in that course we basically spent two years learning Java and it was great we didn't learn anything fancy but basically everyone could come out knowing how to work with the computer and I I believe like four of the people in that 20% course are now in the Bay Area okay interesting so after that I spent a bunch of time after that from grade like grade 11 to 12 I I didn't do any official programming course but course courses but I did my own work mhm like in my free time which at the time I was making games until I realized I would never be able to publish any games because back then Android App Store and stuff didn't exist and I wouldn't be able to afford like making boxes to sell the games in the shops so after I gave up on games and did my two years of military service and then went to MIT and from there on I learned how to do programming for real awesome awesome so when where was that school uh uh this is Anglo Chinese School in Singapore okay yeah that's awesome so actually so you didn't really start like this you know three or five years old like you actually did in your teenage years and kind of uh it's it's it's interesting right like uh uh so uh and I guess you know we kind of need to wrap up so I'll just ask you since you kind of arrived at this projects by thinking of many years and then converging and accumulating kind of result uh what is in the pipeline like what should we expect from you in a few years you know like what are you thinking know about right now and kind of kind of uh you know recent things around manite and fast barers what do you think will be next it's hard to predict what will be next so far my Approach has been entirely greedy I will see something that's immediately obvious I'll go do it and I'll sit back and wonder what I'm going to do with the rest of my time and then I go back to work at p on python at Dropbox um my so the the reason I've spent so long working in Scala in particular has been is because I think that Scala as a language has a really elegant like core to it has a really elegant core to semantics for example in Scala basically everything is a name or a function calling or a name calling a function not quite but almost and much more so than even in other functional languages like more things are functions in Scala than they are in hcll or F or other languages and they have this really elegant core in Scala that sandwich on the top by really crappy superficial syntax is really confusing it sandwich at the bottom by a really slow compiler that's a that's a huge pain to work in and sandwich at the sides by on a in my opinion not very good libraries like especially given what they could be like compare using a library in python versus using a library in Scala the scalar version usually has a million more types a huge amount of confusion and doesn't actually do what you want WR python does what you want immediately yes um so I think that if I want to get the place where I where I'm happy with with programming this is probably the closest I can get because Reinventing the core semantics is really hard MH Reinventing libraries is easy you can write as many libraries as you want if you're not good you throw away write another one Reinventing syntax is actually also relatively easy Reinventing a compiler I don't know but so far I think that there we can bring Scala to place that's much much better than we are now and for me personally I think that by working on Scala I can bring my own personal like happiness programming much further than I would be able to if I had started my own programming language or if I had put equal amount of effort into python or Ruby or JavaScript I think that Scala is probably the most promising in terms of making my life better all right this is super exciting know thank you very much for sharing and we're looking forward to your talk sure thank you thanks [Music]