SF Scala: Eugene Burmako Interview
Recording: SF Scala: Eugene Burmako Interview
hello everybody I'm Alexa Carrboro the organizer of chef Scala and we are back at Twitter for the fantastic meetup which is a part of the cognitive framework festival which is a week-long series of the tops all around the city Scala and spark are key parts of this called language of data pipelines and heroic refuge in Bamako I guess software engineer Twitter and the author of Scala markers so for those of us with the talk audience EDM can you tell us briefly what our column address okay sure thanks for a great question so it's called macros it's a language feature that was introduced in Scala to them the support compiled on metaprogramming which is basically a fancy way of saying that macros are functions that can talk to the compiler they can analyze the code that is being compiled and do interesting stuff with it generate more code apply advanced validations and so on so forth so please check out my talks for more details so you know people through the programming they seen multiple kinds of macros right so you know our like grandparents of the see markers and ctp proposed democracy and then people in loose have their own macros and and when Scott matrix appears in in scale I remember was horror and terror and people saying like how is not Enterprise ready yet we need to do certain up some stuff why I guys going on to the deep end and like playing is more like programming like spirit stuff we need to really get this stuff into the enterprise but that didn't work actually scholars markers too cold we see a lot of use of this you know nobody is afraid this is going to like it makes color more academic so people love it so why do you think this became such a useful thing well as I mentioned in my talk actually today I think macros they go along the lines with the headline of Scala which is basically more powerful users mm-hmm thanks to micros is become possible to achieve well previously will be milestones of expressivity and type safety that were previously outside reach and as a result where people people who use macros thank you library authors that write macros they can express their ideas in a very concise manner that's one of the coolest opportunities with macros provide one of the example here would be in all the rise of library called shapeless mm-hm which allows you to do generic programming this to write functions in normal Scala that process different case classes whatever the shape can say gets the name shapeless in a generic fashion so for instance for all business objects for all case classes in your application you would be able to generate a serializer without you know having to duplicate this logic again and again okay and kind of to the nature of scalar markers right so obviously you know CNC lots of markers and loose but different would make color markers kind of better for scholar alright well when it just started the PFL as a PhD student I knew about this exotic language on a.net platform called nebula and I was very inspired by the idea of compile time meta program as a result I just came to Martin Luther Steve the creator of the language my professor and said well Luke Martin we need to do macros so from my side this was a pretty easy but actually had to motivate this sample because column as you said is a production language you know rushes into enterprises use the over the Silicon Valley why are macros useful for sonic language well the special thing about Scala Macker is that they can talk with the compiler and with the scholars type system and as a result in comparison with with macros which are developed for an untyped language or with C macros which are basically text substitution now macros provide a high-level model of compiled and meta programming as a result we have quickly mix features for instance materialized errs which it was also covering in the talk which are about generic programming basically generating code for all possible types and being able to inspect fields all those types that's all those fields and so on and so forth this is something that's impossible in this clip possible and see and even impossible in template Haskell I think that's a signature feature of Scala macros the fact that they can communicate with the type system interesting and so Scala is typically used with details right and so I think it's very strong figurative Scarlets a scalable language and one thing I've seen people use repeatedly in spark they switch from sparks equal to two just Scala right and so that used to be a big problem with UDS in things like high right like you have to write your you know your functions in the program language and plug into this weird DSL to some very specific good right and and and yourself simplify this can it talk a little bit how macros basically make it easier for people to create yourself yeah sure so there actually has been a dissertation for one of my former colleagues at EPFL that talks about disciplines but also I'll try to condense it so basically Scala step system is very powerful and so it can accommodate a lot of compile time meta program enabled without macros so when you're doing a type level program and this is actually a form of compile time at the program even though you have to use implicit which is basically around the beltway of expression complex logic still you can do it and as a result one simplification to expect level programming it's using macros to express your logic is pay for its color code one advantage of macros is that they allow you to simplify your DSL and that's pretty cool another thing that people often use is additional validations which are unavailable in the type system so for instance a classic example that they keep bringing up over and over doing my taxes with poor macro so spoors it's a project started by Heather Muller also my former colleague at EPFL and it's about very fine closures that they don't capture stuff that you don't want them to so closures that you want to send over the wire if they capture I don't know a Wikipedia on your machine you definite don't want to send it over the wire right as a result of the macro you can go through all the parts of the closure all the abstract syntax trees all the nodes that compose well that this code is composed of and check that we don't capture you know the sweetie PDAs or something like mmm so just to sum it up first of all with macros it's possible to make the type based part of the SLV there and also with macros it's possible to go even beyond types to see exactly the structure of the code that's part of the DSL and I think these are pretty novel opportunities for the accelerators definitely over the years many of them have taken advantage of market sounds great in unison cement AFL and academia and now your solution theater I wonder how does this transition work right so on the one hand you kind of deepen research you're on top of research you appeal advanced cutting-edge language features at the same time you are at the heart of basically the biggest scholar company in the world a public company right with biggest color based probably you know biggest immerse call developers how how the distance vision worked for you you know how can you compare this to works all right well great question well first of all the transition it was not this abrupt because here at Twitter I still do a lot of research so thanks you know the generosity of Twitter actually reached an agreement that they can do 50% of my diamond open source projects call open source projects of my life and as a result what I'm currently working on is the project that we call Scala meta this is the open source foundation for better tooling and it's basically my job description here will better to semantic tools that can understand the meanings of the programs who talk to the type system and the compiler so here you probably can see parallels between my previous research this so as you can see it's not that different so previously I was pushing the boundaries you know with color markers with this feature that's built into the compiler now it's tools still the same that it's like higher orders column all the organization because litter is big enough right so you have enough kind of primarily level so we can go metal level right yeah so actually we have an entire department called E engineering effectiveness which makes sure that the old skull developers here skull developers we also handle Python get well quite a diverse bunch so we make sure that everyone is operating it well their optimal effectiveness and as a result definitely there's need for better tools in Scala still we as a community suffer from this problem and here's where we can make a difference with applied research okay so you mention open source and community and of is the cart open source and I recall that you know when you just started with her you spoke at the scalability conference which was held November and this remember were coming back scale by the bay which is even bigger conference there you know sculpt our village but we talked about empty and data pipelines to talk about data right so I just wonder what was your experience speaking it's color by the bay why would you recommend people speakers calibrate that what kind of talks do you think most useful to you and other folks in the community any persons of all the conference alright so indeed it was a great event in November so actually I was giving a talk exactly the next day after I moved to San Francisco this is pretty crazy but anyway I digress well all know it was a super streamlined event and I was very glad that it was organized at Twitter at my new employer so we actually had an entire floor dedicated to us speakers including hearing these I enjoyed the event very much because it was quite diverse and also a lot of topics so in the community they were presented there so unlike some conferences which are focused on academia or you know maybe on some super practical things like AG light band stack this was a really great mix and personally for me it was interesting to know the about the advances into I was giving a talk about Scala meta blade macros macros again met again and it was great to learn more about you know the adventures in the mono repo style development given by my colleague as to who the Twitter and the well in general it was a great opportunity to socialize with decision-makers in the community for instance Adrian Moore's Scala compiler team team lead from wideband was there I think mile seven as well yep he cannot yeah it was it was really cool to be able to talk directly to them and to see what's coming up next because actually that's a rare opportunity even though while we're here in the Silicon Valley very heart of what some would say skull development you know industry Scala based industry but still it is rare to chat with these people face-to-face and well we definitely love this opportunity thank you thank you very much for this description so we kind of look into the future what would you like to see more of the conference right how community can make it even better and how do you think we can make it matter better for Twitter because I work with different boxes whether I want to expose different pieces of Twitter so what would be from like a meta level Scala meta most useful for us to do both out from the outside and from Twitter side all right let's see I think well for me coming from you know the research community because in edge of skull development I think it would be great to have more coverage of for any concerns for instance that is pretty big and the last week in Copenhagen is called as Copenhagen Marcin under skip the creator of language announced the first preview release of Daria so everyone's really curious about what's in there because it contains a lot of new features it will be great to get coverage of something like that mm-hmm also on the tooling side while I'm the tooling person it would be cool to get different perspective from the communities because well in the Silicon Valley a Silicon Valley is become from the biggest users in the world and I presume that every such organization they encounter some problems because well scale in Scala t-yong you know just 10k long scholarship that would be something interesting personally for me and well as I mentioned before it's great to socialize with people like Martin Luther's key already and Moors and if you could get them on board that would be wonderful I well I really enjoyed the opportunity to socialize with them because I used to work in next office - Martin desk is office and now we where we get to chat together that would be a cool opportunity but I hope you seen this and we are really welcoming you to San Francisco any time so a dream is local so I'm pretty sure we're going to get that in it's very just on the Scala tooling so kind of end to end a tuning round scholar right from engineering effectiveness IDE and their life cycle of them to kind of organization deployments close the control testing right so the whole thing I think it is actually very kind of key part of scala culture rather those people really care about this stuff so I agree that I think I'd like to see more of this and I know for sure the having of the right folks will take it very serious when we talk about how it develops its harsh ecology and other things right so I'm pretty sure we're going to get some of this stuff but if you guys want to submit Botox on the tooling please do so this is piece to open scaled up by the bed Ohio please submit Dre talks and hope to see you there and thank you Thank You gene that was really really good thanks Alex Thanks [Music]