funconf 2013, Jamie Allen: What You Need to Know About Lambdas

funconf 2013, Jamie Allen: What You Need to Know About Lambdas

Recording: funconf 2013, Jamie Allen: What You Need to Know About Lambdas

anyway so this talk is a replacement talk again all apologies Mike slen wasn't able to present today so um I'm going to give a quick talk about what you need to know about lambdas in order to do them right so I am Jamie Allen I'm the director of Consulting for types sa um I've been working for types day for I don't know 15 months now and and I've been programming with scholar since 2009 uh so I've been doing a lot of work with actors and and functional programming and I've learned a lot of things about ways you can shoot yourself in the foot especially when you're dealing with you know production so I want to share some learnings about how to do them and not get into positions I've been before first off I love functional programming this is an important thing for me to say because some things here are going to be somewhat controversial when I think of functional programming I don't think of the Scola Zed type of functional programming that's a way to do functional programming it's a mechanism for doing category Theory through Scola but it is in functional programming in and of itself at least to me my basic definition of functional programming is the concept of immutable State you know programming with nothing being mutable referential transparency which is the concept of you performing an operation over some data and not changing it by accident to get a new value so the classic example that Runar bison uses is the string buffer inside of java if we use a string buffer to get a string value and then we say reverse on the string buffer not only do we get a new value of the reverse string but the string buffer itself has been reversed this is not a side effect you probably wanted and that's refer referentially opaque we want to be referentially transparent we don't want the data we're operating on to change and then finally functions as first class citizens we want our functions to be like any other type they can be passed around they could be used as a type itself they're just normal things in our ecosystem so that's functional programming to me now what is a Lambda a lot of people don't actually know what a Lambda is all a Lambda is is a function literal when I talked about a string if I put my name inside of quotes that's a string literal right a function literal is just a function that has not been bound to a name no vow my funk equal a function it's just you know a literal that has no binding whatsoever so that's good youd think maybe maybe it's not but it's merely an implementation detail of functional programming you can do functional programming with first class function and never touch a Lambda it's entirely up to the way you want to code so here's Java 8 I spent a little time playing around with job 8 and I tried to figure out how they were going to be using lambas and you know it's better than what we were doing before and our imperative style where we would have to Define some collection that we were going to put values into and then we do a four each and we'd maybe do a filter to figure out which ones we want to go into the new collection right but it's still pretty verbose you have to do these Imports because you don't have something nice like predef or package objects that you get in Scala to uh avoid this kind of extra stuff you have to streamify your collection see first I'm going to create my numbers list here and I'm just going to put three values inside of it but to actually apply a Lambda in Java I have to do this stream call which is going to allow me to apply my Lambda through map the map transformation right and then when I'm done I have to manually collect them into the new list it's not an implicit Behavior like we get with scala's map you know that when you do a map you're going to get back a new collection of the same type right you're going to get if you were working with a sequence you're going to get another sequence back if you work with the set you're going to get another set back what's inside of it may be different but you know you you don't have to worry about this extra step here but here's your Lambda it looks kind of familiar those of us from Scala the you know it's not an equal Arrow it's a dash Arrow but you know perfectly readable to anybody who's done Scala before in Scala the exact same code is slightly different we don't have to have the Imports and we can use the object extending app to you know start up our application just a list of one two three and then we map over them right and I don't use the underscore syntax here because usually when I'm presenting this I'm showing it to Java Dev devs and underscore is confusing to them understandably it's confusing to anybody new to Scola typically so just so everybody can understand what's going on for every number number plus one right closure how many people are programmed inclosure wow okay personally I like lessons you know I don't mind writing scheme for things like doing sicp whenever I was learning that uh book but the prefix syntax where we are we are going to um do prefix type operators right prefix notation where we put what we want to do before the actual operands always messes with me this is closure it's very simple code you may wonder why I'm doing a print Lin here when all I wanted to do was map over my values 1 2 3 well if I don't it's lazily evaluated and I don't get a result so literally I could put a print line before and a print L after but I would never see this line executed unless I somehow forced it to evaluate interesting little detail about closure but you the syntax is not quite as readable as Java in Scala but if you get used to it a lot of people like it then we have J Ruby and J rubies is actually pretty simple too all you have to do to make your Ruby run in the Java you know on the virtual machine is say this required Java that's it you can put perfectly normal Ruby inside it and here I Define my numbers 1 2 3 and then I have to do this collect do and my Lambda is right here with this n and n plus one for every n add one nice so what's the problem why are lamb is painful I have about seven reasons why they're painful to me first of all they're not reusable we're defining code that only lives at the call site right so it's got a limited lifespan and and later on if we want to reuse it we can't because it's only where we were using it before it's not bound to anything that we can reference oh well it's not testable in isolation I have seen people write incredibly complex lambdas multi-line lambdas that do all kinds of funny stuff resulting in a value great how you going test that you've got to call the method in which it lives and then you know all the other Behavior inside that method is now called in as part of your test it's hard to test just what the Lambda is doing right maintainability how did we know what our lambdas were doing we had to read them we had to look through every line of the code that's in inside of the Lambda and figure out what it's doing there's nothing in there that says this is a Lambda to do X or Fu it's just read it and figure it out I'd much rather be able to look at a name that tells me what's going on lousy stack traits is anybody ever come into this whenever you're using a Lambda what do you see what kind of information does it give you about that block of code it's got to do this thing called name mangling because it doesn't know what to call it there's no way for it to infer what the Lambda is doing so they literally call this in you know compiler devs name mangling and it's very apt because if you have 15 lambdas inside of one source file you're going to have a non-fund dollar sign zero nonf fund dollar sign n and you you have to figure out which one you're talking about you have the source line number to point you in the right direction what if you put a lot of functionality all in one line you got to figure out where in that line the exception was thrown from what part of it was the problem so I don't like that oh that's I don't like that very much um y oh remind me to touch my computer every day okay so lousy stack traces this drives me bonkers because I come into this situation all the time where people ask me for help I don't know what's going on and I go in and see the stack Trac and then usually it's a really big lamb that they written as well so let's look at the stack traces this is Java 8 as of June and yeah here's the code that I'm using to cause a problem it's basically the same thing you saw before where I was going to get a number plus one but instead I introduced a divide by zero because I want to make this blow up and here you see the name mangling Lambda dollar sign 0 more of them I have in there the more of a problem I'm going to have but I also hav to find a bug line number is one that's not even remotely possible in this code because of all the stuff I would have to put around this like just the public static void main you know that would be Source line one so you know they're somehow defaulting to a value here and and you know they argued with me about this a little bit Oracle because they were in the audience when I presented this at Jack's conf here and S Santa Clara and they said no that's not happening and I showed them the code and I ran it and showed them and they said oh so then they had to go fix it hopefully it's done now with the next release but uh this is unacceptable you neither know which Lambda you're talking about or the line number you'll never figure it out but by the way the sa I'm sorry sorry does it actually increment them though like the first L Lamb's index zero and then one then two yeah because they're going to have to be like Sam's right so um here's Scala again the code's a little more concise but it's really not better as far as the stack Trace goes first of all the stack Trace is bigger and you see here nonf fun dollar sign one means nothing so at least we give you the line number here the playground 23 that's useful closures is particularly honorous I mean it just goes on and on and on but a lot of it has to do with the lazy evaluation but you do see here that the function has been name mangled to fnc1 and you can see the line number where it occurred sort of um it doesn't tell you the class that you might think it says core. clj that's not the class I wrote but you can generally figure out where the problem went wrong and also the the top of the stack Trace is reflecting the closure um stack frames because of the lazy evaluation J Ruby they actually do this really well I was pretty surprised theirs is a very simple little stack trace it just tells you where the problem was and what the problem where the problem occurred and where what the problem was and as I understand that this actually came from a lot of the problems they were having with users who were seeing terrible stack traces and so they fixed it this is a great job it's also really difficult to debug if you write a oneline Lambda right how many people were using say Scala IDE version two or intellig uh like 10 where are they now they're 11 or 12 right 12 yeah maybe 11 back then you could put a break point on your numbers mapcore +1 but it wouldn't take you through the iteration of the collection right that was really painful and the only way you were going to be able to do that kind of debugging was to take your Lambda and make it multi-line you know brace line of code and Brace and then you could put a break point on the uh the Lambda itself and it would show you the iteration because of the source line Fidelity on the jbm now both Scola ID and intellig have fixed this thank goodness but it's still not such a bad idea to put the Lambda on its own line of code so at least you see the line number more specifically and break out the code to be more explicit as opposed to a whole bunch of things put together because then you know what part of it broke right as opposed to where in this big line of code some languages Scola use placeholders instead of names you know that underscore Java just announced that they've reserved underscore for this usage as well but they haven't said that they're going to do it specifically they see some value to it I think they think that might not be such a bad idea but then what do you get for your variable name when you look at it in a watch window name mangling ay you know who knows what they're going to call it and you have to figure out which of your variables you're trying to talk about here so you might know by type that might be a good pointer to it but you know the more complex things you're doing in a Lambda the harder it's going to be to figure it out so real quick digression this drives groovy people a little nuts they call everything a closure and in some respects I could see how that's valid on the jbm we have the ability to close over anything that's final right this is a rule of java and therefore enforce of jbm we kind of have to follow this in Scala as well our compiler can do some stuff for you but at the end of the day to close over a value it must be final that only means that we can't reassign it it has nothing to do with where this is on the heat we could change the value in the heat if this is somehow mutable right it's just like the idea of making a new collection in Java final but you can change what's inside the collection right so closing over over stuff is generally dangerous and this is what closures are usually referring to closures with an N not a j that means you're writing a Lambda but it's closing over external State and a lot of people use the terms interchangeably it's not that big of a deal I'm not that wonky but if you want to know the PLT programming language Theory difference that's it whether or not you're closing over something and in this case instead of saying that I'm just using a number literal plus one I'm referring to a value from outside the context of the Lambda I'm closing over it so closing over state is dangerous and in Scala we have the ability to use lambdas in a lot of different ways we can define a future with a Lambda right you give a future body that's a Lambda just by saying future open brace blah blah blah blah blah Embrace close over stuff you have no idea when that's actually going to execute that can lead to raise conditions very easily if you're using any kind of mutable State inside of a closure think about what you're doing give it a real good think as to whether or not you want to be doing that referencing something mutable from outside the context of your Lambda can really bite you so what is our solution we want to maintain our ability to write programs in a functional style but we don't want to get ourselves in all these troubles with lambas we could use named functions instead of using a Lambda we just bind the function to a Val some name B Fu it doesn't matter so long as it has a name that you can reference and reuse and test against and you know put break points in and it can't close over anything that's a good thing right sort of here's an example instead of using my Lambda inside of my you know what I get past to the map I'm just going to find it beforehand now also the way I've written this exposes um something bad inside Scala it would be better for me not to be saying this add one to value with the underscore here because this is actually going to evaluate differently than if I just said add one to Value passing the function by name and not referencing the individual value explicitly believe it or not that has a difference and I didn't even know that until I did this never thought about cuz I don't write code like this typically I wanted this to be more clear to anybody looking at this how the value was being applied lesson learn don't do that what this ends up doing still I'm sorry why is that because what does the difference this ends up being like a new instance of a function applied every single time as opposed to one instance of a function uh yeah didn't know that so what's the correct thing no value no um and I think part of that also incorrect I'm sorry it's not incorrect it's just that you get another anous class there less efficient less efficient and here you see the output I get from this is still saying a nonf fun even though I gave it a name of bad function in this case why don't I see bad function here in this case this is specific to Scala as well we have this concept inside of our compil code called scoping and the scoping of our function means it doesn't have access to its name somehow and I pointed this out to our compiler devs and said you know this is kind of a problem and they're like yeah this is not the highest priority thing under play at the moment compiler speed optimization is so but I mean there's a work around for it so I can understand now has anybody ever heard the term Ada expansion yeah all that really means is we're taking a method and lifting it to be a function we're just saying that we want a method call but we want it applied as a function to our higher order functions so our higher order functions are things like map flat map filter um what's another one contains right where you're saying something that allows you to apply a function to the values inside of your collection without using any the category Theory terms we're we're using this lifting to just take a method an ordinary method that meets the contract of what our function does and apply that to each value so methods is functions in Scala if we use a method we get some of the benefits we really wanted in the first place the scoping rules that were showing that terrible non fun stuff with the function value suddenly aren't in play but we also have something that's a perfectly ordinary method that anybody knows how to read and deal with some people new to Scala can be a little confused by this syntax because they haven't yet learned the concept of a transformation from int to int or int to string wow a function's type is defined by the input values and the resulting value right it can be confusing for people at first if they're dealing with methods they don't have to worry about it it's just a method it looks normal to them and when we use methods as functions what syntax should we use well here's a case where I've still done a def instead of a Val what I've really done here is only changed the way this is evaluated when is a Val evaluated whenever you ask for it no it'll be lazy uh a vow is evaluated when we create a class when we instantiate the class vows are immediately evaluated as much as possible Right AAP is evaluated when at the call yeah so in this case not only am I I I am getting what I want in my stack trades but every time I call bad function I'm reevaluating to a new function value which is expensive more expensive than we need it to be so it's probably not the way to go in the case of um wait oh another thing depth is not stable that's that's what I'm saying as far as when you get the evaluation uh but it's better to stick with our simple method syntax here right and then I can figure out all right well first of all if I have to do a lifting I can do it once and I get my bad function syntax as well I don't have to see any kind of a nonf fun business you do see it if you look a little bit lower but you didn't have to focus on that and in this case no I still did this I have to fix these slides so the benefits we can't close over variables except for things to find at the class level you could still close over that you probably shouldn't anytime you have anything at the class level those are the sort of things that you have to worry about thread safety right you want things to be passed in you want things to be local to the scope of any method that you're doing as opposed to referencing something from the this of a class instance right it's a general good rule of thumb for dread saate programming we want the internal variables this is actually wrong internal variables aren't operands not all of them some are passed in as operands others you can Define inside of your method as local variables to scope for that in that case they're only defined Within the stack right the stack frame of our method call they're not they values that might be on the heat but the reference them only exists inside the stack frame that's good we get better stack traces as we saw we saw the name of the method we wanted it's more debuggable because typically well if you put your break point on the method call you still get that iterating through every time it goes to the method you are going to see your breakpoint hit as opposed to the older idees which didn't allow you to iterate through or show the application for every single value it's uh more testable not more testable than a function a function is just as testable a named function but at least you can write all the unit tests you want against your method easier maintenance you have names you can give valid names to your function as opposed to people having to read through all your lambdas to figure out what they're doing and you get reability so you don't you don't have to be uh worried about being dry repeating yourself so keep Lambda usage for the smallest expression we all have to find our threshold for pain am I saying never use a Lambda no that's silly if you're doing underscore plus1 why wouldn't you who wants to define a method to do underscore plus1 but anything that requires multiple lines or any kind of complex computation might be a good idea to pull that out be able to test it by itself at the very least language creators including us at typ safe we've got to make this better really I don't think that Java developers know entirely what they're getting into here I got just sitting there and trying to do a flat map on you know some Joba collection that I had I I had a a list with two lists inside of it I try to do a flat map and I got a TI bear that yeah I've see scolar tight Bears all the time and I guess I've gotten used to them they suck they got a little better but they're still not great I saw the Java one I was like holy this is existentials and stuff like that I was like I don't know cu the type system isn't quite as exposed to us when we're doing Java right we don't think about how object is the upper bouns Paul Phillips was talking about this a little bit in his talk maybody saw that object being the upper context bound or upper bounds of our of our type um you know bounding well jav doers don't think about that they're going to start seeing it I was a little surprised by that and on top of that what are they going to do when they start having to write multiples of them whenever I'm teaching I always tell people start with the four expression right and if it turns out that what you're doing is really simple then maybe you can take it back to using just map you know if you want but writing a for expression is an easier way to think about composing your operations when you have multiples of them and Java doesn't support anything like that I think they're going to find out real soon that they want to uh that's it it's actually a pretty short presentation so back up the few slides the good example this one yeah cool yeah and don't do this I like I'm going to rewrite this to have JavaScript inside of it now with notor that'll be fun it's fun to just play around with JV eight actually and just see what they're doing anybody see the different things that they're putting in like you know Defender methods now we're going to have in essence traits in Java if you see Adrien mois he's the head of the compiler team he's running around here today he uh is more excited about these Defender that's coming in and our ability to leverage that for how we encode our traits it's going to be big for us because now the jbm will actually support what we've been trying to do he's more excited about that than lambas we have a unique problem though anybody know which platform you know Scala is currently targeted at java6 when do we make the move to Java 18 because when we do you know we're not exactly going to be able to support 16 17 and 18 at least we don't have the resources of Oracle so it's be really difficult for us to say when we retarget we've been asking the community finding out what people said and most people are like just stay on 16 please okay but then we're still in those tradeoffs that Paul was talking about where we're doing round pegs and square holes we want to be able to leverage some of the stuff that the Java platform is making but we may not be able to if the community wants us to stay Target targeted at 16 and 17 so we'll see no decisions have been made about there was a talk about 212 being targeted at uh 20 but I nothing's been officially decided they also have optional coming their concept of option and initially there was some debate about whether or not they were going to allow mapping over an option you know this this concept of fur is it going to be out yeah now they're doing that they are actually going to support that in Java that's nice um so you'll have Defenders Defender methods which are you know traits you have optional you'll have lambdas they really gun them for us yeah uh so I didn't follow the the Java 7 or Java 8 development much but I the the first example you showed where you had to convert to a stream and then collect at the end yeah I mean why didn't why didn't they just add map to list was that um I mean it seems like that's the obvious way to simplify that yeah you know I I I wasn't at the language Summit and I would have really like to ask them this week my guess is separation of concerns that they really did not want to introduce the functional application of you know lambdas and functions to collections except through a limited interface okay and you know what um Paul kind of alluded to something whenever he showed the 34 methods on sequence in a collection because of the hierarchy of the scolar collections hierarchy starting at reversible once and working its way down yeah he's not happy with that and I think that's what they're trying to avoid here so it's more orthogonal than it is in Scala where Scala it's a a strict hierarchy right throughout the collection system traversible defines the map you know the flat map and the filter and then all the collections have to meet that contract um there may be cases in collections where that's not relevant I mean some collections don't allow you to walk to next or anything like that right um so they kind of have to do that like hashset right what is next there's no order or anything like that right so you know is it's it may be questionable in the way we've done it and they may actually have the better answer and another thing is uh maybe they're not changing the API of the existing collections you know yeah so yeah I'm just wondering if it's something that they're going to yeah added in 1 nine yeah I don't think 18 is going to be the really interesting release I think 1 nine is going to be I think Java 9 is going to be really interesting to see what they do to to clean up what's going to happen here because this is going to be a case where they're just going to drop a bomb on the Java community and I'm not sure it's going to be adopted uh I don't know it's going to be interesting to watch you know is is Java 6 the Windows XP of java platforms might be uh especially with red hat now saying that they're going to continue supporting it in Li of Oracle right yeah I mean as I mean I know people who like Scala as a Java Plus+ like like C++ is a better C so these this functionality going back to it kind of to then they see it as like oh maybe I'll go back to Java now maybe uh you know I I don't know we're not particularly worried about it yeah and typ safe we a think it's going to help people understand the concepts we've had in our language for a while um and that might actually improve our Market where you know more people are going to be adopting it because they they understand them better now um but we also think we do a lot of other things really better like they're going to put completable future into Java 8 maybe I don't know if it's guaranteed this is another thing dougle is working on uh where you aren't going to have the blocking future right in Java right now you have two interfaces to get the values out get and is done and get is what you use whenever you've got one future you're running is done is what you do with a busy spin to figure out if your multiple Futures are done which one's done when and when you can start moving off and doing work based on that right um you know we've had fully asynchronous since SI 14 um you know that's ported back to Scola 293 so that's exciting we think it is and we'll see if that sort of thing is the Innovation Java can C catch up with so Java also getting implicits right kidding yeah I was say yeah you know there's a whole bunch of little things that inside of Scola we we we we disparage and you know things like implicit like implicit parameters you know ch's going to try and do things and they're going to find out they're going to have to be extremely explicit up front and have very long parameter lists to support the functionality of distrib work across threads and timeouts and stuff like that where you know we allow you to sort of say upfront you know how you wanted apply you know I don't I don't really know how this is all going to shake out but I'm not terribly worried about it Java eight fun Java nine that's going be really interesting so and they were worried Oracle was a little concerned about me giving this presentation cuz there were a whole bunch of them at Jackoff and uh they came up to me and said hey you know can I see your slides cuz I'm not going to be able to attend got to run back to the main campus and stuff so yeah sure you know they didn't disagree with anything I said and I think they do want people to hear some best practices they're going to have lifting as well they're going to take methods and lift them as functions so that's good uh but yeah they they know that uh this is a big marketing push for they they're really trying to get the community excited about it they they really don't want people running around saying this sucks which I'm not really doing I'm just saying there better ways to do it all right well thank you