The Future of Functional Programming

scala.bythebay.io: The Future of Functional Programming Panel

Recording: scala.bythebay.io: The Future of Functional Programming Panel

So this conference called Scala E. So we used to close it with a panel on the future of Scala and every year Scala is prospering. So we kind of will up the ante and make it the future of functional programming. I am completely under qualified to lead this panel so I'm just going to be here to moderate the social dynamics of the questions and ensure the fairness of time. So I just want to kind of state two points which can be a starting thing. So I think we live in very interesting times. On the one hand there is the race of ignorance and JavaScript surrounding us everywhere. So people stood as low as to do IoT in JavaScript

So IBM published the APIs in JavaScript. So the world will be hacked and all the medical devices will go wrong and people will die because of this. It's already happening. People actually hacked the diabetic pumps and demonstrated how this is done and repeatedly flushed toilets in remote houses and things like this. So basically we have on the one hand basically every subsequent generation is further removed from hardware. So people use the program in C and the JVM. Finally the program in the graphical terminal which is a browser. So we have subsequent generation kind of less rigorous

On the other hand we have this rise of sophistication in functional programming. People can do more of this. So kind of my kind of uber question is how do we convince vastly more numerous masses of programmers that types are good for them. So we kind of have a backlash against types in general not to speak of functional programming. So on the other hand the objection against functional programming and strong type systems is that it's hard. And we all have to overcome it in various ways. So I would say if the answer is education then how do we teach it? What are the issues? So I would just kind of plant these two seeds. And the rules will be so everybody will introduce themselves and kind of set their relation to functional programming

And then feel free to basically pick up on points and have a conversation among themselves. And I would say maybe 20 minutes into this we will open the floor to questions from the audience. And if you like sternly subject to some of the points by the panelists feel free to line up and ask a question right away. So at this point I'll just let the panelists introduce themselves. Sure. So I'm Miles Sabin. No, I'm Mike Pilquist. I'm subbing for Miles

So let's see. Functional programming. Many, many years ago I was in undergrad and my professor introduced me to ML, standard ML. And I fell in love with it and I went to her and said can you be my thesis advisor and teach me this stuff. And I went on a long journey through lambda calculus and pi calculus and forgot all of that and did like hibernate and JPA and big Java stuff. So many years ago I got into Scala and like, you know, there was this whole renaissance of those ideas and really enjoyed it. I've been applying it at Comcast now for something like seven years and have been very successful with it. So I'm John Degos and I was born into this world an imperative programmer sadly

Basic on a Commodore 64 and I spent more than 20 years writing imperative programs in more than a dozen programming languages. And working in code bases that were in many cases millions and millions of lines of code maintained by dozens and dozens of people over decades. And then I discovered functional programming. I'm like, wow, this is how software can be designed. And that basically converted me over to a true believer now. And I spent a good chunk of my time advocating for techniques and functional programming. Cool. I'm Paul Snively

I'm a principal engineer at Verizon Labs working in purely functional Scala. So if you try to tell me that functional programming is not practical, I'll just laugh at you. So I'm not going to bother having that conversation. The thesis I am going to put down here as kind of a stake in the ground is that functional programming is not fundamental. And I sincerely hope that all of you follow up on that. Thanks. We got a special mic for the Twitter people and former Twitter people. So my name is Oscar Boykin

I'm at Stripe now. I use functional programming mostly in the context, pretty much exclusively in the context of data. Feature generation, machine learning. Those are the kind of problems I work on mostly. My background, I actually haven't been programming professionally very long. I was an academic for a good period of time. And I started, I programmed from a young age as well. And I heard someone mention once this thing like Haskell

And he said something about it being this like very mathematical programming language. So I thought it was pretty cool. And I looked into it and I thought it was awesome because I really like math and stuff. And then when I went into, we decided to go into industry, Twitter was using Scala as a functional programming language in those days. And I knew that. And it was something attractive to me. I was able to get in and work on a lot of data projects. So that was kind of like my background and how I got here

Hi, I'm Jillian. I'm on the Finagle team here at Twitter. So Finagle is a protocol agnostic RPC system for building distributed systems at Twitter. Pretty much all of the services at Twitter use Finagle. How can you think of your service? Well, you can think of it as a function. I'm sure you all love that. And then on top of that, you can layer various filters. We think of it as a function from a request to a feature of a response

And then there are various kind of operations you can invoke on that. So it presents a really nice functional interface for abstracting way a lot of concurrency and distributed systems problems. I'm Stu Hood and I'm also at Twitter on the build team. So my history with Scala kind of via imperative did a fair amount of Java and other imperative languages, PHX. PHP and Perl. Slightly shorter history, but I made the transition into JVM languages and then via closure having functional but not having types is kind of an interesting thing to think about as well. But one of the types and ended up in Scala for that reason. And I've done a few fairly large systems at Twitter in Scala and just hugely enjoyed it

Now I'm more on the build aspect of it. But I'm sure I can find ways to disagree with some folks. Okay, so now I will invite you to disagree with each other. So, or you know, if there is no disagreement then I will ask questions. So it's up to you guys. What do you prefer? If you have questions for each other, you know, or I will ask Paul, what do you mean by functional programming being not fundamental? What I mean is that if you look at typed functional programming, and there's this thing called the Curry-Howard isomorphism that relates it to logic. Then you look at logic, and you look at probability theory, and study for example E.T. Jane's probability theory, the logic of science

You realize that logic is also not fundamental. That probability is an extension of logic. And then you look at the relationship between probability and algorithmic information theory, and realize that probability is not fundamental. And then the next thing you know, you're looking at dissipationless computing, and hardware is getting purely functional in order to not dissipate heat, not to lose heat. And that's crucially important these days. But that's because of algorithmic information theory. That's not because of functional programming. So what I would like to see functional programming evolve to, since we're talking about the future of functional programming, is I'd like to see us quit constructing false dichotomies, for example, between the value level and the type level

Those are just two stages. So I want more stages, not fewer. And I want a unified type and term level language. I shouldn't have to use a different language to program at the type level than at the term level. And I want it to be information theoretically aware, I guess. So for example, I want a hardware description language that will let me build dissipationless circuits. And those will be expressed exactly the same way as I express a type level program, or a value level program, or whatever. But I think the fundamental thing is algorithmic information theory

And you can get logic and probability and types and everything out of that. I actually, I did a lot of work on quantum computing and reversible computing, so I don't want to totally spin out in that direction. But I'm a little bit skeptical about like, like the actual impact of that. So I would be interested to know like, like, why, why, because you can't really have dissipationless computing, which is nice and functional, in the presence of errors. And errors are going to creep in and to keep them out, you're going to have to do work, like thermodynamic work or otherwise. So I'm not actually that bullish on it, like being a thing. Do you think it, like, is it really a thing? Not yet. Yeah, but you're right

And it's interesting that you mention errors. A long time ago, relatively speaking, I guess, there was actually a paper about a lambda calculus with error correction called lambda zap. I don't know if you're familiar with that. But yeah, you might want to look that up. But you asked this question, like, what's going to make it easier to learn? I would say, I don't know if you guys have seen this paper. There's this paper from the San Diego group, I'm often bad with names. And I think the title of the paper is something called ill type programs usually go wrong or something like this. And it's providing type witnesses for, like, type errors

So a lot of times when people are learning functional programming, they've got all these types. And the compiler says, you know, it doesn't compile because, like, this type is not exactly that one or whatever. But they don't really, like, want to get into all that. But if they're working with... So I guess it was someone said it this way. When you're working with Python and your program crashes, you kind of know you're responsible. Like, Python proves it to you. It's like, you pass this and, like, I've got a string here and, like, you know, the string is, like, hello world

But you're adding it to a number and, like, that doesn't make sense. So it kind of... Python proves to you, like, you were wrong. But people have this idea because humans are so, like, into delusion that, like, it's actually the scholar compiler's fault. My code's actually fine. I'm fighting with the compiler. But it's kind of wrong, right? But, like, it's not, right? Usually. And so these type witnesses..

I'm optimistic those type witnesses could be, like, you know, your program doesn't compile. The types are wrong. And, for instance, if I had passed X and Y and Z, it would have busted this way. And that might be a way that people kind of get on board. Yeah, that's interesting. I'm also... you mentioned type and value level programming. And I do think of Idris and I..

Idris the only language... not only language. I think of the most popular languages that is merging those systems. And I still do not know what the practical implications are. You know, frankly, like, for real programs in the real world, am I going to see some effect here? And so I'm excited to be convinced. I think I have a pet issue that I would love to see more focus on, which is essentially session types and making session types awesome. Just because it's still the case that when a protocol has multiple methods on it, you can just call them in the wrong order. And we can't do enough to enforce that

And I think one area, like, practically where people have been doing that is in Rust, frankly, with the ownership checking. Just preventing aliasing is this huge thing. So, compilers and types, but not necessarily functional, is this other area that I think is very interesting right now. Yeah, I certainly agree with that. There's some interesting work in cryptography and in security generally, for certainly around session types. Also around things like non-interactive zero-knowledge proofs of cryptographic properties done with the type system. So I am actually fairly optimistic specifically about the application of functional programming and type theory to security. But it is definitely early days

Yeah, for sure. Just returning to Stu's point about Idris. There was recently a paper from EPFL that talked about predicate refinement types. And for me it was very interesting because, you know, I've seen the types of things we can do with Idris and maybe the types of things we can mock from Idris with Shapeless and Scala C. But, you know, the predicate refinement types in this recent EPFL paper and the additions they made to Dottie felt a lot more Scala-like, if you will. Made those same ideas, those same type of programs much easier to represent in a way that felt a little bit more natural. Yeah, I'd like to say that I think that while functional programming, I think we're all big believers in its power, it is tremendously powerful. I don't think necessarily though that it's going to take over the industry and achieve 100% penetration

I think that functional programming allows us to build larger complex systems and have tools that we can use to reason about them in simpler composable ways. But at the same time, I'd say a good chunk of software development is focused on low quality, cheap production, mass production. And for many types of applications, you don't need anything else other than that. And in fact, to invest even more into the cost of building some of these applications would just not make sense from an ROI perspective. You would spend too much money to create something that doesn't end up giving its return. So I think functional programming, the best thing that we can hope for functional programming is obviously it's not going to go anywhere. I think it's growing really, really fast. The content at this conference, I think, speaks to that

Just how many sessions here were functional programming related. But I do think that there's an opportunity to take sort of an essence, some of the major concepts and pull them out into other languages and tool sets and make them much more accessible and broader. Lower the cost, lower the barrier of entry and lower the cost of using some of these more key abstractions in order to enter mainstream. And again, I don't think that if history has taught us one thing, it's that one size never fits all. And looking for one solution that's going to solve everyone's problem universally is probably not going to happen. We have more programming languages now than we ever have before. And with the rate at which people keep creating programming languages, that doesn't seem likely to slow down anytime soon. So FP is great, but it may not be the future of 100% of all computing

But I wonder, I mean, I would like you to speak more to that because that was an interesting point. Because like, I kind of, I think roughly agree, probably a lot of people have that feeling that like, yeah, it's like, especially if you try to do something at Idris, the cost to get that method proved correct is like relatively high. Now, is that a fundamental thing? That like, it's fundamentally going to be more costly to your functional program? Or can we like what what steps can we take? Like, you got to right at the end, but you were saying like, you know, like the cheap, you know, like crap software, like we can't afford to make those products with like high engineering. But can we can we lower the cost of engineering such that we can do that actually? I think that's extremely important. And a good point to make specifically with respect to dependently type programming languages like Idris, the cost of using them is very, very high. And if you've ever tried to prove that a sorting function actually sorts the elements in the correct order, you're probably looking at three or four pages of extremely dense code that most programmers would not be able to write quickly if they were able to write it at all. But if you take a look at, for example, refinement types, refinement types basically push some of the work onto the compiler, and they allow the compiler to check in variants of your code. So people can just state sort of ordinary Boolean predicates on on values and have the compiler do the heavy duty work of narrowing in that type and proving that a value is is in some narrower member there

And I think that's the kind of thing that it's very, very simple conceptually, and it pushes some of the computation into the compiler. And that's where we need to go is compilers that are capable of proving things for humans because humans are not very good at proofs. And that's probably never going to change. For both. Yeah, it's just an answer. Exactly. So an example is we have this type called int, but sometimes we want to guarantee that it's not negative. So for example, if you're the apply method on Scala's vector, you can pass it negative integers, and that doesn't really make sense

So refinement types would allow us to say we only want an integer that is greater than zero, or even an integer that's greater than zero, but less than some other value passed to the method. So I'll put in a plug here for a library for Scala called Refined that actually does exist. You can download it today and use it today, and it has all kinds of awesome predicates like this, including, for example, you never have to write another email regular expression again. There is actually an email address type that is correct. You don't have to think about that anymore. You don't have to go to that full back page of the regular expressions book that has the email regular expression. That's just one example. Another really cool thing about it is, for example, if you do say int greater than five, and then you pass an int greater than ten to something that takes that, that actually works

The compiler is smart enough to figure out that any int greater than ten is greater than five, and it'll take that just fine. So it's super, super nice. I highly recommend it. With respect to the education thing, that's a really good, important question. And I do think that there's an opportunity to better align what we do in the software industry with what is going on in education. So a lot of critics of education have talked about the need to teach computational thinking, for example. The Curry-Howard isomorphism tells us that an algorithm is a proof. So it sounds to me like what they're saying without wanting to say it is we need to teach logic again

Logic used to be part of the standard curriculum, okay? It used to be part of the trivium, right? So this is a very old discipline, obviously. For some reason, we don't teach the trivium anymore. I don't know why. Call me crazy. Seems like a good idea. But at the very least, you know, logic as a formal discipline, I think should be reintroduced, because that's directly how you get to computation. And failing teaching everyone how to interact with this, I think there was also some mention of having multiple layers within some program. And there's a principle I've kind of tried to apply in a few other places where you, similar to having a DSL at a high level, you have portions of your program that are simpler because you've made the sandbox that they are playing in as safe as possible via types

So types at the boundary, I think Adrian made the good point about definition site variance rather than use site being a thing that pushes it to the writer of the library rather than the caller. And so more and more of that. I think it's important. Like Scala has the benefit of being mixed mode in some regard. But, yeah, I don't know if it looks like an explicit switch or if it's just there are features that you generally don't have to know about if you're a consumer. And I think at Twitter in particular, I think we can say that we have very large code bases in which some people are complete experts and other people are day one Java programmers or other, you know, coming from JavaScript. Or writing JavaScript half the time for that matter. And I can't say that we've seen a huge failure there in terms of having a good gradient from complex usage to simple usage

So Scala seems to be getting some things right there. I just wanted to ask one question about very dangerous development occurring in data. So personally, I need all of your help, you know, to help with that dangerous development, which is the rise of Python for machine learning. And so, I mean, I'm an advisor to Galvanize, which is an educational organization. They have masters in machine learning and they have a boot camp. So what happens is, so the field of machine learning, data science and AI is extremely hard. So basically, there are multiple boot camps, multiple programs which are cooking data scientists. And so they basically teach very good, reasonable people data science as fast as they can

So what they typically do, they pick Python as a non-language. They need to teach you k-means in some kind of pseudo-code-like thing. And so they essentially need to pack a lot of algorithms and concepts from TensorFlow and things like this in a very short period of time. So what happens is they completely skew towards algorithmics and completely meet not even data engineering but software engineering. So basically, the immersion class of data scientists know how to do a very specific step of math, kind of off-the-shelf algorithms and some knobs. And they're not trained in, first of all, how to integrate it with other pieces of the data pipeline. They're not trained in standard software engineering techniques, right, on continuous delivery and testing. And so, on the other hand, in Scala you have these beautiful abstractions, right? And I think Twitter actually was on the leading edge and right with the summing word and algebra and Stripe is doing some work there

So the question is, how can we convince the world that this is the better way to do data with also all the best practices from software engineering? So obviously, I've thought about that a lot. And I mean, honestly, in my opinion, I mean, you're not, you're going like the answer, I was going to say, which is like, I'm not sure that you necessarily do. So I actually think there's like, data engineering can be something very different from like data analysis. For data engineering, data pipelines, it's much more similar to like software engineering, where we probably do get a lot of value out of these things. But when you speak about machine learning in particular, they have a really nice advantage of machine learning because like you don't like, like just write some machine learning code and then out pops an algorithm. You're like, oh, I hope this is great. And I won't really look at it carefully. Part of that pipeline already has its own like testing and validation

So like if like maybe you wrote a bug and like you didn't know it and that's why your algorithm, like your, your test error isn't very good or your prediction error is high, but you generally know it at least. A lot of software programmers, like they don't even know their codes wrong, you know, so types at least kind of rule out some ways of being wrong, even if they didn't test everything. And, you know, ideally we might get the world where the types would rule out all ways of being wrong, right? But in the machine learning world, they're almost like more like experimentalists in that they have this ground truth that they keep kind of coming in contact with. So I think it actually raises like the competitiveness of Python in that area. So it makes it like relatively hard for I think tight functional programming to catch the usability of Python in those areas. I don't think it's easy. So Twitter has done that. They use it in some ways, but usually data scientists aren't like pumped about it

You know, they're, they're usually like, I like using R more, but I guess I need to make this like, like the tyranny of these software engineers are forcing me into data engineering practices. So I guess I'll do it, but I don't know. It's interesting though. There is in fact a very good machine learning library in Haskell called HLEARN that really exploits the algebraic structure of things like distributions and so on and so forth to do very rapid cross validation, to do very rapid introduction of new samples and so on and so forth. It really, I haven't seen as much discussion about as I think it warrants. I've gone so far as to threaten to port it to Scala, but just, you know, nobody's actually said, yeah, I'll pay you to do that yet. So, you know, anybody, any takers? So it really is the case that these, these weird abstract sort of algebraic category theoretic concepts are relevant to machine learning. In particular, they're relevant to performance, which is a weird thing to say about, you know, functional programming and types and Haskell, you know, and all that kind of stuff

It's not generally known for that. But there it is. And I would love to see that intersection emerge, maybe emerge from the Haskell world to the Scala world or something like that. So just it can get a little more visibility. I think the pattern of high and low level and having a sandbox, I think, begins to apply there as well in the sense that if you've made your sandbox sufficiently shallow, that you don't have very much untyped depth. You can have this API and the Python is just this tiny wrapper around it, hopefully. You're not actually creating your own data objects. You're working with data objects that are already typed and then you're just bouncing off, you know, bouncing off an iteration until it works

And then the submission is the compile and there's just not very much to compile. And like a crazy argument might be what if you could run Scala or if you just if you just use dynamic for a whole bunch of stuff at the edge, would some of that flexibility be reintroduced still in Scala, write it like Python. It's not super typed, but you're in a fairly shallow area. Now, I wanted to say that I think that Paul's point is is really good here. And that is functional programming hasn't had a lot of time to work on machine learning. But some of the stuff that we're seeing come out of FP is really, really exciting. But that said, like in order to use one of these languages, you really do have to be a software engineer. You have to know quite a lot about software engineering and type systems and so forth

And that's not going to happen for for data scientists. For many data scientists, that's not going to happen. It's going to be more use case driven to Oscar's point that in some use cases, yes, when you're engineering software as part of some production pipeline productizing your data and that type of stuff, then, yeah, I could see some of that shifting into functional programming languages. But for the other type of stuff, I think the barrier of entry is too high in order for a non full time professional software engineer to pick up everything they need to know to learn a language like Haskell effectively. Do we have and I don't know the answer to this question. Do we have a sufficiently good notebook for for Scala? Yes, okay, we do. Four or five of them. Yeah, four

Yeah. At this conference, we had three, right? So we had Spark notebook, we had Jupiter with new Scala edition, and we had Zeppelin, which I think are pretty good. So people are picking them up. Yeah, I just wanted to add to John's point. In the grand scheme of like evolution of industries, we've only been doing functional programming for a little bit, right? You know, in a sense, you know, Ruby and Python and JavaScript did for the function what Scala has been doing for the Monad, right? And that's popularizing and bringing it to a much larger audience than perhaps was used to seeing it. And so I think it's kind of, you know, difficult to say that like we have this brand new sort of nascent industry in machine learning and deep learning techniques. And we're expecting them to like sort of catch up with everything that's like cutting edge, state of the art, you know, FP techniques. That might be a little much at this point

And we don't really know what the best way to do it is at this point either. I'd like to invite folks from the audience to line up and ask your questions if you'd like. Just so the mic is there. So you can just come to the mic and we'll see the material ask you of questions. Please introduce yourself and then ask a question. I'm Scott Calvert, developer at TripIt. And I'm wondering, we're talking about the ease of use. I know Paul Chisano has been putting a lot of work into his Unison system, which frankly I'm still trying to catch up with FS2 and everything else

So I haven't paid much attention to it yet. But I wonder, you know, addressing ease of use and learnability if you've had a chance to look into Unison. I don't want to stop anyone who has. So if they do. But it does remind me I think we should probably talk about Elm if you've looked at Elm, which is a little bit maybe related. But if you want to directly discuss Unison, I'm only glanced at it. Well, yeah, really briefly. I do think that something that Paul has really tackled that a lot of us have overlooked in functional programming is visualizability

There have been other attempts before at kind of a concretization of the data flow through functional systems and the relationship among functions and composition and all that kind of stuff. I haven't had nearly enough time to spend with Unison, but I am, I have to say, really glad that Paul is sort of returning to that tradition and trying to do it right. Yeah, go ahead. Well, I was just gonna say, I mean, I'm surprised I didn't think about it. I'm sorry I didn't think about it sooner. But Evan, the guy who works on Elm, he has this great talk, actually, which I would recommend to everybody, actually, back to your question, like Let's Be Mainstream, where he talks about this, like, how a lot of the FP adoption problems are, like, self-inflicted, that there's a lot of cultural issues of, like, where there's, like, you know, maybe a lot of, you know, pissing contests, for lack of a better term, of, like, you know, who knows the most obscure things and whatever. But from JavaScript, who are from, like, a very, like, untyped world. And they see a lot of value, and they're able to kind of, like, get on that on-ramp and do something

And so that, to me, I think, is a case for optimism, that the techniques are making impact. I do think the usability concerns of programming languages have not been given enough consideration in order to substantially improve adoption. And Elm is an example of what happens when you have a language that's focused on usability. You can get rapid adoption really, really fast by people who don't have a lot of background. However, I would say the flip side of that, especially in the case of Elm, is that Elm does not have that transition path from being a beginner tool to an advanced tool. It's strictly a tool for a beginner. And you're eventually, actually quite soon, going to run into limitations where you have duplication, and the language doesn't provide any means of allowing you to abstract over that duplication. And some programmers won't notice it for a while

Some will notice it right away. But I think that to solve the problem, it's the same in user interfaces, where you really want a beginner interface, and you want to be able to switch that over into an advanced interface. And people do that with keystrokes and lots of other tools, you know, hiding stuff and whatnot. But it's not been done consistently for programming languages. The adoption of WebAssembly will rise to the initial? Now, the question was, will WebAssembly give rise to more functional programming principles in front-end development? And I can't say for sure. But I can say that, sort of as a compiler writer, you know, myself, that targeting JavaScript is actually relatively easy. Targeting WebAssembly, you have to build a lot of your own sort of machinery. So it's possible that won't have a substantial impact on the number of languages we have that target the browser

I just want to get one last thing in before we let Rob ask his question. But on the topic of visualizing functional executions, perhaps ironically or not, the Java side of Twitter recently plug open source something called nodes, which is anyway. But the idea is to compose nodes, which are functions, which have explicit inputs, which is very interesting and different from how you would generally use futures or monads in general, I imagine. But, yeah, you have explicitly declared dependencies. I think names are required on a node. And so you end up with an execution graph explicitly laid out. And, yes, it's all futures all the way down. But you know exactly what's going on

You get default logging and metrics and all sorts of stuff. And it hasn't seen a huge amount of uptick yet. But great blog post out about it. So worth checking out. I think that's kind of an unusual place for that to have come from. Hello. Okay. So this question is kind of related to what you were just talking about

I've sort of come into the Scala conferences. I've seen sort of a progression. If you came to a conference maybe three or four years ago, people were explaining what monads are. And now you can pretty much talk about them if people know what you mean. Last year was the free monad craze. And everybody was talking about it. And they still are. And at this conference there were three people who talked about fixed point types

And that's not anything I had heard about at a Scala conference before. So there's kind of a natural progression. But, well, I don't know if it's natural, but there is a progression. And, but so... It's deeply unnatural. Yeah. Something that concerns me is that, you know, as, as the abstractions become more general, right, and as the types become more expressive, it gets harder to learn. It gets harder to absorb, hard to explain, harder to learn

And, and I wonder how far this is going to go before we sort of have a conventional notion of what our comfort zone is as industry programmers. And where is it going to settle down if you have any predictions about that? It's funny you mentioned that given that I just talked about algorithm, algorithmic information theory. Because, and by the way, there's another name for this, which is Kalmagorov complexity. And the standard text is Lee and Vitanye, and please go read it. In particular, if you're engaged in machine learning and you haven't read Lee and Vitanye, you're professionally irresponsible. Sorry, there it is. Okay. So, one of the things that we learned from algorithmic information theory is that compression is learning

Okay. And if you can learn an axiom, that's what you, that's really what you're trying to do. You're, you're trying to take a bunch of data, do inductive reasoning on it, and compress that down to some law. Right. And that lay, that law may not be absolute. In fact, in inductive reasoning, it never is. Right. You never have absolute confidence in something you've learned by induction

But once you get an axiom, then that axiom is going to be of a certain size. And, and of course, by the Curry-Howard isomorphism, that axiom is a type. So you're right. Your types do get more complex. They do get longer. I mean, I mean, when I say, whenever I say the word complex, just insert the word Kalmagorov in front of it. And just, just as a kind of a shorthand. Um, so yeah, your types do get more complex in the Kalmagorov sense

They do get longer. They do get bigger. You can't compress them anymore. And there get to be fewer and fewer and fewer of those. And arguably fewer and fewer in people who can understand them. Um, so one of the reasons that I'm so interested in multi-stage programming languages and doing machine learning through multiple stages and learning these axioms and actually having them become types at different stages is so we can say, okay, freeze. We can't just take a snapshot of that. Package that up somehow and make it available for other people to use without having to understand it

Otherwise, there's really no point. Um, I think it was Alfred North Whitehead who said that civilization advances according to the number of things that we can do without thinking about them. Um, and I think that's, if we don't apply that to the notion of machine learning and types and all this kind of stuff, then there's really no point. I think there's one point, um, that's, that's really interesting and that is the way compilers work. The information flow is, is sort of wonder one dimensional. It goes in one direction. And almost what Paul is hinting at is what if you could write something in the compiler could tell you very deep properties about that by compressing down the information in the implementation, for example. And I do think that we have a tremendous ways to go on tooling to make some of these things, especially more complex types or accessible

I also think that if you've been using monads for any period of time or, you know, even functors, you, you've run into cases where you're like, this is not powerful enough. It is not abstract enough. And in fact, like the functor that you know from libraries like Katz and Scala Zed isn't even a real mathematical functor. It's a very special kind of functor that's, that's far from general purpose. And the interesting thing is you actually run into programming problems that require you to generalize these things to their category theory equivalents. And that's, that's, you know, like, like Rob said, it's, it's a trade off because it requires perhaps more investment to learn about something that's even more abstract than anything that you've seen before. But at the same time, it opens up the door to tremendous new possibilities in, in programming. As some of which we, we haven't even discovered yet because people just don't have libraries that allow them to easily express these kinds of constructs

And I think the programming languages that we have today are not particularly good at expressing them. One can imagine a different kind of programming language that makes it much easier to express these ideas without all the sort of mechanical noise that comes from what these programming languages want you to do with your code, which is in many cases not category theory. Is, is there room for more syntax to represent common, common types and type patterns? Um, I think one example that, uh, I know I don't actually don't know the full opinion of, but async await as, uh, syntax for this common pattern, um, has seen a lot of success in a lot of places. Are there more types that should really be represented as syntax because it's like a little bit easier to rep, if not wrap your head around and not fully understand, but, uh, learn. I'll, I'll, I'll just say syntax matters. I, so I, so I, the syntax semantics boundary is unbelievably hard to get right. You know, it just is. If you're a language designer or library designer, whatever the case may be, that's really, I think, um, where you wind up spending most of your energy if you're trying to help your user, uh, who is another developer

Um, you, you want to make the right thing the easy thing. And, and that's a hell of a lot easier said than done for, for sure. Um, a point I want to make about functional programming and types in general, uh, sort of following up on, on what, uh, on what John said, is that, uh, we're asking people to make big investments of time. And, and we have to be honest about the fact that we're making, asking people to make big investments of time. The justification for asking people to make those big investments of time is that once you learn these things, they stay learned and they stay applicable. Um, we're asking you to learn universal principles. We're asking you to learn more mathematics. We're asking you to learn more logic

Um, and remember, mathematics is the language that humanity is evolving to describe how things relate to other things as unambiguously as possible. And it's been tremendously successful for several millennia now. So, you know, I, I just think it's a good idea to, to, to, to, you know, to, to try to master as, as best you can. Yeah. I mean, just to reiterate that, like, you know, there's an economy of improvement that we all have in our jobs, right? Um, we, we, we, we haven't, um, adopted functional programming and, you know, miraculously, uh, get to spend three times as much of our, you know, uh, work time improving our skill set. Um, what's energizing for me is that we are now improving our skill set and things that are transferable, that are these universal properties and not like vagaries of, um, session loading and hibernate, right? Um, which does not scale it to any other technology. Um, so I think it's really, really neat that, uh, we're in a time where we're starting to see these, um, you know, sort of fundamental properties, um, reach the enterprise. But I mean, I, I think you, you talked about, like, the various concepts that got introduced at various stages and then, like, you know, you can imagine, like, there's just this, like, you know, like, river delta fanning out of all these things you're gonna need to know

But I think that, like, you could also imagine, I think there's a lot of pruning that goes on. Like, a lot of folks work at Twitter and don't really, like, stay up late at night talking about moon ads, but they're very comfortable working with future and option and for, and they just kind of get it. And, like, that pays its way, you know what I mean? Like, that, that abstraction really, really worked. And I think free is really having an opportunity to do that in a way that you don't really need to describe, like, what does free mean in that context? You can even just call it, like, a pattern. Um, but, like, you, you write down your things, you do it, and you get, like, an abstract interpreter and, like, rather than doing, like, you know, mocking, you get to do this thing. And then people are like, yeah, that sounds pretty cool. I don't need to, like, worry about it. So I think that not everything kind of graduates to that level, and maybe free does, and, you know, I don't know what the, maybe the next two or three things are, but I'm not so sure that, like, I think there's, like, a middle path, I guess is what I'm saying, where things kind of get adopted, where, without you, like, really understanding them that much, and you get a lot of mileage out of it

I think we have a couple more questions. Hi, my name is Tiwakar. So, I've not completely jumped onto the FB bandwagon for multiple reasons, because, like, there's Swift, which is only getting functional programming paradigms now. Android doesn't just have Java 8 now, so you can't really do it. So, my question is, one question is, like, if you could state an example where someone forced functional programming on the pattern, and it was, like, bad. So, like, maybe an example would be something that, like, I studied iterative programming, whatever, and begin with. So, every problem I try to solve, I think that way. So, maybe if people are learning functional programming first, what problems would they solve, try to solve, like, incorrectly? Like, like, it's, like, recursion, and you can do recursion iteratively, but what is better to look at recursion, right? Like, so, maybe in that sense

And my other follow-up question is, the adoption of functional programming, like, which parts do you think will play the biggest part? Like, you have the IDE, maybe trying to figure out things, there's a compiler, and then it's, like, whoever's developing APIs. Like, they're developing APIs in a functional-friendly way. So, which of these three do you think are going to play the biggest role, or? Yeah, so, that's my two questions, really. I think a lot of people still screw up loops when they immediately get into functional programming. Like, the idea of applying the right filter and terminating early, as opposed to just continue and break. Like, sometimes that's syntax that makes sense to people, and it's not that they were taught that, and therefore they just need to unlearn it. It's actually sometimes much clearer. So, that's definitely a case, I think

I don't know if people would agree. And finagle, finagle kind of sits on this, like, boundary between this nice functional interface that we present, and this, like, iterative, very, like, low-level kind of imperative style. So, often, we sort of have to strike a balance between doing something very clean and functional. For example, like a foreach. Or do we do the while loop, this is on the hot path, like, we need the performance, we can't afford that extra object allocation. So, sort of, it's like that dichotomy between the two. My opinion on that would be there is no problem which is not better solved by functional programming. is a strict superset of imperative programming

So, you can do everything that you can do in the imperative setting, only you can do it better, because your functions are referentially transparent. And they always return the same value for the same input. And they're just values. They're just, you're always passing around values. So, you can do the while loops if you really want in a functional programming language. They look different, but they have a lot more benefits than doing it in an imperative programming language. So, I roughly agree with that, but I think there's something else that gets bound up together with functional programming. There's like functional programming, and then there's like types and logic

And then there's like abstraction. And these are like, they're very related, but they're not like all the same thing. So, in an industrial setting especially, when you like this stuff, there's that tension of like, you know, back to machine learning, like, call it regularization there, of like, how much, how tight you want to make that code so that like, you've minimized the code. Like, you've minimized repetition. You've used the best abstractions. And especially in terms of like cost and bringing people into the team, it might very well be better to like, I don't know, like, like, example for us, at Stripe, there's a breaking point where we weren't really using cats internally, even though we have a lot of people like, you know, Eric's there, you know, who like, who would like to use it. And then we're like, you know, I don't know if we can really educate everybody to do it. Like, it's not a big deal

And it was like, in one week, like, Traverse came up like four times, like someone like, how would I do this? And it was like, this is the same question as that. But it's like, it's Traverse, right? And then we were just like, okay, screw it. We're putting it in. Here's Traverse, right? But when it comes up four times, maybe it's worth it. When it comes up like one time, but you have this super abstract expression, and then you call it what would like concretely once. Like, that's where I feel like it's like, it's not really clear it's paying its way in that case. Yeah, just, so I want to go back to something that essentially, John already said, but just to spell it out even more explicitly. You know, let me pound you over the head with a microphone

A lot of people do. I've literally had people say to me, you know, functional programming is not practical because you can't deal with state. And I'm like, if we couldn't deal with state, we wouldn't be able to write software. The truth of the matter is that we have tools for dealing with state, even mutating state in place. Yes, we actually go into that memory cell right there at that address and go bang, bang, bang, bang, bang. But we can still do it referentially transparently. Same with IO. So I'm referring specifically just to, again, to be super explicit

Yes, there is the ST monad, the state monad. There's also the ST ref monad. That's the one that actually bangs the memory cell in place. Bang, bang, bang, bang, bang. Wait, wait, wait, no, no, that's not exactly true. Yes, it is. So, let me see. So what, I would argue this

What is happening is you are referentially transparently and like immutably with all the beautiful abstractions, building another program that you kick down the line that eventually goes and like that program itself, like its interpretation. If that program is functional, then like, we were talking about this earlier, like, you know, you get all into functional programming, you do everything beautifully, like, we'd like to have the applicative do, like, for, everything's great. And we work really hard, so we'll be back to an imperative program at the end. And so, I feel like, like, I don't know, it's like, I, I, I, convince me that I'm being wrong about the fact that, like, in Haskell, when you, when you return an IOMonad, you still haven't done anything, you've built a program. That's true. And the program will be interpreted in what, like, a human would have written in an imperative way. And when that, when that program is interpreted, you're gonna twiddle some bits. That's true

But that bit twiddling wasn't exactly what we call the, that's not immutable functional programming. That's false. Explain that to me. Say it more than just, it's false. Explain how. Sure. So, if you, if you were to look, let's stick with ST ref. If you look at the implementation of ST ref, and actually, I like to refer to Runar Bjarnason's old blog post in which he shows the implementation of ST and ST ref, there's a var

And that var gets mutated by the update function. It happens to do it in a, a computational context, in a monadic context. And that makes it referentially transparent. But, yeah, the bits get banged on the same way they would if you were to just bang on the bits directly. All we're asking for, slash insisting on, I'm a fascist, I admit it. Okay. All we're insisting on, is that that modification of those bits be done according to the Monad laws. That's all we're saying

is that you have to be able to do this in such a way that it composes with the other things that are pure. Referentially transparent. I don't know if we're all saying that. Because I think he was bringing up this point. Like, he wants to write a for loop. And like, so most of us would kind of maybe feel like it looks a little ugly. I work with Eric Osheim who writes like beautiful abstract. Every time he's got a beautiful abstraction up here and right as soon as he gets inside

And so here comes a while loop. Okay. Here comes a bunch of bars. Right. I don't really, I don't love that style. But, you know, I love Eric. Yeah. But, I'm not sure if most of us would educate our coworkers that way

It's kind of like he knows the rules so he can kind of break them. Right. It's still referentially transparent. I agree with you about this fact. Right, right, right. But, I still would say it's not exactly what we would call like right in the church of functional programming. You know, there's something unorthodox about that behavior. You know, it's a little bit of the..

But, I think one of the mistakes we make, if we're going to talk about how those of us who do functional programming have not represented it as constructively as we might be able to. One thing we should acknowledge is that making those granularity choices about where to be referentially transparent and where not to be in the interest of performance, in the interest of understandability when somebody has to read that source code. I think that's a real issue. I'm not saying you should write everything, you know, it's like the only way you should mutate is with ST ref. No. Sometimes you do, in fact, need some little local mutable collection and you go ahead and bang on that and your referentially transparent interface is exactly what Eric is doing. That's perfectly legitimate. All we're saying..

Just give a piece of chance. All we're saying is pick the point at which referential transparency is going to help you build correct software. And if you know how to write 5,000 lines of concurrent Java code and can build a referentially transparent interface on top of that, first of all, you're a better programmer than I am. Secondly, God bless you. know, I don't have a problem with mutation. I don't have a problem with IO. I want to know at what level of granularity that becomes referentially transparent. And the answer never is not acceptable

So I have one comment for John's comment. So you said functional programming is a superset of everything and it's a superset of imperative programming. So then is imperative programming also functional programming? Then there's no versus. No. So all imperative code is also functional programming, then there's no pi. Yeah, it's absolutely not. Imperative programming is not functional because, for example, things aren't values. But you said functional programming is a superset for everything

Yes, which means that every single thing you can do in imperative programming, you can do it in functional programming in almost exactly the same way, but everything is a pure function, everything is a value, and you have an easier ability to reason about that. I don't know. I agree with you, actually. You're both wrong. You can. No, but my other question is, does functional programming use imperative programming? So, but like you just said, like you could take and literally translate using, you know, ST ref or whatever you like, any Java code to literally, like, line for line. It would be, but it would still be, like, that wouldn't make it, like, great code. It might still be, strictly speaking, referentially transparent, but it might be very difficult for a human to follow those references because there are all these, like, monadic mutations

Yes, and I totally agree with that. Functional programming is, in a sense, too big because it does contain all of imperative programming inside it, and there's no guarantee that the functional code you write will be any easier to understand. However, because functional programming is bigger than imperative programming, it has lots of other wonderful things in there that we can use and that we do use for maybe 70, 80, even 90% of our program. So we still have that 10% that's imperative, that's somewhat nasty, that's a little hard to reason about, but expressed in a purely functional way, and then we're able to benefit from all this other good stuff that comes in with functional programming. Thanks, guys. I think we have one last question, and we are coming to the end of the hour, so make it a really good question. No question. I'm Sergei Winitsky, working for Workday, and I have a really good question

So... Each of you work in teams where functional programming is being used, but obviously not all your colleagues came to it easily, and maybe yourself also had some reservations. What were the incentives that made your colleagues switch to functional programming? So maybe you can share. I mean, I think in particular with Future, by far the most common monad that we have in use of Twitter, just the native parallelism that suddenly, by declaring a thing a particular way, inherent parallelism is kind of amazing. So that's one of the things that I think draws people and makes it worthwhile for us to teach everyone at Twitter how to do this. I think there's no question that we gain benefit from that. Also with error handling, like large distributed systems, maybe you're making a series of concurrent calls, like gathering back that data, what if one of them fails, et cetera. We don't want a series of infinite callbacks and whatnot

It's very ugly. And then functional programming really makes that pipeline incredibly clean. I would say that I remember maybe, you know, maybe 20 years ago, people were saying it was like really impossible to program like at scale, many, many computers, highly, highly concurrent operations. And then like MapReduce and other paradigms have come along to show that like that's actually not really the case. You can program tens of thousands of cores rather effectively. And I think that that paradigm, even when not expressed with a functional language, I mean, it's a deeply functional paradigm. And a functional language is just an amazing match for that. Almost no one looks at that and feels like that was a bad thing

So, I mean, like Spark, even if people don't like do Scala elsewhere, they love Spark. So like Twitter, they're scalding, but it's the same thing. I think for us at Verizon Labs, I think a big thing has been the ability to test individual functions in isolation and, you know, just REPL driven development. And that's especially important, I think, when you're working with colleagues who come from primarily a dynamically typed interactive language background. You know, they're used to firing up a REPL and just typing stuff in and going. And I think, you know, next time I see Howie, I need to slip him a 50 or something. Because, you know, to be able to pull up the ammonite REPL and not even have to use SBT and just say, load this dependency, load this library, load Scala Z. I have literally sat down with colleagues, fired up a terminal, fired up the ammonite REPL, fired up Scala Z, and shown them some code that they could then just use interactively and sort of wrap their head around

And that makes it just a huge, huge, huge difference. Well, I'm a CTO, so I cheat. I only hire functional programmers. But that said, I do think that when I'm talking to other developers who work at other organizations, I do emphasize the benefits. Not the fact that it's some shiny new thing that's becoming hot and, you know, a buzzword and all that's true. But the actual benefits. And the benefits are composability and reasonability. And really, really strong guarantees on the correctness of your software

FS2, for example, allows you to build really complex streaming applications with guarantees on resource leaks. And if you've been a Java programmer for 10 years, you know how hard it is to make and live up to those kinds of guarantees. And we have, like, libraries, parser combinators, which can allow you to describe a grammar exactly as you would describe it if you were writing it down on paper, but which actually parse something and do it maybe incrementally with good error reporting and all this other wonderful stuff. And then we have command line parser options libraries, which are capable of you declaratively specify the syntax of the command line arguments you want to support. And you get free documentation out of that. It auto-generates documentation. And it will not only parse it for you, but give you back those well-typed structures. So these are really practical benefits

And, of course, the testing benefits and just all of this stuff combines together. It's a really solid package. You don't have to oversell functional programming because it does a good enough job on its own. Yeah, since everybody took a turn in order, I'll say something, too. You know, I've trained a lot of people in my workplace, both in functional programming and Scala. You know, I think people, you know, it's unique about your question specifically that, you know, what incentivizes people or really what's, you know, people's motivation for really trying to understand functional programming. And I think a key point is that everybody has different backgrounds and different motivations for why they're interested. Some people really are challenged by, like, the puzzle aspect

Right? Like, you know, put a monad in front of me and I'm going to play with it and figure out, like, how I can plug it into different things. Other people really are more motivated by, like, this really makes my concurrency life easier because I've, you know, debugged deadlocks for the past three years. And so I think, like, really finding and understanding everybody's backgrounds and seeing what's interested, you know, what they're interested in and then tailoring the message to them rather than sort of going with, like, a, you know, one-size-fits-all approach to training and learning is really important to adoption. Thank you, guys. I think that probably is a good note on which we can conclude this panel and this conference. Before we go, I want to mention a couple things. So, basically, everybody comes up to me and asks when the videos will be published and where are they. I think I mentioned the keynote, not everybody was there

So, basically, follow, we are at Twitter. So, we will do everything through Twitter. We'll tweet about it. So, go to at ScalaByTheBay handle. In the profile, first of all, there is a link to the feedback. Please leave your feedback about this conference. And then we'll tweet the, essentially, when we publish everything. will go to Functional TV channel on YouTube

If you subscribe to that channel, you'll just receive all the videos as they're uploaded. And we're pretty good at kind of quick turnaround. And we'll additionally tweet about it. And we'll also tweet the location of the shared folder where all the slides will be published for those speakers who are willing to share the slides. Some companies prohibit slides, surprisingly. But we'll do a pretty good job in kind of splashing them into the videos. Anyways. So, please follow the Twitter for all the announcements

Also, please submit your talks for the next conference, which will happen about the same time. Hopefully, Twitter will have us back. In that case, you know, it will be in November. So, the CFP will open early. We'll open it in May. We like to have everything in by July so we can spread the word. So, submit a talk and hopefully we'll see you back. And if you're local, at SF Scala

Thanks, guys. Can we get a round of applause for Alexei? Yeah. Thank you.