Scale By The Bay 2021 : Austin Huang, Introducing Hasktorch

Scale By The Bay 2021 : Austin Huang, Introducing Hasktorch

Recording: Scale By The Bay 2021 : Austin Huang, Introducing Hasktorch

all right so um thanks very much for having me my name is austin wang and um in this talk i'm going to be introducing hass torch uh which is a haskell library for tensor math and neural networks and now before i dive in though it's worth asking the question you know uh where does a language like haskell play uh fit into um the world of machine learning because if you look at it to date um there hasn't been that much of a presence of functional programming for machine learning applications so um why is this something that's potentially worth looking to in like a design space that's worth exploring and to answer that i think um part of that comes from a framing of um you know what are we doing when we work on machine learning projects machine learning applications and we need to kind of unroll some of that our assumptions underlying that and um so as one starting point to think about is there are these stats that get thrown around um you know so this one's from venturebeat that you know 87 supposedly 87 of machine learning models never make it into production right and it's worth asking you know um why is it that that failure rate is so high in um in doing in sort of making machine learning model uh making machine learning add value um to businesses and to be really kind of successful in transitioning from research to production i think part of that stems from the way we frame what we're doing when we work with neural networks when we work on machine learning projects and kind of the typical conceptual view goes something like this right so neural networks are a machine learning algorithm and they're one among sort of multiple machine learning algorithms like svms k-means logistic regression and um and so neural networks kind of fit into uh the space of machine learning algorithms which we study in this field of machine learning which is in turn a a kind of subfield within artificial intelligence right and so we think of um uh neural networks as being sort of interchangeable with kind of other algorithms this kind of are our kind of conceptual framing of what we're doing and i think this this view it's not wrong but it's sort of it misses a big uh aspect of uh what we're doing in uh in machine learning and um to get some sense as to kind of what what's missing there let's look at something where that's actually had a major impact on machine learning um this idea of these transformer architectures uh neural network architectures that have really kind of in the past few years transformed the way uh no pun intended uh we we do natural language processing and right now are in the process of also um changing uh uh having a large impact on computer vision as well now to understand this um if you if you look at something like a transformer architecture um it's not really so much a novelty in an algorithm so you could write you know an equation that sort of reflects the computation of the transformer architecture end to end um but that's not all that enlightening right and that's not how you arrive at um at ideas like this right um if you kind of look at it it really looks more like a program with sort of different uh things that are intended to um be composed together to do certain functions so for example positional encoding you could describe it as okay you're um doing this sine cosine computation but um really kind of the intent of positional encoding is to kind of push the positional information of a token or some piece of data forward into your vector representation that then gets consumed by um subsequent uh um uh pieces in your architecture and so it's much more useful to think about of of in architecture as something that is a system that that sort of corresponds to a program and so you're kind of uh putting these programs together one thing that's a little bit different between neural networks and sort of traditional uh programs is that um the final computation that you do is learned so um you can think of a machine learning model as something akin to a curried function where um it is a function that sort of takes parameters as input and returns a function that performs your model inference this model inference maybe the prediction or the classification that you want to do and you arrive at those parameters that you kind of plug in to your model through a training process that training process um is optimizing some objective function known as loss and takes advantage of the fact that um these uh your uh your inference function is end to end differentiable using um using this uh uh automatic differentiation procedure so um you use the automatic differentiation on your model you can uh get gradients and you use that in your in your training procedure so you can look at this kind of whole endeavor of machine learning as sort of writing programs uh that are optimized on data in this way so we see that kind of have this shift in view that when we're defining a model we're kind of thinking about it um not so much of each architecture being say a different algorithm but um each out each architecture being sort of different ways of composing sort of functional blocks to arrive at sort of new ideas for architectures if we go sort of one level up in abstraction these uh models get trained through some sort of training procedure and there again when we talk about sort of different training procedure and sort of innovation in model training it also uh corresponds to innovation in building systems so uh so for example our as we get to kind of these larger and larger uh language models that um we use in an lmp we need to have sort of systems that uh make the sort of paralyze sort of the model across uh multiple devices or paralyze the data across multiple devices we have sort of new innovations in sort of uh federating the training to uh preserve privacy um in which case the sort of the training becomes a distributed process among multiple workers we can also have kind of different ways of combining models with each other through things like distillation transfer learning and all of these are sort of different ways of sort of composing uh new sorts of training processes we can also insert things like simulators into the loop of the data uh into training processes as um as sort of data generators uh for the training process and um so you can have entire physics engines game engines sort of rendering engines be part of that training loop and so all this to say sort of the training procedure is also all about sort of composing kind of systems of uh of of combining data and um and different and how you use that data into uh that optimization that we talked about when you go to sort of machine learning applications machine learning applications themselves are distributed systems so we start from kind of our model development process to train and uh we then embed our trained model into some sort of uh runtime usually there's some api that wraps the model and the way you pass kind of a model from uh training to production is to say take a serialized representation of the model computation and then uh put that into an api endpoint right and often you may have some edge consumers of that so maybe your api service is uh serving a mobile app or a web app that does something with computer vision or you have some embedded system the model can also go uh siri uh can also be deployed all the way to edge sometimes um to do the model computation say in the browser or or on some embedded device and finally you have some feedback of data coming back and uh from these services and and coming back to update your model so um you can think about sort of different uh machine learning application those sort of special cases of this so for example uh something like a large language model api like um gpd3 or you have some service for codex um you can talk think about okay i have this large model and i i operationalize it as an api you can think about sort of um autonomous driving systems where your model gets sort of pushed to this embedded uh edge runtime and it's supported has some supporting apis to um to capture feedback data or you can think about sort of privacy preserving api where you might actually do sort of training on your client and so um all this is to say that um when we think about uh progress in neural networks in research and application they all sort of correspond to innovations in system composition right so we talked about the neural networks themselves are systems that are described by differentiable programs right and new ways of composing uh such as transformers are are where a lot of the innovation comes from um new ways of thinking about teaching machine learning models are are all about sort of building systems which compose data producers and different sort of learning procedures and finally to pull this together into an application is all about sort of integrating models into distributed systems spanning multiple runtimes so zooming back out from that original view of kind of neural networks is sort of like one algorithm among sort of multiple machine learning algorithms we can kind of shift our view to kind of think about neural networks through the lens of an emerging programming paradigm right and uh it we're what we're doing on a day-to-day basis was we're building systems around this emerging uh programming paradigm of um sort of differentiable uh functional programming right and uh a lot of where the innovation is coming from is the through the ability to create systems and uh by composing systems in different ways uh we arrive at new results uh new kinds of models we've arrived at sort of uh new kinds of systems new kinds of application capabilities and um so here's where we come back to this idea of typed functional programming because type functional programming is all about giving tools as tools to make systems that are composable uh be able to encode invariants in the type system and um and it gives us a lot of sort of expressivity to create systems and a lot of times when we talk about machine learning we don't acknowledge this fact that um at the bottom of everything we're doing is we're building systems uh out of these uh uh using this uh differentiable programming approach and has torch's goal is really to kind of enable type uh functional programming in uh this system design space so this is all all very well and good so this kind of high level sort of philosophical discussion of what we're doing when we do where we're doing machine learning now let's get into the nuts and bolts of how do you make that practical because there are definitely a lot of practical issues to solve um at a first cut um there are all sorts of you know hardware backends right so to really train these neural networks you need gpu support and and then on top of the gpu support you need a huge swath of different mathematical functions and um that all takes a huge amount of resource to build up that stack from the bottom up right and um so the approach we take with hass torch is to pay feedback on uh pi torch and for those that are not familiar pi torch is really the probably the most popular uh python library for uh neural network researchers and it's built out a lot of these computational backends and uh library uh functions and automatic differentiation capabilities are already built into pi torch not only that pi torch is separated you can kind of think of it as separated into sort of this python front end and the c plus plus back end and the availability of that c plus back end um gives us a starting point uh from which we can bind to the c plus backend and we can build uh higher level sort of native uh type functional programming abstractions and explore where that takes us um in in in implementing these machine learning systems so uh there's a deeper dive on this by uh our collaborator uh junji hashimoto from icf a couple years back on sort of uh this idea but um at a high level we start from uh what's called lib torch which is the pi tour uh c plus uh library and for the low level part of the library we have a code generated haskell to uh do the raw bindings and also take care of some memory management and then on top of that we can build out sort of the higher level api that end users would use to build models um this api is kind of split into two parts with kind of two different purposes the untyped tensors the idea there is really to just have parity with sort of pi torch functionality so to be easily be able to import sort of new functions and and um and operators that are available in pi torch um but leaving tensor sort of as an opaque object then there's um type tensors or now torsten's been working on gradually type tensors where there's a an explicit type level representation of the tensor shape and there's a lot more sort of type level exploration of type level computation of what can you do when you are sort of explicitly representing tensor shapes in in the type system and so on top of these apis then you can build sort of training procedures models and then embed uh those models in applications so what does it look like to work with uh hash storage so coming back to this view of sort of machine learning as functional programs um it it we're able to kind of have uh model implementations that look a lot like this sort of conceptual model of machine learning so for example here is a simple um mlp so these are just basically stacks of uh linear layers um and uh really what you're doing is you just uh you implement that exactly and sort of that first parameter there is really the the parameter state that gets passed in and it returns a um it returns a function corresponding to a multi-layer perceptron computation here's kind of a simple description of a variational autoencoder and again sort of the first parameter that sort of curry is the is the parameter values for the uh for the neural network and it returns a function that performs the variational autoencoder function here's another another example is where um you use the native haskell sort of composition operators to um to build up uh neural networks as compositions of function in haskell right and here's a more uh complex example this goes back to that transformer architecture i showed in the early slide and one thing that's different about this is this is using uh torsten's work on the gradually typed api and um you see there's a little bit of uh type level trickery going on where it's marshaling sort of type level values into value level values around shapes and things but overall the computation is still kind of reflecting the the composition of the model architecture here right so we talked about kind of these the the inference computations the parameters themselves can be represented as algebraic data types uh fairly naturally in has torch and these can be nested in one another so that you can compose neural networks of out of modules of of of kind of uh sub components that are also themselves algebraic data types we can also take advantage of generics to do a lot of automation of boilerplate for example for parameter updates and things like that so that you don't have to kind of rederive set this boilerplate for every new model that you implement there's a lot that you can get for free with uh using um haskell's generics and finally you can have um common machine learning patterns models as type classes so for example this forward function defines the ultimate kind of inference computation for a model and you can abstract that out into a type class so you can might write code that's sort of generic across um across different models that are kind of uh writing against these sort of type type class uh interfaces uh and uh i mentioned uh the um one of the things that neural networks depend on is this uh capability around automatic differentiation and automatic differentiation basically takes any sort of function that's within the library or composition of functions and for a given data point gives you the gradient at that data point and then you can use that in your optimization procedure so that we get for free uh by binding to libtorch i know there's a lot of work in sort of functional programming world to make sort of the representation of automatic differentiation more explicit and that's something we can uh explore in the future of uh having that kind of automatic differentiation procedure explicitly represented but for now we can sort of piggyback on the uh capability that uh lib torch provides us to get these uh gradients for any um any sort of lip torch function or any sort of composition of lip torch functions and then sort of putting data together to train a model so training model is basically just becomes uh folding over data there's a little bit under the hood here where this run step calls the automatic differentiation and updates parameters and gets a new set of parameters you update the parameters and that's really kind of your your basic trading procedure of course you can start from this and get a lot more complex in terms of going back to kind of these examples of federated learning of connecting training procedures to uh generative processes or things like around synthetic data or simulators but this is kind of the basic structure of a training model and these examples so far are primarily for the untyped api um for a deeper dive on a kind of a more in-depth tutorial i encourage you to check out this literate haskell tutorial by matt whitman for now most of the tutorial is around the untyped api but as we do more work on that gradually typed api that torson is working on there'll be more material there too right so i've talked a lot about sort of the kind of basics of what it's like to work with uh has torch um i want to kind of go up one level we talked about training a little bit um to what it might be like to to start building applications uh uh machine learning applications right and so we talked about some components of the of this uh in the earlier slide where we these applications are basically you kind of push and pull these um these uh serialized models into a service runtime into edge so what what's that like uh in in has torch uh one thing to be aware of here is that there's uh a lot of tooling around sort of representing models as as serialized artifacts and uh we can piggyback on that in order to be able to interoperate between languages so we can take a model that's uh trained in pi torch and we use it in in haskell and so that gets us around kind of this issue that you know even if we have has torch maybe it seems like it's prohibitively expensive to train this computationally intensive large language model um but in actuality we can sort of take the weights uh load them up and uh and and integrate these capabilities into um into our haskell programs um so to get a sense of how it's doing that sort of going back to kind of our um representation of models you can kind of think about um these uh separating model into sort of parameters and computation and the parameters are represented in the pi torch world by the state dict that state dictionary that has sort of all the model weight values and the computation is uh you have this torscript intermediate representation that you can use uh to represent kind of a model computation so there's kind of different ways in which you can mix and match so if you're working entirely within haskell in your training models with hash torch you can always use basically saving and loading of parameters and that can let you kind of go from being in sort of your training mode to then loading the parameters at uh within your say servant api or other web api to uh then um load the model and run the model computation once once it's been trained if i'm going back and forth between python and haskell i can take advantage of this tort script function ability to to extract out the state dick sort of parameter values as well as a computation graph and if all i want to do is say use haskell for deployment it's actually quite easy to just um with this kind of one liner load a load the torch script intermediate representation of model and basically call the forward function and then i can immediately have that sort of uh pi python pi torch model uh running in a haskell application if i want to do a little bit more with the model like let's say i want to uh fine-tune the model a little bit more i want to train it a little bit more i want to introspect the insides rather than loading the computation as sort of this opaque uh intermediate representation object i can also parse the parameters uh into a kind of haskell uh has torch model and this is what it should do if you want to kind of pull out sort of different uh values or if you want to kind of update the model some more uh you can you can take this approach um so kind of uh pulling that all together you can uh this once you load a model then you can use it uh just like in any other any kind of standard um api so here's an example using servant and then calling out to uh servants uh to stand up a web api and then calling out to hashtorch to run sort of the model inference on sums uh some parameters that are specified through the api interface right um if you're interested in kind of this aspect of sort of integrating um models into apis and building applications around them uh you can take a look at this small sort of mini tutorial series under uh examples model serving there's a sequence that goes through some of this uh in more detail so i want to kind of before i wrap up talk about sort of some of the ways that has storage gets used and primarily it's it's by a a great community that's sort of exploring these two different directions right these two directions of one is what are some of the things we can explore that's at the intersection of machine learning and programming languages how do we kind of uh put type systems around uh neural networks and how can we kind of express neural networks in new ways and second is uh developers that are interested in sort of building uh machine learning systems using uh the tools of uh functional programming and so we've had uh lots of uh community members and collaborators that are exploring pretty interesting things so kiara had an europe's paper around program synthesis um jesse did a um uh google summer of code with us around uh neperian functors um torsten's been exploring the space of gradually typing your networks and really kind of looking at sort of pushing the boundaries of haskell's type system by uh expressing these uh really large uh language models like bart and bert and representing all these shape relationships um using the type system of haskell um i've been personally i've been working on uh using has embedding has torch in uh applications and one that i recently open source and it's sort of a pre-alpha stage is um a a sort of a personal knowledge integration platform uh that's one of these kind of second brain knowledge gardens and one thing that's different about this is uh this application is meant to be self-hosted and the idea is to have the the api is written in haskell and the idea is to have something that's um self-hosted and enables these kind of machine learning integrations around nlp uh in a way that um can be sort of personally owned as a as an application uh and part of that is using sql lite as uh storage media so if you're interested take a look at the github repo for that so with that i just want to you know call out if if people find this interesting want to contribute definitely join our slack you can take a look at our website at hashtorch.org you can look at the main repo and as well as a project template so um yeah so thank you for your time and i'll uh i'll uh definitely be happy to take questions and and uh i'll see you in the um in the meeting room afterwards as well you