Scale By The Bay 2018: Tongfei Chen, Towards Typesafe Deep Learning in Scala
Recording: Scale By The Bay 2018: Tongfei Chen, Towards Typesafe Deep Learning in Scala
you could you hear me or okay today I'm going to talk about towards typesafe deep learning in scala so to war scheming is not that ready yet okay I'm a little bit of self introduction I'm Tom fetch and I'm a PhD student at Johns Hopkins University I work on natural language processing so I use deep learning all the time in Python so I'm not very happy about that so that's what I do so I wonder how many of you have actually done deep learning here okay okay so I'm just in a nutshell what is deep learning it's just a kind of a machine learning technique where we have this core data structure called tensors or it's just a fancy name for multi-dimensional arrays it's not the mathematical tensors so we could have an image representing represented as a three dimensional tensor or three dimensional array where you have three axes their width height and color channel so for example we have an 100 times 100 image so in memory is gonna be a multi-dimensional array of 100 times 100 times 3 what is 3 being red green and blue the three colour channels or in natural language processing we have a sentence here the cat sat on the mat each word it's going to be mapped as into a fixed length vector we caught the word embedding in in natural language processing so this sentence going to be represented as a 2d two-dimensional tensor or matrix where you have two axes one axis is the word and the second axis is the embeddings so deep learning usually is for example in computer vision it comprises of this stacked layers of small functions like convolution rectified linear units pooling layers and fully connected later softmax and whatever so the whole thing is a is a composition of smaller blocks so we and given this whole this huge function we can learn you buy existing data so deep learning in its essence is just function fitting the easiest case in machine learning could be probably be linear regression where we learn a function that map's M dimensional vector to an n-dimensional vector where the function is y hat equals ax plus B where a is a matrix and B is a vector this is probably the easiest function v fading problem and to a more fancy function fitting problem that's the machine translation where we train a function that map's a French sentence to an English sentence that's still a function that we're going to fit let's look at a linear regression thing so this is the function f where this this this thing's got a computation graph where X is the input it's you do a matrix vector multiplication by the parameter a and then you add it with the vector B to create Y hat that's the predicted Y that's the function f that we're trying to learn in this function this block we see that we have two parameters a and B so the smaller to fit is composed from smaller building blocks with parameters in here we it's composed from multiply and plus and add and we can train it by gradient descent with respect to a loss function where here the last function usually in linear regression we use this is a mean squared loss or l2 loss that we minimize the distance between the expected output and they and the predicted output that's the Y hat and Y so this so we connect the Y hat the predicted output to a Y that is the actual output the fold output to produce a loss losses the distance between these so we're gonna minimize L so this is a function so we're gonna we're gonna minimize the values of a and the minute minimize the value of the loss function L and to get our train parameters a and B so this is um what basically deep learning is it's just a function fader with birth parameters so we can take differentiate different differential differentiation against in industry and in academia also we're going to we already have a lot of common deep learning libraries and use people use tensorflow from google pie tours from from facebook MX net originally from CMU now it's acquired by Amazon one thing and this dinette from Carnegie Mellon cafe to China and so people wonders if you already have so much to so why am I doing this the problem with all of these is just this is in Python all of this is just the main languages in Python which I'm just not happy about so let's look at why Python we're in my opinion is not a very good fit for these problems for example just look at example in tensorflow look at the linear regression problem if I write that intensive though so first I start with a thing called placeholder that's the input node in the computation graph so M is the size of the vector and this is the internal type the type is specified it this way as a parameter s just this is just yeah very strange and X is going to be a tensor flow node and this is X and now we start with a Y and this is the old output Y so we start with a parameter it's a tensor flow called variable and we initialize with some some normal distribution B is also a variable and then we start applying our network ax is multiplied them and the predicted output is ax plus B and our loss is the we divide we subtract the predicate the predicted output from the real Y and we do element wise square and then reduce some that's the l2 loss so this program see there is just no types whether it is a input or it's an intermediate result or as a matrix or as a vector or this is a scalar variable this is not a vector this is not a matrix so everything intensive wrote is just they belong to the same type called tensor so we don't know the actual shape or size or dimensionality of that at compile time or or Python there's no compile time so at runtime you will just fall into various type errors from my own experiences when I do PI program Python everyday I run into like type errors every day spend hours debugging these this is a more complex example this is in PI torch this is actually a copy from my own work in school so so variable it's just a something similar as tensors an intermediate result so are we going to specify that the types of this tensors in this way so this is a natural language processing thing we'll have a batch of sentences each sentence the words and each word has an embedding so you can see here we apply this transpose thing after this what is transpose is just swapping to access 0 and then access 1 this effectively changes the semantics of the of the axis of the tensor axis but this is not reflected in the type so I have to write something like a comment here to let me to keep me track what is access 0 what is access 1 so and here like unscrews is to insert and new axis at the time at the index 1 so this actually makes the tensor having one additional dimension and this is not shown in a type either so this whole thing you you have to keep track of this the types of tensors yourself and recently there's have been a talk in in a natural language processing conference that people talk about what is the good software engineer engineering habits for producing these and one of their habits they they encourage that everyone do is to add type annotations after each after each of these expressions so this is not ideal we want we want the compiler to track them by itself so we don't need to remember them so so the pythonic approach starting from numpy is everything belongs to one type tensor and dra whether it's a vector or matrices or it's a sequence of vectors sequence of matrices or it's an image and video a word or sentences everything is just a tensor so how many app how many axes are actually in the tensor it's just 0 dimensions of scalar or it's a 1 dimensional vector two dimensional matrix or higher dimensional or what does each axis stand for we just don't know so the programmers have to track the track these by themselves they try to remember them by hand by heart so I'm actually these people who write Python they can actually do this I cannot yeah I kind of remember these I need static types so that's so that's the motivation so I started my own side project that it's called Nexus that's here that we try to implement a typeset version of all of these so our goal of typesafe tensors is we have a tensor of shape of a type like this in scala so axis is called it is an axis descriptor that describes the semantics of each axis so it is a tuple of singleton types so each axis is going to have a singleton type as its label so we know what each axis is and all operations on tensors are statically typed the result type is known at compile time ID can help programmers if you if you are operating in compatible tensor it's just compilation error and if you use IntelliJ or things like that there was going to be a unrest underscore a wavy underscore under underneath your codes to show that you have you have done something wrong so for example let's look at the these examples but first this is image so this image and the types in in this type safe tensor notation it's going to have a type like this it's a flow tensor with three dimensions in a tuple with height channel width hi-chan are three types our three singleton types they're just label phantom types they're not that instantiate or anything or like a sentence here is gonna have a type workflow tensor of word and embedding so we're gonna have type safety guarantees on these only operations that make sense mathematically are allowed by the compiler for example have a tensor of a and tensor of a so if I add them okay we can add them the tensor of a and tensor of a B so this is not a double because their shapes do not match in some language like R this is actually doable they broadcast this to the same shape of this they copy the tensor of a to the B times to let make it the same shape and add it up I think that's just a very implicit thing underneath and snot that's dangerous and actually I'm if I if I am allowed to quote the Zen of Python explicit is better than implicit yeah but all these people they're just doing everything implicitly and for example I have another two vectors a tensor of a and tensor of B they're both one dimensional vectors but they don't add up the compiler doesn't allow them because the semantics of the vectors are different for example this is a tensor of with this is a tensor of height still semantically it doesn't make sense to add them up so still there's a compilation failure or matrix multiplication to apply matrix multiplication on two vectors doesn't make sense failure if we apply it on two tensors with the same shape doesn't work it only works if the second axis of the first sensor is the same as the first sector first axis of the second Matrix only in this way that this works or let's stay look and look up this thing's this is called a sum pooling or reduce some so if I pop the three dimensional tensor here I want to sum over the the first axis this is a zero first and second sum over these two that have reduced it down to a two dimensional tensor how would I do this intensive flow I write TF dot reduce some X dim equals one at this points to dip and in pi torch like it's called tourists are some X Deming was one we don't like these so in for example I have an X that's that's of type this I would write some along be X the sum of some along B is an operator that sums over that axis be it results in tensor AC if I do some along D it doesn't work because D is not in the list of the axes a be seen a tensor X so this doesn't compile so how do we actually implement these type safety guarantees in Scala these are non-trivial type safety so we're going to use at a type level programming come abilities of Scala to do it to do this um first I'm writing everything as tuples here to pose looks better than heterogeneous lists but underneath everything are heterogeneous lists I'm just using the shapeless on generic and to blur to transform them for and back forward and back but um so we're going to deal with with a shape H list underneath so we can do type level computations on top before we get into how we implement these type safety guarantees was first going to look at how how we could produce a type safe computational graph that that uses general generalized ADT's in scala so professor artists agree outer tsuki yesterday just implement introduce the scholar three format of enums so i'm gonna use them here in the code in a my code is not actually in them it's a co trait and then case classes but enums looks better so we have a symbolic of x that means a symbolic variable of X that's in the computation graph that it's its gradient is going to be computed when we do training so we can have several cases there's case input of X that's the place holder intensive row that we feed something in there is a parameter of X a parameter of X has a var here it's because it's it value keeps changing at every iteration it's it's a mutable state and we have a implicit value here so grad means it's differentiable this this tags this is a type pass that says that this is differentiable it has some operations in here like addition and subtraction that or like multiplied by a scalar or things like that very similar to a vector space we have to have that to do parameter update Const is a constant so it's a constant here and we have apply apply nodes are if we apply a symbolic variable of X using an operator for X to 1 the results and symbolic variable of Y that's an app note there's a apply of 1 apply to up to and we support up to up 3 so this is a expression tree or like an ast that's that we are used to formulate the computation graph here so let's look at what is up one up one is a operator or a function from X to Y but that's differentiable how we do how do we encode this so up one is that apply function give it a symbolic symbolic vector of X result in symbolic number Y what does it do it it just wraps it in a apply node it just creates a wrapper around that and had a forward method and a backward method this is the forward method that computes that actually confused to the value and the backward that is this so we get the gradient of X using a gradient of Y and the function itself for example this is up to we have a forward and have a backward one a backward two so we propagate a gradient to the first input and the second input using this mathematical formula formulas you write the implementation implementation of these in these virtual methods so in forward computation what we actually are doing is given a symbolic vector a symbolic variable of a we try to get the value of a we compute the value of a so this is an interpreter in this is a functional interpreter so with cats it's you can write this as this this is a natural transformation so it's just interpreting the computation graph so we start from X we've given a value of x and y so we propagate all the way through through L using an interpreter to interpret this graph to get results or it's a backward computation we would compute a gradient we start from the last note that's the loss because we're taking the gradient of of the loss with respect to a and B so we do it in a backward way so we a backtrack from this graph so let's introduce some little thing about operators and modules so operators are modules are both functions are both symbolic functions that map symbolic effects to symbolic of Y options operators have for in a backwards so the computations execute in here itself but module is so here this thing by this thing in this gray box can be abstract as a module so it has internal structure and up is like atomic note that these are both symbolic functions so now let's introduce the core part of my work is about how can we encode these type safety guarantees in Scala so these are polymorphic symbolic functions so as we have just introduced an op op XY it only applies on one type X but for example addition it applies on images applies on sentences if applied on any matrix with different shapes and whatever so as long as the two operands share the same shape the shape they can be added so how can we abstract over X here so we do something called poly function this is very similar to a shapeless holy one every every use shapeless so this looks like this so a Poli function has a ground function that grounds that given type in X Y produces our actual function that offers an X but Holda function itself does not have any type parameters but how can we ground this to function we can only ground this if an implicit F can be found is is that if the compiler can prove that F X Y is there then we can apply this operator on symbolic X to get a symbolic of Y so this is the core thing so this is very similar to a case start aux-in shapeless if you are familiar with shapeless so this is the signature we can apply this function on symbolic of X as long as we can find an F X Y so F X Y is an arbitrary type level predicate so this is compiling as proving so we are proving that this operator can be applied on a type X if that is proved then we can proceed with compilation so implicit F X Y is found that is preposition F X Y it is proven so we can encode any type constraint we want on type operators into F because the scholar has a very expressive type system so let's look at how these things are actually done this is a polymer for operators just just a something that we can add an implicit not found here on F so it produces really good completion error error messages so if you add something that's not the same type the compiler would report this operator cannot be applied to an argument of type in X so let's look at how it's actually I'm done talking some bad so ad we had a type level construe a logical constraint and type is as long as this X is differentiable we can take the gradient of X then we know we can add X and X into X that's the that's the logical constraint we want to impose and type system how do we do this write an ad with an ad F X is grad we require an implicit that is differentiable so we returns that f of X xx that is a that is a the proof that we can add X and X to result in X so that's forward as X 1 plus X 2 this add sign here is there because we have grad and backward so addition the backward the the gradient is just 1 so it's just the gradient derived from from upstream so that's backward - this is add let's look at something more interested as matrix multiplication so we have two matrixes matrices that can be multiplied whenever the second axis of the first matrix coincide with the first axis of the second Matrix so what we want to do is we have a tensor type TR so is real type tensor K is a type class that says that T is a tensor type over r RS float or double or half now if we went to on a GPU so for any of these types we know that we can we can do a multi matrix multiplication over ta b and TBC to get a TAC that's the logical string we want to impose on a type system so we'll write this as long as we have T of is 10 Ryo tensor and we result in that type of ta be to TBC to tac and that's forward make sure to multiplication and this backward computation that's the loss another example would be access renaming sometimes we want to rename an axis for example we wanted we reuse axis of a tensor Force different use so in X if there's a dimension called a we want to change you to be how do we do this so we want to we for this operator we won't have a type constraint that is given t as a tensor doesn't require to be a real tensor it could be in tensor or boolean tens or whatever so if we have a axis of a if we removed you and add V or results in a B then we could say that we can rename ta to t be so we implement a replaced I'll ox this is a something that proves that if I remove you from a and then add V I got B so this something you can write in a shapeless style thing this is a type level constraint or let's look at a some along axis this thing we want to implement for example like this this is a 3d tensor of ijk if we want to sum over J will get I K so one dimension is just gone so we want to encode this type level constraint is we remove you from a will get B so if we sum along ta along the axis of you will get TB so we do this in this way so we first get the index of you in in a this n is that n type of is U and then remove this axis of n to be so these are all type level operations in scholar so this index of and remove add are my own additions to shapeless in shapeless H list these are not implemented so let's implement index off so what is index of we know that so index off is a type level function that we can use to get the index of a specific type in H cyst so we know that X is the zeroth element in ex-cons T and we know that if X is the ice is the eighth element in T then X is the I plus 1 element in H cons t this is a recursive thing okay so we do in this way okay let's be fast um I said that's that's not enough time now um so we're gonna do a native C and CUDA integration um so like this so this is PI torch PI torts uses MK open class and CUDA underneath and then generates a bundle dynamic library library that's like died lib or s o in Linux and Mac so we generate a bridge to Python we plan to do the same in Scala so there's the same thing but we have a type class over here so this type this type class could be implemented with C negative things using a swig or kind of like Java C++ binder yeah so and we are supplying it to you could also because Java Scala runs everywhere scholar runs at a JVM so we can use torch MMX net through a bridge to run on on using torches not underneath implementation in Scala or if we do scotch is we have tensorflow Jas as the backend probably you can ask is either discard native using the raw torch C code underneath so I don't have enough time lapse of Whitman's quick so a neuro networks in a lot of natural language processing tasks that are going to have dynamic structures so this we have a sentence a sentences of variable length so and in a lot of senses we do our nn recurrent neural model on top of that so this is basically a fold operation or a scan operation in functional jargon so in this framework we implement this with a fold or scan instead of like in Python they just do it in arbitrary functions or in a syntactic setting the cat sat on mat we know where this is the syntactic tree there's a sentence this is a noun phrase verb phrase we can do this tree cat a morphism on top of to get the something on top this is the standard model for sentiment analysis in NLP or you could even do a normal machine translation for example this the Germans in a TAS house is client so we do this as a forward and the backward and then together and then unfold to get the English translation the houses is small and this is the end of sentence symbol so this is a variable architecture the state-of-the-art seek to seek architecture from neuro machine translation this is about like a thousand lines of coding in Python but um from if we do in Scala we know functional programming this is a very functional program this is scan left scan right zip with and this unfold so yeah so this could be actually written like I I think we can actually do this we can write a neuro machine translation in less than twenty lines of code in Scala yeah if we have these primitive operations ready okay so is there enough time left okay so um there's also this engineering issue of static and dynamic computation graphs so there are tens of aphaia MX named when they first produced a whole graph and then interpret it and there's a PI torch trainer and glue where they the per office while you while you build your graph the values of the graph are actually computed so one of this is eager and this is a completely lazy you could do that think of it that way but this has this problem because because I've said before there are in in in natural angle processing you have variable lengths so these things are hard to code in static graph because every graph is different so but this is easier in dynamic graph but that it's not lazy as eager sometimes we don't like that so in our setting will produce a views will produce a user control via evaluation could be either strict or or eager or lazy because we for for every symbolic value can put a value function into that to get the value out of it but it requires a kinase a interpreter a computation instance so whenever there is a computation instance we can get the value so we'll get the value that's strict or if we don't it's lazy so we can have something like this so this is the architecture so passed through a layer a sigmoid activation function another layer and a soft max output so and then we compute a cross entropy of the loss so given the input and the output so in this given Clause there's an implicit computation instance so in this block we can do a lost start value but out of this block you cannot so every computation here is just lazy and every conversation inside this block because of me strict so we can switch between the these so it's just like it's I think it's a very cool idea okay so to make it fully work we need to we want to do automatic batching complete GPU support I have preliminary GPU support this garbage collection because these GPU members are allocated on the GPU and JVM doesn't know about it so we have a new manual garbage collection and probably distributed learning through spark so that's I'm gonna talk I'm just doing this I mean I'm this is just my side project if anyone would join me I would be happy thank you that's the exactly the kind of talk we want to see here where you're taking something in substance type level programming and something practical like deep learning and glue hands together we do have time for questions so okay yes I aware of that actually I have contact with him we have some some correspondence with him but his idea his uh his tensive ro Scala is just complete it's just a simple wrapper over tensorflow that's its API is exactly the same as the Python API so it doesn't have these types of tensors yeah yeah that probably could be done that could probably could be done but I I have a problem with deserializing Onix into my system because mine is typed and onyx is not so I'm not sure I think I can port my thing to onyx but I'm not sure how can do it reverse way reversible probably yeah probably but right now I'm using a torch as the under underlying implementation yeah so I bind a torch to JVM and Costco's torch okay all these checks are done in compile time yeah at runtime all these type signatures are erased they're just a tensor underneath or is in the porch case everything is erased to a long that's a pointer to the GPU memory torch so the PI towards seek or I don't use the PI torch Python wrapper inside pi torch is a c++ core i'm using that so yesterday we saw a talk from the foundation guys in which they were wrapping in that case pascal yes so in the same memory space using a facade yes it sounds kind of similar yes to what you're doing with that C chord yes that's right and there is also a new project called Haskell torch that wraps torch in Haskell using the similar things yeah and it's also a pretty typesafe yeah yeah I know I know I I write Python every day I feel the pain so I guess that I guess that people would eventually and know that this is a I think is a better way to do everything [Applause]