data.bythebay.io: Marek Kolodziej, Hidden GEMMs: How Optimized Math Libraries Work
Recording: data.bythebay.io: Marek Kolodziej, Hidden GEMMs: How Optimized Math Libraries Work
Hi everyone. This talk has a bit of a fun in the title hidden gems as you may know from the general description of the talk. Gem here stands for general matrix multiplication and these gems are really hidden because I assume most of us who do machine learning spend a lot of time using canned solutions such as MKL by Intel or CuBlas by Nvidia and I was just curious you know what these guys actually do to make things perform and because sure we can use these libraries but at the end of the day there lots of benefits from understanding how to make any numerical algorithm performant both on the CPU and the GPU so I just dug into this a little bit and decided to share what I learned. Um and um I not going to spend any time on marketing or anything like that but I wanted to say that we're hiring we're looking for Scala developers the standard Scala stack of Akka Play Framework Spark etc. We are definitely looking for a lot of front end people security engineers systems engineers etc. and C++ although the C++ group is mostly in Dublin but in San Francisco every everything else. Okay without further ado let's focus on the content here. So I'm going to talk about optimization on the CPU quite a bit
Um from just a naive triple loop order N cubed implementation of a matrix multiplication all the way to cache aware algorithms. And you know my main interest these days is a computing and so I'm going to talk about GPU optimizations as well. Um the reason I'm putting more stuff in the slides that I'm not talking about is because I'm adding more detail for reference if you guys want to check out the slides later. So, why bother? Again, you know, we can just use MKL or cuBLAS or things of that sort. Well, I really do believe that even though matrix multiplication is a very simple algorithm, a lot of the learnings do translate into other problems. And to give you an example, um I went to the GPU Tech Conf in uh in uh San Jose in April, and uh Nvidia just announced that they were adding a graph library for GPUs. But hey, you know, before uh CUDA 8.0, there was no graph library by Nvidia to do graph processing. So, yes, you may have had, you know, cuBLAS and cuFFT and cuDNN, etc
But if you were dealing with graph problems, you didn't have an analogous library. So, there's always something on the frontier that may not be canned and provided. And so, from that perspective, I think it's good to learn what what one can do from scratch. And uh you know, matrix multiplication itself is obviously extremely useful uh uh you know, from affine operations in deep neural networks to uh uh convolutions for convolutional neural networks, although you could do convolution more efficiently using a fast Fourier transform. Um you you can use uh gems in matrix uh factorizations, and here's an example of using a gem in uh in LU and um Cholesky factorization, for example. So, yeah, it's pretty common, obviously. Um Um I'm not going to go through the basics of matrix multiplication too much, but because I'm sure you remember that if you have an m by k matrix and a k by n matrix, the inner dimensions, the number of columns in A and the number of rows in B have to conform, and then you get an m by n matrix, and you multiple uh you take the m's row of a and the m's row and column of b and you get the m comma n element in in the matrix c, right? So, um here are some specs of the machines I used. Mostly I did the computations on this laptop actually, just a standard Core i7 CPU, and there's a basic Nvidia GPU on this laptop, 384 CUDA cores, but this is I think the Kepler architecture, so that's not the newest architecture
Um but the final demo that I'm going to do at the very end is based on the Nvidia Titan X, which is pretty much a state of the art for double for single precision computing. Teslas would be good for double precision, but people are actually going into half precision right now and even 8-bit computing, so honestly I don't care about precision anymore that much. Um anyway, so here's the naive C example. Um uh you know, just a triple loop. We iterate over index I, which is the number of the row index for A, matrix A, and J, which is the column index of matrix B, and K is the that inner dimension that has to conform. So, very simple, right? We just accumulate these values. Should be really trivial. And because it's C and we want to be performant, you know, we're probably okay using void functions, which is really horrible if somebody is a functional programmer, for example, but hey, you know, this is massive, it just needs to be fast
Is it fast though? It's not. This is actually quite slow. It's 1,200 milliseconds. Um I'm going to actually give you some comparisons because it's really hard to, you know, think about that as as a value without actually a reference to to other measures. So, well, just for comparisons in Java that's going to be twice as slow. Um, obviously, the JVM gives you some overhead, but hey, 2x is not too bad considering that in Python, if it's pure Python, it's going to be 180 times slower. But, you know, it's an interpreted language. It can be forgiven for that
That's why we have NumPy. And speaking of NumPy, we get 74 milliseconds. So, why the heck did my code in C take 1,200, right? This is very disturbing. Because, yes, most of the computation here is happening in C and not in Python, but still, that's kind of disturbing. And MATLAB and Octave gets us 20 milliseconds. So, okay, 50x faster than this stupid triple loop solution. So, what is going on? Um, I can't really get into too much detail of modern CPU architecture because that's a whole separate topic for a few hours, but if you guys are interested, there are some really interesting sort resources here. There's a very good summary by Don Lu from Microsoft, and uh I forgot where Matt Godbolt is, but he's a very good engineer
He has a great understanding of of Intel's architecture. But, you know, all standard processors, whether Intel or ARM, have instruction pipelining, branch prediction, cache prefetch, superscalar processing, SIMD or single instruction multiple data, out of order execution, etc. So, you cannot really think linearly the way your program is actually written, even if it's an imperative program, because the processor that just doesn't work that way. So, I'm going to talk about some of those details in a moment. Um, yeah, so if you guys are interested in pipelining, this is really the best resource over here. Uh, one thing I definitely want to mention because cache locality is pretty critical for this particular problem is that uh Uh, it only takes us one millisecond, one cycle to access a register, about three cycles to access L1 cache, 14 cycles for L2 cache, and 240 cycles to access RAM. What what does that tell us? That anytime we have a numerically heavy algorithm, the vast majority of the time is going to be spent the processor the processor is just going to spend waiting for data. It's not going to do any computation
Um, it's bandwidth bound. It's not compute bound. So, the processor is just sitting there. Um, and uh for those of you who remember remember electrical engineering, um, you know, DRAM or the main RAM is actually relatively slow because even though the the solution here is very simple, it's just a simple capacitor, you have to refresh the capacitors, you have to have analog-to-digital conversion to measure the voltage of the capacitor, etc. Whereas SRAM, which is the what how the cache is built, actually looks very complicated, but this is digital logic, so it's really fast. Um, and unfortunately, this digital logic uses a lot of power, and on top of that, it's it takes up a lot of space, which is why we have kilobytes of cache, and we have or megabytes at the most, and we have gigabytes in main RAM. So, we have to live with these tradeoffs given that the technology that we have right now. And, you know, just to illustrate how important cache is, if you look at these 4 megabytes, which fit on a particular processor, this particular processor has a 6 megabytes of on this machine has 6 megabytes of L3, but let's say this particular processor in this demo had 4 megabytes
We see that the difference is between, you know, 5 and 10x for cache misses because as soon as we we reach the capacity of the cache, we're down here, right? So, so we definitely lose a lot of performance. And up here is registers, so definitely there's difference between registers and cache, but once you get down here, we're already out of the cache. And so, the processor is mainly waiting waiting for data. However, as you can see, random access is still twice as half as fast as as sequential access, which means that the CPU is still doing a good job at least prefetching the data. And this is actually another point that's important. Um so, how can we speed up that stupid triple loop? Well, first of all, if we just reorder the the loops, this code these three lines are identical, but we just reordered the nesting of the loops. So, now we have K and J, not J and K. Um this means that we're first going over the rows of A, um which is where we would get cache misses because C is a row major program programming language, unlike Fortran, which is column major
And so, um and so, accessing an individual row is much faster because you're utilizing the cache. So, this is this um index moves slowly, which is good. But then, we also want to move across the uh um we want to fix the rows of B of the second matrix, and we want to go across the rows of the B matrix. So, that way both A access A matrix A access and matrix B access is by row, and that actually allows us to have more cache locality for both matrices. And guess what? We suddenly went from 1,200 milliseconds to 160 167. That's a 6x speedup, right? That's becoming reasonable. That's still bad, but it's already, you know, only half as slow I mean, half as fast as as the NumPy implementation, but it's not, you know, it's not terrible anymore. Um that also tells you that GCC is a pretty stupid compiler because the fact that you have to reorder the loops yourself means that it didn't do a good job optimizing
Um, and I'm going to show you that Intel's compiler, ICC, actually would have figured that out for you so we wouldn't have to do that manual tweaking. Um, and um, the next step, uh, which is pretty common in both in GPU and in CPU computing is tiling. You can say, "Okay, instead of taking one, uh, row of one matrix and one column of another matrix just to compute one element, we can split this computation into multiplying this row, um, by this column to get this element for this part of the matrix, uh, storing that result in the register, then doing this computation, this row times this column, um, and then adding, uh, that to the result stored in the register, and so we basically split that computation into subsections, and that allows for even better cache locality." And this is becoming really horrific, right? I'm not even talking about like Haskell versus C++. I'm talking like even for, for, you know, for C programming that is becoming really messy and nasty and, um, pretty awful, but there are benefits. So, at first it would seem that there are no benefits relative to loop reordering because, uh, the loop reorder and the tiling is about the same. If we actually the micro benchmark and average it across many iterations, it would probably be the same. Um, and they're both much faster than the naive, but they're not really different from each other. However, this is because GCC does a poor job optimizing
If we switch to Intel's compiler, then suddenly we see, "Hey, Intel already optimized the naive implementation. We didn't have to reorder the loops." So, same result with or without loop reordering because the Intel compiler was smart, but our tiling solution actually gave us another 2x. So, now you may have noticed the NumPy implementation was 74 milliseconds, we're at 56, we're faster than NumPy. So, that's pretty cool. Um it's still slower than the GNU Octave solution because of ATLAS, which is a self-tuning library that is more cache-agnostic um or MKL in the case of MATLAB running on Intel CPUs. So, you might get additional boost because, you know, Intel's spent so much time tuning that library for different matrix sizes, so they actually have different subroutines for different matrix sizes. So, that's just a crazy amount of work, but we're already getting reasonable here. Then there's this issue why people still use Fortran
I think that's kind of interesting just as a side note. In Fortran, basically when you're dealing with pointers, Fortran assumes, "Hey user, you know what you're doing. You're not going to be accessing the same piece of memory from two pointers. And And if you do, you know, I'm not giving you any guarantees." That's why Fortran can hyper-optimize the code, right? In C, because you can get race conditions and and and, you know, some non-deterministic result by accessing the same pointers in memory concurrently or doing out of order execution on the same point of memory, in C it's basically defensive and it always synchronizes a block of code and says, "Hey, you know, I'm preventing out of order execution." And that is causing a lot of slowdown, and this is why Fortran is faster. Now, in C there's this keyword restrict, and so you can theoretically get Fortran-like performance, but I still think that Fortran's actually faster. Obviously, anybody who grew up after the punch card era probably would not like to write Fortran anymore, but I guess it's good that some people did so we can have fast linear algebra. Um and so the thing with the with pipelining is when your when your CPU is actually executing some piece of code, it's already decoding the next instruction, it's fetching the next instruction. So the pipe So the pipeline allows the processor to always be executing something because it's already doing the decoding in the meantime
It's not waiting. The problem is if you have dependencies in your computation, you're going to get what's called the pipeline bubble. The pipeline is going to stall because it's going to say, "Okay, the next instruction is uh depends on the result of the computation of this instruction. Therefore, I need to wait to see what the result of that instruction is." And uh this is very common, you know, there's not much we can do about it, but sometimes we can, especially if our code is, you know, has to be very performant. Once in a while, we can do some crazy stuff to optimize. So one example is loop unrolling. You know, here we have this accumulator and if we have a for loop because in the next step of the for loop, we actually depend on the result of the sum accumulator from the previous step, uh this actually stalls the pipeline. So that's very bad
That really reduces uh CPU performance. You can say, "Okay, what the heck can I do about this, right?" This is as simple as it gets. Well, you can have two accumulators and it um um update the uh the uh uh loop counter by two and then you have two stalls, but the stalls are every other step and then you actually have more opportunities for out-of-order execution and stuff like that. And then you just add them together. Hopefully, a smart compiler would actually do it for you, but sometimes the compilers are not smart as I showed with the GCC example. And uh in fact, if I uh pass this code through GCC, the assembly shows me that this loop exists. I compare I compare and then I jump to if not equal and then I just move two values and add them. So, so obviously it's purely sequential code
There's no no optimization here. So, GCC even screwed up that unrolled solution here, which is pretty awful. If I use Intel's compiler, it does vector operations, and it basically does kind of a reduction here. So, it it it does element-wise addition here, and then it does a reduction of these additions from before. So, it actually does parallelization of about 4x, and that actually that has major performance implications. So, if you're doing C programming on on the Intel architecture, really use that compiler if you can, because it makes a big difference. Of course, you know, cache awareness and stuff like that doesn't change the um the uh um uh complexity of the algorithm, and there are algorithms with better complexity. So, we're talking about the cubic order, and there are algorithms like the Strassen's algorithm, which is order 2.5
The best version of that algorithm is 2.5, or Coppersmith-Winograd, which is 2.37, right? Much better uh complexity than than cubic. However, the problem with these algorithms is, as you know, big O doesn't always matter, because with Coppersmith-Winograd, you would have to have matrices that that are of the size bigger than the RAM of of even the biggest machines for that uh for that uh complexity to matter, because the constant factor in that algorithm is so big that the complexity improvement doesn't make any difference. Um and people do crazy stuff in these libraries often times by basically um having different different function calls for different matrix sizes. So, it's really kind of a black art, but I think that the general idea of using caching better, for example, and uh and pipelining are are pretty become a applicable to other problems. so I think it's an interesting uh insight. And this particular book, this the famous CLRS book actually does cover some of these matrix algorithms. Okay. Um and then if you're really hardcore, you can do intrinsics, so you can do essentially what looks like assembly in C
It's It's basically going intrinsic vector operations in in for different architectures like AVX for Intel or Neon for ARM, but this is really unnecessary in my opinion. You can do just fine without this kind of junk. And then because this prevents your code from being portable, so you might as well have written assembly at this point. Um there is something though that is platform independent, which is fused multiply add. When you're doing a dot product of two vectors, you you have a step where you add the result of of a multiplication. So you first multiply then you add, and so you can fuse that in one operation. Most CPUs these days, both GPUs, Nvidia's GPUs, and Intel processors, and and ARM processors actually have the ability to do fused multiply add in one operation, so that cuts the number of computations by 50%, which is great. The problem though is fused multiply add is less accurate, so you have 0.1 * 10 - 1, and that it gives you essentially zero for for this float, but if you do fused multiply add, there's a bigger precision error
So if you have computations that that require a lot of precision, this particular optimization is useful, which is um basically where adding this FMA argument to to GCC, and that will work on ARM, Intel, CUDA if you're using NVCC the NVCC compiler for Nvidia, etc. Um um, and you know, most compiler manufacturers actually do say, "Hey, don't don't do don't use the fast math option because fast math will actually crash and burn if you're doing precise computations." But, if if you're okay with that, that's actually great benefit. And uh another example from Intel, it does the um a vectorized uh multiply add, so you don't have to vectorize yourself, which is great. Okay, finally GPUs. I think you're probably more interested in GPUs these days because of deep learning and stuff. Um, so um what are GPUs in general, you know? Um the general uh philosophy behind GPUs is that they're high throughput processors because you have hundreds or thousands of uh threads running in in in individual arithmetic logic units um instead of you know, let's say four to 16 threads on a on a CPU. Um Um but because uh you can only fit so much logic uh into these chips, um they have a much uh smaller control units, so they cannot do things like out-of-order execution and pipelining and branch prediction and stuff like that. So, they're very good for number crunching, but they're they're really terrible for arbitrary logic
So, uh so um but if you have a high throughput uh problem that uh that can be high latency, um then that definitely is great. And that is definitely the case with most numerical processing, right? So, linear algebra, deep learning, um uh digital signal processing, stuff like that. Um Also, because GPU computing is mostly for number crunching, um uh uh GPU programming is actually lower level than traditional C programming even because you have the ability to, for example, programmatically access the L1 cache. It's called shared memory in in Nvidia's terminology, but it's really on the same chip as L1, so you can you can programmatically decide what's what goes into L1, whereas on a CPU, you know, the it's a black box, right? So, you have to, you know, play with with uh with things that I showed before, but you have no guarantee as to what the cache eviction policy is. But, the flop benefit, right, the number of floating point operations is huge. So, if you look at Intel here, you know, it's basically an order of magnitude less than than the latest single precision um NVIDIA GPUs, and um there are even faster ones these days. Um so, clearly uh for for, you know, algorithms that train for weeks, such as uh you know, AlexNet or something like that, doing that without a GPU these days would be just ludicrous, right? Training a convolutional neural net of that scale. So, um so, we do benefit from the flops
Um but also the memory throughput is much better because uh the GPU is uh split into many streaming multiprocessors, which means that um you know, each of these multiprocessors has its own memory bus, and and the the memory throughput is also much better than than in the case of the CPU. So, yes, if you look at the sort of simplified representation, the GPU has plenty of cores, but the control unit almost doesn't exist. Um whereas on the CPU, the control unit actually has more space than the cores, but um but then the the multithreading is much more limited. So, low latency and low throughput, high latency and high throughput. And this is pretty much how it looks internally. So, each each multiprocessor has a bunch of cores that execute separate threads, and each of them has each of these multi-processors has a register file and and an a thread scheduler and stuff like that, but all of them have separate memory accesses, which is why the memory bandwidth is so high. And as I mentioned, you know, in addition to global memory, which is traditional RAM, you can actually programmatically access not just Well, the registers are actually not programmatically access accessible, but traditional you know, local variables in a in a function will do and and the compiler will optimize that, but you have access to programmatically controlled level one cache or shared memory. So, the naive computation is very very similar to the naive computation on the CPU
We take um a thread, which takes a whole row of A and the whole column of B and computes one element. Um but before we get to the computational examples to the to the code, you know, the general idea of of using of using the GPU from from your native application is that you first have to allocate memory on the GPU. You have to do a mem copy from the main RAM to the GPU RAM, then you have to launch your kernel, which is your actual CUDA function that does the math, and then you have to copy the data back. So, your application better be compute heavy because the cost of going over PCI Express to ship the data to the GPU and from the GPU is huge. So, if your computation is small, then then your GPU computation could actually be slower than the CPU. So, your your data set be rather you know, it's best for it to be rather large. Or if you can do something like SGD, right? If you're doing some kind of stochastic gradient descent or any kind of iterative computation and you can keep them the data in memory on the GPU and not ship it back and forth, then you finally get some benefit from the GPU as well. So, this is the a very simple um GPU kernel
Um we uh have to refer to an individual uh thread block and thread index. And um and we iterate over K, which is that inner dimension that's conformant between the two matrices, and we finally get the result in this accumulator. We ship it back to the to global memory on the GPU or the main RAM. Unfortunately, it sucks. It's really slow. It's This is uh 280 milliseconds, which is better than the thousand the 1,200 we had for the naive CPU implementation, but a reasonably optimized CPU implementation, even by me, was like 50 milliseconds, and the MKL implementation was 20 milliseconds. So, what the heck is happening? This is pretty bad. GPUs are supposed to be magical, right? So, we have to resort to blocking
Just like we blocked uh or tiling. Uh just like we did tiling on the CPU, we're going to tile on the GPU. But right now, because we actually have control over the level one cache or shared memory on the GPU, we can actually push the data into uh shared memory ourselves, and then it's going to be resident there until we release it or until the kernel goes out of scope. And so, because of that um once we do this uh this direct copy, all the subsequent computations are going to be uh based on shared memory, which is much faster than than the main memory on the GPU. And so, we do the same tiling as we did before. Now, that is unfortunately getting really nasty because now we have to um we have to allocate shared memory, which is the level one cache, a type of memory. We We have to push the data into this cache ourselves. Then we have to sync the threads across the block
Then we have to finally do this computation for the accumulator, and then we have to sync the threads again. So, that this doesn't go out of scope. And the reason we have to sync here is because we're doing tiling. So, this memory will go out of scope after this loop. And so, and so, we basically have to sync before we go to the next step. That's already quite messy, right? GPU programming is really low level. If If you thought C is low level, then this is lower level. This is like almost assembly because you have to think about the hardware a lot
But, we're getting better. So, 193 milliseconds. Then, there's this notion of aligned and coalesced memory on the GPU. Aligned memory means you are accessing memory based on your thread index. If you're accessing the memory not based on your thread index or a multiple 128-byte multiple of the memory address, then you'll require more memory transactions. And normally, GPUs ship 128 bytes at once. So, it's it's a lot of data is shipped. And if you have multiple transactions, you're actually going to lose the whole performance benefit of the GPU
Coalesced means just accessing contiguous pieces of memory. So, if you have several threads together, they should be accessing contiguous pieces of memory. Um So, this is aligned and coalesced because it's on that 128-byte border. Um the the consecutive threads are accessing consecutive pieces of memory. This is all This is not consecutive, but it's these threads are still accessing this contiguous piece of memory. So, that's still is coalesced and aligned. This is misaligned because we're passing that 128-byte border, but the uh memory addresses being accessed are are coalesced. So, So, all GPUs had bad performance with misalignment, but new GPUs are actually stable
They don't care about alignment as much. So, that's good. We don't have to worry about that crap because how will we remember the memory address, you know? On the other hand, um um uh uncoalesced access, meaning accessing random pieces of memory and not accessing uh contiguous pieces of memory means that there will be a lot of 128-byte transactions, and that would basically eradicate the whole benefit of using the GPU. So, here's, you know, 100 GB a second, and you got to maybe 10 GB a second if you're if you're doing striding. And what happens if you're um taking the second matrix, the the second operand in a matrix multiplication, and and going down the column. Well, you're striding because s- uh the C memory uh the mem- the memory model represented in a programming language like C is row major. So, every time you're going down the column, you have a stride of the size of the row. So, basically, this is pretty bad, and uh we'll need to figure out how to improve that
So, now with coalescing, I'm not going to go through the details because there's just no time. Um um we will get another benefit. So, we went from 280 to 190 to 150 ms. So, that's better. Now, there's this thing with uh with bank conflict. So, you can programmatically access the level one cache or shared memory uh in this case. And if you're if each thread is accessing an individual uh address in a different memory bank, then this access is completely concurrent. However, if uh if several threads are accessing several addresses in a given memory bank, then that those accesses will be serialized because there's just one piece of uh there are just two wires going, right, into that uh memory bank
But if different threads access different memory banks, so you can flip the memory access uh to this level one cache or shared memory, then you will get a huge benefit from that. So, this is definitely several threads accessing one bank. You'll have, you know, eight to eight-wise bank conflict, so you'll have eight serialized operations. This is completely concurrent instead. So, it's getting nastier and nastier because we have to think about the hardware a lot again. But, we're now down to 200 to 100 milliseconds from from 280, right? So, we keep improving. Now, this is actually the kind of a linear algebra trick. So, finally we're not thinking about the hardware, we're thinking about the math a little bit
That's more interesting. Um so, what can we do? If we're doing a matrix multiplication, normally we just do this inner product, right? But, what if we did an outer product of individual rows and columns? So, here's an example. Here we have a 3 by 3 matrix and a 3 by 5 matrix, so the result is obviously going to be 3 by 5. But, instead of doing an inner product, what we can do is we can take the first column of X and the first row of Y and get a three by five already, then get the second um row the second column of X and the second row of Y, get the three by five, and do the same thing for the third row and the third column, and then add them element-wise. And we And as you can see, we get the same result from the inner and the outer. And why is that important? Because on the GPU, if you have two operands that are accessing shared memory, that access will be serialized. You cannot have two operands that access shared memory at once. So, what we would rather do is put some stuff in the registers and um and and only have one operand coming from from uh from shared memory
And so, we can basically each thread can access one one element of each row of B. Um and entire tile of A can be in shared memory. And we do as many iterations for each thread as there rows of B. That way we just store one element in a one register for B. We just operate in a in a loop and basically B is in a in a register. One column of C is in the register per thread and and A is in shared shared memory. Okay, this is horrible, right? Because in addition to all this thought process, I actually realized that I had to unroll this loop because I have another loop here and you know, even with C++ pragmas and stuff, it's not going to do a very good job. So, so maybe some loop unrolling was necessary here, but um but if you already know the loop size, then maybe that's okay
Okay, so we're at 55 milliseconds where the CPU speed at least at the reasonably optimized CPU kernel. If we just do kernel only timing, so if you discount the whole shipping over PCI Express, it's 46 milliseconds. Um and so we went from 280 to 48. That's already a big improvement. But, you know, if you're into that kind of low-level thinking, then there are professional expert matrix multiplications for the Fermi, Kepler, and Maxwell architectures that have been implemented by experts. And and they get into many more details about this kind of stuff. Now, if we use cuBLAS, which is Nvidia's BLAS implementation, then we actually get down to 11 milliseconds. So, I already optimized by 5x and they optimized by another 5x
And so, 11 is already better than the 20 milliseconds or so on using the Intel library on this machine. So this GPU is really tiny on this on this laptop, but it already does operate twice as fast as the CPU. Um Um okay, and just the last thing if you if I use the Titan X, which is the top of the line GPU, then I would have gone down to 1 millisecond. So this is kind of an overview of how we went from 1200 milliseconds on the CPU with a very naive implementation to 1 millisecond on the GPU with a very performant implementation on a on a big GPU. You know, a difference of a thousand X is huge if you're doing deep learning, right? When your algorithms run for days or weeks. Um so yeah, this is the overview of the timings and I hope we have a little bit of time for questions. Yeah? Did you see that Google's going to make their new ASIC? Yes, I I heard about the ASIC they call it TPU, tensor processing unit. I was really excited when I saw that yesterday, but then I actually thought, okay, I think this is more marketing than reality and I'll tell you why
One example. Here you you saw three orders of magnitude of improvement and they only said that the difference between the top of the line GPU and uh uh TPU is one order of magnitude. Secondly, Nvidia actually demonstrated at GTC in April that there's a ton of opportunity for improvement on the GPU itself because for example, between CUDA 7.5 and CUDA 8 using cuDNN, their deep neural network library, they got a 6X speed up just by changing the software on the same GPUs uh for a recurrent neural networks and considering hard recurrent neural networks are to uh improve performance for because of the you know, sequential nature, it just shows that improving software is still the the way to go relatively inexpensively. The next thing is the Maxwell architecture gets an enormous Sorry, not Maxwell, the Pascal architecture gets an enormous boost in performance. And the final thing is um Microsoft actually demonstrated their FPGA implementations. So, FPGA is programmable hardware, and FPGAs are normally much slower than than uh ASICs or application-specific integrated circuits. And Microsoft's um FPGA implementation was about as much of an improvement as Google's hardware, but you can reprogram the FPGA. So, you can change the algorithm, and I think that's more flexible
And even Intel has FPGAs built into their Xeon processors now, so Yeah. yeah, yeah, sure. Thank you.