Scale By The Bay 2018: Yunsup Lee, Leveraging Scala to Build Hardware at Scale
Recording: Scale By The Bay 2018: Yunsup Lee, Leveraging Scala to Build Hardware at Scale
you thank you very much it's great to be here today I'm gonna talk about something quite different today I'm going to talk about how we leverage Scala to build hardware at scale at sy5 so by now you're wondering what do I mean by Hardware which part of the hardware am I talking about so today we're gonna talk about how we leverage Scala to build chips like that the micro processors or system on chips or SOC s personally I really don't like the word SOC but that's what the hybrid industry calls themselves Sox SOC Stoops micro micro processors all interchangeable during the top so before I talk about that I like to talk a little bit about some trends in the hardware industry so you know the first trend that I see is compute needs are changing our imagination is endless it's limitless we keep come up with all these workloads that are computationally demanding like machine learning we've talked a lot about that here today at the conference but we all noticed that GPUs are not getting any faster this is the spec int score compared to the VAX 11 back in the 1980s and you kind of see it started stagnating at 2015 when was the last time you bought a computer that was blazingly faster than the previous one so people have been doing a lot of things to overcome those problems first we talked a lot about how we're accelerating the workloads on GPUs GPU is a custom piece of hardware that is more specialized for certain applications some people go all the way to building compute custom chips like Google TPU is a great example it excels at doing one thing very very well but it doesn't do quite well on other things that's all good but what's the problem custom chips are very very expensive to develop so this is the process technology node as you go to the right it's more modern process technology node like 16 nanometer and 14 nanometer and developing a chip on that process takes about according to this graph 320 million dollars this is an exponential gap where there are more compute needs that are rising CPUs are not getting any faster we need more custom chips but less and less than people less and less people can afford it right this is a huge problem so I've been telling my colleagues in hardware that we have to just cop pletely copy success from separate worlds and you know I I think I think there are four things about it why software world excels and the four things are open source abstraction composability and productivity open source is the ultimate form of reuse you know the best way to build a build your application is to not write new code just go leverage all the code that's out there I mean that's the best way of you know building applications and open source is the ultimate form of sharing abstraction is a great way to split your interface important interfaces to the implementation so that whenever a new thing comes along you can quickly you know adapt composability is how you compose all these important interfaces to build a bigger system quickly last but not least productivity is you know that that obsession about tools on how I'm gonna do less and less stuff and you know be lazy about it so that you know I can do more and more things with less effort and less people you know there's four things so the sad thing is every time I go tell my hardware colleague about this everyone says we're doing this too so then I have to tell them a story right so you know Instagram right then they said yes I do know that do you know they were billion dollars when they're sold when there are only 13 people and they're like whoa that's eighty million dollars per engineer Wow that's that's great and then I have to remind them why does a hardware startup just to get started you need at least 14 engineers to get started Oh yeah you know like that's right 14 engineers off the bat you know forget about the billion dollar you know valuation or something this is this is by the way I counted them and I got really depressed and really jealous about the software industry by okay so after after that period I kind of started thinking about this so why is that the case so if you go google Instagram tech stack actually the software world just shares all the technology that they use there's two things ease of development ease of deployment nobody in their same mind today goes and builds Linux from scratch just to go build a web app nobody goes and builds these databases from scratch nobody does that just go go download some package and you get going and couple that with ease of deployment you you you don't need to go build a data center to go build a web app you just go buy some compute and as a matter of fact Amazon gives you free computing at the beginning to kind of lock you in so you know you just can get started and a couple of these two things together you can quickly iterate you can deploy your application get feedback fix the things that don't work and you know then you get a billion dollar company at the end of the day so alright by the way Harvard that doesn't look any anything like this so that's what's i-5 is trying to do sci-fi we're building a tech stack hardware tech stack based on open source technology not sure how many of you guys heard about verse 5 but this is a free and open instruction set architecture that I started working when I was a grad student at Berkeley as a matter of fact we were the upper you know story of the amp laughs who did the spark and that's why we got into Scala so on top of us five we build chisel which is our hardware description construction language built in Scala and fertile compiler which takes these circuits down to industry standard circuit exchange formats like Verilog and of course we leverage Scala and hence the reason why I'm here today it's really hard to find people who intersect between scope and hardware and you know this is my Hail Mary to go find some people so we're interested so that's all the open-source technology but even though you have all the RT hardware components that you need to go build a chip building a chip still is 300 million dollars so you know that's the parts i-5 is is doing we're building a web service to help you go build hardware so I'll talk a little bit about that today too but why why are building all this open-source hardware stack why you know like you know why would you do that so this is the initial team at Berkeley who are doing risk five if you count count all these people it's about 12 people so with 12 people what we did is we defined a new instruction set architecture we did the architecture we did the mark mark architecture we did all the porting of GCC operating system you know build the chip package it test it get it on the board and actually do it you know we didn't have an army of Engineers when I was at Berkeley you know a lot of these companies have thousand people working on one chip we didn't have a thousand people so we had to do something differently so and with dozen people we actually built about 18 for the past six years actually 16 nanometer chip it didn't take three hundred million dollars we we did it quite you know I guess there is some split from the graduate students to but you know there's a lot that we got from Scala too so it's a combination of two things you know but you know nonetheless Scala was a big part of the entire thing so quick introduction on risk five it's a free and open instruction set architectures so what is a is a it's it's the language of a microprocessor all the software that we write has to boil down to these set of instructions so the ones that are popular today are x86 and arm which are all proprietary so there's really nothing we can do you know about it so risk 5 is a free and open version of it so you can think of it like Linux of ISAs so it's a fifth Jenner wrist design arrest stands for reduced instruction set computing from UC Berkeley it's you know Andrew my colleague and friend and I started the project in 2010 these are the four wrist projects that were there before and we're experiencing a rapid uptake both in industry and academia and is a is you know designed for the smallest microcontroller all the way up to the biggest supercomputer and since raise five is a free and open standard there are open source implementations of the ISA but there's also proprietary implementations of the ISA as well and it's supported by a growing software ecosystem all the you know gnu toolchain stuff been utils GCC Nuala GFC are all up streamed in their respective projects we have a bunch of operating systems for it like zephyr linux free and freebsd and a bunch of distro supported to risk v like fedora and debian and the standard is maintained by the non-profit risk v foundation so if you want to make any additions or suggestions to the ISA you can join the foundation and you can you can suggest changes to it so the risk v foundation members consists about 125 plus companies you see a lot of the giants there if your company is name logo is here thank you very much if you're not maybe consider joining you know it's a lot of cool stuff happening there so to show you the state of the software for a risk v I pulled a graph on the Debian web page you see that hockey stick there that's the number of risk five packages ported and that has been done in the past three months we went from zero to 80% of Debian packages port diversify and I'm happy to report that we have more package supported than ia-64 Itanium that that that's you know it's it's not being used today so when there's a new thing a very important thing is education - so I'm also happy to report that the undergraduate textbook and the Graduate textbook for computer architecture written by Dave Patterson and John Hennessy are have all adopted risk 5 so all the undergraduates and graduate students who are coming out of school will be up to speed on risk 5 ok I'm gonna switch gears a little bit and talk a little bit about the hard word that we built at side 5 so this is fu 540 freedom unleashed 540 it's a 64 hit multi course 5 SC so all the chips have this block diagram associated to it to kind of you know specify what's on the chip so on the in the heart of the chip you can see that u54 MC core complex it's a multi core 4 core quad core 64-bit RISC 5 application processor with an l2 cache and a D um controller connected to it there's a bunch of you know peripherals like SD card readers by I screwed see you are GPIO and all that there's a Gigabit Ethernet and the guts of the chip which is you know called tiling is also coming out of the chip so that you can connect FPGA chips which is good for prototyping hardware so you can map your custom hardware in the FPGA and you can connect it with the chip and you can actually write software for it so that you can you can kind of prototype your hardware before you know you're interested so this so this core runs at 1.5 gigahertz it's manufactured in T SMT 28 nanometer we also put this on a development board with 8 gigabytes of DRAM and it's fully functional we also built a expansion board where you know the guts of the chip is coming out so it's connected to this FPGA so that you can pour your custom hardware into it and this Gigi has a PCIe connection over here which there's two slots to it so there's a by 16 PCIe slot and a by 1 PCI slot so you can connect in a GPU card here and a USB card over here and there's an nvme drive on the back so it's it's it's a real machine so runs Debian you know web cam tweets we're at Twitter today so tux cart on the back all works and so this one is running Yolo be three on MVD LA which NVIDIA has open source recently for their edge inference machine learning accelerator so we map that onto the FPGA and we connect it with the RISC five board and all the drivers and all that is ported to it and and you can run these type of things so I like to show you a demo of the board but the cool thing is you've been seeing the demo already this entire presentation was running on the board so Google presentation and you know the web browser and all that works wait I'm waiting for an applause applause I'll prove I'll prove this to you right so it's JavaScript it is not there yet so yeah it's maybe somebody can work on it it's yeah so here's the terminal it says risk 564 see if you cat the proxy PU info you can see its RV 64 ima of DC and its sy 5 in the rocket processor right here so it's a 4-quart machine and you know it's stepped in so the standard the standard software thing just works so I'm gonna go back into presenter it's kind of cool I think alright okay so how do we build this so that was all the marketing speak so let's go let's go see some code so before I tell you all the Scala stuff you unfortunately you have to understand a little bit about what it takes to go build a chip to appreciate what's going on so the traditional approach of building chips the at the lowest lower there is something called GDS - that's a blueprint of the chip it's it has to be manufactured in layers it's like building a cake it's a 48 layer cake and you you know you put a you know you kind of shine light and you take some parts off and you know it's like building a building right so that blueprint looks like this and in this there's all the transistors that are designed into it and you know the reason why the 340 million dollar thing is 340 million dollars is because it typically puts like 20 billion transistors together that's that's kind of crazy so back in the day when Intel we're building 386 and 46 you know you know people drew this like by hand these are polygons and you call that polygon pushing and there's all these rules that you can't put this thing next to this thing and you know all that it's almost like building a permit you kind of like you can't you can't take it in it's crazy what people did that like I mean you know some people still do that so people you know in the early 2000 people a yeah you know that's just too much you know there's something called languages and since you know you can compile things and things like that so I briefly mentioned Farah log to you right so there's a Vera log there and this is a language so there's some state elements it's which you know only counts when the enable bit is high you can see the counter updates and you know otherwise you leave the counter and when you reset when the reset line is high the counter gets initialized to zero so this is better than pushing polygons but this very log language you know initially was developed for verification not Harvard construction so there's all these language parts that are not synthesizable so you know it's like it's like somebody has to look into the code and tell it and tell you like you can't do that like it's not a great language to build hardware because there's all these things you can you can do wrong like why do they have a language like this so please I'm sorry this is the functionality of the chip you have to synthesize and place synthesized into gates and then there's a tool called the place and route tool which takes the gates and sprinkles it around with knowing the design rules so that's automated now parts of it so the entire industry has successfully raised a level abstraction for sure you know that that took twenty years and it was great is very successful but if you look at software it's like I dropped by dry analogies here GDS too is binary like you know that your exe file your L file that you have to run some people can write binary into like your text editor and you can write a working program for sure I mean I I used to do that in PC tools in the das days but you know some people can do that but you know going and touching up the GDS - is basically that and you're thinking like why would you do that then you know there's this assembly language which raises the level of abstraction a little bit more where you still have registers and all that but you know still it's not composable and it's hard to maintain I think Vera log is that and the software world basically figured out a way to raise the abstraction even further like compiled language like high-level languages and even Scala which compiled sentence JVM and all that and you know some people don't want a program Scala like give me a web application so you can go even higher than that so that you can kind of hide all those all that and deliver value still so this is what we're doing that sci-fi you know we're not stop being a verilock we raise the level of abstraction to something called chisel it's our language that we use which is embedded in Scala and we use a compiler infrastructure called fertil which basically turns the circuit into a compiler problem which fertil is also written in scala and you know for some people that's a little bit too much still so at sci-fi we're also building web services on top of all of this so that we can even hide all the Scala nasaw infrastructure so that's what we're doing at SCI 5 so let's dive into chisel first chisel is a hardware construction language embedded in Scala and there's a reason why I said that not HDL which means for Harvard design language and Scott and chisel we strictly have only things that you can do which are correct to go build a hardware so we removed all the things that are illegal in the language chisel is basically basic hardware building blocks in classes in the Scala abstraction like registers boxes and wires and we're using the modern programming language to productively put them together that's it using parameterize types object-oriented programming functional programming and we leverage static typing which are all functions of a modern programming language and it's great so with all this we're basically doing meta programming for hardware and the nice thing about this is the meta programming language and the construction language are in the same language and since it's embedded in Scala you know one of the core values of Scala is to let other people build higher level of abstractions that couple to language chisel is embedded in Scala and we let people do that on top of chisel as well so as I promised here's some code this is a ripple carry adder for you know for those who took a digital design class you know you remember how to build a ladder its 4-bit adder here it consists of four one bit adders all the inputs for each bits go in and the carry you know kind of crosses along and your son comes out so this is chisel code as you can see it's kind of it looks Scala II there are things that are defined by the chisel three package like module input bundle IOU int you know wires and things like that and we at we overloaded this colon equals function on these modules to facilitate something like a domain-specific language so this you know is all instantiating for adders and you know you declare wires in between and you connect them with a for loop so the act of executing the code will have a data structure of the circuit in the memory which we serialized down to Verilog so this is how chisel works but you know you probably are looking at this code and you're like this is my scarf and we we acknowledged that you know like a lot of the people who come from the verilock land this would be the first code that they would write in scala and in chisel and they be like yes i use a for-loop you know and and that's great so - there's multiple ways of writing this and slightly generalizing and parameterizing it and making it more you know more more scala ii so i've generalized this into a prefix um which you know we know that there's a list of things and the output of the thing is you know you you you do the operation on all the things and so for example if you had a adder in as your adder for all the summons this would be the output you know i'll put zero v70 up it won't be 0 + 1 0 +1 +2 so generally speaking this prefix um definition would look like this you could give a sequence of summons and you give a associative operation which you know operates on these types great so there here's a concrete implementation the prefix um which implying the ripple prefix um which basically implements this this structure over here you you you know do this operation and then you call a recursive function to to you know write all the layers down note I have not used a chisel type yet it's parameterize types T right so you can actually give a Scala type to it and it will just work fine a slightly more complicated version which is faster with less layers is doing this parallel prefix um or dense prefix um where the first stage you do it with the offset of one you apply the operators the next stage you do operator for every two two lines over here the next one will be four and then you stop when you know the offset is larger than the lit the length of your list so here's the dense prefix um implementation you shift the offset byte by one every time every stage and you do the associated associated operation and then you just call it recursively and you're done and you can use all this to go implement the adder that I just showed you so note I didn't really have any chisel types now so I'm gonna make it a bit more concrete now with the actual which is all types so this example will basically select the first data valid data element off of this list of valid iOS also is still parametrized so valid IO brings a valid signal and the date the bits the payload of it and basically I want to examine all the Velvets and only return the first valid element is a circuit that I'm building here so so I provide the method of the prefix um that I want to do in this case it defaults to deep dense prefix um then I register all the data and then I called the prefix um with an operator that operates on the chisel datatype and with all the valid and I basically shift that and I I find a first valid one and I that through a MUX so I didn't have time to draw this pretty so this is my handwriting here so assuming that you have four elements over here the the valid types come in here's their registers of the all the ballads and here's my dense prefix um with the or operators and with all that you know I shifted down and I I you know flip the bits and and it so that I get you know one hot signal over here I can put it into MUX and I'll get that first one so that's you know how we use Scala to go build you know cool hardware and output of that is this the you know the in memory data structure of the circuit and this is the IR and intermediate representation of a circuit so and we call that fertile flexible IR for RTL so what happens is there's a front end which is basically chisel chisel execute and it generates a F IR RTL internal data data structure and then we push it through a bunch of transformations to get it down to something that we can emit into a verilock format so if you look at that you know it looks like a traditional compiler where you can put a bunch of transformations of your own when you're transforming the circuit into some barrel lock so at a high level it looks something like this similar to what you know software world does where there's a bunch of projects and bunch of libraries which is which are written against this particular language there's a front end which breaks it down to I are like LV Mir and then you do a bunch of transformations and when you're done to mid code you will map it on to a back end like xyx arm or wrist v so similar to that we have a bunch of projects that are going on so rocket ship is the processor library written in chisel which by the way I searched get up today all the trending scholar project it was number 25 on get up so that's kind of cool there's other you know library like chisel utils and things like that so rather than just wrapping all this around inches an implementation what we do is we our front end I mean we have a language it's chisel and we have a chisel front end similar to claim which turns into fertile and there's a bunch of transformations that run and at the end of the end of the day it will map into a certain back-end which today it's fair along so and all the fertile infrastructure is also written in Scala so to give you an example here's how a transformation would look like so this is a pass which runs on a circuit and for every module that I match I'm gonna call the squash empty you know you don't function so this is a very simple example but you know you can see a lot of the things you could do which is which are orthogonal to your program I'm sorry the circuit so so those are kind of important things that we use to build Hardware chisel are infertile which are all in Scala and I like to kind of show you the web application layer that I just talked to you about so for folks who want to design risk five cores who don't want to deal with the Scala program you can just come to our websites call the core designer and there's all these kind of functionality and optionality that you can go configure and then you can go review and build and this deploys to Microsoft Azure cloud where we have all the right tools and at compile Scala you know runs all the tests before there's a delivery ready for you so that's what we do for a core designer and we're not stopping there by the way this is sci-fi chip designer which is building your own custom chip so you can load some templates with certain I blocks IP I know this is a foreign word it's called inner inner electoral property it's a block that you have to pay money to get so I call it IP blocks so there's a bunch of IP blocks that you need on a chip so there's a bunch of third-party vendors that we work with to onboard their IP blocks on therap platform so that when once you want to go come build a chip you can come to this website and you can you know select your your bits and pieces and you know do some configuration and you had review and build and we will take care of the rest of all the things that you've seen you know synthesis listen Rao making sure that the design rules are met you know manufacturing testing packaging and we'll give you a working guy back to you so we're building this with a bunch of partners in the industry so we're building this entire stack and we're integrating all that into a web application so I have to tell this thank you very much for working on Scala it's really important to us where we bet the entire company on Scala as a matter of fact and you know but the other thing is it's really hard to find people who know Scala and hardware at the same time you know you know if you if you know Scala you know more and if this thing if you think this is cool hardware is simple we can we can work on that so you know please come work at sci-fi I joined the revolution and thank you very much I'll answer [Applause] that's right absolutely right so Western Digital has announced that the Martin thing does CTO of Martin Western Digital has announced that they plan to move all the billion cores that they're using to risk five Nvidia has already announced that all the microcontrollers that go into the GPUs will be all risk five so that was about one and a half years ago so you know I don't know exactly but my suspicion is the GPUs are coming out today will probably have risk five processors in them so the particular customers that we work with who have already announced their product is Kwame has a SmartWatch which has our microcontroller in it there's a company called voodoo who builds SSD controllers with it and nvme drives and things like that so it's and there's a bunch of projects that are happening all over the world so which you know probably will come out soon so Brady's first leader of why exactly Scot the first part and the second longleaf this color force misplaced something it doesn't right so the first question is why Scala I don't really see your good reasonable here using Scala second question is sorry I was thinking of the first question I answered the first question every ask the second question so we started out with Ruby when we were doing all this oh wow this was a photo for vision board here and then you know we were talking to the folks downstairs who were ant lab guys you're doing spark at the moment and they basically sold us on Scala they're like why are you using Ruby you know like Scala is comes with a great runtime it statically typed you know there's the the way that the syntax works encourages good-looking DSL so I mean it looked great and I'm not regretting that choice at all the the actual benefits of using Scala doesn't come from the leaf level modules actually there's a lot of program ability that goes in when you're interconnecting the entire thing because a lot of things change when you interconnect and you know we basically built a Scala framework to go deal with all that and it was probably impossible to do with any other language maybe I don't know the second one I remember now is you know do we cache the output of the generated hardware ah right do we catch any errors yes because Scala again is a statically typed language there's a lot of bugs that we don't even see before we go into you know elaboration there are obviously errors in the microarchitecture hardware design itself which we cannot really debug in Scala but I think there are ways to go do that and we're certainly open to exploring those possibilities [Applause] what's next for side five is the question so the reason why I started sigh five is when I was at Berkeley I felt extremely lucky to be there because we had free tools and we had free you know space to go tape out chips and with that I was enough to be Dane you know dangerous enough to go build my own processor and you know compilers and all that and it was a blast you know I I knew nothing about building hardware before I came in I actually was a software person who wanted to know more about hardware and I felt extremely lucky that I got a chance to go do that but if you look at hardware today since its 320 million dollars there's only certain places around the world where you can learn how to do that I think that's bad for the hardware industry and the numbers actually back that the hardware industry is only a four hundred billion dollar industry when the software industry is a trillion-dollar history you know there there needs to be more developers and when was the last time you met somebody from undergrad who was so passionate about hardware not you know it hasn't been a long time it has been a long time yeah so the ultimate thing for force i-5 is to build this open-source hardware development ecosystem where a lot of the people will use Scala to go build hardware and it turns out the new people who have hasn't really learned véra log and things like that they have a easier time learning building hardware and Scala it turns out so I think there's a lot of things we can do there and you know my final goal is so that anyone in their world to people who are determined their garages can go build a chip startup that's really what's i-5 is going thank you for the question [Music] better partner yes if we say hey try to know oh you're quite familiar with the process all right so the question was how are we dealing with the ecosystem with chisel because the output looks very different so every time somebody comes to me and talks about you know named wires with T underscore and u underscore things I ask them when you debug your software do you look at assembly code and most of it some people do yes I I do - I'm not trying to offend you or something but most of the time you can use high-level debuggers and other type of tools to go debug do your software and as you said that's some of the problems of chisel which is a new language where certain ecosystem things are not there which will help you debug all these things so I showed you the whole stack of partners that we're working with and since chisel is getting some traction in the industry some of the commercial vendors have started to wonder what they should be doing about chisel and chisel generated code that said you know I think we're not there yet but I think with more and more people using the language and using all the existing libraries and having like a ecosystem and a development network and I think we'll get there so the question is what type of customers sy5 have and what are they working on with us great question so chisel is a very radically different way of building hardware and you know for established hardware companies who already have a well-established flow and things like that it's hard to penetrate with chisel right away there are some new projects that are starting at these big companies who also use chisel so I think those folks know we're engaging at that level but for folks who have a more like a traditional way of building chips we we work with them at a at the verilock level so they're engaged with their web products so they would come configure their cores that they would use in their products and we're building some chip templates the kind of the generic form of chips which are very configurable with you know certain companies so you will start seeing these startups using our services - right absolutely I'm sorry micro processors oh I I love that question by the way so right so the question is what are we doing really you know what do I think about side-channel attacks and how are you going about it right so the side channel attacks you can consider that a bug but it's also featured oh I didn't know we had a screen saver it's a bug and a feature of an out-of-order processor of the underlying fundamental architecture of microarchitecture of how we execute code and it's a big problem it's it's a hard problem how do we still get all this performance well you know not leaving all these breadcrumbs and the really cool thing about verse 5 is that we have a forum where everyone can work together to go solve this problem we're not limited by Intel we're not limited by arm I mean they have a lot of money but they only have so many people working on the problem I really think risk 5 the really cool feature about it is everyone speaks the same language and everyone can contribute ideas that work so this is a time where we have to declare emergency as a computer architecture community to go work on this together but we can't work on on a proprietary instruction set architecture that's why a lot of people around the world who are you know world-class experts on this topic are coming to the risk by foundation to work on this together so I believe where's five will solve you know will mitigate the problem better than anyone else and to kind of prove that point you know there's this gnarly subject in in microprocessor and it's called memory models it's about you know it's kind of reasoning about all the loads and stores and orderings and how other processors look at it I mean it's it's a very complicated mathematical problem and the respect foundation at the very early stage most of the times Intel and x86 - is an afterthought you know they built some hardware and they tried to formalize what they built and risk by was since it was a open is a all the world spritz on memory model who came to the risk by foundation and we wrote sped specifications we wrote formal models we check them each against each other and all that so that is the proof of why open ISAs have a brighter future than others so welcome you to come to the respect Foundation as well you're going any other questions well with that I hope you can go you know convinced intense your friends to that type of is a very exciting company thank you very much [Applause]