Scale By The Bay 2021 : Panel : The Hardware vs Software
Recording: Scale By The Bay 2021 : Panel : The Hardware vs Software
all right let me introduce you vitaly uh and so this is the uh classic panel of our scale by the way conference welcome everybody who is joining us uh for the panel uh you know we have a history of really interesting debate panels and vitaly was the one who actually introduced this format we had several debates in previous years including you know what will happen if everybody will weather will everybody will move to the cloud and functional programming machine learning you know whether that will take hold and i think actually it doesn't i think we are moving to the cloud uh so today the topic is hardware versus software in various aspects i think ai is a big theme but also there are many others and again vital is our fearless moderator so i'll introduce vitale vitale basically build salesforce einstein which is which a combination of software engineering and data science and so this is the way we like it really because it's it's basically a scalable first uh application of ai where focus is on high quality engineering strong without programming spark which we love right and kind of multi-tenancy and so forth and uh now vitali is the ceo and founder of forest totayai uh and uh vitaly will moderate this panel he set the rules and i believe he will let the panelists interest themselves so vitaly take it away lexi thank you for the uh nice introduction um so folks for those of you who have never seen this um kind of format before we're gonna do kind of friendly panel a friendly debate between two teams of two um arguing a kind of a emotion and then we'll see which kind of one of you were moved the most from kind of one side to another by kind of the arguments that were made the history for this pale um this debate is our other previous debate where there was a comment made about hardware versus software and we decided that you know we should continue this conversation a year later so as you folks know um where all the increased cloud computing on one hand it seems that um kind of is being abstracted by compute apis on the other hand there is a resurgence in a lot of uh amazing companies and development um in silicon and there is a definitely now a sense where something that was thought about previously as a commodity becoming is actually a very interesting play where people are taking notice so that sounds like a making of an interesting debate like i mentioned we have two teams of two argue for and against uh motion that hardware or compute is like electricity people will not care how you get it and arguing for the motion meaning that hardware will be abstracted are alex o'connor and david cantor maybe david you can start by introducing yourself absolutely so uh i am one of the founders and the executive director of ml commons for those who are not familiar it is an open engineering organization that focuses on making machine learning better for everyone we do the industry standard ml perf benchmarks which i had the privilege of leading the the inference ones uh we also build large open data sets and i'm thrilled that we have two of them that are going to be released at nurips this year and then we also work on best practices to sort of eliminate the frictions that you have in using machine learning today and uh perhaps particularly relevant to this i have a long history in sort of semiconductors microprocessor design other hardware related things so naturally i was stuck on team software yep we like to keep it interesting so and your partner is alex alex can you please introduce yourself hi everyone i'm a senior manager for data science and machine learning at autodesk in particular i work on our e-commerce and customer empowerment team so we work in personalization natural language processing and the like and we run our support and assistance chatbot eva which is firmly on the uh side of uh software as well so i have uh i'm looking forward to the opportunity to do it awesome and the side arguing against that is that actually hardware will have its resurgent moments that will become more and more important are anna goldie and brian cantrell anna and perhaps you can go with introduce yourself hi i'm anna goldberg i'm a staff research scientist at google grade and i'm a co-founder and co-lead of the machine learning for systems team there where we focus on machine learning approaches to various problems and systems optimization and chip design and you know recently we published a paper on reinforcement learning approach to chip floor planning that i'll talk about later today awesome and also anna you know did not mention that she was also going to recognize this here as one of the 35 innovators under 35 by the mit technology review and brian you were not recognized as one of the 35 innovators under 35 so what do you have to say for yourself uh the tally i actually was i was in the tr 35 in 2006. so probably for detroit right i'm saying this i'm saying this year this year it's tough you know they unfortunately they've got a tr-35 they don't have a tr-47 unfortunately so um i'm considered to be out to pasture um so i'm brian cantrell i'm the co-founder and cto of oxide computer company i'm very happy to join anna on the pro hardware side of this fierce debate um and to watch david get uncomfortable arguing a position that doesn't necessarily believe in um the uh i we are oxidized oxide we are building a rack scale machine um bringing the innovations from the hyperscalers to the mainstream enterprise on-prem market with the big thesis that jeff bezos is not going to own and operate every computer on the planet so excited to be here oh wow we already had kind of the i think the 21st century goodwin law already if like you know how long does it take to mention jeff bezos so uh we'll do it um in kind of a panel of intro so there will be a first uh step where basically every site will introduce kind of their opening statement about five minutes uh per person then we'll have some questions from the audience and for me and eventually we'll have some you know closing arguments and hopefully some you know friendly debate in between so the side arguing kind of for um david or alex who wants to go first uh uh alex yeah let me start um go ahead david all right so i believe i've got five minutes so i'll trust you to keep me honest uh on timeline yep uh so uh i want to point out so i actually studied math and economics at the university of chicago unlike everyone else here i i actually don't have a background in cs except for a minor uh so i'm gonna actually take a tact from my economics training and so i claim that actually ai is not too dissimilar from a wide variety of other technologies that are frankly to our ancestors magical things like electricity flight automobiles right and uh or real-time uh ad bidding for instance um uh and so you know if you look at all of these things the way it evolves you know you can look at the history of flight like we started out you know and there were people designing planes that were like a bicycle with wings that would flap its wings like a sparrow and you know as it turns out that actually doesn't work and so the arrow you know like the dreamliner works on a slightly different set of principles as does the 737 max right and so you know what happens is you have a natural maturation where the way a lot of these technologies work is initially they are you know very raw and exposed there's all sorts of things that matter what plane are you on the safety ratings are very different the ranges are very different but as time goes by you ultimately get uh the beauty of software actually is sort of forced by economics which is you have layers of abstraction right and that happens due to the economic system and you know it happens in software as well because it's a good engineering practice but if you look at electricity today right most people probably 90 percent don't actually really think about where their electricity comes from right now if you're in california it's in pg e and you know we're really only reminded of where it comes from when things catch on fire in napa valley uh but you know realistically that layer of complexity generation distribution uh you know inspecting right of ways that's all hidden right and so if we look out 10 15 20 years if we want ai and ml to really be profoundly shaping our world that's where we need to head right so the mechanism for that is again layers of abstraction and so you know if you want to look at like take some of the most remarkable capabilities out there say google's t5 model um you know which is my understanding bigger and better than gpt3 but gpt3 will work right macy's and goodyear will never train gpt-3 right we are going to have a small number of companies with that capability that focus on it that have the systems understanding to tackle that problem and then the rest of the world will kind of collect that bounty downstream right and so you know if you were to look at it today you might say it's going to be transfer learning or fine-tuning or delivered via an api right where you know maybe you know you just send to the cloud you know your image and then it comes back and says okay you know ups just dropped off a package go grab it before you know your neighbor does or whatever right and so that's sort of where i anticipate that ai and machine learning will ultimately evolve which is the complexity around hardware and low-level things like math kernels and like what are your hyper parameters for training all of that needs to get hidden same thing for the entire deployment pipeline right you know even if you have a trained model like you know what anna and her team produced for rl for chip design ultimately you know getting that deployed is a very non-trivial endeavor and so removing that friction is actually what's going to unlock a lot of ai capabilities you know everywhere and again you know a lot of that is okay we need to take this model we need to get it to you know run in the right place we need to check and make sure it's not going to do something crazy like uh you know uh uh accidentally identify that i'm a criminal and send me to jail you know think you know not that a chip design algorithm would do that but you know you've got a lot of standard processes that need to be resolved and all that's orthogonal the hardware right this is fundamentally a software problem and then i think the last point i'd make is i'm a big believer in what i call data centric ai right which is that ultimately the way we drive up the capabilities of ais with bigger more diverse uh data sets and again this is something that is orthogonal to the underlying plumbing right if you want to have a self-driving car the key is getting data on a car operating in winter right which you cannot collect in san francisco because we just don't get snow right so you need to get your data in new york city and again that's not really about hardware for ai except insofar as there might be sensors involved there so you know when i look at the the fundamental challenges to really making ai go from like magic that no one understands to magic that operates in the background and everyone takes for granted you know which is ultimately what we want you know it's it's about software so i i rest my piece and thank you david right on time er so brian or anna who wants to go first for this uh side arguing against go for it anna okay i can start off um so i guess it's always great if you can abstract away the hardware but i think the problem here is that you know all this progress that we've been making in ai and machine learning has been fueled by advances in hardware and the effective compute that we get from hardware is a function of like the fit between the algorithm that's running on that hardware right and so therefore we can't and we unfortunately cannot ignore the hardware that that we're running on like we've been running various like experiments on you know how can we do hardware software co-design and it's clear that you can get multiplier improvements in performance if you do a better job of this like co-optimization task and so i think that the future very much like requires that we focus on the properties of the hardware and the software together maybe brian do you want to add to that sure yeah i uh totally agree um and we're we're definitely team hardware software co-design here um and um you know my view is if you look back at the last uh 40 years in hardware and software uh hardware has been moore's law has dictated the direction that hardware has and and has dictated the revolution in hardware uh and i think the most important revolution in software has been open source um which sounds trite to say but the the older i get um the more i appreciate how important soft open source software is um it's been interesting for me to go back and read of course i lived through this so i i should remember this but reading tales of software development in the 90s feels foreign to the modern software engineer because so much of the of the trials and tribulations of software development in the 90s was due to proprietary everything compilers were proprietary operating systems were proprietary databases were proprietary you couldn't do anything without consuming proprietary software the open source revolution has been extremely important for software moore's law has been extremely important for hardware we are on the cusp of those now flipping i believe that open source is going to become extremely important for hardware and i believe that the end of moore's law is going to be extremely important for software so um both of these mean that the other is going to have to learn from the from these previous revolutions i mean if you look at to take the open source revolution to to hardware um we talked a lot in the in the 80s and 90s early 2000s about reconfigurable computing and people kind of forgot about that reconfigurable and computing is actually really important the problem is those tool chains have been entirely proprietary and is exactly right about the ability to unlock these order of magnitude gains when you design hardware and software together but in a world of proprietary xilinx a proprietary altera or formerly proprietary lattice and trying to actually do this i mean i feel like i'm going back in time looking at this proprietary software that is expensive that it doesn't work it doesn't actually allow us to deliver the benefit of that um i feel like the most one of our most unsung heroes in computer engineering is claire wolf and the work that she did to reverse engineer on the the lattice bitstream format and allow yosis to actually allow open source tools to generate a bitstream for an fpga uh this is a total breakthrough that people don't really appreciate we are now seeing this the i think another breakthrough that people don't appreciate open sourcing of blue speck um a really interesting hardware description language um that is it's a small demographic but i think an important one uh blue spec doing for hdl is what rust did for for software development uh and then at the same time the the end of moore's law i agree with a lot of what david is saying about the need to get more and more data but the end of moore's law means that we are actually going to need to focus on the efficiency of our compute and i think you know look no further than than the m1 and the interesting hardware software co-design that apple has done there to achieve great performance but great great performance per watt um that is the real a real breakthrough there and we have not yet seen that the data center um i believe that's coming all of that is coming with hardware software co-design well thank you anna and brian and alex um you know uh when did you wrap this thing for us sure uh thank you so i think it's it's great to see such like uh insightful debate already i mean we're shortly into this and it really does make sense to me to think about though what actually the impact of this is at the end of the day because both anna and brian have done a wonderful job of explaining why at the margins of performance and scale there are these useful ways to think about how we can fine-tune these things and to me that all reflects on what david said earlier about this idea of electricity generation i'm sure it's super fascinating to get into the experience of you know the blade design for uh turbines and it's really important for us to think about those sorts of aspects but ultimately you know the real impact of machine learning is not about the uh you know the choice of specific hardware or the choice of specific stack it's about this understanding that it's really uh you know it is a flow of data and a flow of automated decision making and speeding up of business processes that actually is the impact of machine learning and hardware is extremely far from that that experience for most companies and most practitioners and getting further and further away because of the power of open source and because of the power of uh you know the commoditization of ml the you know pervasiveness of ml features throughout throughout the environment so to me i think this is the point you know you know the the optimizations of how it works at the at the end of it that the impact of that is getting more and more distant from day-to-day practitioners and more and more distant from the experience that people have we're all using uh you know this um kind of metaphors in the same way but what we really need to think about is the fact that you know the biggest challenge for ai is really not in uh you know as i say those marginal performance gains it's really about that question of you know which companies can leverage their um their data which companies can transform their business processes and they will increasingly seek to do that from uh you know main providers from people like uh from large companies who have the resources to look across scale to perform those experiments themselves because we're moving from that home inventor that you know oh i wired the thing together myself put it onto the rack and built it up this is a cloud journey again you know from the 90s where people were kind of installing racks and made out of lego in their carriages i mean i hope no one is doing that anymore right instead we're gonna we're gonna use cloud provider of choice and gpu of choice or whatever tpu or xpu of choice from that on and i think that experience also exists at a now software level which is i people will move away from training their own models or even perhaps fine-tuning their own models and instead you know expose their data to a central provider and you know make leverage that and that will be increasingly important because as companies it allows companies to focus their effort it allows organizations to say these things that are all the things beside what i want to do as my core competency i can take best advantage of along the way and i think that's an essential thing to think about as we think about this experience the other part of it for me is the disconnection between um you know the place that i train the place that i develop and the place that it runs so you know oh nnx all of those sorts of technologies they are handling it i'm very happy to allow all of the much smarter people than me in the world to abstract all of that stuff away from me permanently and i will just be able to say let's make it go uh uh let me just you know make it go away for me let me not have to worry about that in the way that i i do today and i long if i never have to install an nvidia driver on a docker image again i will be a happy man so for me this is very much the key point about it is that you know um essentially what we're thinking about here is the real impact of ai is data the real impacts are the real value of that uh is is the fact that open source is about um a strategy of coalescing effort and providing many eyes and many hands on the parts that should be shared and none of those are things that really matter about the hardware uh so much as they do about the ability to abstract away those concerns in an essential way and for me that's what i would think about on this on this matter the um you know what we're seeing with this is you know the what we don't really understand i think yet and what we will begin to understand is what this idea of pervasive machine learning is going to do to uh the developer experience as well we're beginning to see this idea that you know it's it's moving away from this idea that i think machine learning is a separate role from uh you know or is a separate thing that you know exists in its own box and is becoming part of every piece of software that you have to develop and intelligence is embedded that way as well that's also going to change the tools that's also going to make it need to be more abstracted away from the hardware the way all other software is the way databases are and the way thread management is and everything else like that well thank you alex um and again there were uh very interesting points uh made here uh by each side and now we're gonna move to kind of a question answer where you know both sides can um ask each other hold on can i can i interject moore's law is not dead well you can start with that statement and go for it so yeah why do you think it's not dead oh okay so uh uh one of my hobbies is semiconductor manufacturing and device physics and and so actually there was a great keynote from pat gelsinger yesterday pointing out that moore's law is not in fact dead now i claim moore's law is you know like uh you know brian you said you would be eligible for tr 47 so perhaps this statement will resonate but i'm going to suspect you are not in as great shape as you were at 20. and moore's law is sort of in the same way you know at 50 it's just a little bit slower than it was in its vigorous 22 25 year old days you know it when it takes a drink and falls down it takes longer to get up but if you look at the innovations that we have coming you know you've got things like gate all around transistors you've got stacked cmos you've got new materials like i'm actually quite confident that we're going to see a fairly significant improvement in hardware performance now it may not be uh in the same orders of magnitude that we got historically but i think we've got another decade to go and i think you know the important thing is that as software folks you can probably step back and identify that and from a hardware software co-design perspective right if we write all of our code and see and tune it well we can probably wring out orders of magnitude but despite that you know the uh most common language for machine learning is uh python which is uh not performant to put it mildly because you know the reality is that you know and this is what i've always said about hardware design is when you look at things like caches versus scratch pads right a cache is basically an acknowledgement that i want like ordinary programmers to be able to get media and good performance out of it and so you know i i think it's anna's point which was that you know ultimately there are leaky abstractions and you know you can't fully isolate hardware from software is in fact true right i would not deny the power of hardware software co-design but at some point that performance delta drops from orders of magnitude to 10 20 30 percent and then it becomes a question of you know am i willing to drop i see anna doesn't agree i can't wait to hear the response but you know i think if you were to talk to any development manager many of them routinely dropped 20 30 40 performance on the floor all the time for the sake of getting things out the door quickly right i mean and you can just see that like people don't do security which is a freaking legal liability because it's like no we just need to push out this feature and please ignore the fact that like all everyone's social security number is now a public record you know so well i think let's maybe let brian respond uh well you know feel free to go for it and i think you and i are both going to take a take an opposite approach here you are going to go first or uh do you want me to take a swing you can take a swing first uh yes we got a real debate on our hands now um so uh the good news fatality we we we have uh we we've got a real debate here um moore's law is indisputably slowing down so um there's just no question about that um and yes gofets are a thing um yes we are i mean i think citing a gelsinger keynote as the reason why moore's law will continue um i mean there's uh there's a lot of uh counterpoints to be had on that um the uh intel themselves failed to deliver 10 nanometer for in fact we still don't know what happened with cannon lake right cannon lake was i know i i i've joked that the the conflagration at cannon lake sounds like a hardy boys mystery um but we actually uh don't actually know uh why intel for years failed to deliver um and now we've got a hunch they fail to deliver because it's actually really hard we're pushing the physics really really hard um the i i feel obligated to say this may i imagine my co-panelists know this but you and the audience may not realize this that when we talk about process we talk about nanometers of the node of a a seven nanometer node five nanometer node three nanometers with canon lake um those nanometers are that's a made up number they don't that is not actually um that's an expression a rough expression of transistor density it would be much easier if they talked about millions of transistors per millimeter per square millimeter but they don't or cubic millimeter um they um so nothing is actually uh three nanometers in a in a three nanometer process or five nanometers in a five nanometer process um the the question is what is moore's law is not an absolute expression of the maximum density achievable by humanity it's an expression of economics it is the it is actually cents per transistor transistors per dollar that is indisputably ending it is certainly slowing down and the uh and i i i'll just make one other point and hand it off to i'll tag out for for anna to get her licks in but the when we talk about the i think both david and alex talked about how uh you know we're building these abstractions and yes it's inefficient but um it's the abstraction that matters i i feel like i'm talking to the auto industry before the 1973 oil shocks about that gas is just infinite and why would we ever bother to even think about the actual efficiency of our engines why would we bother to think about the actual pollution that's coming out the tailpipe it doesn't matter it's going to go on forever it's like well and we learned in 1973 and then again in 1979 it actually doesn't go on forever these actually are finite things and when we i you know at some point we are going to have to explain to our children and grandchildren how we used our finite fossil fuels and lit them on fire so we could solve math problems and sell them to one another um that's going to be uh that's going to be embarrassing um i think that that in in many regards is kind of the the proof of work that we see in cryptocurrency it's an embarrassment um it is it is implying that these resources are infinite they are not they are finite and the way to get to drive better performance and importantly better efficiency and by better efficiency i mean not unconscionable inefficiency is going to be with hardware software co-design as moore's law slows down anna yeah so go ahead i just want to call out that we already have the topic for our next debate cryptocurrency is an embarrassment hold on is that a debate well it depends on who we'll find for the other side but yes anna please go yeah i ahead i completely agree like whether moore's law is dead or dying it's certainly not keeping up pace with the demand that our machine learning algorithms have for additional compute so if you're looking at some of the data in terms of compute consumption by ai models they're doubling every 3.4 months so even back in the day warsaw wouldn't be able to keep pace with that and if you think about like you know how can we reach our goal with ai like if we wanted to get like a you know one percent error rate on imagenet we need many orders of magnitude more effective compute so like we're just not going to get there without hardware software co-design and you know in some sense like additional compute is just a very clear fuel for progress like if you see those scaling laws papers by open ai it's just this smooth power law the more compute we have the more data we have the more parameters we have the better the performance and it's just um we just like we need to um do better in terms of our software hardware co-design in order to be able to achieve that and sure like you know maybe python is like the top language right now for machine learning practitioners and researchers but really it's just because most of the most of the computation is taking place in these like calls in the tensorflow or high torch apis to hardware that have it happens to run pretty well like you know these matrix multipliers do well on current hardware but um really if you look at say like you know sarah hooker's hardware lottery you know there's a lot of machine learning approaches that could be extremely promising they just don't happen to run well in today's hardware and if we really if we want to unlock the potential that ai and machine learning has we need to be able to be more adaptive to these potentially more promising approaches awesome so uh alex i want to kind of hand it off to you also kind of respond but let me kind of try to upload the conversation a little bit so we're not going to drag on whether you know moore's law is dead but i want to kind of offer a different perspective i think a branch and vmware so one of the kind of you know great jobs i have as a founder of a startup and brian you probably you know share this one and maybe there as well is that we're also the procurement officers for the laptops for our teams and as a procurement officer i have to say that when apple announced their new m1 laptop like everyone you know came in and like suddenly you know something happened to their laptops and they want you know they spilled coffee on it accidentally all in the same day so beyond kind of um you know everything that is going on is like there is genuine excitement about the kind of those advancement in silicon so i would also be curious kind of why i think you know maybe it's still you know doesn't matter it's all you know going to be abstracted anyway so i you know i think it's great i remember telling me uh someone explaining to me what the extraordinary suicide rate of blackberries was the moment any company authorized iphones back in the day it was exactly the same uh same thing they astonishing how they all the screens would spontaneously combust at that time um i think uh m1x is a perfect example though of the kind of story that we're kind of the kind of story that we want to make here which is honestly for most users i think it's a theoretical right like for them the you know how many of them will actually connect collect the fact that there are ml related cores on their uh hardware to the experience that changes not a lot how many developers again maybe there'll be an advantage there but ultimately the important point is about how the experience changes and what type of software we're developing and how better integrated it is for the end user and for the developer who's making it in my mind i think that is the and the ability for for companies to do that is where the real like interest is in this in the real abstractions i don't think anyone is arguing realistically that you know we we have the perfect hardware and we never need to move away from that ever again i mean that that's that would that's a slightly crazy kind of uh concept i don't think it's a realistic one i think what's important is to ask where is the real value and where is the real like innovation uh that actually is going to be transformative and i don't think that that right now is uh is going to be in uh the kind of elementary uh aspects of you know fine-tuning the the the uh the hardware i think that it becomes we're at the point now where really we need to move from the laboratory and move into the kind of industrial kind of context level and that's really where it's about the businesses it's about the data and it's about that sort of velocity and all of the ability for us to do the things that the car companies were not able to do around sharing standards and sharing uh value is actually what we can do and you know the to me i think this you know universal adoption of um you know the the abstractions is actually and you know the disconnection from the hardware is the is the freeing of the practitioner and the freeing of the user from concerns that really shouldn't exist that's this is also going to make the the kind of alison davis uh a point and then brian and nana i would love to hear your thoughts we are now so many levels of abstraction away and i would say let's say the kind of the more common you know software development even you know think about like no code is like a thing that we're we're talking about um so there is just so much specialty that needs to go all the way um kind of to to really start understanding hardware like we're now at an age where you know people don't even know sql and they're using orm as like an abstraction layer right so uh so kind of brian rana kind of what is maybe the kind of the strongest case that actually um besides and i think maybe um david said you know or alex i think it was alex sorry that there will be definitely on the kind of margins uh there will be folks that you know care deeply about it but for the vast majority that actually um you know they just won't care um i think another like uh issue that we haven't discussed yet is you know climate change right with this increasing like this exponential increase in in the consumption of compute by various like ai algorithms the carbon footprint is also exponentially increasing and so depending on the fit between the software and the hardware that it's running on that the power consumption could be radically different and you know it could be that using a certain type of hardware is like basically consuming a blood diamond like you are mortgaging the future of our planet and users cannot be free from the you know caring about that if they're a responsible human being yeah i i absolutely agree with that and vital i mean it's a very good question in terms of of how do we i mean we we invent abstraction to allow people to do more work with less lines of code right that's what it means to be at a higher level of abstraction but when you are able to do that just to anna's point you are able to inadvertently consume far more resources than you actually should and you know for i think for much of my career i view this as a bit of a paradox about how how are we going to resolve this because how are we the software drives us to these towering layers of abstraction but we know that we we lose efficiency and i i mean i guess this is going to be very on brand for me to say this but honestly before rust i was becoming a bit dispirited because it felt like the abstraction was only headed in one direction i think the rise of rust is really important because it allows people to deliver much higher performing artifacts that are um that are really very very sophisticated without necessarily needing to have all that sophistication themselves so i mean when you one of my early experiments with rust um now coming up on three years ago um was um wading in with something i'd implemented and see and re-implementing it in rust and i was shocked when my rust outperformed my c and why is that well it outperformed my my c because in rust if there is no balanced binary tree in the standard library in order to use a in the standard library all you have is a b tree well a b tree is actually a much better data structure than a balanced binary tree from a memory locality perspective and a cache line is now looking a lot more like a page coming off of a disk and that is that increase in efficiency is something i didn't have to be aware of that as a programmer i didn't have to really understand the nuance of that difference um it's the the powerful abstractions of ross that allow me to pull that bee tree off the shelf and just use it and i think one of the great myths about rust honestly is that its rigor gets in the way of the ability to create something quickly rust allows you to create things i think really quickly but things that that operate with a much higher level of performance at a much higher level a much greater level of efficiency so um i view that as a really interesting template for the future i don't think anything's going to be rewritten and rust um but i think that it now opens the door to new kinds of languages new kinds new ways of thinking about the problem rust shows us that there is a a third path where we can allow software engineers to operate at a high layer of of abstraction but deliver a high-performing low-level artifact that's pretty interesting awesome so kind of to wrap things up and david i'll kind of maybe give you a little bit more time but let's kind of uh finish it and go into kind of the uh third section so you know please respond to what brian um and anna uh said also i would like to just uh kind of to go off uh anna's comment and ask you david why are you willing to kill our grandchildren before and let them use python that is exactly what i want to respond to so first of all so go for it and also let's wrap up why people should actually side with your side of the argument yeah so so first of all i mean i'm i'm uh uh you know the energy consumption and the impact on climate is a significant one but i think looking at it in the framework that you have is very much a fixed mindset that precludes potential growth right which is to say is doing matrix multiplication via pi torch or tensorflow really the right way to be doing things you know when i look at some of the things that i'm most excited by it's things like active learning that maybe allow you to cut the amount of data and therefore compute you're doing by orders of magnitude and essentially to train smarter to do inference smarter and so you know i'm not willing to bet against human ingenuity right and if you were to look in the 70s right at a lot of these sort of energy-related arguments it's like oh we're going to have peak oil or all these other things the reality is that economics will tend to break a lot of these things and by break i don't mean eak i mean ake right and so we'll sort of smoothly kind of decelerate on a lot of these and i think what we'll see is right yes we will be camped in how much power we can consume for compute and that will drive innovation on many other dimensions right but the people who will spend their time fundamentally delivering those innovations whether they're folks at google brain or or whatnot will ultimately make those available from a few number of people to the masses right and so i don't think we have a future where it's either we get ai that is magical or we set the planet on fire i think we can do those both but again it requires a huge amount of innovation at the software level right above the level of the hardware to return to my point and i certainly hope that's true i don't i don't want to see san francisco catching on fire regularly that would be a serious problem awesome so you're going to we're going to get magical ai and light the plant on fire that's right no no no no no no no no i'm saying i'm saying we don't yeah so brian since he could have started maybe let's kind of from everything you heard um can you just wrap up and you know tell the audience why you know you did such an awesome job and your opponents did the shitty job of explaining their position uh well i i think that that that both my uh my teammate here and i both have seen the power of software hardware co-design we have seen the inefficiencies that that can be addressed um and and i just i can't understate this enough that the the rise of open source is is a really critical revolution of this layer of the stack we have um part of the reason we have not seen the gains that we might have otherwise seen and because we've been talking about software engineers being able to to develop in soft logic and fpgas for a long time but a big part of the reason it hasn't happened is because the tool chain is terrible or has been terrible um and i think that we are i think that that's going to change i think that the indisputable end of moore's law is going to accelerate that change as people have to get more creative there's still there there is uh this is the end of moore's law does not mean by the way that soft logic is going to save us or that that uh custom design is going to save us a software hardware site isn't going to save us it's it's going to mean that we are that we're gonna have to go to these techniques to deliver a more efficient system um so it's we are not gonna continue to see the gains that we have seen um but we are going to be able to develop much more efficient systems by thinking of the hardware and software together and i think that more and more people will blur that boundary between a hardware engineer and a software engineer i think that's going to be in a very productive way more people will care about this interface are going to have to and i think that some of these real brute force techniques of the past and a lot of this historical al mldl that relies on just massive amounts of data for somewhat middling results i think we're going to begin to question that and begin to really look at this through the lens um of efficiency as we looked out 10-20 years alex uh you know please wrap kind of your position for us yeah i think that essentially this comes down to the thing that you know all of the arguments about uh the value of uh what our what's the best word our uh co-debaters have said it has has really focused on i think a kind of a desire to reverse this trend of abstraction away in a way that i don't think is productive ultimately you know the the value of abstraction and the impact of machine learning and the impact of all of this is in the software because it's in the experience that the users exp like have at the end and for me you know we can look at the power of abstraction in the way we see going back to the car analogy the ability to replace a an internal combustion engine with an electric engine and then defer the problem of power generation to the scale out there and you can only do that when you have abstractions if if i had to completely relearn how to drive to drive an electric car then we're going to run into the same problem day to day here if we have tightly coupled hardware and software at the kind of team level or at the company level where everyone has to worry about these things then we're never going to be able to could conduct the sort of mean shift the sort of substantive change in attitude across the industry that's going to be needed for us to do these things i'm very happy for people to continue developing hardware i'm very happy for people to think about these things but i want it to happen in a room somewhere else i don't want to have to know have to have my own nuclear generator under my desk because or i have to have my own solar panels everywhere because frankly those things don't scale as well as they do for me to uh to rely on a grid the ml grid is going to be there i want to tend my data and i want to put that out that way and i think ultimately all of the stuff about um defining the power of open source has been about finding the type of abstraction that it turns out people can pick up and use and not worry about the internals of and i think that to me is where the the future of this is and that's why it's about the software not the hardware and it's about the data and the people not the uh not some sort of a reversion to a mean where i'm going to have to break out my soldering iron every time i want to uh do some text classification awesome and anna and last but not least please you know bring it home for us sure yeah i think i wanted to respond to david's point about um like you know we shouldn't bet against uh software like innovation and i i guess my response is why not both and with the fate of the planet at stake i think we have to do both harder software co-design well awesome uh so folks uh you know like mentioned before that was a kind of you know a great way to i honestly and again nothing anything against the outer panels that are going on but it was just you know so pleasant to see you all trying to you know kind of bite your lips before kind of responding in a while so i think like hopefully at least in speakers they enjoyed it in the audience as well i definitely enjoyed it thank you very much and we ran the results so apparently pre the debate it was exactly 50 50 split and after the debate every single team convert every a single person from the other team to their position so it's a tie and you know job well done and i think now um admins i think do we need to go to another room for the q a session or how does it work before we leave can we just say thank you to vitaly for excellent uh moderating and uh supporting the debate 100 uh yeah so i wish we had a little bit more um kind of annoying so is there a special chat that we need to be on or oh is that the the email they sent about hold on let me see if i can drag that up so guys if you um uh i think we can take a few questions here if there are questions for us uh in the in the discord uh and um uh for more detailed q a we can move to special chat so i think you know maybe um we can take a few questions here and then then go there uh and uh i actually mentioned for you guys uh right so i kind of uh try to keep it it's very hard to kind of you know you know kind of keep this question but um so one of the things that i've seen uh right happening in this across our communities right and we're not talking about it this this is what i'm really interested about so so if you look at the rise of the clusters and ai so gpu is kind of the elephant in the room if you look at you know what's happening right so nvidia the whole rise of nvidia right is basically due to gpu and this is what started as kind of a kind of a component of a pc a gaming uh kind of apparatus suddenly kind of broke through into the mainstream there are many many factors right so for instance you know cryptocurrency in mining right and then ai and rendering so what what happened and i wonder if you know if you might agree with this or not but it really it is driving a lot of the economy right so so nvidia became basically an economic powerhouse it threatens intel right so kind of the cheap design you know apple is doing that cheap design so we have massive changes in the world economy because chips became a very important economic engine right and even now affecting the fate of the planet and actually you know for instance just recently that adrian cockroft who was a vp of open source at amazon now moved to uh actual unit of amazon which cares about energy savings so his new job is essentially sustainability meaning sustainability for aws or so there are senior folks are moving into kind of leadership roles caring about this stuff but what what really started this are these you know chips right which is becoming kind of at the heart and so as software people we we're not really seeing that we we we're on the receiving end so how i've seen this happening right and i led the team which around the the rise of deep learning so how we experience this like this gpus how do we do how do we go about them there is cuda now you need to install the kudo library right it's not actually running everywhere like how do i install code i cannot install it on a regular mac event like i need to have the gpu and video gpu and so forth so so to software people uh this thing came through kind of pigeon mail right like we got we got you know rumors about gpus right we got some strange kind of uh ripples on the surface of the ocean and so we're kind of very extremely reactive right and so i wonder uh if you guys have opinions because in my mind it's actually driving enormous amount of change but in the software community we're not talking about and harder people just pushing these things out the door they care about like let's have more kernels let's have more you know um flops so so why is that that we you know it feels like we really in software community we just have to like figure out right so on the one hand you know to say it's a power of abstractions and we don't care it's easy but in effect we do care we're extremely uh powerless actually we are receiving what so when i'm i wonder like when brian builds his data center and there is like this enormous cpa is going to change everything for instance so are we going to be kind of presented with the fact that now we have to do everything distributed systems differently and we have to learn everything and so that will actually right it's like if we think what what matters the most who is in the lead does it mean this hardware is actually in the leader mode so i wonder if you guys can share an opinion on this i have a bunch of opinions here um man where to start um you know i think it's an interesting thing that if you ask bill dally uh who is the the cto of uh nvidia right sort of what matters for ml he would actually tell you it's not cuda right it's tensorflow and pi torch on the training side right and and so you know the uh rise of the gpu is that first of all there was volume economics behind it which was video gaming right which is you know accelerating uh uh approximations of light transport uh which happened to resolve uh in vector in in vector processing essentially uh and then you know uh my friend brian capanzaro sort of managed to connect that up to python to the machine learning world and then imagenet came along right and imagenet showed that if you get enough data and you get enough compute suddenly you can get these qualitative leaps and capability where you know the the computers are beating the humans right and uh you know the gpu certainly was the vehicle for that but you know uh the days of like a lot of people you know monkeying around in cuda in order to do their ml are you know to a certain extent gone right it is about tensorflow and pi torch and like even it it you know google i mean i i don't work at google but you know uh right the whole mapreduce paradigm is there to insulate people right jeff dean can't answer every query by himself memes to the contrary um and yeah so uh but i think the beauty is actually in connecting up the hardware with the software i should be clear that i am actually a big hardware software co-design proponent myself i mean i guess i would i would disagree i think jeffy and ken personally answer all the queries but uh in terms of anyway it turns out you were bringing up imagenet so like if you look at say a model like efficientness which is one of those sort of state-of-the-art vision models yeah like it just the utilization on current hardware is just extremely poor so like i i think we really can't ignore out of curiosity does anyone here actually know the particular like primitives that are in imagenet that don't map well uh to you know like that's why it's a verbal convolution ah okay yeah yeah that makes sense like there's like five percent of the flocks and they're like sixteen seventy percent can i interject for a moment to explain the the the mismatch here yeah so my understanding and correct me if i'm wrong is right if you look fundamentally at something like a tpu right the power comes from having a very large matrix multiply array it's about 128 by 128. uh a gpu is a bit smaller but it's still i think 32 by 32 depending on the data type and so the issue is with the depth wise separable convolution you're no longer actually doing a matrix multiply you now have a very sparse matrix multiply rather than dense and so you know really what's happening is you know most of what you're doing is multiplying zero which is a very stupid thing to do because everyone knows the answer of multiplication by zero right and and so you know we have this very fundamental mismatch and i actually wholeheartedly agree that you know if you ask me the two biggest things that i think will drive ml performance in the next five years is advanced packaging and heterogeneous integration and then being able to efficiently exploit sparsity that's really correct especially the early layers have very few input output channels so they're using such a small percentage of the systolic array and that's why yeah yeah brian do you want to maybe uh answer the question that came up in a chat in a in a broader form yeah so the i mean the question the challenge i thought was a very good one was um why is there no general-purpose fpga in our laptops that we can um why have we not seen and then i think there are a bunch of economic reasons for that really um i think that part of the problem that we've had with fpgas again is i don't know it's very unbrand for me to say it but um these tool chains are terrible i mean we you know we were uh evaluating a a uh looking for a secure fpga which is basically a contradiction in terms um there's essentially one vendor who is who even attempts to make a secure fpga um and we literally couldn't get their tool chain working at all if it had been open source we would have just fixed it and the fact that it was proprietary i mean it's this flashback to how bad software was 30 years ago when you had to go deal with a vendor for everything um and that has been a real serious impediment um and i that era i do think is ending and we will see a real rise in reconfigurable computing and soft logic um and i think we will see um will we see it in a laptop hard to know because as a development vehicle a lot of economic constraints there as well um but i think we are going to see some of this promise um deliver itself um over the next five and ten years as we get these fully open tools i have a question for you actually so um what are your views on cgras right because you know the difference between a cgra and fpga is the granularity of configuration right and i feel like you know that's one of the interesting things in the ml space is if you look at many of the accelerator startups right a large number of them are building uh what are known as coarse-grained reconfigurable architectures so you know uh uh if you were to take sort of like the apex of this it's like cerebral systems which has an entire wafer with hundreds of thousands of cores with reconfigurable routing between them uh and by reconfiguring i mean sort of software reconfigurable right so it's not quite choose seven bits or choose 16 bits right you give up you know one level of complexity but you know you still have much of the power i'd be very curious about your perspective on those yeah i think it's very interesting um full disclosure cerebros and oxide share an investor so um eclipse ventures is funded both they're very much assembling so um just keep that in mind when i when i say they're a terrific company and just know where that's coming from um but i think the the cerebros approach is that that disclosure made the the distribution approach is very interesting and i think that the i mean what they've done is just uh incredible with wafer level silicon um and the um i think what's interesting there too is what we are if you look at why is amd besting intel right now in server compute it's because amd believed the chiplets when intel was a chiplet disbeliever it's kind of it's nuts to me that one could not believe in chiplets but um they very much didn't amd very much did and the the rise of chiplets to me does allow you to change that dial of of reconfiguration where you're going to have a template and now that chiplet is going to have a fixed function and now you're going to lay that down on onto a larger package and the kind of the the flexibility that that affords you now i think that the the the flip side of all that again to be back on brand is that the is you end up with these very complicated uh machines on the die the die itself now looks like a data center and it's uh really problematic when that's all proprietary right now there's an excellent timothy roscoe keynote at osdi this past year really recommend that folks watch that where he talks about something that those of us in the industry know which is that these cpus have become ridiculously complicated with many many cores that are not the cores that you see many many hidden cores and i think part of delivering on the ultimate promise of that is allowing other folks to reconfigure that differently to get transparency into the decisions that have been made there and the software that is running on those on those many courses long answer to the the question david but i think that the i think there's a lot of promise there and i think the ambiguity in what we consider to be reconfigurable computing from a gate level through a through a hard logic level a core level i think that that ambiguity is healthy and important and it's going to be really interesting yeah well um uh all again i would like to thank everyone for being here for i think what alex it was like the longest session ever in scale by the way it was in fact so long that facebook no longer exists so during during the session you know facebook has now changed its name to meta.com so anyone who's interested in checking out or has any final statements about the rebranding but other than that um again it was a joy pleasure alexi thanks you for inviting us to this conference this was fantastic i want to thank all of my uh co-panelists and our moderators and and instigators this is actually really fun it's a lot of fun absolutely excellent thank you very much my friends we should talk intel's 10 nanometer over a drink where there's no microphones thank you guys yeah and i must say that david and i we kind of you know were chatting only really we anticipated actually one of these name changes because we you know i actually mentioned the company uh behind the number temperature i didn't actually know about anything which is going on and then i realized that's what's happening so uh thank you again guys this is this is amazing so now if folks who can uh uh check out uh our special chat there is a button uh for q a so if you guys use some of you want to go there uh we can uh go to special chat or so on the you should have received the link to there uh on them uh for for viewers there is a button called q a and so that actually goes into the special shot environment and definitely go there and see you know there are any folks there who want to ask us questions but that ends the official part of the panel thank you very much guys and we'll see you in special chat you