LLM Avalanche: Panel: Performance

LLM Avalanche: Panel: Performance

Recording: LLM Avalanche: Panel: Performance

foreign but remember this is the biggest shortest most technical Alum conference in the world so far so I think we're doing a good job uh once again um I'm Alexa Carver from one of the co-organizers of this and the only local organizer around the area AI the oldest technical AI metal stars as little bimitop uh maybe already 10 years join us bay.aira.ai and also around the conference called scale by the bay it's in this 10th years the technical conference about software engineering cloud architectures and data pipelines for nml so we care about AI at scale right we don't care about the in one notebook we care about it at web scale and we're going to have it back in November so scale up by the way.ei that's our conference and also in other life I am director of Open Source science at ABM connect researchers and open source developers to accelerate science so this bundle is about performance right and uh there is a lot of meanings of performance there is performance of models themselves there is a performance of the results there is performance of llms as benchmarks there are performance of LMS in Enterprise deployments so we'll we'll kind of tackle all of this in different ways first let's uh let our panels introduce themselves uh begin the chip and I'll call the Llama for you but why can't I have it uh I'll tell you this is going to be a mechanism of moderation I'll I'll let you know in a minute but for now you can have it hey my name is Chip so um we actually have an abaca farm so yeah so we have like about 22 of those and they're very cute but it's not my full-time job unfortunately so for my full-time job I work for clipart AI which is Unified data for real-time machine learning and of course today so we have to mention AI so here for real-time Machining and AI oh no worries thanks Chapo um I'm Greg um I don't have a llama Farm but I went to a llama Farm recently it's called alpacas by the Sea it's in Pacifica it's absolutely wonderful it's like the best thing you could possibly imagine my day job I'm a CTO of lamini we're language models on your private data so we help you take an existing language model and then specialize it and then deploy it on your your private data uh David Cantor I'm the uh one of the founders and the executive director of ml Commons uh our mission is to make ml better for everyone and we do that through uh benchmarks and data sets uh Greg actually helped uh lead create one of our uh best data sets which is a three terabyte uh speech recognition data set but since this is an llm conference I'm going to tease you all by saying we have a gpt3 speed Benchmark to see who can train gpt3 the fastest with their systems and unfortunately the results are secret until 9 A.M tomorrow morning so stay tuned wow that's tough to top that um yeah so my name is Ankit I work on model serving at databricks particularly on gpus and these days a lot on large language models so yeah I'm super excited to chat with you all today by the way I was told there was going to be a llama but I didn't I thought it might be a real one it's just like a the llama's like the Talking Stick from third grade you can only talk if you have the Llama it's like a lock well there you go oh okay um hey everyone my name is beyond I'm the CTO and co-founder of a company called sourcegraph uh some of you might be familiar with the code search engine uh that we built it's a search engine for code helps developers understand large code bases but the reason I'm here is uh we recently built a coding AI assistant called Kodi that makes use of some of the underlying abilities of source graph so it uses a code search and code intelligence apis to essentially fetch contextual Snippets that then are fed into a large language model so we're doing kind of like the in context learning we're using a variety of large language models underneath the hood to build this tool and and kind of integrating it with classical developer tools and in a variety of ways I might have my llama back I will explain in a minute why so we actually try to bring llamas here right and so if you remember uh in March we had the biggest AI in the top of the world AI so as a local Bay uh by area organizer I was the co-organizer of dragon face with Clementi Lang so we brought llamas to the exploratory we tried to bring them here what happened is in the city of San Francisco you could not have live animals because a hundred years ago Ringling Brothers Circus animals escaped and they've bitten people and they stomped on them so in the city of San Francisco you could not have llamas why you can have them Exploratorium it turns out this under Port Authority so next time we'll have a llama event with live llamas we'll do it there right for now how I'm gonna use this so this is a new experimental device so if we have very tight panel so if one of the panelists is going on and going on the Llama is going to snooze like this all right all right maybe maybe not we'll see I don't want it to be like the main thing all right so performance uh less the following uh everybody please define performance of a lamps as it really matters to you what is your definition of performance chip I regress it in here um so um for me performance um I think there's like um so I think for me I think the salary three things we care about um accuracy latency and cost and that's what performance means to me like how to like optimize for the three key metrics like whichever works for you like sometimes maybe like okay it's just okay I'm very nervous it's approving yeah yeah so um so yeah optimize the three metrics like for some use cases like you might be like okay when I'm being like things uh a bit slow so you want to obtain my phone being cheap and correct but like if it's not sometimes it needs exact latency then you might need to sacrifice either correctness or cost it was a good start um let's see performance so many different aspects of it I think one thing that um I I see people running into a lot related to Performance is I just don't have a big enough GPU or I don't have enough gpus to run my language model that if you want to take you know let's say one of the um open you know llama based language models maybe it's 20 billion parameters 10 billion parameters you probably need a pretty um recent GPU like an a100 and those things are really in short supply so I see a lot of people just blocked on memory capacity of just not being able to run language models so I I think uh chip had a excellent definition I'm going to have to add on to it I would say uh time time to solution for training is a performance metric you might care about and then uh in terms of accuracy right like one of the things that's obviously uh a lot of people pay attention to these days are things like robustness and many of the AI safety related concerns uh hopefully the community will be able to boil down to some sort of performance metric um so but that was an awesome definition chip yeah I've got to say the same thing I think Chip's definition was great I'd love to add robustness to it I think for me performance is about value um you know you've got a lot of different options out there uh small models big models proprietary models uh and and really you know when you think about running one of those models you might optimize it for batch or latency and that's about what value means to you in your application if you're running an application where it's really really critical someone sees the answer immediately value means you need the absolute lowest latency but if you're you know running something where it maybe not doesn't matter as much then value means I need to have as few gpus that I'm running this on as possible so maybe you want as much concurrency as you can get so you know kind of speaking a little bit more to what chip was saying actually right like there's a bunch of different variables to optimize yeah so for us it's quality latency cost so basically what chip said um and also piggybacking what uh Ankit was saying I think the weighting of those three factors differs depending on the use case so for something like code generation uh that's that's a case where we actually find that a cheaper faster model that's stupider uh works better because you don't want to be sitting there waiting multiple seconds for your code completion but for something like q a when you're asking a question you can tolerate a little bit longer latency although even there we think that there's a trade-off so you know like gpd4 I think is is the the best model in terms of quality of the output but we find that at least for our use case the latency that you spend waiting for an answer can sometimes be uh you know it it's uh it's not the right trade-off uh for for a use case where you're just asking questions about a particular code base you know can I add actually one comment so um actually specifically for inference like to to narrow to drill in on latency a little bit like I think one issue is uh there's different levels of sensitivity like you can talk about latency in terms of averages you can talk about it as like p50 P99 right and so there's going to be some applications where it's that absolute tail latency that matter right and uh then there's going to be some applications where average latency matters and then there's some things where you're batch processing and you're like oh latency what's that right and then it's as you say how do you use this few resources as possible right and actually like speaking to that right latency in terms of P99 p50 is super important to reason about and it's hard for llms because your latency scales of how long that thing wants to keep talking um and so I think that's actually a really cool opportunity for co-design between training models and serving them because you kind of want your model to be concise and that's why people have done work on like telling gpd3 to be less wordy and things like that or more worthy if they need it but that is an important factor in how much you're spending I think I decided to jump on that do you think it's just like cultural difference in it how people prefer it longer or shorter or like depending on the use case whether they prefer longer or shorter responses yeah I think it's on the use case um so like for summarization you probably don't want it to Ramble On too much because then it's not summarizing um but if you're brainstorming then you certainly want to get as many thoughts out there as possible so I had a very interesting experience in Washington DC recently and I was meeting with a bunch of lawyers and uh I was talking to one of the lawyers and and I said you know one of the things I've noticed is that in Law Journal articles uh people perceive quality is linearly correlated with the length of the article is that true and and the woman said oh yeah that's absolutely true Long papers are good and I'm like have you ever met a technologist because like we prefer bullet points you know not long papers like you know some of the most impactful papers like you think about like Shannon's information theorem like those might be single digit Pages um and so I think it's not just like the context but also the the people you're interacting with right if you're trying to write your college essay you probably have a particular page Target in mind uh but yeah I think in general like shorter outputs or probably better but then maybe you want a really big context window up front to scan over you do you wish you had a stop generating button for lawyers at times just kidding I have an embarrassing story I've only actually used a Jeep an llm like maybe a handful of times it's what we're talking about generation um just a quick quick performance tip the cost of generating tokens is actually higher than the cost of the context like the input to the model so every token that you're generating because the model has to read in all of the weights to generate every subsequent token it's actually more expensive to do that generation than it is to to process a long document thank you I think I think we touched on a lot of points and also I really like you know you guys you should have a conversation right like it shouldn't be just like me asking questions you know so and we will reserve the type of audience questions in there so please think of your questions right we'll have probably about 10 minutes in the end so uh I want to piggyback on what Ankit said right so inference versus training right and actually let's kind of explore so I talked to the mattress right he runs one of one of the biggest communities mlops community and uh he did an online conference on elmaps and I asked him like what questions did come up right what trade-offs people people see so here's some right so in performances fine tuning versus International retrieval versus large context window right obviously that's I think we have kind of advances in all of these three areas right then there is a question you know small models versus large models do you have to have a large model obviously a large model maybe criminal parameters but obviously it's harder to run harder to use for inference harder to train in it's like for Value right like you said for your metric of performance do you really need this maybe the overall value given the cost and the size is lower so I want you guys to talk about like your opinions on the architectural features of models for performance in any order well I think Greg you know made a great point right which was that the input context is is less expensive than the output generation right so to that point if you're practically trying to build an llm application right now your best lever is to toss as much relevant input context as you can as the model will let you and see what it hopefully that it comes up with a good answer um and you know smaller models will do worse with that bigger models will probably do better with that at least in what I've seen I don't know what others have so I think there is kind of this like step function thing that happens that we found like so generally we like to optimize for lower latency because uh for end user use cases like code q a and code generation the latency matters a lot but when we're answering kind of like a high level question that the user is asking about the code we do need a model that at the very least has this kind of uh Chain of Thought ability the ability to kind of like synthesize multiple pieces of context into a coherent answer there's some models that are like too small to do that and in that case it like if you try to do the context stuffing approach that you just described uh it it cannot be not good for the end user despite the model being cheap so the latency Point's really interesting um it's it's actually really interesting like why we're all here we're all here because we have these giant models like we're here because we have gpt3 and gbt4 and the performance of these things is really amazing um I think actually everybody's been using language models that are low latency for a long time if you've ever used like spell check and like Microsoft Office from like 1990 it was a language model and it was super fast it was just not interesting like not interesting at all in the same way that like gpt3 and gpt4 are so I think one of the most interesting things actually about performance with regard to language models is that we're on this trajectory that takes us from um the Microsoft Office spell check two gpt3 gpt4 and that it keeps going we actually understand some of the theoretical like reasons why these models are learning and they just keep going so I think one of the most exciting things for me is that these are the worst models like as good as they are these are the worst models we're going to see in the next decade I mean I think one of the other things that's interesting and just for context Greg and I helped Define ml perf inference and and one of the things uh there is you have a bunch of people competing on how fast can you make inference and there's like so many techniques and I get the impression that when we look at the high end of these models like you know we don't know what most proprietary models are being deployed on right whether it's barred gpt4 or whatnot but it's super common that you can trade off a very small amount of accuracy for an order of magnitude or more reduction in compute and potentially in memory footprint right like it's very normal to be using a 16-bit floating point or 8-bit floating Point although we don't have a standard for that yet and there are people who are looking at you know 4-bit either float or int right and so each of those you know gives you sometimes a factor of two sometimes a little bit more and but there's like hard systems engineering that goes into that or you know dynamic batching and other things and so like one of the things that I'm super curious about is actually how many of those techniques from sort of system optimization have been thrown at inference for these large language models I suspect it's not a lot because I don't think there's really you know sort of a push button flow yet for any of this yeah so you know that's you know to plug myself a little bit that's what we're trying to build at data bricks um I think that we're still super early both from the research point of view and the systems point of view um as a former systems person myself I think there are cool adaptive batching type of techniques that people have tried but I don't think people have gone very far um or deep in terms of how they're doing those um if you actually look at how these models are executing on multiple gpus those are traditional systems approaches to doing matrix multiplication so if you you know studied matrix multiplication in 2010 a lot of those same techniques are are being applied just in these large language model libraries um but you know speaking of the points around four bit eight bit I think we're just it's it's such an exciting Direction because it almost speaks to and you know it's not scientific to say this but it almost speaks to like maybe there's some level of understanding that's being achieved in these weights that that isn't you know super super granular due to like particular weights that they that exist and um I think it's really really early days on how people are going to build these models like I think GPD 3.5 turbo for example is a distilled I mean the rumor is that it's a distilled version of a much larger model and that could be a really exciting direction that the open source Community goes do you want to Define distillation for folks who may not know sure yeah like uh maybe a quick explainer is just that distillation is the process of taking a much larger model and almost having it teach or or train a much smaller model so that you can kind of take some of the insights that that larger model has and hopefully try to bake that into a smaller model and that's that got 10 000 foot view on distillation there's a lot of really really deep work here but um that's kind of what it is so I think like can I ask you guys like I want to hear questions about so like just do like a lot of techniques a lot of things to like discover and optimize for inference right but who would be like the good people should do so Who can capture the opportunity because I do they have to work for big research Labs people actually said to this big model or anyone can spin up like a random model at home and like experiment with those I think the rise of Open Source has been uh really interesting recently I think um as you know others have alluded to here the um there has been this space of Technologies like low bid low Precision training um low Precision inference um and there's been this rapid innervation so there's this pool of different Technologies one of the really interesting recent Technologies um was released by deepmind it was called chinchilla so this was actually an adjustment of the size of models that used to be way too big to fit on anybody's machine like any real like hobby is like your laptop chinchilla adjusted the appropriate size of models down by about 20 times so it finally got down to the point where you actually could run it on a workstation you could run on a cloud computer and this is actually enabled open source developers to release you know many iterations on top of that to start including some of these optimizations like low-level C plus optimizations like a mixed 4-bit and then higher Precision 32-bit inference engines so I actually do think that there's a thriving hobbyist Community or open source Community that's been relatively recent like even within the last six months yeah I I plus one to that I think there's like you know still a gap between where proprietary model inference Stacks are and where the open source Stacks are frankly um if you know if you look at these larger gbd4 style models people are using very Advanced Techniques like mixture of experts that frankly no one has done a good job of in the open source Community yet but if you look at things like ggml and faster Transformers there's really really vibrant communities and I think it really originates from hobbyists and and they're moving really fast so I think it's quickly kind of coming together as as open source tends to do it also depends on the use case right so like if you're looking for a language model from which you want to extract embeddings for something like information retrieval where you do the thing where you command compare the the distance between you know two text documents in in the embedding space of the model those models tend to be super small and you can optimize the heck out of them and actually like it's what we found is it's you can get a much better model for information retrieval by taking an open source model and fine-tuning it than you can get from any other proprietary apis for code generation that's uh that's an application where we think like the the open source models are basically on par with the proprietary models that are available right now because again these are typically smaller faster but Dumber models um I think the the kind of like remaining Frontier at this point is kind of the more like chat based you know logical reasoning models where I don't think that there's any open source model in existence unless you know you all know of one that compares comes close to gpd4 in terms of reasoning ability I think gpt4 still wins but the open models are getting close like the Mana core Falcon you know family of models or like at about Google bar level of performance which is amazingly good if you imagine like you know compare that to what we had about a year ago so is a hardcore systems guy though like I right I mean sort of the the one of the points is you know given Moore's Law given advances in algorithms all of these things right if you want to skate to where the puck is a trillion parameters seems really hard today um I sincerely hope that within my lifetime we'll be doing that on our phones with milliseconds of latency right and so that's kind of like we should be thinking about that I think Greg's point about you know Microsoft Word has an embedded language model that runs every single time you type a character right like imagine a world where that model is comparable to gpt4 do you think there will be a Moore's law for infant speed um okay so I have like some kind of uh non-conformist views uh Moore's Law is a little over 50 years old uh is there anyone in the room who's over 50 years old please raise your hand this is not a discrimination thing but you know so you guys are all still alive you're looking pretty healthy but you might not be quite as fast as you were when you were 20. and that that's kind of how Moore's Law is doing uh I have a lot of friends who work on device physics and there's a tremendous amount of investment to make sure that you know the Next Generation process technologies will be faster um and the other thing is you know a lot of companies uh have just phenomenally talented Architects and you know one of the things to me that's like really beautiful about ml is it is a full stack optimization problem right you can optimize the data right so like there was a great uh demonstration by graphcore that if you sort your tokens into similar lengths you can get about a factor of 2x speed up in training and now everyone does that in mlperf right that's just purely a data manipulation trick right you can use faster training algorithms you can use better fft libraries better compilers better chips larger systems faster silicon so we've got a huge number of tools and as long as there's like this promise out here you know sort of the job of the industry you know whether you're purely at a software company like databricks or an an ml Focus company like how do I pronounce your company name right is yeah go with the llamas right but is or you're a silicon company uh is like how many of those tricks can we throw to add a lot of value thank you so I think uh now I'll ask one more question right and and then we'll ask audience questions so I think Beyond actually asked this in our prep right let's kind of uh ask how this performance related architecture so I think you mentioned search right so the results right what we care about mostly currently done with prompt engineering but there is actually interesting emerging ways to interact with LMS which are not prompts such as if the DSP firmware from Stanford demonstrate search predict where you give examples of what you want then you define the search space right and then the thing goes and finds and then it predicts what you want and I think with the open air functions we see similar way right so you give data sources so I wonder and that determines both the performance as a result but then also architectural so I wonder what do you guys think you know our current ways to interact with our lamps affects the performance as we Define it where can it change if we Define if we change the ways we interact with the lamps obviously like in you know in the future you might be with your phone you're not going to spell everything maybe we're going to show it right I wonder if you guys have thoughts and obviously Beyond you know you have experienced this maybe you can start with what you mean by this with Cody yeah so you know just for the context uh the way Cody works is we essentially uh when the user asks a question like hey how does this project work or maybe like write some code that shows me how to use this API what we do is we issue a bunch of code searches and uh kind of like code intelligence requests like final references and go to definition to pull relevant Snippets sort of mimicking the way that like a human might go about understanding a code base and then we place all those Snippets into the context window of the model and then we ask the user questions so this is just an example of what some people call in context learning it's also called retrieval augmented uh uh retrieval augmentation and so essentially what we're doing is composing the language model with other systems and so now the overall performance of the system is not just the language model latency but sort of like the the added latency of this back and forth that happens between the language model and these other systems and if you take that to the next level I think what a lot of people are starting to do is play around with like agents where like you you basically have a language model and you say here are a set of tools or apis that you can access here are a bunch of examples how to use that open AI functions you know here's a templated set of functions that you can invoke to fetch additional information that you might need to answer the user's question so it makes it a very multivariate uh question this question of performance yeah I mean I think retrieval augmentation is super popular these days because current large language models are pretty bad at telling you the truth if they don't know the answer um and so um that I think is in some ways papered over by kind of asking it to think a couple of times about the different hops right as the DSP paper does it basically gives this example of someone uh you know for the audience generally where you might ask the language model a multi-hop question and the language model would then like maybe asked a couple of times like hey what is uh you know if I'm asking how many stories a castle has uh which Castle is it right and then maybe it uses that to generate a better response um I think accuracy is going to be super important for a lot of use cases and so the trade-off is probably worth it for those but for others maybe it's less worth it like summarization for example and right now at least like the agent stuff too like the accuracy kind of degrades over time because like every step along the way is a chance for it to mess up so you look like compounding the rate of an error kind of goes to one the longer your chain of reasoning uh is and it's also super slow so like I think uh I don't know like one question for kind of the folks working at the lower lower levels like do you think we'll see performance improvements in the next like year or so that will will actually make these multi-step workflows more tenable from an end user point of view and maybe like when we see like models are specifically geared towards this like agential kind of like tool tool use I think we will see specialized models um I think one of the best examples of specialized models is the move from gpt3 to chat GPT where gpt3 is trained uh just on the internet data like imagine just a stream of Reddit comments or so on um you know you may get what you want you may not get what you want but if you just take a small amount of data like maybe 50 000 examples of questions and answers and then fine tune the model on that it can actually generate like a much more coherent like just better suited for the task of question answering responses um so I think we'll see the same thing where there's the potential to see the same thing for tools of actually taking models and then specializing them so that they don't just you know do what they would do with a like a random redditor would do um with one of these tools but they're actually proficient using the tool so one of the observations I'd make is that you know we've been developing software for a really long time and we have a lot of tools in principles that we sort of operate by right composability is one um introspectability and so one of the things that's that's very interesting is in the ml World a lot of these things have not yet titrated in right there's no GitHub for data right there should be right you know can you imagine so some of us did develop software free GitHub and it was painful right but like you know uh so all of those tools that we have in software there are analogs in the ml world and so like the question in some sense is okay now that we have the ability to compose what does that mean and I think you know Greg you hit the nail on the head which is we're going to see specialization you know the ideal system like for Source graph I don't know but I would speculate you probably want an array of models with different latency right and different trade-offs and there's going to be a curve of you know sort of accuracy versus performance and you're going to want to select the right one or the right Ensemble of ones we have three models actually right now right yeah and so the question is you know is there a future in which maybe you want seven and then then you know the bigger question for the whole industry is how do you shift all of those curves in the right direction and uh you know I think as we get more advanced tools we'll sort of see a lot more specialization whether it's on the latency Axis or you know what if you had a model that was specialized C plus plus only in one python only thank you David so we have five minutes for questions maybe one of the guys can give up the mic for the audience and share with others thank you David Jasmine will bring you the mic for audience questions um How do you how do you see the um I guess are there any models that are are worth um kind of fapping onto a chip uh getting a Foundry to to create a processor for and uh how do you see uh risk five is fitting into that and I guess is that do you think that's going to slow down model development but I'm the Silicon guy okay um so uh I'm gonna just paraphrase the question uh is it possible to take a specific model and like make it ship that is hyper optimized for that model that model alone so it is possible to do that you would probably get enormous performance gains and then you would have to retrain the model and you'd be in trouble and you would have a very expensive piece of silicon that's no longer worthwhile um there is a constant trade-off between sort of the general purpose nature of silicon and specialization if you look at every chip that is out there they are all general purpose to the extent that certainly they can run a family of models there are some that are optimized for instance for inference using attention it's one family of chips like that um but that's really um it's a complex economic trade-off and the other thing to keep in mind is you know building a Cutting Edge chip is pretty expensive and so oftentimes you want to amortize that over a large Market which means multiple flavors of models so really quick one um I thought that's what Nvidia did I thought they built a GPU that was just really great at running Transformers and that's what's powering all of our language models I think like even before I go it goes an intermediate layer took a while right they might run like resnet or like GPU I think if you want to optimize for how to run GPT models on GPU right and it's part of that same trade-off right like you can go all the way deep and right code that runs exactly one model and will be super optimized or you can go the compiler route and it will be less good probably I think it's probably too early to like make a chip or something because even if you look at the software stack you know a new model like Falcon comes out has a slightly different attention mechanism and all of a sudden you're out of luck and that's a tough situation to be in I think like also like on top of that I think like um Transformer is very very sticky architecture but we do not know how long it's gonna last another point I think like a lot of now today people are tuning uh so I I have a little beef with like the way we'll say like 100 billion parameter because like there's a whole world like dance versus a sports model and when I think a lot of people are in a dance model but I do think there's a lot of Promise in um sparse model and when we reach that point like number of parameters doesn't mean much anymore so actually I mean like a really interesting open question that I'm not sure I have an answer to but is you know when we started working on ML perf you know call it five years ago Transformers and attention were sort of coming into their own and we're replacing rnns um will we be using attention as the basis of most of our ml five years from now I have no idea very hard to predict and like related to sparsity I am a believer in sparse in the long term um one of the things I'm I feel like we haven't cracked it though I feel like we haven't figured it out yet I I know right now there are more than 5 000 research papers that are written about different sparsity methods but our state-of-the-art models that we use do not actually use sparsity foreign hello so I was wondering you have been talking a lot about inference and the performance in inference but how important is the performance in model training or fine tuning in something you are concerned you spend a lot of time money on it or what do you see in the industry so uh there was a great paper from David Patterson at Google and so for those who don't know he's sort of one of the fathers of modern computer architecture and he talked about the balance of training and inference at Google and training is about maybe a third of their workload and inferences about two-thirds more generally if you just step back you know training is similar to compiling code it matters a lot to developers but realistically for most people no one cares about the speed of compiling code what matters is the speed of running that code in the long run inference will dominate training what I will say though is that if you are optimizing training and you're training a massive model if you're doing a huge pre-training job you may literally not be able to get the gpus you need to do your pre-training job if you're not doing good job optimizing your training stack so I think it's actually quite critical to have a really good optimized training stack Jensen always says the more you buy the more you save laughs thank you I think it will come to a close I just want you know uh each of you to kind of leave us with some parts that was in in your own work where do you focus on performance right where do you immediately kind of think of performance as you as you implement the alarms latency for sure all we are all in with like how to make things run faster I mean I think it's quality it's got to be um the end application has to actually work and you have to get it to that point before you can focus on the other pieces I'm greedy I want high quality low latency and then I want to run it on an embedded device that sucks down like five milliwatts yeah it's not just you actually I also want a magic lamp in like five wishes uh yeah um well I would I would say in an Ideal World all three of those but um I think in the long run uh concurrency will be the hardest thing for people to get right um and figuring out a way to keep concurrency high but latency low is tough cool so I'm I'm coming at it from the application Level so the ultimate performance metric for us is developer time saved uh so it's like how much time can we save our users uh the developers who are creating the software that powers the world so to the extent that we can make all this stuff like way faster performances or I should say speed latency is is a really uh key feature thank you everybody let's thank all the panelists [Applause]