Bay.Area.AI: What's New for Python Developer Infrastructure, Omkar Salpekar & Elias Uriegas
hi everybody I'm Eli urius I'm an engineering manager at meta uh currently I support the pytorch developer infrastructure team and this talk is kind of centered around what's new for pytorch developer infrastructure it's a generic talk we didn't really know what to talk about when we first submitted it so we kind of submitted it like this and so I hope you guys enjoy all the uh different topics we want to hit on so first and foremost I want to introduce the pytorch developer infrastructure team there's a couple of members of the team here uh today uh this is at an offsite that we had recently where we got to actually meet in person for the very first time after working together I think we formed the team around four years ago uh so our first time ever meeting in person alt together so really really awesome so what is the mission for developer infrastructure so our mission has kind of changed over the past couple years so I just wanted to kind of highlight the kinds of things that we've tried to do uh for 2022 2023 and looking ahead so first and foremost 2022 was a year about achieving stability uh we wanted to get to Green CI with kind of state-of-the-art flaky test detection and disablement uh we find that when Developers trust CI uh it makes it a lot easier to kind of push changes quickly uh on top of that for 2023 our Focus was really on going faster so how can we provide a quicker time to Signal uh through through things like Target determination um and then moving forward we're looking into how we can integrate generative AI things like large language models into our tooling for better debugging and treeing uh before I go into tooling that we've built over the past couple of years I kind of wanted to go over some high level goals and how we build our Tooling in the first place uh so first and foremost we want to have a high confidence uh and Trust in our test testing infrastructure what we find is when people trust the signal that's getting to them as developers they'll likely be able to push changes without having to worry about actually causing regressions as well we wanted to have easy to use tools that provide visibility and stability so when you're able to actually identify where a failure comes from you'll be a able to uh more likely debug and be able to fix it and at the end of the day we also wanted to maintain High Velocity with reusable components so with our tooling what we all what we always find is that we end up having components that we can reuse in a lot of other places and so we end up doing that uh first and foremost so building things out with that kind of in mind and at the end of the day this all kind of coalesces into increasing developer productivity so ensuring that we can ship the best things at the right pace with the certainty that they won't break anything uh ideally uh so to go over some of the tools that we've built over the past couple years uh hud. py. org is kind of our Flagship Tool uh this is the one where all developers of pytorch can actually go see uh signal uh as the as it occurs on the repository this particular view is a timeline view of all of the commits on pytorch pytorch Main uh and it kind of highlights uh failures as they come along the on call for pytorch pytorch actually ends up utilizing this tool to be able to tell when failures start to occur and if a revert actually needs to go on we also built out a view for people uh when they submit a pool request to actually be able to find signal quicker uh what we found is that through different tools like Circle Ci or GI of actions they didn't always provide the uh right amount of context for uh failures and we wanted to be able to provide a kind of unique experience for pych developers so that they can get the quickest signal the fastest way as well on top of that we collect I want to say probably the the most amount of metrics of any open source project uh right now when it comes to CI signal uh basically every interaction that you have with GitHub on the pytorch organization we collect the data for that and so we just have a large data store where we can run all kinds of queries on top of it and be able to kind of extract a lot of information and we basically make almost every decision off of that uh data and we also have a dashboard by the way everything here that you see is open source you can view it on github.com pytorch test uh so it's all there um another thing that we built out that we thought uh actually really ushered in a new age of pytorch developer productivity is at pytorch bot merge uh essentially what it is is we wanted people to be able to merge on GitHub for historical context the way that pytorch was traditionally developed was by merging code in through the uh Facebook monor repo and that code would get reflected back out on GitHub uh last year we went on a big project to kind of change our development model to be able to merge on GitHub first and then be able to ship those features into the Facebook mon repo afterwards and so essentially we called the project GitHub first because you merge on GitHub first uh and a lot of things came with this as well so when you actually submit at pytorch bot merge here what goes on in the background is it spawns a workflow that runs a script that'll actually do things like merge on green and it'll actually do some uh higher level things like do flaky test detection and just ignoring a flicky test and workflows as they occur which leads me to my next thing which is flicky test detection and disablement uh this is one of the things that we've built over the past couple years we understand that flaky tests are kind of the bane of of almost every organization's existence uh what what we wanted to do is we wanted to have a system where if there was a flaky test that was detected on Main of the Pyro repository we didn't want that failure to kind of propagate to every single pool request as it came through so what we did is we built a system where where uh we actually have a bot that will detect FY tests on Main or on PO requests submit an issue to actually disable them and then it will be globally disabled across every single po request across every single commit that's running at that time and will not affect the people's ability to be able to land and merge and you know get good green signal if they have it three more things that I really want to talk about for developer infrastructure that are like super important to us ossi infra is the Beating Heart of almost all the things that we do every single job runs on a cluster of about 3,500 nodes at Peak uh times uh that contains a large list of Hardware a large list of operating systems uh we basically support all major operating systems we support all of the Nvidia gpus we support AMD gpus we're looking to support other gpus as well as they come on board so osci INF for is a way that we can kind of meet the needs of the growing AI Community uh and be able to test on all the platforms that are important to everybody here as well we built a bunch of reusable workflow components over the past year uh we have an understanding that not that machine learning researchers aren't always the best at writing CI and you know CI guys aren't the best at writing machine learning research and so what we wanted to do is we wanted to be able to give people reusable components in the form of GitHub actions workflows where they can write something as simple as a bash script that sets up a GPU that allows you to kind of get access to the hardware that you need and you don't have to think too much about how it gets set up as well we provide a ton of nightly binaries and we do a a ton of validation on those binaries in order to be able to actually ensure that when you receive a binary from pytorch it's actually what you would expect it to be so to talk about that I'm going to invite Omar who's also on the developer infrastructure team to speak about that cool uh thanks Eli um so yeah everybody I'm Omar uh I'm a software engineer at meta also on the py dev infra team uh before Dev infra I worked on um pyro distributed with some friends here um but yeah I think uh you know Eli kind of described a bunch of tools uh that we build to make the developer experience on pytorch better another big part of Dev infra is getting a healthy correct and performant version of P torch in the hands of our users um and this is done by our release infra and we actually release at pretty large scale so on any given day we probably run around like over 500 build test and upload workflows um and that number is is large because we support a pretty sweeping build Matrix we have to uh build both wheels and condo binaries across different python versions uh Cuda versions Rock them for AMD GPU support uh different operating systems different CPU architectures Etc and we don't just do this for the main pytorch core repo but on for on the order of 10 other projects in the pytorch ecosystem so extrapolated to over the entire year we're publishing around 200,000 uh pytorch binaries so in the past we had a lot of duplicated logic across all of these different projects we didn't have any standardization in terms of the platforms that they need to report uh and this made actual official releases really difficult so we decided to change that and we created these modular reusable workflows built on top of G of actions um and really the purpose here is is that we wanted any project in pytorch whether existing or new to kind of get up and running with their cicd in about 20 lines of of config code so all they have to do is kind of interact with this interface um it'll set up a clean build environment for you um into whatever platforms you opt into build the binary um it supported a number of hooks for custom pre or post build steps that you might want to run specific to your particular project uh you can run any arbitrary smoke testing to uh verif like validate the health of your binary um and with just this any new library uh can start publishing binaries um they can be compliant with the entire pyro ecosystem and they basically have a hands-free process for running not just their nightly build process but also their official releases um so this is all the interface that a project maintainer has to interface with but what really happens under the hood so we have a nightly cron trigger uh and what that does is it takes all the commits made to the project over the course course of the past day squashes them into one commit and pushes that to the nightly Branch uh of the project uh this triggers a number of workflows and the each project has the flexib possibility to opt into which workflows they want to support so at this level um we support uh wheels and condo binaries across Linux Mac uh Mac M1 and windows jobs and we also have some early support for Linux arm 64 Wheels um and IOS and Android binaries so once these jobs are triggered the first thing they they do is they kick off this Matrix generation job um and this Matrix generation job ese uh essentially specifies the sub jobs uh according to the python versions and Cuda and roam versions that each of these platforms need to support so these sub jobs are all kicked off and then they're dispatched to our self-hosted AWS cluster uh using GitHub actions and so each of these sub jobs also contain their own kind of unique Hardware requirements for example the Cuda build jobs will require some GPU instance uh Windows build jobs will require a Windows instance and so on so our self-hosted cluster supports all of the these different SKS and this is the same logical cluster that we use to run all of our CI jobs on as well as all of our benchmarking jobs across the entire ecosystem um each of the machines in here comes uh you know preconfigured with our custom Amis were appropriate uh for Linux jobs we build custom Docker images and all the build testing and uploading happens inside uh of those containers so uh at this point you know you have your build environment set up you run your build uh you uh do any testing that you want to do um and then for uh cond binaries we upload them to our uh pytorch nightly Anaconda Channel um and for Wheels we upload this to our self-hosted piie index uh which is backed by S3 on the same AWS cluster and this is the back end for the download. p.org uh site that you can access and so that is all just what happens for one uh project in the ecosystem we actually have to do this for each project and it also needs to be dependency aware because you know for example torch Vision nightly for a particular day will depend on pie torch nightly for that same day so we can only start building torch Vision once P torch nightly is built tested and uploaded so we do this dependency aware staggering across the entire ecosystem everything gets uploaded um and after that we fetch them all in for some ecosystem wide validation workflows to ensure uh that all these binaries work well together and we run some metadata checks as well um so yeah that's the story of how pior binaries are essentially uh built and distributed to all our users uh usually with a pretty high success rate as as Eli mentioned we track all of this data so if anything goes wrong we're we're constantly figuring out where those issues were um and if anything does seep through all these lines of Defense uh there'll probably be a get up issue within 5 minutes from somebody in this room maybe I don't know um so yeah that's that's the story of relise um so this is all great right we have tooling to make development on a rather large and complex project easier we have well- defined systems for cicd uh we have significant automation to get pytorch to our users um so what's what's next for us there's been a lot of activity uh and research around large language models for software engineering applications um at the Pyro conference we introduced a lot of new tools and features for uh where pytorch is enabling innovation in large language models so we're thinking of how how we can close the loop and use llms to improve pytorch in particular uh the development of pytorch so there's plenty of interesting data that we have organically around uh not just the code itself but uh commits logs metadata Etc um there's a large body of research uh including models like code Lama built by uh some friends at Fair in meta um where you know they're performant on things like code completion and multi-line infilling um a lot of these models are small enough to be able to be inferenced on a single GPU um and can easily be fine-tuned on specific Downstream tasks so we can use this data and these fine-tuned llms to at the end of the day provide an improved developer experience for for pytorch developers um so let me share some numbers to motivate a specific problem that we have on pytorch CI to to any push to a pytorch PR we run around 2.3 million tests and you know that that's a massive number and it takes 4 hours for any developer to get full signal on any change they've made to pytorch which is uh pretty bad um and and you know this really affects the development experience the the speed at which you can iterate on your changes to pytorch and if we extrapolate that over the course of the entire year we're running around 1 trillion tests um on pytorch C in the entire year right and that comes out of somebody's budget so intuitively most changes don't affect all the unit tests being run right uh and all these unit tests are repeated across this entire build Matrix that we support so it's clear that we're overrunning tests um we thought how can we take some information from the code being changed in the pr to run only the most relevant tests and this process is known as as Target determination in many build systems um it turns out that the pytorch architecture is good for developer usage good for building new features and things like that but it's pretty bad for traditional Target determination there's lots of complex interdependencies between different modules uh lots of python code Pi binded to C++ generated code uh you know Cuda kernels Etc um and we've tried doing Target determination with pitor in the past we've uh tried hard-coded rules where if you know changes are only made to one module don't uh run tests in a different module we've tried um explicit dependency graphs in the style of like the buck or basil build systems uh We've also tried uh some simple her istics like past failure rates and for one reason or another we've we've all had to basically walk back these attempts so we we had this idea uh let's pose this as an information retrieval problem right we can take the traditional two Tower model um approach as as inspiration um which you know is a classic Model that's used for many search applications but instead of taking a user supplied query and searching for the most relevant documents we can take a code change and search for the most relevant unit tests and so we employed uh the code Lama 7B python model um we parsed the as of the the pytorch codebase identify all the unit test functions and all their uh the functions that they transitively call um and we use code Lama to generate embeddings for each of these test functions uh and with this you know we create an index of the unit tests uh so when a new PR comes in we can run similar as parsing for the pr find each function that's changed by the pr use the same model to generate embeddings and then compare the pr embeddings with the test embeddings using something like cosine similarity so at the end of this process which really mirrors the two Tower retrieval approach we have essentially a rank list of unit tests that are uh you know from most relevant to that PR to least relevant and over time we can get to a system where essentially we filter out the the least relevant tests and as we build kind of more confidence in this model only run the most relevant tests and the the idea here is that the code embeddings will be most similar to test embeddings for Relevant tests and they will score higher so we we prototyped this and the early results are really promising you know we've uh tested this on a number of sample PRS uh it's very good at detecting similar tests the tests it flags as least relevant across a number of sample PRS uh are in fact completely unrelated um and indexing and retrieving uh after extensive optimization can be done in reasonable time bounds so here's one You Know sample PR that we tried this on uh it made a change to to P like torsch distributed um fstp fully sharded data parallel um it in fact it changed the init function uh of that module and so we find that the most relevant tests that that the system identified were all fsdp tests uh and the tests that it found as least relevant were in Onyx jit Funk torch name tensor and you know other modules that had you know no interdependency or no connection to to torch distributed at all so we're continuing to iterate on this uh to prune out irrelevant functions um adding context about uh like transitive dependencies in the sequence that we encode and potentially doing like dimensionality reduction for um the the embeddings themselves uh and we're working on rolling this out uh to run on PR so that we gather data about like which uh kind of like where the gaps are in terms of this model especially because uh you know there's no concrete Benchmark uh to to verify the performance so um I guess to tie this back into our overall like llms for developer tools Vision uh I think there's really two high Lev things that we want to get out of this right as we continue to iterate on these models we're going to find Opportunities to both improve pie torch as well as the Upstream like inference code that we use um and the second thing is to do the actual kind of thing that the model does which is to provide an a better end endtoend developer experience for py devs so uh for the first thing right we needed to do a significant amount of end to-end optimization of the system to to get it to perform well so one of the things you know we started off using hugging face uh to uh inference the code llama model uh because we're using this as a embedding model we don't want the last layer activations we actually want the second last layer activations which gives us the the actual embeddings um hugging face uh does not expose an API that lets us do this in a granular way it actually gives us all the intermediate activations of the model which can cause ooms really easily so we shifted to using the actual code llama inference code which didn't support this so we added support for essentially fine grained uh like being able to retrieve activations from any arbitrary intermediate layer um this still ran into out of memory issues uh on our a100 machine um and that's because uh the code llama inference code made the implicit assumption that we were using the model for next token generation so it had this KV cache right which was was mentioned the previous previous talk and so when you're using this as an embedding model you're not actually calling the forward pass in the loop so there's actually no purpose for having a KV cache so we're working on upstreaming a change that uh enables a no KV cache uh option for uh using Code Llama Or llama 2 as an emitting model um and this is in addition to many other changes that we made for as parsing and other parts of the system um we also see llms as only a part of the solution but not not necessarily the full solution there's a body of of interesting research work around uh looking at correlation between Pairs of tests whether they pass together or fail together with very high frequency um and and other such heuristics that could be combined with the scores from our retrieval system um and and we're exploring how we can best combine them to to filter out the least relevant tests um and we have an additional set of of use cases in the pipeline for example identifying the exact error line in large log where you know if you have a large number of tests or a very complex build like you you do in the case of pytorch you might have like a multi megabyte uh log file um or for doing flakiness detection or air suppression right these things kind of prevent you from Landing code into pytorch should prevent you from doing releases um so if we can uh kind of use the similar techniques to to autodetect when when tests are flaky or when jobs are flaky um you know that would also be a very um tangible quality of life Improvement and another we could possibly do in the pipeline is to do um generation of of unit tests so we're looking at all these Downstream applications uh again the the purpose is really twofold to find Opportunities to to improve pie torch and make it kind of a better tool for uh doing both training and inference for for llms as well as providing a superior developer experience for for pyro developers and yeah with that happy to take any questions yeah so having had this experience um it's like mind-blowing that you could use embeddings to to deal with the test I hate tests so that makes me really happy but like seeing what you've see now what do you think about starting a software project from scratch and ideally you want to avoid your test being a mess but in this case you say don't worry about it the model will take care of it right it'll figure everything out for us is that like a terrible idea or could that be a plausible Way Forward yeah I feel like uh there's no good way of answering but uh yeah I guess I think the difficult thing is as as the project grows to an incredible scale and I think we all have like firsthand experience with this like as the number of commit scale to you know where you have like a thousand commits a month uh or you know have you an incredible number of tests and folks kind of depend on uh every commit by commit you know being correct and being able to be built from Source you almost get to a point where you want to over test so that you don't break people and maybe you can clean up the mess around like you know excessive CI time later um if we were building this up from scratch like it is possible that maybe going the explicit dependency graph approach may have been better but I think it's possible that that style of development might actually have made it harder to develop on pie torch and and so maybe this like post-factum way of cleaning things up is not bad yeah yeah when will you be doing 3.2 you're doing that yet um is this for for like wheel support or for like uh torch compile support you got like three versions of python there's also specialized versions there's like one that begins with an M I can't think of but it has specialize fixed uh C code inside of it so that it runs Inc incredibly fast yes Rems it C speeds basically I see um 312 do you know uh we'll probably look to support 312 within the first release of the new year for 2024 uh that's probably the earliest that we we'll be able to get it in as far as alternative python interpreters go we probably won't support those unfortunately it's just a matter of like where we choose to spend our attention yeah thanks and uh so and also to note there uh three2 support for torch compile will always lag through 12 compliant pytorch binaries uh cuz that's a significantly deeper integration that we need to do yeah yes thank you for a great talk I think it's a really Noel way of using L to do this kind of test uh prediction uh but one of my concern is that um like the result of L output is kind sometimes kind of indeterministic so um for example in traditional CI test group when I push a PR and get a test result I have a sense of security that is passing all the tests that it needed and it's going to work in production but if you do the prediction there always this kind of possibility where like some tests um un captured by this similar research for example like it happens very frequently for example in our company data code base where uh we modify something and something very far away like doesn't seem to be related at all it's breaking um but for example based on the similarity seach test might not be uh detected and the people may come to us after PR so um do you guys have you guys uh consider that um part in your similarity search approach um and uh is there any like thoughts to address this give for examp develop a security where after push the code we know it's going to work after the test yeah um so I guess there's there's three things uh that I can say to that right the the first thing is that um what the system gives us is a ranked list of tests right so we can determine some kind of threshold that we feel reasonable um right we can be as conservative as only filter out like the last 10 tests right and then go from there in terms of like iteratively becoming more and more aggressive with filtering um so so that's like a a lever that we always have to to tune in terms of like how many tests we want to filter up after that um we basically run the entire test Suite on the trunk of Master uh all the time right so even if something is missed by a PR uh everything will be run on the main branch uh it'll go into that HUD likee system and somebody will go and revert it if it broke something in those tests so this is essentially a little bit of a trade-off we might run like fewer tests on the pr maybe there's a slightly greater chance of you breaking something and getting reverted but at the end of the day we will make sure that trunk on on a periodic basis is still healthy right right and then the last thing to say to that is like we you know we have that same issue right we have a monor repo as well so we have this issue where you know we can merge something and something like many many hops away because pytorch has a massive blast radius uh might break and then you know somebody comes after us saying you know like you've broken everything you know everything is down like please revert it so so you know we have the same issue and to to some extent like at at that massive scale like you kind of just there's some amount of like proactive work that you do to ensure that enough reasonable testing is done ahead of time and some amount of reactive work where you ensure that okay if something breaks you revert it quickly and and keep trunk healthy right that's that's just a a trade-off that needs to be done past a certain scale yeah to expand on his answer as well uh for uh number two when we do trunp test we also do that as part of our land validation so whenever you do at high merge it'll actually trigger trunk test which test before actually mering that into so actually a prerequisite into running a Reb version of your PR against all of the tests that would normally run it's more like a signal kind of seeing whether it's uh your test is your PR is likely to work and then before merg we have a fullprof test before we actually basically so yeah Target determination will probably mostly work at the pr test level and then when we actually do land validation before we merge it into that'll be the full test how about general question so I come from strongly class English background and so I had this experience like we had a big rubal base was Dynamic language and move to scholar majority of test like checking number of parameters the right number are the types correct so all this goes away when you have a type language so I wonder like in the pical Bas what is the mix of C++ to Python and like how many of these tests are basically because python of dynamic language if you for instance prep everything to type hints how many of the test will go away is this a reasonable approach to decrease the number of tests yeah so I think um over time the trend towards like being more pythonic is the general trend of the pytorch code base vast majority of tests are in Python and Target determinations is only being run on python tests at the moment typically pytorch unit tests are or pytorch tests are less unit tests and more integration test so they test things like are the types right less and more of like okay if I run backward on a tensor with this shape like does it return the right inputs or you know testing specific operations testing whether process groups are initialized correctly testing whether graph tracing uh is done correctly there the tests are typically more of that nature than you know these like very uh common best unit test practices style tests answer your second question question around what the what the libr is moving more towards I think Mark s had a really great talk as part of the keynote yesterday where we talked about pyth py moving towards more python code I actually think 2023 was the first time the majority of the code in PCH was actually python instead of C++ uh if you're aware of torch. compile all of the backend stuff for torch. compile is actually written in Python stuff us yeah at least that's so yeah the basic IDE that moving pych towards a more strongly type language is probably not in the cards at least in the near future uh but it is it is something that people have talked about and the typing thing is a very hotly debated topic even amongst pip cor so yeah interesting how frequent you building that embedding index that you use for the retrieval so as people adding the new unit test how would you make sure you're not part of your yeah so I'm working on uh kind of this like selective reindexing mechanism so that you can kind of like after uh PR is made to master you can figure out like which tests were changed and only update those parts of the index um so basically like reindex it selectively after he's changed to the code on Master what if people introduce a new Tes thing so if if like so that that will be covered by one of the heuristics right if you introduce a new unit test that should be run no matter what um on the pr okay then wouldn't that be expensive every time people adding few tests you go and rebuild the whole Ting space or is it an incremental type of so per PR it will be an incremental thing and we actually so after all the optimizations we're able to build the entire index in 3 minutes um for the entire code base from scratch so the selective one will be significantly cheaper yeah thank you this is guys