SBTB 2023: Nikita Shulga, How to help yourself and ML community make PyTorch better.

SBTB 2023: Nikita Shulga, How to help yourself and ML community make PyTorch better.

Recording: SBTB 2023: Nikita Shulga, How to help yourself and ML community make PyTorch better.

[Music] hello my name is Nikita and I'll be talking about how to make Pik torch for yourself and for Community better and I think that's kind of an interesting topic and uh thank you for the introduction but I don't I would not consider myself a machine learning engineer because pytorch is a machine learning framework but you don't need to be a machine learning engineer to be able to contribute to pytorch so it's much more broader project so quick synopsis of what I'm going to talk talk about so what if you new to ml what if something is broken with your program what if you have an idea how to help what if you want to help and how we test the changes in P torch so let's start from the beginning I don't know how many people are familiar with what pytorch is great I think uh then I probably have to can skip the slide but quickly if you want to start with python if you haven't done not everyone raise their hands you know it's a great time you can just boot up collab run some tutorials and you'll be rocking with machine learning so do it also don't forget that you might have a powerful ml tool right on your laptop or in your gaming PC and most of the ml framework support that which is a great feature of the Nvidia and AMD gpus anyway so what if something is broken don't ask questions there is Google but if something's really broken try to reach out to community so pitor have like uh two major uh feedback channels when people just want to ask questions is discuss. p.org or questions Channel and slack on the other hand if you believe that something is broken with spy torch especially if you have a program that used to work before and no no longer working now file an issue the main way of communicating with spy torch developers is through GitHub issues uh like people might think that you know oh if I know somebody from I'm going to send him an email and this person will reply most often that's not the case I probably check my email once a week I check GitHub every day three or four times a day so not and not just myself right lot and lots of developers like check GitHub issues so that's the main way of communication like what is wrong quick intervu so you submitted an issue and it's getting ignored you wonder why well maybe it title is not very descriptive maybe you put a screenshot of text and the screenshot of a really really small font size maybe you never mentioned your system configuration or you you submitted this report not to the pyto repository so first and foremost pyto repository is pyto PTO and GitHub and please try to write descriptive issues so how to report a good issue first of all try for reproduces with latest P torch uh currently uh I I don't know if like a lot of projects do that but at pyos we try to fix everything on trunk but we rarely fix something with older releases and that's it's an absolutely horrible horrible horrible regression so if you want Community to fix something in software what was released two years ago you can try to argue but your best option is to say you know it is broken with the latest release um another is try to get small reproducer um ml is an interesting field when you can run for days and days and uh you know finally something crashes or run out of memory and you immediately rush to submit an issue it feels great but it's not very actionable to the people who are looking at it um another so if you can have a smaller reproducer submit it if you have a collab notebook that's great if like just a piece of code also great also run collect environment so collect environment is a tool we built into ayto distribution that allows you to collect basic information about your system namely what is your python runtime version what's your pych version what Hardware accelerators you have and what other ecosystem libraries you have because sometimes other libraries can interfere with py or sh can interfere with other libraries if possible submit the full logs very often people say you know like I run my program and here's the three lines of logs like memory memory out of memory or I try to access like a tensor outside of boundaries not very helpful having a bit bit logs before that would help us to identify what's going on and if possible when developers ask you questions try to follow up so how the same process looks from oh sorry did I okay so how I can probably talk about if you have an idea so if you have an idea about how to improve by torch issues is also a way to communicate about that so you submit a feature request issue saying you know I have this idea this is why it's great but try to get like a good explanation why you think it will be beneficial not just for you but for the ml community and if changes something fundamental start with a special sub repository called request for comments when you try to like engage with more than one individual or more than one entity trying to find out what to do and yeah like answer questions if you say oh I want this like great Optimizer and then don't provide any details why it's needed people probably would not pay too much attention so the same process looks from the PTO developer Community point of view um like every day we have a person who looks at all the incoming issues and assign labels those labels notify people who are interested of monitoring the label about what is going on uh when label is assigned those people who have more knowledge about uh this particular area of the project might ask questions or might have an insights or might have an ideas but about like what is going on for the so-called high priority issues we mark them and we have like a group discussion about are they really high priority and how we can quickly address or mitigate them because if there is a workaround even though it's unpleasant sometimes software have bus so it's easy to say it is workaround and we will fix it later so now that we like done with fun in issues what if you want to contribute to community in some ways first of all thank you very much it's rare gift and everybody will appreciate like addtional help you can start by participating in discussions you don't need to be an expert you don't need to like you like good intention is always needed right and then you can learn and other can and help others um helping down breaking complex uh problems into an easy one if you like an intermediate Emil practitioner and you see that somebody submits an issue which is very complex and you can probably help them identify and narrow out to an easier problem and last but not least Le like if you want to C start coding we Mark issues again I started with labels uh we Mark issues with like a actionable or good first time issue and it means that if you want to try your hand that contributor at py torch then maybe that's your opportunity and it's the name suggests like pytorch is python about 50% and about 50% C++ so you don't need to be like a system level engineer to be able to contribute and especially with like a new features like torch compile which is very pythonic so almost every's implemented in Python you can Implement features that show tremendous uh performance gains and purely in Pythor so C+ plus knowledge is not mandatory so if you want to submit a p request what to do or maybe what not to do like it is very important that your P requests have a good description fixes blah is not a very good description uh 300 character long string is not a very good description if you want to fix some bug that was reported previously other regression test we love testing because otherwise if somebody fixes something then somebody later not because of the matters but because of the lack of testing we'll break it again and that's the worst thing that we want to deliver to the community is a software that used to work and now it doesn't this is why in the beginning I said that you know like please if you have uh like progressions reported right away one PR should try to fix one problem like do not try to aggregate everything into one p request and I don't think it is specific to a pytorch uh Community it is kind of a general rule of many open source projects because if you expect somebody to review your changes it is better when they have a clear idea of what those changes are doing and it's easier to reason and understand and provide feedback if you just making modification to one feature one function also referring from submitting of big PRS or new features without discussing it with the community first because you made subst substantial investment of your time and knowledge and probably money into writing this change and nobody cares about it it is upsetting but the reason maybe other people don't care about it is they don't know how great great it is and they don't know because they haven't seen an issue participa in discussion and also like why Community sometimes reluctant to accept new B code contributions because it's very easy to contribute code but it's easy but boring to maintain it and maintenance is something that becomes significant CH part of like responsibilities that b developer Community have to deal with and don't worry about finding the right reviewer we try to find you one and if not you can look at the people who modified this file recently and try to mention them and say can you please look at the PR but also please don't be discouraged if you don't receive a timely feedback unfortunately sometimes uh like developer Community is overwhelmed by dealing with the incoming issues or working on the features that we want to present for the next Ries so let's talk a little bit about a PO request life cycle which I think is pretty common for the pie torch but there are a couple of Kats so you propose a PR as a regular review cycle starts when the people that are assigned to look at your PR or just the people who contribute to PTO regularly look at the uh add the contents and make a suggestions about what needs to be improved like how to add the test maybe suggest like better function name um point out like some error checking uh I don't know time constra considerations or memory and so on and so forth in the background CI started unfortunately we cannot start the CI by default new contributors you ask why like it is a common feature uh this is something that changed in the last probably three years when people start to abuse freci for their personal purposes right we test by tortun gpus so essentially you can train a small Network for free this is why unfortunately we no longer allow like any PRS to be tested on our system because that gives you access to lots of gpus but again once you contribute to pytor you will be regular contrib and we will trust your PRS more and U CI will be running um also we cannot run all the CI all the time so we introduced like we we love labels we love doing lots of things in open and we use labels as I mentioned before for issues uh classification we also use labels for selecting which uh subset of tests should be running on uppr so to add additional labels uh like to additional test and you like slap ACI flow/ something label I will talk about it later and uh give will seeo testing test running uh when review is obtained merge process can be initiated by talking to the boat why you don't press a button first of all we want to make sure that all the testing is finished and uh also sometimes uh I'm going to talk about it later we you can have red on your tests but your PR is still mergeable this is why we want the bot to monitor the progress of the testing and whenever the testing is done like do it automatically unfortunately after PR is it's not 100% guarantee that your work is done there it can be reverted and I can talk a bit I will talk a bit later about the reasons for reverts so P request life cycle so failures as I mentioned not necessarily the author's problem the reason could be that you based your PR on the broken uh commit when you started working on the change one way to avoid it is we have a special label called viable strict which contains only commits that we know that all the extended testing has been run on and they green so if you start on those commits it's very very unlikely that you'll be affected by some um issues that crept up from the broken trunk um uh there there is a tool called Dr C well it's not a tool it's just a comment on the issue so whenever you submit an issue we have a bot the same B that will merge your commits that will look at your commit and whenever you receive a signal it analyzes the reason for success or failure and give you a summary so for example if TR was broken you can go into Hub look at the commit that your PR is based on check all the like failure signatures and compare it with the ones you see in your PR but it's long and boring so we build a boat for you that can tell you like you know this can go as planned we have a one flaky fader we run it it's green and we have a couple of tests broken and drunk but again doesn't seem to be a problem so how do we as like a p developer Community aware of this regressions that happens post land so again we build a tool called HUD which gives you like a bird eye view of the status of C on the every commit so you can see in the row it's individual commits like there's a time it was committed hash some title who committed it and then there is some testing right so the column represents individual tests well that's the LIE this column represents an aggregated test I I can show you later like how many tests we actually run every PR and another way like if you want to like but this gives you a bird I view right so you know that something passed or failed and if it failed you just see a red X not very actionable so you can look into individual PR and again we try to highlight not the job that failed but actually individual test name and give you a command that you can run to reproduce a failure on your local machine um another quick question is why we not running the art why we're not using the artifacts with build during C for releases the reasons for for that are twofold one is when we're building for C I would want to build the binary that quote unquote works on my machine so the binary that works on the runners that we allocated for ourselves that have a certain software pre-installed that we depend for the um py to work but when you want to build a package it's not as easy because you need to deploy something called chromatic package so like uh pie standards say that your package cannot depend on any by six libraries on Linux and similarly on windows so you cannot say oh if I want to accelerate with gpus I just assume that user will install that GPU runtime you should package all of your GPU runtime that you depend on into your package B but another reason why we like run continuous deployment separately is because when we do continuous deployment we want to build it for more Hardware configuration than uh we're doing during CI so for CI I only care about one GPU flavor that I'm going to run my C on but when I want to give you a package I want the package to be available on all the gpus that like currently on the market well with some Kats um speaking about release Cadence so we try to have like a quarterly by releases so 2. one was released I think like months ago we tried to do one or two patch releases so when we release the minor release we can notice some regressions from the previous release and we say well it would be good to be patched so we release one two patch releases between every minor release and we also have nightly build so if you want to try latest greatest P torch you can always try nightly at your own risk though because we can introduce regressions but again we build nightly of this viable strict Branch so technically it should be green so why do we build all this complexion like complexity it looks like GitHub already provides you like developers a great utility to do everything you want uh so like by this is all needed uh byor review structure historically is not very flat byor is well I wouldn't say it's an amalgamation of many projects but it have different parts that have like different strong code owners and the code owners can delegate the review sorry review privileges to like uh other developers we don't want to interfere with that but G does not have a good mechanism of saying you know if only this folders are modified these people can review but if that folders modified that people need to have review so we need to build our own like system on top of that and this is handled by both um another reason is we just cannot run all the testing all the time it takes too much time and frankly it's too expensive because again ml testing needs to be run on gpus gpus are not easy to find and again if you run everything sequentially from start to finish it will take more than a day to test a single pull request and developers don't want to to wait that long um also we need to validate that bytch works not just on Linux but on Windows and Mac not across all the hardware like not across all the accelerators but at least for the CPU Runners um and the code for all this is uh stored in by Tes infr and I think with that I open for QA but looks like everyone started any questions so so how often you guys pretty much automate all the CI process so if I P request everything has been automated or you need to wait to the next to request no so you like whenever your put request is approved and merged like it will be available in the next nightly build but yes otherwise you need to wait for the next Ras to get your changes in rights yes so if you uh look at the top level MD file for the repository there is contributors MD and um some that contains more links to Viki Pages there's also a couple of fantastic uh like one of the core contributors at Young runs a fantastic uh blog about like contribution to pyto where he discussed about like how certain parts are um implemented and that can help one to learn what they need to do if they want to contribute to those parts and again U I think like like uh the question you're asking like is there enough documentation if one feels there's not enough I think that's a great opportunity to contribute and again uh the success of the open source project is not necessarily about code contributions but about all the ecosystems around it so if people are willing to contribute more to testing to uh documentation to like community in general just answering the questions that that that's great any any else yeah yeah sorry uh I work with torch audio a lot do you know what uh opportunities are are things that you're looking to develop in torch audio that I could contribute to are um so again I think the idea is similar you should look into torch audio project and look for the good first time issues so I I'm more focused on python se but uh like I I I bet there are some issues that people want to like some problems that people want to solve uh one I can think of though maybe it's not less of a melon more of a system engineering issue is I wish there was a more reliable way of decoding data format right so I want to be able to use torch audio with MP3 flag vales you name it and always be able to get like a tensor out of it cool thanks I'll look into that sure no okay question I guess no more question yeah thank you thank [Applause] you