A Match Made at Upwork

data.bythebay.io: Alya Abbot, A Match Made at Upwork

Recording: data.bythebay.io: Alya Abbot, A Match Made at Upwork

hi so just uh before I start I want to say please ask questions throughout I think it'll be nice to have the interaction um so just raise your hand if you have if anything if you have any questions um so the title of my talk is a match made at upwork um so I guess um as was just mentioned for those of you who are not familiar with it upwork is uh the world's largest Online freelancing Marketplace um we really believe that in the modern world as a company you shouldn't be restricted to hiring people who just live within say a 50 mile radius of your office and on the flip side as as a somebody looking for work for example you should be able to work for a silon valley company without having to pay barer rent prices um so we're a Marketplace that uh brings together clients and Freelancers from around the world um so on our platform we have uh over 4 million uh clients posting jobs uh over 10 million Freelancers um and there's over a billion dollars uh worth of work getting done every year on the platform um so the big question for a big question for us that I'll be talking about today is how do we how do we match up those clients on those Freelancers how do we uh put together uh the 10 million Freelancers looking for work with the 4 million clients who want to get work done uh so as in the real world uh the offline World um the matching process generally starts with a job post so uh the client will uh post a job um Des describe what what kind of work they're looking for um so as you can see here the work on our platform actually ranges very widely so on the left we have uh somebody uh looking to have a logo designed for 50 bucks on the right we have somebody uh who's trying to hire uh data scientists to do Predictive Analytics and data visualization um the the budget they have in mind is $50,000 uh they're looking for real experts so you really have a big Spectrum both of the types of work and um kind of the scope of the project so everything starts with the job post um and then Freelancers can come in and apply for the job and hopefully uh the client can find somebody uh who works great for them that they can hire um and so a key metric for us is uh what we call the fill rate which is the number of jobs where the client uh is able to hire somebody over the total number of jobs posted so where how do these Freelancers um show up while they they search for jobs on our platform and then kind of a key metric on the freelancer side of the of the spectrum is what we call the W rate which is out of the applications that a freelancer submitted how many resulted in that freelancer getting hired so this is essentially how well directed are this the freelancer's efforts and um the focus for for my talk today will be this piece of how Freelancers actually find find jobs in the platform um so we provide a tool the the job search where a freelancer can enter any kind of keywords to uh to look for jobs so often those might be skills like PHP or Java they could be the type of work like if you're looking for logo design or for data science and then we also provide uh lots of different facets that the freelancer can use uh to filter down the jobs so for instance maybe somebody's looking for entry-level work whereas another person is looking for expert expert work uh so this kind of uh user-driven uh job Discovery presents a challenge where the Freelancers aren't the job the right jobs aren't always seen by the right Freelancers um so a lot a big piece of this is that um the search is very timely right so job gets posted now client wants to fill um hiring process actually much faster on upwork than it is um in offline um and so if a freelancer just doesn't happen to maybe come to the search at the right time they might miss a job that would be perfect for them so a big question for us is how we can help Freelancers find the right jobs uh and we have a number of ways we can uh try to recommend jobs to Freelancers um particular one I'll talk about today is uh on shown on the left which is a job digest email where uh we can send send out email um on a daily or a weekly basis uh to Freelancers with personalized job recommendations for them uh of course that begs the question of okay you know we have a venue for sending out recommendations we need algorithms that will actually tell us what recommendations to send to each user so one really nice thing about uh being a being at upwork and being working on this platform is that we really have the full uh scope of data we know everything that's involved and and how uh the client and the freelancer come together and what happens then um so it's starts with uh Impressions so Freelancers finding jobs uh the Freelancers might click on jobs they're kind of interested in they'll apply the jobs they're excited about uh we know which uh Freelancers get interviewed for the job uh who ends up getting hired we know a lot about how the project actually goes so how many hours The freelancer Works each week uh at the end whether the client and the freelancer are satisfied with the experience we collect that feedback so we kind of have like we really capture the full spectrum it's not like a recruiting service where they might know something about how the users interact with the site but they don't necessarily know kind of what happens afterwards did the person where did the person apply where you know was the employer satisfied they're kind of missing that piece and so having this full full funnel lets us model different pieces of it and kind of ask ask the questions we're interested in um so the two that I'll focus on in this presentation are these two pieces here so one uh which jobs a freelancer is interested in so if a freelancer sees a job will they actually want to apply to that job and then two uh which I'll get into to in the second half uh which jobs actually need more applicants so first looking at um this uh piece of uh which jobs are freelancer is interested in I kind of want to highlight that this is not kind of a a god-given right like we we need to decide what it is we want to model and that's this is a conscious choice that we made uh for example we could have tried to model the full spectrum right we could have tried to say if if a freelancer sees a job will they have a good outcome working on that job and that's that's what our prediction um but that's really something where it's going to be really hard to understand and hard to model and hard to come up with features because there's just so many things that can go wrong on that path um so we picked something which is kind of an intuitive model it's something that's user Centric well why are we showing you this job well because we think it's a job you're likely to want to apply for and we also we have lots of data so uh as you go down the funel funnel the data gets smaller and smaller and we have a lot of data on this kind of left left half of the spectrum so how do how do we model it how do we predict how do we identify uh which jobs uh a user will be interested in uh so our model model is uh content based um so basically the kind of main things still we look at the main kinds of features are looking at um the job that's being posted and comparing that to the jobs the freelancer has applied for in the past um so this kind of makes sense so we look we can look at things like is the title of the job similar to jobs as freelancer has applied to before uh is the category similar so if somebody's applying for lots of uh logo design jobs are probably not going to be excited about a machine learning job uh and then what kinds of skills are required and do those match the the Freelancers previous applications and the skills in the Freelancers profile um and so this is kind of the heart of the model uh there actually two other pieces that make the model perform a lot better that I think are kind of interesting to think about um so one is basically a prior on the freelancer side so for this particular freelancer out of the jobs they see what fraction of the jobs they actually choose to apply for um and then similarly we have a prior on uh the job side out of um the Freelancers who saw this job which fraction uh chose to apply for the job so for example we have uh jobs that might be very appealing on the platform like the $50,000 uh data science position we might also we also have some jobs that are less appealing like somebody who wants uh 15 press press release was written for 10 bucks total uh so including these two pieces how selective the freelancer is and how appealing the job is actually gave much better performance uh when training the model on on the match um so we have a model that's it's just a simple logistic regression model that does pretty well um it actually does quite well with just a small subset of features uh which is what we ended up putting in production and uh then of course we don't just want uh performance in this abstract Au sense we want performance in our in our real Marketplace so we tested out the model in gen for generating uh a job digest email that we sent out to our users um and had really good results getting uh an open rate and a clickr rate that's uh twice the industry standard for this kind of email um also uh getting seeing real impact on our metrics which is are things like the fill rate where which I mentioned so which fraction of the jobs posted are able to hire uh and then on the freelancer side uh increase activity in terms of applications and also in terms of actually getting hired for the jobs so there's a piece uh of the of the real question we're asking that I've been abstracting from so far which is it's not just which jobs are relevant for which freelancer um the question is which jobs should we show to which freelancer right now so for example we might have a job that gets posted and doesn't for one reason or another doesn't attract a lot a lot of attention gets very few applicants um and the client isn't able to hire so we want to realize that early on right we don't want to wait until the client has uh given up and walked away and we want to send send Freelancers to this job if it might be relevant for them on the other hand we might have a job where um the job is posted and the client gets some great applicants starts interviewing them and now a bunch more people come in and this is actually not a good experience it's not a good experience for the client who might be already in the process and now they're overwhelmed because they're getting more and more people showing up and it's hard to filter and it's hard to figure out who to talk to and it's also a really bad experience for the Freelancers uh because now they're applying for a job uh but it which is they're very unlikely to get and it's really like it's really a waste of their effort so we want to be mindful about which jobs we're actually directing to people people towards um and ideally we would have just a handful a few great candidates for the for the client to choose from and we can look at this kind of more more visually more quantitatively so on this chart uh along the x- axis we have the number of um applicants for the job and the Y access has a number of uh great candidates uh which were identified by our model to be a good fit for the job and the colors show you the fill rate uh for that particular spot so uh given that number of uh total applicants and great candidates How likely is the job uh to hire somebody does the chart make sense or there questions mean so every every square is what average so every square is the average for jobs with exactly that many applicants and exactly that many great candidates after like I think I guess I looked at a week after the job was posted posted the edes might be weak yeah so the edges have less data but it's a minim it's there's a minimum cut off of at least I think 100 jobs per square you have great candidates that's the labeling by your model yeah that's a that's a separate model yeah so people can apply even though they're not pre candidates right right yeah um yeah we don't these are Freelancers who kind of deciding what what they want to do um but we have a model to tries to identify the best applicants candidates they may be informed by of the positions that are available by by your email where yeah so currently the So currently the the impact of that model is more on the client side so when we show a list of the C of the applicants we highlight some as being uh recommended by our system yeah what not a great cidate so that's a whole separate model the that actually I let me flip back if I can um I showed maybe it's not worth it actually so in the funnel that I showed there was a kind of an error that pointed from applications to good outcomes so that's where that model lives that that model tries to predict out of the applicants who's actually going to get hired and have a good outcome for that job um so yeah so um this kind of just gives us a sense of of some of the things that are going on here so uh one thing you can notice is that um going up are has a lot more impact going than going to the right maybe it's it's hard to see on this screen but um getting more great candidates actually increases the likelihood of hiring a lot more than um getting lots and lots of applicants so you can see like along the bottom uh theob those jobs never hire never have a very high chance of hiring um even as if the total number of applicants is large and then kind of another piece of it is that getting more applicants is much more important uh for the jobs in the bottom left corner that don't have a lot of people uh who have already already applied than it is once you already get a bunch of applicants so this is kind of just uh to get a sense of the data we can also think about what we can model to make the predic to make the right prediction about which jobs to select applicants for um so what we have uh is a model of future hiring so we can't quite figure out so what we really want to know is which jobs would hire if we sent this candidate to that job but aren't going to hire otherwise um that's really hard to know so what we do have data on is something that's a related question so here we have a job that's getting posted and uh people are coming in and applying for it so um here we have time so first there's one candidate another and another and so on and say in the end uh the client decides to hire this particular candidate so what we have for the model is um a label based on whether the client will hire a future applicant right so at the beginning uh we're here uh the candidate who end up getting hired hasn't applied yet so the answer to the question is will will this client hire somebody who applies in the future is yes whereas at this point uh the applicant who got hired has already applied and the answer to the question of will this client hire somebody who applies to the future is at this point no right she's already applied so that's uh what we train a model on um and it it looks at features like the size of the current applicant pool how long has the job has been up for um as well as the clients prior history and trying to make this prediction so do you know at which point the client looked at the given applicant we have that data um I guess that I'm not I don't think I use that in this particular model but we have all the interaction data and actually one piece that's relevant right is all another way that a job might un the client might not end up hiring is they might kind of walk away from the job and we can see that so um this client activity piece is looking at how long ago did the client actually interact with their applicant pool so we can try to get a sense of are they still actively trying to hire for this position that's a good question um so I just I should just show you two pieces of the system we've we've uh been we've put together based kind of on all this data we have about uh clients and Freelancers in our two-sided Marketplace um um I just want to mention that there's a lot of further complexity lots of lots of other questions uh we think about um so uh one that's maybe already come up a little bit is um okay you know we have these uh applicants um which ones of them are actually going to do a good job on on on this project um we also want to think about uh Freelancers more holistically so not just what do we want to do locally for this job right now but how do we help our Freelancers build build their careers on the platform uh one thing I mentioned briefly is that of course not all jobs are created equal so they range from this $50 logo to this $50,000 data scientist position and if we had you know we'd love to fill both but if we had to pick one somehow we St to care more about filling the large JB um and then beyond that you can start thinking about the value of the client right how do we keep clients um happy with the platform and using the platform and these questions kind of build on from there so there're sort of I think it's just a really fascinating um place because that we have so we have data about so many things and just a lot of things to think about in terms of what we optimize and what we model and how we make all those pieces fit together um so that's that's it for my talk and I'll I'll take any questions which point of view are you getting paid and which side P right so I think I didn't actually mention our um business model so uh we basically take a percentage of all the transactions on the site um so um formally it's kind of we're charging the Freelancers but in some sense it doesn't really matter it's kind of the a fraction of the money that's flowing through the platform thank [Applause] you