data.bythebay.io: Eric Colson - Synthesizing human and machine capabilities
Recording: data.bythebay.io: Eric Colson - Synthesizing human and machine capabilities
thank you Michael so if you've been reading mainstream business books these days you are aware of this theme that machines are going to take over the world and leave us humans without jobs uh others such as Elon Musk and stepen Hawkings Bill Gates they go on to say that AI poses an existential threat well this is not going to be a talk about the contention between humans and machines quite the opposite this is about combining the unique talents of humans and machines to create new capabilities so let's start with a very familiar machine learning test that of the recommender system so this is a screenshot of Amazon on the top we're probably using a collaborative filtering algorithm here and on the bottom it's probably an affinities algorithm very traditional uh machine learning Le techniques and we as consumers have come to rely on recommendation systems for navigation and Discovery when we shop online but when we go to stores we get a very different experience we get a human a knowledgeable sales associate that helps us and it's a very different experience for us now there's many that will argue that this thing on the left which is powered by machines is the digital version of that on the right which is powered by a knowledgeable human but they are not alternatives for each other you cannot even say that one is better than the other it really depends at the task on hand allow me to demonstrate that so looking at this audience you all appear to be mostly humans um so I'm going to ask you to perform two tasks so a little closer a little closer so when I reveal the first task as soon as you know the answer just raise your hand really high so I can see it I need to count all right so ready for task number one find the ion vectors in that Matrix I don't see did you no one okay nobody can do it so there's usually one person that raises their hand thinking they can do it and the truth is you can all do it if you've ever had a course in undergraduate mathematics they teach you how to do this by hand not on a 10 x 10 matrix usually a one by a 2 x two or a 3X3 and this would take you many hours to do it's a lot of rope calculations so you're going to get very bored doing it you'll probably make a lot of mistakes along the way but a machine can do this flawlessly in milliseconds all right let's let's try task two I think you'll find this better suited for your particular talents just raise your hand High when you see it find the leopard print dress okay we have hands going up a few milliseconds and you guys are on it and you weren't confused by the fact that the dress is actually blue and the leopard print dog didn't fool you either now machines are going to struggle with this it doesn't take them a long time to render a answer but the accuracy is going to be inconsistent unless they've had blue dresses and Le print dogs in their training sets it may be hard to discern but as humans we have the powers to improvise we have knowledge of social norms and we have the ability to rate relate to other humans in fact that dress is not even leopard print at all that's actually Jaguar um and but we know when somebody ask for a leop dress that jaguar and even cheetah would probably suffice so we have two very different processors with different skill sets and if you had a that had to do with a lot of rope calculations saying say finding igen vectors in a matrix then that is a task that's far better suited for machines other tests that may rely on social norms or ability to relate to other humans that may be better done by a different type of processor a human processor so this notion of human computation is not terribly new it's been around for quite some time um this is a picture of um and then I 1940s these were people whose job it was to add up long strings of numbers they called them computers quite literally if you said I needed a computer in the 1940s they would have said well we got Judy available or maybe Bob right we didn't introduce the prefix digital computer until the 1950s or so um so this was the early day computers um and now today we use human computation but in two very different ways number one we don't use them for roach calculations we use them for the things that only a human can do and number two thanks to Connected devices we don't have to gather them up in rooms like this we can connect to them wherever they are in the world and any time in the day um and that way you can build them into your software workflows you might have a machine test that chugs along executing rope calculations then hits a point where it needs some human judgment it'll route a task out to a human gets its answer back and keeps going along its way so with programmatic access to these wonderfully distinct compute resources you can do a lot of things you can combine them in new ways to achieve higher levels of performance there's precedence for this uh the game of chess L held as perhaps keep close got it uh leld is perhaps the ultimate U uh demonstration of human thinking abilities well in 1997 the world-renowned chess player Gary KAS broth lost to deep blue a supercomputer buil bu by IBM now this was inevitable it's just a matter of time the human mind is no match for The Brute Force methods and the calculation speed of a supercomputer but something interesting happened after that there emerged this thing called freestyle chess this is where the chess player can use a device of his or her own and this is usually a much more modest device than a supercomputer but something to help with the calculations and it turns out that the two of them combined can beat a supercomputer Casper off and himself did this he engaged in freestyle chess and he said that with a machine at his side to do the calculations he felt that he was freed up to focus on the more creative aspects of the game so the two of them can combine to produce an outcome better than either one can do on their own and it turns out there is commercial applications for this too for combining machines and humans um this is Stitch fix this is where I work it is e-commerce but very different do we have any Stitch fix customers in the audience okay we got a couple yes we do men's clothes now as well um so at Stitch fix like I said it's e-commerce you can buy clothes from us but it's so different there is no shopping at stitchfix you can go to the site stitchfix.com but nowhere are you going to find a browse page a product page or a search box you're not even get recommendations there on the site at Stitch fix the customer does not pick out the merchandise and this is the value proposition there's a lot of people that don't have time to shop they don't know what's currently in stock or what might look on look good on their bodies or what's appropriate for their age they want a service to do that for them so this is how it works if you were to become a customer of Stitch fix you first thing you have to do is fill out this pretty lengthy style profile this is where you tell us all the things about yourself your height your age your weight your preferences for fit preferences for style you can even write a a free form note that describes elements of your lifestyle that may be pertinent you can also make a Pinterest pinboard full of examples of things you like and you want to share that with us the next thing that's going to happen is an algorithm is going to run one that leverages all of that information you told us about and it's going to pick out five things for you and then it's going to ship them to you sight unseen you have not seen anything it the first time you'll see your merchandise is when it hits your doorstep you can then open the box and experience them in the privacy of your own home with your own shoes and wardrobe and get feedback from a loved one cuz you're not going to drag them to a mall these days so so that's how it works um oh and you also don't have to keep anything you can send any or all back you just pay for whatever you tell us you'd like and they going to keep send the rest back we take of the shipping both ways so that's how it works it is like a recommendation engine but with a much greater commitment because we have to pay for the delivery so there's a lot of companies that use recommender systems to some they drive incremental sales to others they are the means of Engagement to others still they are the primary vehicle for Discovery but it's dish fix this is our business model 100% of what we sell is through our recommendations so it's extremely important to us also unlike these first three companies we have incremental cost when we get it wrong right for these we've all had goofy recommendations on Amazon and Netflix you kind of shrug your shoulders and move on but for us it's devastating we have the cost of shipping both ways we got the cost of inventory being out and we've got a pissed off customer she may not be shrugging her shoulder she may been counting on this stuff for some event so it's extremely important to us so we're going to have to use all the processing we can get our hands on the machine type and the human type so this is how it works we have our compute resources here machines humans and then also we have um we do our own delivery our own Logistics um but the two of these combined machines and human compose the algorithm we have a styling algorithm that's distributed across machine and human Hardware now we have all the machines we can get our hands on thanks to Amazon or AWS for humans we've had to amass our own Army of compute resources we now employ almost 3,000 human stylists and these are uh professionals these are folks skilled in the areas of fashion merchandise this is not Mechanical Turk these are skilled tasks all right but machines and humans have very different work styles to to coordinate their work we're going to use a queuing mechanism right machines are nearly INF fatigable and very fast humans need braks and they're a little more slow so we're going to use a queuing mechanism to coordinate their work so that over there's a customer and this is how it works so if she wanted a shipment of clothes all she has to do is pick a date when she wants to receive that that creates a shipment request uh which needs to go in one of those cues we actually have a machine algorithm that determines that the cues roughly correspond to distribution centers throughout our country and they may have better or worse inventory for her or we may take into account the proximity at any rate they pick one and it places it in the queue then the first thing we're going to do is route that shipment to machines for processing because there is a lot that we can do with the data that has to do with rope calculations so we can do things like um we're going to perform the M algorithm M for machine so we can do things like PCA and SVD which find the directions and the data that explain the most variation we can do things like Matrix factorization that finds latent attributes that might predict what somebody's going to like uh mixed the effect models that capture interactions between product attributes and the customer attributes and then there's uh neural networks and deep learning this is largely for the image and Text data so all these things might entail quite literally millions or billions of calculations so they're far better done by machines effectively what they're doing is they're taking all the inventory we have and they're going to calculate a relevance relevancy score in the context of that customer so it's going to eliminate a lot of the merchandise and then what's left it's going to rank order by this relevancy score and return it back to the queue but we're not done that yet we need to do more processing we have this other processor that we can leverage so we're going to Route it to one of those humans now humans are far more heterogeneous than their machine counterparts so we don't just want to pick one arbitrarily there's differences there and perhaps there's one with a higher chance of success with that particular customer so we have a machine only algorithm that picks one of those not go to the one on the right there now now once it's assigned to her she's we're going to drop off those machine generated results in this very nice looking interface so we can present all the information to her and she's then going to perform what we call the H algorithm H for human and this is all the things that only a human can do things like improvising or curating applying her knowledge of social norms they're going to foster a relationship with the customer ultimately she is going to reduce it down to exactly five things and she's going to put that back under the CU from there a signal is sent to our Logistics system our own we do our own warehouses and they're going to turn that information into real products they're going to do the pickpack and ship wrap it up beautifully and get it over to the customer so it's about doing more it's not just machines and not just humans but the two combined to achieve higher levels of ability and that combined with the convenience of Home Delivery adds a lot of value to her so the assertion here is that a styling algorithm s composed of human and machine resources is going to be better than a styling algorithm composed of either human or machine alone that is we're banking on additive results um or perhaps additive with some synergies so this assumes that humans and machines both contribute in a non-zero way and that those contributions are different so this means we have to be very careful with our training now training machines is pretty straightforward we have a lot of literature out there things like back propagation and cross fold validation feature selection regularization we do all these things to help train our machines but training humans is a little different there exists this H algorithm again H for human that runs on human hardware and they've each person has built up their own algorithm over a lifetime of experiences and observations and it's hidden to us but we can reveal it through this custom interface we've built we can subtly change the information that we show them and then assess the outcome of their decisions and in that way we can ensure that their contributions are always positive and that they're always complimentary or orthogonal to that of the machines so there are a lot of benefits to combining humans and machines this is the short version of this talk so I'm going to jump right down to number five which is specialization so as long as held that the division of labor is the source of all economic growth but in this case we're sharing our labor with machines and that concerns us but what is it that we get to do more of where is it that we humans specialize well in the case of Stitch fix because our stylists don't have to do the routine work the rot calculations they are freed up to focus on the more creative aspects of the game and that's important because occasionally our customers write in very personal notes such as this one she says my husband is returning home from my a tour in Iraq he is disabled I would love something for a very special date night now our stylists are very much real human beings and they can't help but be moved by this and they are often compelled to provide more than their styling Services sometimes they write back they may think the customer for her husband's sacrifice they may send flowers or a little gift they focus their time relating to a customer in a very human way so in essence the machines are enabling our humans to be more human and that's all I got thank you very [Applause] much thank you uh we have time for perhaps one or two questions thank you was very interesting um I also work in impation uh systems and I have a question now that you have the C Factor inside so what kind of metrics or things that you use that are from the machine to the human and then away to the customer cuz usually we measure right and usage of of the side things like that but then uh a human can get a list and for some reason reject all of it or I mean so what kind of things do you measure for this good great question so um the question was how do we measure machines and the humans um so machines easy enough if you've got things like Au that can the standard features or standard metrics uh for humans they also have their own metrics um in not only what they achieve but how they interact with the machine results we do look at things uh common uh from taken from search um mean reciprocal rank how far down the list did they scroll to find their things and when they do that does it work out well for them or is it not a good thing so we give a ton of feedback back to the humans and likewise humans provide feedback back to the machines it's a great question all right I think do I have time for any more uh yes have time for one more question got one more I sell your hand first I have it right you the machine delegates to a arul of human right and then human suggest Styles how do you measure performance given of the machine and human given that it's chained right how do you know the machine did a good job picking the right human and whether the human did a good job in general that's prob need a little B easier but yeah my question let's see if I got the question correct is when we assign the task to a human we have to we have an Al that picks which human to do it how do we know we got that right and the answer is through controlled testing you have to do if you have a hypothesis say that this customer is a mom and that stylist is a mom that's a great hypothesis that maybe matching them up based on those characteristics may do better you will only know the answer to that question if you sometimes assign moms to moms and other times assign moms to random stylists and then can compare AB where you're controlling you're only changing exactly one thing and then you can uh get your measurement right okay thanks everyone I will be out there if you have further [Applause] questions