data.bythebay.io: Demetri Spanos, Challenges Applying Traditional UI/UX Principles to M.L.
[Applause] all right are we set with the mic all right uh so thank you very much uh for that uh lovely introduction hello everybody uh my name is Dimitri I'm going to be talking today about some challenges that I have faced uh applying traditional UI and ux design principles to machine learning products um and I want to emphasize that this is uh me attempting to distill my own personal experience uh and I say that for two reasons um one is that I want you to take whatever I say with a grain of salt I'm not presenting this as you know this is established fact I'm presenting this as this is what I've managed to extract from um my experiences uh but I also Point want to point out that we're at a stage in development of the field where really this is all we have right we have practitioners telling each other stories about what works and what doesn't uh we don't have an established body of knowledge I can't point you at a book that says here's how you can go learn to do what I do at some reasonable level of quality right uh and so the fact that this is a personal story uh or you know a Snippets of personal stories is is relevant to the bigger Point here about uh the evolution of the field um so anyway uh I've spent uh 10 plus years since my PhD uh doing endtoend system design for machine learning products and machine learning features of products uh working both on the algorithmic side doing the mathematics and computation but also working on product and feature concept work working directly with UI and ux designers uh and thinking about how uh the capabilities of machine learning can be distilled and refined in such a way that they're more accessible to um you know more people right um so uh anyway I've worked you know in lots of with lots of different kinds of companies and many different roles um I I want to emphasize that while I consider what I do design work I am not a capital D designer I didn't go to design school I don't I'm not particularly proficient in visual design uh I have worked closely with many designers and I think we Face many of the same challenges um but uh I want to I want to make it clear that I'm not here to tell you like hey this is a really effective visual layout for communicating something about a particular machine learning phenomenon okay so um here's what I hope you will take away from this at the end first that there are recognizing the fact that there are many time tested design principles for software and this may seem so obvious as to be not worth saying uh but I think it's important to think about how we got those principles because they you know we didn't have them 100 years ago we have them now where did they come from and how can we get more of them right um it's worth thinking about that uh I hope that you will be persuaded that we have less such stuff for dealing with products that incorporate automated learning elements um and uh so if you're persuaded uh by my claim that we have less such organized knowledge uh what can we do I mean are we just going to sit back and not do anything about it presumably not so where where can we start looking for Clues um and then I'm going to talk through specific principles that I found useful in my work uh that perhaps you will find useful in yours so let's Dive Right In design principles what are they and why does anybody care about them um so design is a big word right it it means many different things to many different people um within the scope of this presentation I mean designed to be uh you know what I I consider a formula of three parts understanding to one part of choosing that is a designer has to on the one hand understand the creative medium in which they work but must also understand an audience's values and an audience's mental models so that's the three parts understanding and then the one part of choosing is I then as a designer on behalf of my audience have to make good creative choices that serve their values uh and that are accessible to their mental model so they can understand what it is that I've done for them um and hopefully you know hopefully I've done a good job for them but ultimately if they don't understand it uh that's not doing anybody any good so that's what I mean by design uh and you know that incorporates visual design interaction design uh lots of other pieces of design that don't yet have names uh but that's what I mean by Design so then what's a design principle um a design principle is a compact piece of knowledge that can be easily shared that helps you identify problems or opportunities in advance that helps condense shared wisdom for reuse so that you know not only experts getting together can say oh hey there's this thing that I do and it's really great but the you know newer generations of people who do this work we can give them something that they can say oh yeah there's that thing that they told me and maybe it's kind of useful right uh because right now the you know if I were tasked to train a junior person I would not have organized compact knowledge to pass on to them I think that's a problem uh I think you know we we can do some things to fix that um so anyway uh I hope you believe that we do need this because machine learning product are full of hard choices and I'm going to talk about some specifics as we go through the talk um but really it's full of hard choices right because we're asking people to put their trust in something that they don't particularly understand in most cases uh and the you know human factors in design design trade-offs there are very complicated um so trying to solve that those hard problems without the guideposts that uh you know traditional visual design for example has uh well it's it's a hard job uh and I would like for my job not to be so hard so um that's what a design principle is here's an example from visual design maybe the most widely understood design principle uh usually is called the proximity principle uh and it states that spatial distance mirrors relatedness or stuff that's nearer together is probably more related all other things being equal um this is a widely understood principle in uh among visual designers it informs all graphical layout it's easy to discuss and to debate and maybe you can say oh hey in this case we can make an ception for whatever reason um and most importantly it frames conversation both for debating the merits of a design but also for educating uh educating people in techniques for design so that's a visual design principle this is an interaction design principle uh and this goes by many names I call it the predictable button principle but you're all familiar with this um you want to convey the outcome of pressing the button before the user presses the button right um and this supports the user being confident in using your application in having a sense of agency and the speed with which they complete tasks um and you know this is a principle that should inform all interaction design um and you know when this is violated people get upset right if you go click on a button at some e-commerce website and it doesn't do what you expected it to do and there's money on the line you're going to be upset so anyway this is an example of an interaction design principle um so there are lots and lots of design principles out there right again design is a very rich field uh I couldn't possibly do it justice even if you gave me two hours um there are lots of different books books like this one universal principles of design uh this one don't make me think a classic in web usability design uh there are many many many books right these are just two that I picked that happen to sort of fit in um but I'm not specifically advocating these two as better or worse than anything else uh the important point is there's lots of literature out there right literature that's available in conventional bookstores you don't have to go find a research monograph you can just go find established condensed knowledge that can help you get started solving your problem um and you know we we don't have that in machine learning design um so in that big c of condensed knowledge uh I've pulled out one sort of meta theme that I think is important to examine in the context of Designing machine learning products because that's where I think things get a little bit different um so this meta theme is uh what's usually called mental model alignment uh and the idea is that when uh when a user is presented with some artifact they might try to use you know a web page a software application whatever right consumer electronics device there are three models involved there there's What's called the implementation model which is the nuts and bolts of the actual machine right it's the code if it's software it's the gears in the car it's the you know it's the the physical or digital stuff that actually makes the experience exist at all so that's the implementation model there's the presentation model which is how that physical or digital artifact is presented to the user uh and the presentation model I I like to think of it as uh being an an invitation to use it in a particular way right it's the presentation model says hey you I can help you do XYZ if you're interested in XYZ perhaps you would like to to try try me out right um so the presentation model is is about how a device advertises to potential users how they might use it right and then there's the user's mental model uh and the user's mental model need not be related at all to the other two right the user's mental model is how the user comes in with expectations uh and prior beliefs thinking I think that it should work sort of like this or it kind of looks like this other thing that I used maybe it'll work the same way or I have no idea how this thing works and I'm not even going to try to figure it out right like these are all examples of user mental models um and I would say that 80% of of uh the principles that I found in the design literature stem from this desire to have the presentation model match the user mental model as much as possible right um even if it's very hard to do so um and you know for a long time for example with word processors it was hard to technologically create the fiction that you have a sheet of paper on your desktop and it doesn't go away when you close the application that it's transparently saving in the background and even if the power goes out your document isn't lost um you know there's often lots of engineering that goes into creating a presentation model that matches the user mental model uh but it's very important to do this work even if it's very hard um so okay that's the meta theme that I claim gets twisted up a little bit when you start thinking about machine learning products uh because there are some unique characteristics of machine learning products and and products that incorporate machine learning features that uh I think are are worth examining uh in terms of the relationship of the user's mental model to the Design's uh presentation model um so at the end of the day a machine learning product or feature is full of what I'll call surprise and delegation uh and these are sort of you know abstract placeholder names but hopefully you'll you'll get a sense of what I mean um with Mach and um let let me back up for a moment I'll get I will elaborate on what I mean by surprise and delegation in a moment uh first let me uh make my my main claim which is that with machine learning products model alignment between the user's mental model and the presentation model isn't just hard sometimes they're actively intention and I think that this is something that doesn't tend to happen with other kinds of products and the reason why they're actively intention is because when a user goes to uh an artifact that's equipped with some kind of learning capability the user is essentially asking to be surprised right the user doesn't go to the machine saying tell me something I already know the user goes to the machine saying tell me something I don't know something that's interesting or valuable or surprising right uh the user is asking uh in the act of using the machine learning artifact uh the user is implicitly asking to be challenged at least a little bit right um so you know I mean I think it's worth thinking about the fact that uh the user is coming in with the you know in the frame of mind of I need to be told something or educated somehow or informed how do I how do I balance the need to potentially update the user's mental model with the fact that in order to get them to use it at all I already have to be pretty close to their mental model right um so let's elaborate on that uh just a little bit more um surprises fundamentally challenge mental models right like you might even Define surprise as something that CH challenges your mental model um so you know a user comes to whatever say um stock recommendation uh application that uses some kind of machine learning pattern matching whatever the user is coming to the device saying tell me something I don't know so that that's important right but it better not be stupid right and what do I mean by that um let me give you this this metaphor that I found really useful in thinking about how machine learning products and features should behave so think about these uh you've all seen these like beach Hunter uh metal detector things right so imagine you're a user who's just bought one of these things as a treasure detecting device but you have no concept of of magnetism you don't actually know that what it's doing is having you know having an electromagnetic interaction with a piece of metal right so you're walking along the beach and you're using your thing and it it pings and you dig and you find some buried treasure in a chest and you say great the thing works I'm really happy right it told me something that I didn't know and that was valuable information but then you walk you know 2 m further down in the sand and it pings again and you dig and you find you know old beer bottle caps right um if you don't have some pre-existing knowledge of how magnetism works you'll say well this I mean this is not treasure I'm really confused now because the first time it found me treasure and the second time it found me garbage right so you know how how can I understand this device that sometimes gives me treasure and sometimes gives me garbage when I don't have the you know the physics knowledge to understand the metal detector or the mathematics knowledge to understand a machine learning device um right so that's how surprise can challenge a user's mental model um so let me walk through a couple of anecdotes here uh I was working uh doing doing some client work um a imagine a company that sells some um some physical widgets online they track transactions as they come in in real time they're worried about chargebacks and you know whatever the usual things that you're worried about as a as an e-commerce retailer they wanted a machine Learning System to flag transactions as being you know reliably of high quality or maybe a charge whatever right they they had several different pathologies that they wanted to be able to diagnose um so you know we built them a machine Learning System uh and it found a very strong correlation with a particular column having a value that was supposed to be impossible right so you know they thought that it could only be from 1 to 10 and it was 50 or something right um and this caused a lot of initial distrust because they said they you know look the the database schema says it can never be higher than 10 the the software is stupid right um and this is a a completely natural response even if it's frustrating for us as designers this is a completely natural response um and they later found out that the the statistical calculation was completely correct it was actually due to an upstream change from a third party data provider uh and it was actually providing very valuable signal that they then used to detect uh good versus bad transactions so this is an example where not I mean this was not finding bottle caps right this was finding treasure but the user thought it was finding bottle caps um so anyway that's one anecdote here's another one this is a little bit just slightly more technical um if you designing a product that somehow relies on a covariance matrix run away um covariance cannot be mapped onto a common person's mental model at all I've tried for years it just doesn't work um so we were working on a project to do some price estimation based on know whatever tracking some features from from text descriptions multivariant regression uh a non- analyst end user who wants to be able to slot in additional columns into the prediction system uh and those of you who are familiar with multivariant regression know that if you add in more columns the coefficients of the previous columns can change right this is just what happens with covariance matrices uh but users are completely reasonably perplexed by this right because their mental model of the coefficient is how important is this variable right so how you know and within that mental model is it Reon able that adding more information makes the other thing somehow less important uh yeah I mean you know to us we understand how that works right but many you know I think it's perfectly fair for someone to say that doesn't make any sense to me don't Design Systems like this for me this is stupid right um so anyway something to consider um yeah I have one quick question not lose on topic so that was regression where you have VAR right what if you addas model and for or you know Advance where even worse yeah because sometime variable can go up and down depending on the so if you look at uh UI research on for example chunking and people's ability to understand recursion people can understand you know there like there are the two heuristics right there's the five plus or minus two items that you can keep in short-term memory and then there's the uh one or two levels of nesting that most people are com comfortable in interacting with if you give them a decision tree with seven levels they just tune out I mean they may say they understand it but they won't understand it but what if I don't show them a tree I mean this is a real problem which we are trying to present I show them the top seven important variables and that might change depending on the data right so you need to come up with a way to report it so that the variables are stable and you have to change the mathematics to do that you can't use the mathematics that comes out of a textbook because the mathematics that comes out of a textbook tells you those coefficients need to change you need to find a different way to quantify and report that information so that it's stable because if you I mean if you report unstable information to your user your user is reasonably going to say this guy doesn't know what he's talking about or this software doesn't know what it's talking about it changes its mind every five minutes right I believe we had a question over here as well it's a little one extra layer so what happen talking yes expain I I I have cut out a slide about uh very very bad behavior that users uh undertake with uh how they use confidence intervals I I'll tell you afterward anyway uh people also very bad behavior very very bad um okay so uh I talked to you about surprise let me talk to you now about what I mean by delegation um and I I you know I claim that impactful machine learning invites delegation right that if you if you've designed something that's not useless then presumably it's going to change someone's Behavior because that's the nature of what what these things do right they give advice they give guidance they say do this not that this is better this this is worse whatever right um so so either you've designed something that is useless or perceived to be useless in which case you know bad luck but doesn't doesn't really matter or you've designed something where You' suckered someone into trusting it and now you're affecting people's behavior right um and you are you've created a machine that makes de facto value and policy judgments right and you know this is this is something that has been explored uh by the New York Times you've probably seen this article about can an algorithm be racist uh about uh creditworthiness and uh I can't remember what the other anyway good article you should read it um anyway my claim is that this invitation to delegation and the implicit value judgments that are being made by these systems it's actually different from conventional automation that when a user comes to Conventional automation their mental model I think is is something like okay this is a computer there are some rules someone put the rules into the computer and then the computer runs the rules maybe the rules are stupid but then I take it up with the person who put the stupid rules in the computer right um here you have something different because you don't have a human who's putting in the judgments right the human is putting in some mechanism for creating judgments but the the judgments are coming from the machine and from the data uh and I claim that this is really is a different kind of thing from the perspective of Designing a human experience uh okay so uh here's another anecdote uh so we're working on a predictive sales lead prioritization system um the you know you have a thousand sales leads you want to know which ones you want to call today um and there's information like what sector is this potential client in uh this NL system rightfully detects that a particular sector is extremely unprofitable in terms of salesman hours versus uh Revenue brought in so it pushes everything pushes all of those down to the uh bottom of the queue uh at the end of the month when they do their diagnostic reporting they say holy cow we didn't sell anything to you know Healthcare or whatever the sector was right um and you know the salesperson gets an angry call from his boss and then I get an angry call from the salesman uh and then I have to explain like no look like it's actually really unprofitable and the salesperson says you know you're right but I can't take that to my boss right my boss is not going to accept me telling him that the machine told me that it's not profitable so I Didn't Do It um so you know uh take of that take from that what you will but I think you should take something because there's there's something important there that has not been I think has not been deeply explored by other branches of UI and ux design and it's something that we should explore sometimes season I do exactly the same work I me we look at accounts in so sometime challenges okay let's say em one particular Marketing in Europe there's not much value but you might want to still consider it because in future you might generate so it's a investment risk you taking same applies to one of the if I may if I may draw a cartoon outline around what you're saying uh this is consultant mindset not product mindset right Consultants can debate and explain and educate products can't right like you you can educate a little bit in a product but you can't you can't try to have a nuanced conversation with a user in a product right um and so much of how machine learning has come out to commercial application has come from you know an academic mindset and a Consulting mindset uh and I think we really need to like we need to recognize that there's a huge cultural bias there and we need to try to work against it because with that mindset I think we won't design good products um like you you can't rely on telling them look this is like eating your vegetables you'll understand someday and you'll thank me it it just doesn't work in products um oh so let me move along because I'm a little behind schedule um um so here's the important Point none of these people I've told you about are stupid even though they caused me great frustration none of these people are stupid this is what a real user's mental model looks like out in the wild right now this is in 2016 what you're facing if you're designing a machine Learning System that you expect to be used by by nonsp Specialists right um and you know it's tempting to think that they're stupid um but at the end of the day it is mean it is wrong and it's useless right just throwing up your hands and saying these people are stupid well okay suppose I believe you now what right like we still have a product to ship so you know good job like identifying the idiots but we still need to do something right um and you know like in really in addition to being mean and wrong it really is useless so you need something better than that um and you know if I can try to condense um sort of the opposite of that like what what I think today's users have to the extent they have mental models at all uh it's these two things right small harmless predictable Surprises by that I mean you tell me something that I don't know but it looks like something that I already know right so if I know for example that uh college students particularly like buying my product then if a machine Learning System tells me oh hey like uh you know adults who do online adult education also like your product maybe that's a bit of a surprise but it's a predictable surprise right it's like oh okay well it looks like a thing I've seen before so that's cool like I thank you machine I trust you because this already fits into the mold that I have so great right so that's what I mean by a small harmless predictable surprise um and then the other is don't do anything I wouldn't do right uh and that that goes back to the salesperson with the you know the machine deprioritized all these leads and got him in trouble with his boss okay so we have all these problems uh we don't have an established body of knowledge so where where should we start looking right um so uh there are many many different ways you could try to approach this right um there is some limited directly Ono design literature it's almost all from Don Norman author of the Design of Everyday Things he has this book called the design of future things uh so you can read that but it's very abstract and doesn't have many many concrete conclusions around recommendations uh however there is a rich adjacent literature in Psychology people who study the perception of uncertainty the perception of risk the perception of chance uh and there is lots of stuff to mine there uh if you're willing to do a little bit of work to you know adopt it to your own domain so I'll put this here for a moment this is someone you don't know but you really really should know uh this is uh Professor Garett gigerenzer of the max plank Institute he's a professor of psychology um he does lots of work on how humans make practical decisions and why that why their mechanisms tend to be close to Optimal in the context where they work right and so I like this is not his quote I'm I'm sort of boiling down like literally hundreds of papers into six words here or whatever but his main message is that humans are not stupid they're just very contextual um and you know so he's explored different decision characteristics that people use studied how people evaluate evidence studied how people manage risk uh how they evaluate probability so if you only read one person's work it's this guy um really and he has lots of very accessible books in addition to his research papers so that's one option uh let me give you an example of something that I got that I thought was interesting from his work uh so you may not know that so we're all familiar now with probabilistic weather forecasting right it says you know 82% chance of rain tomorrow um it turns out there was there's actually a cultural Gap that probabilistic weather statements came to the United States almost 20 years before they came to Europe based partly on gigerenzer's work and his colleagues uh they did user psychology research and asked them if I tell you that there's an 80% chance of rain tomorrow what does that mean to you uh and they got a huge spread of answers uh ranging from it's going to rain on 80% of the land to it's going to rain for 80% of the time to if you ask 100 weathermen 80% will say it will definitely rain to anyway like you should read his work really it's I mean it's it's valuable it's entertaining you should read it um so anyway um one of the messages that I got out of that that was valuable was uh making relative likelihood uh s uh relative likelihood claims and action suggestions rather than trying to St State something about some abstract truth so instead of saying there's exactly an 82% chance of rain instead say it's 20% more likely than average and you should take an umbrella because you know you wouldn't want to be caught out right um and you know I've applied this almost that exact thing in in designs that I've done it's really very valuable and there's a ton more stuff that you can learn from him um there are right okay so I had this is almost exactly the the 80% situ ation uh predictive sales application salesperson gets a batch of leads they're rated at 80% chance of conversion and then they call me and complain well only three out of the five converted and that's 60% not 80% right and you know people don't have uh people don't have any intuitive understanding of small data set variants and you know you have to design with that in mind uh I'm going to pick up the pace a little bit because I want to leave a few minutes at the end um so there are many many others worth reading uh Conan and tersi on rational heuristics arieli who has work that directly conflicts with gigar renzer and so that that's an interesting combination to to examine uh this guy gell who's looked at in intuitive probability assessments many many more we can talk about this afterward but uh there are many so let me talk about some concrete suggestions things that I think are you know in the direction of design principles that we can use in machine learning products uh and there are three partially overlapping principles design for audit don't prescribe persuade and limit your surprises to the content not to the Behavior Uh so design for AUD what does that mean it means you make why did this happen completely obvious so Netflix does this for example this item was recommended to you because you watched such and such movie right uh Gmail does some similar things um so this serves two purposes one is that if the user sees a surprise that they don't like they can say oh okay I understand procedurally how it came to me so it's demystified right it's not like you know lightning struck me from from the heavens and now what you know how do I not have this happen to me again uh it also gives them a way to reason about their own future in using the device right it makes it easy for them to speculate what else might happen if this is how I got this recommendation what other recommendations might I get you know good or bad um and this is often in conflict with using better algorithms because you know I can't in all honesty give someone a good explanation for the why behind how a neural network makes makes a recommendation right like I can walk them through the mouth but that's that's completely useless um so this is something serious to think seriously about uh in what kinds of algorithms you choose for making recommendations to people um and you know often often this human human question is in conflict with the larger performance goal of let me get the absolute best statistical model I can um I'm going to believe this because this is a serious Point don't just just pay lip service to this principle it's very easy to cheat right it's easy to think that if you put a little piece of text that says our statistical model estimated that you know this piece of real estate is worth such and such dollars per square foot uh you may know what application I'm talking about here um this this is not anywhere in the user mental model just stop it right like this is not an explanation this is dodging an explanation this is saying magic like you know elves did it right um and you should seriously think about if you can't provide auditability maybe you need to rethink what you're doing maybe you need to rethink the interface between the value you're trying to provide and and how people are going to consume it um because you know like at the end of the day no one wants to be using mysterious devices right um and and we would like to think that we're designing things that are not mysterious so uh if you find a situation where you're compelled to put mystery into your device then you know maybe that's a sign that that something else is wrong right um so don't prescribe persu uh this is a compliment to to auditability if auditing is why did it happen persuasion is why should you trust it right uh good persuasion takes the form of clearly interpretable facts that the user will perceive as pointing to a conclusion so for example if you recommend to someone we think you'll like this movie because 50 of your Facebook friends like this movie very much the user may agree or or not but you're making a goodfaith attempt to persuade the user the user can say yeah okay that's a reasonable argument like I may not agree but that's a reasonable argument right whereas if you try to prescribe something to them hey watch this movie um without an attempt to persuade it's it's ultimately it's dehumanizing right it's saying look I know better than you you don't need to worry you're pretty little head about how this happened just do it right um so right persuasion is about inviting the user to agree with what the machine has learned um okay let me pick up the p a little a little bit more um so right the final principle limit your surprises to content not to behavior what I mean is if you think about machine learning as content-driven adaptation right where the content is the data and the behavior is the actions that you recommend um surprising content is usually okay I mean there are caveat right you can make people you know there are things that they might not understand and reject as outside their mental model but it's usually not catastrophic surprising behavior is not okay right uh and you know one example of the surprising Behavior was you know deprioritizing all those leads that you know some of them should have been should have been addressed in order to satisfy the boss's demands right but you know where you draw the line well It's Tricky um uh I am going to all I'm going to say here is that one principle I found useful is that content ends where consequences begin so giving people facts and Analysis that's fine I call that content as soon as you say now do this you're you're stepping over into behavior um and uh to respect the the conference timetable I'm going to plow through these pretty quickly um right okay uh we we can talk about this uh a little bit more afterward if you uh if you are so inclined so let's wrap up um let's come back to the initial idea that mental models are a you know a big theme in design thinking and they they tie together many different design principles um and that I'm arguing to you that for machine learning um user mental models are roughly where mental models of desktop operating systems were in the 1980s right uh people are just not yet comfortable with the kinds of things that they're going to need to be comfortable with and so you need to design in a way that that gives them a smooth on-ramp um and you know there are more and more non-data people using ml powered products daily and this is really good right this is it's an exciting time to be alive if you're interested in the kinds of stuff that we're interested in so uh let's not ruin it by failing to do these things to actually come up with domain adapted design principles that we can share and debate um to have an honest and empathic view of what a real user mental model is and not to to resist the they're just too dumb kind of thinking um and you know to over over the course of years I guess put together professional practices that go beyond a single person reporting personal anecdotes to a bunch of other people hopefully someday we'll have something more codified and organized that we can pass on to the next generation of designers uh and with that I am done minutes two minutes okay so we can take a couple of questions I guess or on okay oh okay sorry just so wonder what you think right it's BEC more multi the algorithms are definitely becoming more and more complex you could offer users a lot of flexibility models and parameters and how to but one of your slides was all about simplicity so yes how do you balance between giving someone really something that the most accurate state of the art performing complex crazy deep learning multi-layer model and lot of flexibility with giving them something that does fit their mental model that you can explain them if you think the tree is a problem right that is really pushing well it's not my opinion that trees are models this is well [Laughter] documented um so uh let me answer your question with uh kind of a spin on an anecdote you're probably familiar with the Netflix prize my summary of what happened with the Netflix prize is that Netflix burned $2 million of prize money something like 10,000 hours of PhD level labor two years worth of of development delays in order to beat linear regression by 10% on root mean squared error which doesn't even correlate that well with user satisfaction um so you know and in the end they didn't even deploy the the restricted boltman machine right so uh I think it's it's useful to remember that when you're trying to evaluate trade-offs between uh simple and and complex models um the fact that we can put a number on better performance doesn't mean that it's actually a better model right uh just because the neural network achieves a lower rmse you know sometimes that means yes like go use it and that incremental performance is really important um and you know sometimes it's just not worth it the you know the human cost is too high I mean that's an overly dramatic way to say it but uh it's not that far from the [Applause] truth