Scale By The Bay 2021 : Adarsa Sivaprasad, Continous integration of ML products and UX design
i want to talk about so we've been hearing um a lot about i know a product specific to how we build for scale and uh what are the uh different methodologies and what are the you know both the research side and from the product building side what is happening in that spectrum so um i'm trying to take a step back here and uh look at it from a broader perspective of what are the design decisions that we are taking here and what are the decision um what are how these decisions are affecting uh long-term uh product life cycle as well as um larger topics i know um so in the um last decade um it was uh companies which calls themselves data centric or completely data-driven companies that was putting out ai or machine learning solutions um on a very regular basis and as part of that product but now um with the advancements in ai and also um more and more uh products becoming uh going through the digital transformation journey ml and deep learning algorithms are increasingly being integrated into the products so um the conventional product design also needs to um adapt to others data centric product design if we look in the last decade concentrated on data warehousing scalability of pipelines for big data um or more recently interpretation through kpis dashboards and more recently like right now we are in a stage where uh from this uh level of uh dashboards and uh interpretations uh making uh business interpretations out of data we are um talking about model explainability um data privacy and data democratization at the same time there is also a need to look into aspects such as biases within the business from the business point of view as well as users um who which need to be built in by design into the product so that's the attempt uh that i would uh do in the in the talk uh so before starting uh this looks at two major aspects one is what is machine learning and uh what would that mean on a product level and at scale and the second thing is how we traditionally or currently we look at user experience so um tom mitchell gave the definition for machine learning as a computer program he said to learn from an experience with respect to safe and experience e with respect to some class of task t um if it performs a task t based on a particular performance measure and improves uh the experience e um [Music] this is a very mathematical uh formal definition but the the idea is that uh basic idea is that it learns from experience it empathizes with the user or the people who are interacting with the system and it improves over uh time based on some performance metrics right so um like from the mathematical point of view uh even in uh deep learning or neural networks um things like i mean um there are theorems like universal approximation theorem which theoretically validates that this is possible and uh so when you are putting a system to learn and uh on a wide variety of interesting data sets and functions uh how do how do we make sure that uh or track what the system is learning similarly um on the user experience um domain i mean taking these two domains are as two separate things uh user experience uh talk about uh the now uh what what is the experience or the aspects of a person's experience when they interact with the system so on on both these tasks uh also in current uh framework our user experience deals with the research around the product uh prototyping um or creating uh information architecture communicating effectively what is the interaction and works mostly as a separate entity in the outs on the product design point of view but if we only looking at the two definitions of the um of ux and ml a few things are common so uh experience how do we learn from the experience how do we empathize with the experience that we that the system is being exposed to and obviously um the human centric part of it right um understanding what a feedback means uh when uh when does a person like a product or does not like a product um and uh so the um i'll try to touch upon all these three aspects um as we go along um so software uh life cycle uh whether it is like whatever be the um agile or whatever workflow that we are um using are typically um has a approach as shown right so uh there is a need uh to build and um so it goes through a design sprint uh where uh it it is uh looked at from how what are the interaction design aspects of it uh or what is the user experience uh sorry user interface aspects of this and a product team that sits between a development and the user exp the design spring is the only communication so the requirements are given or handed over to our development team that works on developing it and even say there are still few systems that evolve over time like some data products that does you know aggregation of uh certain matrices or telemetry they um there is a loop uh going on there but it would still be a loop that stays within the development so only closing of the loop that happens here would be a learning through the learning if we decide to add an additional feature into the product but uh on a traditional software system uh wouldn't fail once it is deployed as long as the requirement of whatever was given into the product does not change but that's not the case with the machine learning system so as the the model is built for a specific data set and the underlying uh characteristics of the data might change uh the model and uh suddenly one day it wouldn't work or um there is typically a test case um failure would not flag uh or passing all the test cases that does not necessarily mean that the ml model would stay in prediction so um looking at that more in in a ml life cycle um there are probably like two phases where there is a poc phase where the business understanding and data understanding is a closed loop where when you are when we are trying to formalize the use case that uh is being worked upon and then there is a model development sorry data preparation model development and evaluation stage which again forms a loop of iterations of improving upon the model so until um recently modeling was completely something a data scientist or an ml engineer um more typically a data scientist would do the initial ml frameworks um say um like even even uh theano or cafe were designed for data scientists ml frameworks are now evolving uh towards more uh like uh taking the example of keras or python they are uh becoming more um user friendly to um user um two software engineers so but um all the the other aspect of it is um the um ml as a service or ai as a service has also made a lot of difference in the way these systems are being used in a traditional software engineering project so the other aspect of uh this is um the automation of the solutions automl is also i mean apart from the usage and integration of ml solutions into a typical into a normal software development lifecycle uh there is also uh the uh automation of mlc auto ml platforms that are being integrated right from say continuous development stage to the deployment mlaps everything is catching and uh even decisions like um whether a human is required uh if for example uh um taking the example of self-driving car you know uh so whether a human uh is needed or not is also a decision we are leaving the uh letting the system uh take uh and yeah so i i want to go into a little bit more depth of what could be the different possible um things that the you know the ml model is being affected by certain decisions that we are taking at the design level now um this this would be going you know a little bit more into the diagram that we looked at earlier um typically recommendation is a very um most commonly or most wanted integrated tool into most of the projects hence this example uh the idea of idea is to seek to predict a user's preference now um the data required for a recommended system could be coming um from different places and the need for a recommendation is also a question so is there a problem that can be solved would be the first question here and say uh take the example of a content uh recommendation uh platform so uh increasing the time on the app and uh to determine that that's the kpi that you want to work with is a decision uh decision making uh at a product level now in the proof of concept stage uh where which involves data data set preparation or the exploratory analysis uh the a product owner would be involved but maybe not a lot of decision design decisions are being taken there but coming to the implementation uh design point of view um a lot of in if we uh consider this in a traditional um sdlc uh phase uh the implementation design uh would be a completely technical decision but there are um aspects of uh front-end design like uh say collecting the feedback from the user uh and how would that feedback improve your model over the time so that's definitely one decision where the ux and ml definitely has to work closer and probably not something that can be written prior to you know building the solution like it cannot be written on a design document and then go into development and never go back and check check on it the other uh thing would be once deployed into the production uh the performance because ml models require continuous performance evaluation uh even uh certain aspects of say a bias observed in the recommendation need to be fixed these are not something that you fix only at a modeling level but also from the right from the design point of view i mean from the design phase of it so going back to what we uh talked about experience empathy and human uh being human-centric um what would be so uh if i take the example of uh when we stick to the example of a recommender system um what are the different aspects of which comes here right so error the human uh mind reacts to uh an error um in in different ways so um what i mean here is um misunderstanding of actions uh a very good an example uh famous example is when an enter key and a return key are placed close by and somebody clicks on um every time somebody clicks on the enter key they mistakingly also click on the return button so the user could interpret a situation like this as their incapability to use the system whereas this is a design fault similarly in a recommendation when somebody is served a recommendation i and which is the uh one which comes on as you know recommendation one two three four so it's it's on the uh person who uses or interacts with the system their interpretation as to okay yeah i i am getting this recommendation because i have viewed certain things in the past so this is an error that you i mean if if that is not the case it is an error that you are communicating to the user in the same same situation the other aspect is the feedback collection so typically in recommender systems there could be an implicit feedback and an explicit feedback so what i mean by implicit is uh things that in the back can be derived based on how the user is interacting uh things like uh average so in the content recommendation uh number of users who have interacted with a content time spent on a content um history of the content that the user has used in the past um used um so these are the implicit feedback and we might use these matrices to build our model the other thing is not changed is it only this one slide or is it getting changed okay is [Music] yeah oh okay okay i didn't realize it uh but okay so i think i'll continue from here so um so i was talking about the error um and talking about is it moving now like when i'm flipping between the slides what is this oh yeah it's moving now okay it's moving on and we are on the second slide the error implicit feedback an explicit feedback slide sure thanks uh so on the um explicit um yeah the explicit feedback would be uh things like uh rating uh that we get from a user or uh us ask um explicitly whether this particular recommendation is good or not no so these are uh again um factors that would be um depending on what is being shown to the user the other large scope of things that comes under this is the biases so i have written few kinds of biases here implicit bias um so implicit bias would be uh that which occurs uh when assumptions are made based on um once uh say mental models or personal experience as to how this would apply so say a confirmation bias is an example of um implicit bias so where the model builder unconsciously uh say processes the data in a way that affirms their own uh pre-existing beliefs or hypothesis so um it's it's it there we have seen uh cases where a model builder has actually kept training a model until it produces a result that aligns with their own um hypothesis sorry this is not a problem specifically to machine learning or software engineering but even in other areas where uh experiments are done ah the other reason could i mean the other bias another bias could be the automation bias so a tendency to favor um so from a user who is interacting with the system uh there would be a tendency to favor results generated by automatic system so as a user i'm assuming uh that the recommender is giving me a result which is most active to me irrespective of uh because the user doesn't have a visibility into what is the error or the accuracy of the system uh they uh are unaware uh that this need not uh the the results that that is being shown to them i need not uh show exactly match the their uh history or tendencies um [Music] the other uh thing could be uh so presentation bias is something um for many years um it is being dealt in the ui ux area where things like a position of the content where was the content placed on the user screen um what was when was it recommended like is it at the end of viewing another content or is it as part of a collection um so that's that that's one uh it could also be um like metadata of the content which is being displayed on the scene uh what i mean is um say um movie recommended system um are we also showing along with the content then is are you showing the name of the actor or uh name of the director or um is there um a familiarity or appeal to the icon that is being shown on the screen along with the recommendation um then um there could also be things like whether uh this say a recommended content is already available in the device and whether the user is made aware of it you know where pop notifications are shown to the user our reminders are given to a user who who has a system say an app installed in their mobile phone or through their email so these are uh biases uh presentation biases which are going to influence the data or the implicit feedback that we collect and interpret or train the model going ahead the other aspect is the um generalizability so um when we uh talk about i mean this this is a term which which is mostly used in the context of machine learning uh as uh like generalization uh would be your model's ability to adapt properly to uh unknown new or previously unseen data drawn from the same distribution as whatever is your training set so there are uh assumptions about the training and test data that is inherently fed into the generalization while uh while there are today in say computer vision or in nlp models that are highly uh generalizable uh the kind of uh biases um both from you know technical uh bias variance uh the uh the spectrum of uh overfitting uh from that bias as well as the biases user bias that we talked about from a ux point of view both these things add on and becomes multiplied gives a multiplied effect when it comes to aspect of generalization when you are presenting those results of the model into the to the user so overall uh looking at all some of the some of these uh factors that we talked about um the ability of a model uh to fit into a variety of function uh which is basically the capacity or um and in effect the scale because uh you would be able to scale your product based on if it's an uh if you are using an ml model inside so the capacity of the model would determine the uh uh scale to which the same product could be used for so model with low capacity will uh struggle to fit into a larger user group and vice versa so relationship between capacity generalization and all the different aspects of you know the collecting data collection through feedback and the error the user introduced and the modeling introduced error are definitely things that are going to be there when we work on building a product around any ml uh solutions um the other thing is um the auto ml so with ai pipelines that are being automated the aspect of design is also inherently in built in built by whoever is designing the auto ml platform so it again makes the visibility of the errors and biases that we have mentioned built in into the system so in this um i mean to summarize what we have talked about so far there is a difference between um i mean or there is a need for the software system development uh to you know go hand in hand i mean or to slowly evolve into ml system development and while uh currently in bc uh like there are uh products which are completely data driven and data centric density data driven but more as data centric and then there are a much larger community of you know domains and products that are going through the digital transformation uh journey and being uh using um going to use ml solutions and come out as product so from a design thinking point of view um this need to be addressed and we need to look at different aspects so um like we spoke continuous integration the development practices which developers integrate code into say a shared repository and we are automating even that process would um will i mean yeah this uh continuous integration or continuous um delivery testing monitoring all these aspects uh would mean a different uh slightly different maybe integration might still remain the same whereas something like a continuous testing of the model because that as the data changes the model version in the production in production changes so uh continuous testing of models um and continuous monitoring would be to areas that definitely needs to be built in into the product cycle along with existing paradigms such as integration and deployment so um to sum up what we spoke so far consumer facing applications have um long embraced practice of you know catering as closely as possible to the native intuition but um the the design thinking is um you know the lack of design thinking or the need for design thinking is um not becoming more and more noticeable in the landscape of developer facing tools um so these including um while earlier there was a bias towards say compilers or programming languages um currently um using a human centric uh framework to uh leverage um the aspects of different uh you know um paradigms of um software development will help connect existing developer workflows into a more seamless experience for um being adapted to the ai solutions ai or ml solutions so that's it for mine you