data.bythebay.io: James Mulholland, Remixing Design with Data
Recording: data.bythebay.io: James Mulholland, Remixing Design with Data
All right, welcome back from the break. Uh, our next speaker is uh, James Moland. Uh James is a manager of uh UX design and research team at uh Platfora and uh what he does and what platform does is making big data accessible to regular users. Uh James has a truly impressive number of patents in the area of UIUX. uh he has a master's degree in human computer interaction and his talk today is going to be on remixing design with data. So please welcome James. [Applause] All right. Uh thanks everyone
Um excited to be speaking here. Uh the convergence of design and data uh is a fascinating topic for us uh especially working on a data product. Um, but as uh as Michael said, I'm James. I currently lead the design and research teams uh at Platfora. I'm here to talk today about how we at Platfora make the process of developing the product more data driven. Uh so to give you a little bit of context to your work, I'm going to give you a quick rundown of what we do at Platfora and then I'll jump right in and show you some of the research uh that we've been recently focusing on. Um so this is uh a shot of Platform. platform is a big data discovery platform that transforms raw data in Hadoop into interactive in-memory business intelligence and behavioral analytics and this is all within a single UI and in the same platform
Uh once the data is actually stored in uh big data uh repositories like Hadoop, uh users can manage data sets and do all of their data preparation interface that looks kind of like this and then uh perform visual analysis uh in the platform or export to maybe some of your other favorite tools. Uh typically we concern ourselves with the very pressing issue uh of UX and data today which Alexi mentioned uh earlier which is figuring out how products can really enable access to data as it makes its way to the screen in a form that people can use. Uh but today I really want to focus on the uh the process itself of being data driven in product development. So when designing a product, our goal is to ensure that the product is functional, accessible, and delightful. Of course, we believe that sound decision-making uh is important at all stages of the product development and being a data company, we are exposed to these ideas daily and we need to design our product to enable others to also be data driven themselves. So it only makes sense that we have a data data driven process for ourselves, right? So let's take a look at the design thinking process uh to see where it makes sense to be datadriven. Design thinking was developed originally at the Stanford D school but originally popularized by uh IDO and consultancies working in design. Uh the process of design thinking is broken down into these five steps
uh empathize with your users, define the problem, ideulate the solutions, prototype from those ideas, and ultimately test your final product. At each stage, it's critical to ask the right questions to evaluate your design work. So, when you're empathizing, you need to ask, who are you really solving for? When you're defining the problem, you need to figure out what problems are worth solving. When you're ideulating, what ideas should we choose from? And when you prototype, is this idea working as we expected? And ultimately, when you test the product, you need to know which enhancements you need to do in order to improve the implementation. So when you think of using data in product design, what do you think of? Probably AB testing, right? That's a pretty common example. It's pretty well studied. There's a lot of products and services that support this process. And it's a fairly straightforward result, right? You compare one implementation to another and you can evaluate the two
It can be a very useful tool, but ultimately it really answers the question, how do you improve the current implementation? So even with this in our toolkit, it's important to remember a lot of decisions that happen earlier in the pipeline. That is, a lot of decisions have been made to get to us to this point. So how many of those decisions have been validated with data? So let's see what happens as we walk backward through the process to earlier parts and look for more opportunities to be data driven. So during ideiation, you want to generate many ideas very very quickly. When you know the problem you're solving, you can come up with different ways to solve it, but you can only implement a tiny fraction of the ideas that you generate. Ask yourself, how are you making these decisions today? Do you have a method or are you just using your gut? There's definitely an opportunity here to be more data driven. So, how do we use data to figure out what to build? How can we choose between perhaps many competing ideas? Early on at Platora, we had a fairly ad hoc practice of talking to customers and their stakeholders to gather input from customers and to rank their priorities. uh those customers would rank each feature set uh as either gold, silver or bronze in a very short list
Uh but ultimately using this as data to really measure what we needed to do was fairly limited. It provided a singular metric that was oversimplified. Uh the rank lists they gave us were ultimately subjective and different for each company. So we really couldn't compare across companies to develop our product holistically to meet the different needs that we encountered. It only mentioned really their important issues. So inherent in the ranking was everything else that was not mentioned that they might consider valuable. And ultimately we are only getting customer input from a business perspective on what people would buy, not what the users want, what the users need, and what the users care about. So we learned about a popular survey method called the Kano model
Some of you may have heard of it. uh helps us pri uh prioritize each product enhancement and feature. Uh the method was invented in the 80s by Noryaki Kano uh or Kano I believe it's really pronounced uh sorry uh as and it's been popularized by product managers to balance feature road maps uh even more recently the Kano model has been adopted by many in the UX community uh for for similar reasons as as what we're doing. So uh here's how it works. You're kind of looking at this uh this chart up on the screen. maybe wondering what it's what it's trying to tell you. Uh, imagine you're a product team and you're trying to decide what to work on next. So, you have all these great ideas
What kind of features should we put in the product? What changes would you make to the product to bring the most value to your users? You pull together a list of great ideas uh for feature enhancements or usability fixes you might want to compare. Uh, and you create visual aids as necessary. So, here we have kind of a small little mockup as part of this survey. So this survey you send to users and you really ask them two questions for each feature or enhancement. Uh the first one you ask them a positive question is what the method calls it. Uh if the feature is present, how much more positive is the experience? So this is going to tell you how much delight is added to the product uh if you add this feature in. And then you ask a very very very similar but critically different negative question which asks how would you feel if the feature is not present? And from this we determine the level of frustration that is avoided by adding this feature. And so here are our results
Uh we ran this with a number of times actually at at Platform now. And uh a couple of our studies here you can see the results of these different features that we have. Um they're kind of encoded in the middle but each of these rows is a separate feature. And um in scoring these we can compare the metrics for each enhancement. The length of the bar on either side indicates a desire for change. The level of uh related uh the level related to frustration is indicated on the left and the amount of potential delight is shown on the right. And if we plot these in a two-dimensional view, we can see a pretty interesting picture. Now, the Kano model actually suggests ways of categorizing the features based on these relative ratings
I've kind of used my own words here to help describe more of the value that we get at Platfora from this study. Uh so if the um feature shows up kind of on the bottom and towards the right mostly dealing with uh frustration but not as much delight might consider that an MVP minimum viable product feature where it's truly expected that this feature is uh in the product but it's not necessarily going to be a point of innovation or uh bringing delight to to the customers. Uh attractive features, however, are somewhat the opposite in that users don't really expect them. Uh they make the uh experience of the product a lot more interesting and uh perhaps even fun. Uh certainly enjoyable, but um they're really not expected and they're not going to make or break a deal with a customer. Now, the blue area is really the area of um greatest importance. We call this the area of focus. This is where users will um indicate that a high amount of frustration is going to be avoided as well as uh actually have a very delightful experience and this is where the greatest opportunity to innovate is within that feature set
So if you have any uh resources extra resources to spend on features these are probably ones uh to invest in. So now this is not just a list of feature demands that you get from different sources. This is really a real quantifiable view of what each feature is going to bring to the table and the overall user value it provides. So in modern applications, it's important to balance delight and frustration in this way. And this gives us a model to manage those trade-offs. the multi-dimensional aspect of the Kano model helps us reveal those trade-offs um that you need to consider in order to sort of meet your experience or or product strategic goals. Uh by asking questions independently uh the the positive and the negative question and across uh different feature sets, we make it easy to compare across users and across their organizations. We learn concrete information about what users value, not just what is top of mind and what introduces a a strange bias
And of course, uh it's centered around user input, which is important. So using the KO model, we found a quick and accessible gauge of the value of any idea that helps us measure trade-offs. Time is of course a precious commodity and being datadriven here helps us invest our time wisely. So let's take a leap back all the way to the beginning of the process to empathize. Empathizing with your users is important to understand their real needs. But you have to make sure that empathizing brings you to the right users or your perception of their needs could be very skewed. Traditionally, only qualitative studies are used at this stage because humans have complex motivations that can be hard to uncover. Even at this stage, however, we can use quantitative data to help validate who we are really solving for
So, personas are a popular tool perhaps most easily described as like a user archetype. Uh they provide a common method of refining knowledge of users into a form that is easy to remember and to communicate. Early on at Platfora, we developed personas. These personas here from interviews with a large uh large user base. When we started in the market, there were almost no other tools uh enterprise tools at least on the market uh for big data. Uh our personas were created with interviews with existing users based on existing business intelligence uh workflows. So we have our persona Louise who shapes the data. Jeffrey and Mary Beth who serve up the data in different ways and different insights and Deborah is a busy VP consuming the dashboards as they come in
So a lot has happened since we first developed these personas. The market has evolved. Our product has evolved considerably and we've now implemented telemetry into every single click action in Platforma. So, we wanted to use this data that we're getting back to make sure that our personas were still relevant. So, using lowle low-level behavior patterns, we take the raw user interface clicks and develop a bottom-up datadriven approach to define and refine these personas so that they reflect a product's long-term use. So, here's what we did. Uh, we took clickstream data from user interactions. uh the click interactions um basically gathered automatically through nightly data dumps of telemetry u from organizations that have agreed to share this with us
Not every organization of course uh is is willing to do that for security reasons. And over the course of two years we aggregated across 1,200 different actions within the system uh three and a half million clicks uh 39,000 uh session uh individual sessions uh from a total of 2400 users. So the process works this way. We looked at all of the possible click streams kind of here. Each of these dots is a different session and we clustered them into common workflows and each of the click stream sessions sort of fits together as part of a a pattern. Here you can see the uh hierarchical tree output from a ward pair wise clustering. Uh so we had a data scientist on board who was helping us work through this analysis and each of these clusters then creates or identifies excuse me uh a different workflow set of activities uh or workflows um that we can model against. So here's what those workflows look like
There's 10 different groups here each a different color line. And in order to be able to digest the data uh as humans, we kind of clumped each of the interactions based on what a user was kind of doing at that time within the app. So for example, workflow five here is green and focuses primarily on data set creation. Uh workflow one is in red and focuses on data modification but also has a significant uh amount of dashboard and visual analysis uh activity. So here's a look at those same clusters broken down at the user level. Each user session is shown here in a vertical slice and the color uh intensity indicates how often they use that particular workflow. Finally, we used a statistical approach also with the help of our data scientists called a mixed model. So that actually sorts and clusters the users by their workflow similarity
This gives our users uh gives us excuse me the behavior patterns uh to match against our personas. So we found that uh these these personas that we identified resembled what we expected to see from our current personas for the most part. So this is Luis mostly focused on preparing data. We found Jeffrey who will process the data and build reports. Mary Beth who's using mostly the analytics related activities. Now it's worth mentioning at this point that uh all of these three users so far are commonly using the entire application. So this is from data preparation processing all the way through visual analytics. Uh which is not something we expected when we first set out uh defining our users originally
And whereas Deborah is a data consumer so she hardly touches the the areas of the app outside of the analytics interface. So we were able to spot these behaviors that generally matched the personas that we defined already. But in addition to evidence of our existing personas, we also discovered a fifth user persona. So one that favored behavioral analytics features and was even more involved in the data preparation. So from this we really wanted to start targeting this new type of user seeing this as a an opportunity for innovation within the product and we knew that this characteristic of using more of the earlier part of the pipeline in conjunction with the later part of the pipeline uh of visual analytics uh was really important. So since then we've actually invested heavily in both the behavioral analytics and the the full stack interactions uh that take you throughout the data pipeline. So we return uh one last time to look again at our datadriven process to help you make decisions. Uh there are many many methods uh research methods at your disposal
Right? Some of them are not even invented. They get created all of the time and we're familiar with research methods like AB testing and working with personas. that um many use data to evaluate the products towards the end of the design process. It's important however that to become truly data driven, we must also invest in using data not just at the end but also earlier in the the design process as well. The future of design is truly to be datadriven and working sidebyside with data experts, we can build better, more engaging products. Um, if you're interested in reading more about the two case studies that I presented here, uh, they were recently published in the proceedings of the recent KAI conference in San Jose and I'll be posting links uh, on Twitter and hopefully linking through the data by the Bay group to to get those out to people. uh if you have any stories or examples of your own work uh working with data scientists. I'm really curious to hear more about designers working with uh with data professionals
I've been hearing more and more as I've been talking to people uh other companies that are actually starting to sit designers and data scientists next to each other. And so I think there's a great opportunity to enhance how we're working together. Uh so feel free to share any of those uh stories that you do have. Um but that's it. Thank you. [Applause] Uh any questions? I think we have maybe two minutes. Interesting talk. One of the things I kind of wonder about the sort of identifying the users in the beginning there was they're responding to a to one product or one basically one interface
So sometimes I think when it's AB testing people are responding because of artifactual things. they don't see the button or they don't there's something artifactual about the the screen. I wonder if they're only getting exposure to one type of thing, maybe how much of that is potentially artifact or other things like how do you compare it to some other control case? That's a really good question actually. Um I think part of some of the control case might be um through the contextual observation that we had in the beginning in the first place, right? We tried to get a a wide breadth of user types and different behaviors with different tools. And so from that, we kind of created a picture in the original personas of what we expected. Now marrying that with what we've now discovered becomes kind of an interesting question because we know there's opportunity to better serve users that are using our product. What it perhaps doesn't tell you is going outside of that that you've effectively created a bubble. So designing within that bubble can be very very effective
Um but ultimately going back to probably more qualitative methods or exploring perhaps cross productduct uh telemetry or clickstream or some way to to analyze behavior across products would probably get us there in a better way. Any more questions? Thank you very much James. All right. Thank you.