data.bythebay.io: Aaron Kalb, Pro-Active: Designing a genuinely helpful SQL interface
Recording: data.bythebay.io: Aaron Kalb, Pro-Active: Designing a genuinely helpful SQL interface
you cool so proactive what do we mean by that well as a little pun so proactive according to rick shin area means acting in advance to deal with an expected a change of difficulty and a pro is an advanced user who as we all know hates any attempt by software trying to be helpful so you know when microsoft is kind enough to offer this help you know sort of Charlie Brown what they'll hear is just obnoxiousness right so we'll do the stuck in two halves for some talk about how being proactive can actually be helpful and then we'll talk about pitfalls to avoid this being how it's perceived so real quick who's in the room who here is a designer okay I got a couple people who is a developer I did a scientist okay it's just like the the valley majority of you guys maybe who bites sequel it has okay a lot of folks that's great so get a pretty diverse group any lawyers here's a lot of lawyers oh cool so I can defame everybody fantastic um who's got a driver's license okay good good number of you who remembers taking their driver's test okay keep your hands up if you think you'd get a hundred percent on the driver's test your ticket today oh very few left for key modulations and whose google maps okay hands up again cool so um so yeah it might give you introduction I did subastas at Stanford and then did a bunch of stuff at Apple missing on Syria but all around language and people and computers and i'm going to use both those bits of her experience to kind of go deep a little bit today so back on us at stanford i was in a lab with the late great cliff mass here on the top right and he was working with this car syndrome references stanley the second-generation car designed by sebastian Thurman and the team there is now doing autonomous cars at Google and Stanley was designed for urban driving and Sebastian's a sort of very future oriented sort of person who excited for the singularity and for you know cars all driving themselves and so am i I think about save a lot of lives in w faster and better clips and HCI and communications professor and so huge trying to figure out what do we do in the meantime right when we still have humans behind the driver's seat we've all its intelligence and all these great sensors and great AI how do we kind of have this work with humans and the dream was this would be an angel on your shoulder but it has perfect spatial attention and awareness and you know it'll keep from having an accident it'll help you optimize your gas mileage all this stuff right angel on your shoulder the other option is it could be you know your mother in the passenger seat all the time right gasping and every stop sign right and so how do we kind of get that balance of like backseat driver versus angel and so we tried a bunch of experiments with basically the when and the how of interventions right so you're at the wheel sensors are going it can tell if you're driving well or not and there's the Delta between what you're doing it would do how can it then communicate with you so we had things that were subtle like a little flashing red light on your window so trying to keep you from cutting people off in traffic right so I'm you know looking over here and then some into my blind spot I see you by fasching light okay i'll i'll stop right it could make a loud honking noise or a pleasant chime noise or it could be a haptic feedback the steering wheel could vibrate or my chair could vibrate right like some sort of video game and we did all these very much different things which is when to intervene you intervene far enough in advance to minimize damage but then people feel like I knew that stop leave me alone right so many different hypotheses we said maybe when you're signaling only then should we give feedback because then you're kind of asking for it right or we said what about the opposite what about if I'm messing around with the radio or the GPS and I'm drifting to another line that's when we buzzed right so anyway it was this is a got me interested in whole idea how do you help people without the attempt to help actually hurting them and making them worse case shut you off and you get none of the benefits so let's continue this metaphor the driving metaphor for this whole talk is going to be driving how does this driving stuff relate to consuming data so when you're behind the wheel of a car anyone remember back in the old days there are paper maps and hopefully they're in a language you can read and you use them to figure out where how gonna get me want to go but you know you know your goal how you're going to do it so you plan your route you figure out when you're gonna leave based on that route and then as you drive you're making a ton of decisions every moment right how fast to go what lane to be in which exit to take should have your high beams on you know you should you signal and you're trying to be quick and get to the right place and maybe be courteous and complain enough with a lot might get a ticket right and you should do a lotta systems that there once so um this is just like I would contend trying to write a query before you start you kind of need a map you should figure out okay which table contains the information that I want and which version of that table is actually the one that's up to date something we're going to be stay out and they're going to be production etc etc right and then as you're actually doing it you got to figure out okay I've got my base table what am I going to join with using what condition right what are we going to filter by which way should i calculate this metric right i can make the some dr said you know whatever this familiar people had this experience should i include hawaii in the set right maybe depends on whether i am on the finance team or the ops team may I include gender maybe that's not allowed if I'm building a you know credit score classify or as data product but it's required for my analytics to the to the government or two more getting something like that so so I think in search we're gonna put the promise of proactivity how can we design an interface simple how we do it on the road and how we could do it in the sequel contacts to help people with the planning and the execution to avoid mistakes and to get they want to go faster okay and then the second half we're going to say okay how do we do that such a way we don't go overboard and it backfires so um there's different ways to change behavior we're trying to do is get people to be better drivers in the kind of crazy highways of their sequel and other data systems and so one way that the powers that be sure to influence people's driving behavior is through the DMV right so for a hundred years the good people to DMV have been serving California and during that time they've written almost a hundred pages in a little more than hundred pages of a bullet pointed rule for how to drive in every possible condition and I think this is sort of a traditional approach and you can think of it as sort of train test trust right and we even have customers who do things exactly this way right they have a big juan manuel here's how you query data warehouse you know never ever ever do a right join with the ball table right and they'll give you multiple choice tests and then at this company they don't give you your credentials to query the database until you've passed the test it's just like the daughters of this model is really amazing and do you think nobody ever gets anything wrong it was company but because i mean we know some of you know design background here um you know sort of a declarative memory that you are kind of cramming into and drawing from when you take the test it's not the same that that you need in real life when you're making decisions in real time under pressure right so thank goodness this is not the only approach that the government uses the able to drive safely they also do things like this right they have things that are salient and simple and just in time in the contacts to get your behavior to change right there when you need it right and i think and this is the approach the relation tries to use of sort of proactive intervention now i'm gonna do a quick contrast here here are three real questions excerpted from the California driver's test take a look who thinks they could get them all right mind percent confidence okay too late you're dead right a truck is just slightly from the left so again thank goodness we have these sort of ways of communicating as well instead of just training testing and trusting will have these sort of salient signals here with the red actionable advice you know because trucks made wide right turns don't pass on the right this is a passenger loading zone wait no longer than four minutes 59 seconds I love this one the best right literally red flashing lights it's all the things that social psychology tells us well well will work but it's a fix to the thing that is developing the school bus minds me sounds so I think if you if you can do both the DMV test the situated intervention do both if you have to pick one I think is a strong impulse to me pick the legend time intervention and in fact for many of the interfaces it get us to drive correctly we only used in time intervention and we wouldn't even dream of doing that otherwise so here's a quick example it's just a newspaper clipping where it shows that for certain stretches of these highways in Maine the speed limits raised to 70 and elsewhere it's it's not right imagine if we took down all the speed limit signs and we just had this but you just look at the map and you can look every time you when I have fast to be going but probably have more accidents and more infractions right thankfully we put signs up where they belong right and we can even go better we can say let's not just consider where you are where the sign is let's consider how fast you're going if you feedback this tailored to you based on your behavior and sometimes it's good very actionable and sailing into if they slow down mine doesn't in this ok so what that means in the data space is we can have warnings and a different types here in bright red letters right there in the seeghe letters is it an older version of our app but you can see here in one example that the data is stale so the answer is not going to be correct and it's not HIPAA compliant health care data so it's not going to be compliant and will tell you that right as you're writing it just kind of like you turn to the the Mirage of the parking lot right you see oh shoot it's it's a accessible space then you back out and move on before you get in trouble or here it's a performance warning I just push to run and it's like training wheels right before I can get in trouble says hey if you run this query it might you know it's in the whole database down so put on the brakes right now people are annoyed by this because sometimes we're wrong thankfully is an ignore button and when you push that we can learn from that feedback and the hope is that the cost of not having to intervention literally your query should one in five minutes runs for two days and you're angry emails we think it's worth it to have this this point but again you know that the data kind of coaches awesome on how to how to tune those variables okay one other example here this is all about again the choice of like DMV trying to pin you from driving unsafely and then you know street signs and things oh one fun thing actually is we talked about the the speed limits you can even go farther if I've never take all the stop lights in the street signs away right so you've unlabeled intersections and you look in your book right and this is oh if you arrive at this intersection which happens to be first Street and Main Street it can be that you know if you're going in the north-south direction right you can do that at 15 seconds past the Minotaur 30 seconds passed a minute right I think you know you can go totally absurd with this and see that you know you're much much better off putting all of that contextualized then then just trying to you know training and test right so that's why it's going to be proactive to be to be in someone's face but you can not only just help people not make mistakes can also help them achieve their own goals faster and Esther GPS is all about right you just say where you want to go and it does the route planning it tells you when to leave it does all of that for you and it reduces the cognitive overload you just focus on the actual driving part and it will tell you you know when to turn otherwise you can relax and if you're doing that you have a chance to introduce of information right tomtom makes this choice to show you your own speed limit so it's one of those whose how fast you're going constantly right and they have your attention because they're helping you drive while they're adding let me drive safely all right same thing here they can show there's a construction zone in this traffic all information we put on interface right so wouldn't it be great to have gps for your data right instead of just a street map did you do all the work yourself so what would that mean gps for your data would mean turn-by-turn directions based on the same things it's based on in the driving world based on where you are the context of what you type what tables you're using you know that sort of thing and where you're going you know if I say hey computer take me home right in this case we have semantic search the user has typed mal and that's a you know could be the beginning of the english word male we should maybe want to type in gender code equals 1 or it could be a country code is the one for a mulcher Malawi or Molly right so normally i have to go consult my map by David Sawicki or a friend to figure out what's the name of the column what's the name of the value and do all that look up right here no map required just say where you want to go it'll tell you how to turn to get there soon but by step but make sounds so um so this is sort of losing sequel syntax and social statistics oh sorry and semantic search excuse me to to help filter the list we can also figure out how to add a rank order options in that list using social statistics so what does that mean think about google trap traffic in maps right you're driving this is in LA if you're unlucky enough to be there with an android phone in your pocket and two hours ago you're moving at 60 miles an hour and now you're moving at five miles an hour and that phones you're laying it back and Google aggregates all of that data okay updated traffic information right so what it can then do is say Oh everyone else is going there you should go here instead to save time in the case of sequel usually if there's two birds the table that looked the same but one hasn't been used for two years and one was used by a hundred great analysts just this morning like that's the one to use so we can do the opposite and say okay where's the traffic and have that guide us so we can filter by sequel syntax socialist and semantic search and we can rank by social statistics that put the the most likely candidates in that order and really help you out right and this goes beyond critically a normal autocomplete right normal if it's saying you typed a couple letters what are the next letters you want to type that would be like saying what if your self driving car says like Clippy it looks like you're making a left-hand turn do you want us to finish it for you all right cliff infants actually very dangerous that could be nice but like way better if you can say oh you're trying to get home let me figure out the hundred different turns to get there and then you know we can that mean you stuff is a little less important cool this is what we just talked about when you're trying to leverage proactivity to make interface better there's a lot of opportunity and promise there you can reduce not just the little you know keystrokes there's a thing all people think about or simple wizards you're writing a letter can I type from your name for you not that helpful right we can go beyond that and actually reduce the time it takes to do research the time it takes to do planning the time it takes to reinvent the wheel r you need to take all the time it's wasted mistakes you could make along the way with little mistakes like slips that easy to recover from because you know you made a mistake but also big mistakes that can cause problems to your whole organization and there might be variants cities may not even notice you made them and so we can kind of nip that in the bud as well right by seeing by baking the best practices into the tool and making each analysts as far as all the analysts combined right and we can prevent cognitive load in context switching be a lot of interviews were without this tool people had like literally like a hundred tabs are open you know Ricky's yeah or skype it's that it's like oh wait which version of this column etc and that also custody nurse dress and waste of time so I think this is a big big opportunity cool mix isn't sent so far any questions on ok cool so now let's let the other shoe drop right clearly there's a lot of promise how to make sure you don't go overboard what are the perils of being too proactive got the promise where the perils the parallel the biggest role i think is is is having the clippy disease right attention distracting hyperactivity disorder right just being in your face but it can even get worse than that this is a text message that I've sent a variant of many many times back and that is ever happened any of you right this is the computer trying to be helpful it says sec lower case is not a word and you are a human who wants to spell things correctly I can't help but notice that the C key is near the X key maybe and just saved you the work and having to accept this edition I'm just gonna do it for you as you push the send button right this happened to me and my boss like so many times and thankfully we both worked at Apple on language technology and so we didn't nobody reported anybody but I think an understandable mistake by a computer can cause a big misunderstanding for a human and so I think the cost of false positives can be quite high so we're trying to balance these tensions I think in the previous section I sort of laid out why you don't want to be too reticent or too timid right if you if you is the computer know what's right with some confidence then maybe you should speak up you know and if there's a way to do something better don't make it too hard to accept that suggestion but it's very dangerous to get cocky we get pushy right to make it so that the person cannot reject your suggestion even if they know they don't want us or to think you're right when the confidence special really is down there right so if your examples again back to iOS near and dear to me here I'm front about a sentence for whatever reason you know it's got we're in the markov room it's got a markov model that thinks that for some reason a positive i don't know why it isn't able to get a singular right there but there it is and of course as soon as i push ? it's going to accept the suggestion with the question mark at the end and if i want to recover from that app to push delete twice for the ? then it'll eps then I push this button to get back to punctuation and I push ? not too bad how much time did it save me what I was gonna spell it wrong you know anyway do you see straight outs here right ively annoying things that happen is a place a building on staff that I used to go to a fair bit and it's called wang and it's like cap plus like anything else it's not an acronym and somehow my phone just got convinced it's all caps and every time I type it i have to like delete the whole thing into it again it's nuts so these things flush appeal didn't we have this like okay cool so i think i think what's interesting here is this is an interface that apple took back from the phone to the mac and on Mac I love it because I'm type I'm touch typing on my keyboard I have the haptic feedback at the shape of the keys so that it can do without looking right and so as I'm typing my gaze is up here and if the suggestion is coming up and I don't want it I push escape and if I'm a couple letters in and it's a good suggestion then I push space and I'm actually saving time right but with here I'm looking down here to type I have to move both my eyes and my finger appear to reject it but it's a little bit too pushy for me some of the time the case was really great is when i'm not typing any proper nouns or like you know s80 words if I'm just typing the most statistically probable sentences and look we're on our phones a lot of people are typing or tweets and facebook messages and and and a text messages but they're saying the usual stuff hi how are you I'll be there in one sex right the huge so anyway in that case you just keep your gaze here you tap it out you make tons of mistakes it's correcting in real time if it had to stop and say click here to accept for each one you wouldn't get that case so i think this sort of a flow based design here is like how do you kind of know how like you ought to be right and then make a choice but how pushy to be there a sense ok cool so over here on the side by the way i'll just point out another approach one that's a little way less pushy and a little more maybe pro finally iOS 6 this is a about when I left Apple they had this great button that I loved the dot-com button and it used context it would only show up when you're in a browser URL bar or in like an email field in your contacts and you just push calm and you get those poor letters for you know for free there's a pattern that some of you may notice me you're not do that you can hold down a letter on the keyboard to get alternatives so the most common cases you hold down the a to get all the accents for you know Spanish and French and whatnot well for calm you could hold it down and if you get org that edu it set it right now I'll go didn't discover that but they didn't have to everyone discovered calm and if somewhat intuitive in Iowa seven they kept the options per ticket calm or any of them in my frequent one you have to hold down your finger on the period key which nobody would think to do I would be holdin on a single key to get a multi key so it was really smart to have calm just right there on a surface because people can kind of figure out the concept because leverage this convention to having to really try so I think we see sort of different extremes here the extreme of saying you have to actively reject our suggestion is one extreme having a thing that you have to actually like read a book to know how to do it is the other extreme right I need you to hold long press like how much time is that saving you right and then there might be a happy sweet spot where you can kind of put a suggestion you accept actively make it so easy to accept it as you can save you time no again we're talking about shaving off four characters is not curing cancer but I do think it's really nice design over here on the side so let's talk about query tools that's why I think some of us are here here sequel pro who is a sequel Oh nobody oh really o people query my sequel at all this is a really nice in many ways desktop client for my sequence i recommend it but it suggestions I don't think are so hot it's a free tool so I you know good for them but they suggest nothing in this context select star space right so if mr. Mark Harper he would tell you that if you look at sequel that runs successfully in the logs what we do literally there was only one thing that can occur here and it's from write probability one but we get no suggestions so if I'm a sequel novice Oh what is it it is it select star out of select star in table I got your my tab stuck if i type f i get an alphabetical list of all possible keywords as selects are false select star fast we all right that's all i want to do it right and i can scroll down dun dun dun down past the bottom and select from if i won so this is a straw man there are other contexts they're much more ambiguous and there are certain keywords or maybe you know a long table or schema name you have that is so long as we only save you a lot of time everything was hard to spell there's definite value to the system and its kind of interests and ered but this particular moment not pretty right if i'm not a pro I don't know to write from at all computing from F is a very helpful and if I am a pro I can type the remainder much faster than I can hit go down to the list right so be really great if they could leverage context and do smarter ranking based on mine now i'm going to do a tool that's really good auto-suggest this is sublime we just sublime way a few great so what is really great so again after x equals i got nothing this context here is it doesn't matter but as soon as i begin typing it finds matches it doesn't just rankin alphabetically to some smart things based on the particular letters I've typed if I build i'm going to use them it's a fuzzy match so if i'm a novice who's made a mistake like a misspelling it will still match things that are close like maybe I accidentally skipped the a somehow I think it's there but alright arrest any mission to pee and I want to type this instead but it's very forgiving and once you see this forgiving fuzzy matching you know the ways highlighting you realize oh I could type in RR lmn right it kind of awards acronyms and things like that right so we're pro usually really figure out that you can use this to save a lot of time might so again it's discoverable and it's recoverable but it's kind of that nice nice medium so one thing both these tools to that made us believe it wasn't even up for debate is as soon as this box appears the top item is highlighted and it really makes sense since Sublime's case because if you're doing all this work to find the most probable thing and put it on top make it really easy to get it onto your onto your page but okay anything and there's nothing after you accept a suggestion or after wise because after punctuation only happens after a letter so I'll get back to that a minute so so elation said let's not wait for a letter right because the place for you to make the biggest impact is not saving keystrokes it's saving the research and the planning and looking at the map right so after on if I know what I want to type I can just type it whether I'm a winner or a pro what's great is if I forget what's the right condition during this on this is the moment you can make a big difference say them a lot of keystrokes an allotted wiki pages but by default we had a highlight from the very beginning like the other tools do this did not go so well in user testing with pros learners are fine they're going slow they're looking up there thinking very carefully what's going to happen if each key and they've loved it send em in a ton of time but the pros they use the two keys that the universal keys for accepting suggestions right return and tab to do things like this return tab right and when you have a suggestion coming up when they weren't expecting it after a white space instead of after an ascii character or in alphabetic character they would get these and searches they don't want and it's very hard to recover now you've got to like hit delete a bunch of times even god forbid move your hands all the way to your mouth so then crows hate the most to like make up for are trying to be so helpful right so um so the new plan is okay let's not highlight this by default if you really want it you can push down arrow enter but so you have it if you want it to keystrokes away if you want it but not going to be inserted accidentally but of course even down arrow some pros you like I use the arrow keys all the time to move my cursor and I use other things to use all the vim commands to like just bounced around like nobody's business right so so we can allow that to write because the interesting thing is when you stop focusing on saving keystrokes and focus on saving cognitive overhead and research time you realize he's the hard part typing letters see there is a hard part that knowing that C is the letter to type because this is the string you want oh and sorry right so even if we make it so that arrow keys don't work we just show you what's available and even something on top but this is just great out because its proprietary but that is that the top it's the one that you'll be inserting you can just read the other options here type letter the first one tell you you know disambiguated and then hit enter and keep the arrow keys preserved to right so i think one of the things you can do to work for pros is to realize well what is going to help them do maybe the biggest savings aren't in the execution phase and the planning phase cool what would that make some sense another quick example of this we tried to be really helpful and then realize if you had to maybe tone it back by being clipping this unfortunately is an old version of the code base I don't have access to right now but I can show you the equivalent from iOS another thing I worked on back when I was there before Sri was what eventually became the little like three word options you have statistically so you know once upon a pep upon I hit I hit time right I'm doing the most probable words in a row and it just one tap each time and it's being so helpful it's inserting the space for me so it's halved the number of taps need to do so far and again these are taps not just keyboard strokes so he's even more useful once upon a time there it needs a couple of hours for that was an amazing and against give you a long way you get the gamma for free right princess but now it inserted a space that I don't want now there are way more spaces in other punctuation unless your comic raised you like me I'm even then right so it wasn't so bad that i went through my life every several you know accepting having the do one delete to then get my cursor I want it to be for punctuation but how many of you know what would happen if I didn't push delete if I just go over to punctuation and hit punctuation character all right take this with you into your lives will be so happy you did you can just hit ! and it will insert it behind the cursor in this space right no they knew of i this is most it's the coolest undiscoverable future in this and I and we try to do the same thing we're like trust us we know sequel right after select we were gonna suggest the most probable column or function in order so you can literally just type return get select return my type for my column return it would insert a comma and the next most popular column and you just keep going maybe with eric keys in the middle and would like right you're the best query right and all use to do is slam me on the enter key people hated it hated it because they're good at typing commas they're psychoanalysts and they'd end up with two commas or no commas official eat it just you know we're trying to save them a couple microseconds and we're getting all this cognitive load we wound up doing is taking a page at ad on Norman's book again I know this is more of a developer crowd than a designer cloud so I'll just say he is these ideas of the Gulf of execution how hard it is to figure out how to do something and then go for evaluation harder to figure out what state system is in and what that means for us is to go back to hear when I push enter or tab if that's the right and this is highlighted what will happen every time is it will insert exactly the string exactly at the cursor right back when half the time it would do that and how the time it would insert a comma before the cursor with the space and the thing where people didn't know what to expect and the amp readability drove them nuts right and so it's better to say what you see is what you get always and yeah it takes some more commas and spaces you know in this very highly regulated environment but it means that you always know what's going to happen and so you don't have this nervousness about the system and you can actually leverage the goods that are there it sounds so these are just a couple little design earnest you went on with this particular interface which is one of the proactive interfaces for one of the absent elation so I'll leave all of us for the questions section but hopefully gives you some sense of how you can make a proactive query tool and some of our bucks you can avoid when you try to get too clever so swimmers this section our goal to reduce the risks of backtracking loss of trust confusion frustration distraction some of its emotional but some of it is quite practical and to do that you need to not be too clever not be too pushy right figure out what are the things you can be really confident of and deliver those and not miss you blow you don't want to miss the forest for the trees but it's so easy as a UX guy that account keystrokes especially if you're a data UX guy I've it you're like oh I want numbers and forget to consider the bottom you're going to count anything right count seconds that's the atomic unit right okies drugs but what about cognitive load cost of recovery you know those things can often add up to weigh more than the time takes to type of space or a comma and the other thing you want to avoid is try not to sacrifice global consistency to hyper optimized for one local instance and if you're going to do that but you've got a good reason and your use your test it and make sure your user tests are always global right because if you test oh let's test you know how fast users can you know fill in the the columns and functions between select and from in the column lists and then over here we test half we're going to add a join and if you're just a quick let me add a filter and you never do a holistic say hey just get your work done in our tool you don't realize that the the cognitive overload are figuring out how is it going to behave in this case actually outweighs the benefits of that optimization and they can optimize in their heads better than you can they're going to going to be doing so I'm putting it together we talk about what is the paradox that the pro presents to us they're super experienced they're better at planning they might even have to plan right think about you know before uber when you had like the the SF cabbie who knew the one-way streets by heart right so not as helpful to give him GPS right as as maybe it is today less prone to error right so these people they're already operating at ninety percent capacity to get them that last ten percent in some ways you've got a hyper optimized right make it so that it's new really great you know if I have to move my mouse to the top of the screen and hit accept suggestion now I'm not going to bother doing it I'll be faster just doing it myself but they're so sure of themselves and so skeptical of technology that if you make a mistake especially if you push the mistake on them and make it easy for them to reject it then they're even more frustrated and there's s pretty to delight them so I think the goal is to try to consider little really look at the flows we have can we minimize these two things at once their intention can we minimize the time it takes to accept a suggestion if you want it but also minimize the probability of accidental selection if you don't definitely intention very easy to accidentally do one or the other but as we saw with the co.com example with this idea of saying you know only highlight after keystroke but show suggestions after white space you know there are opportunities if you're thoughtful to get both at once so that is what I've got today elation calm if you wanna learn more about what we do if you're interested in joining our team and thinking about really tricky design on data problems you can also get Alicia naw Kham / careers I don't have to answer any questions you guys might have thanks the question was any possibility of open sourcing at composed so not not soon this is technology our customers really like Henry really like to get revenue from that but i think these coatings are interesting i think one is we're looking at having this technology be out kind of in the wild having at a university program it research program see what can you use it might not be able to pay for a license is one opportunity and i think who knows what the future will will hold many parts of what we do might be part of projection parts we prior Terry any other questions well while we're waiting here's some best of Clippy those looking for the two images we ended up using this and really creative people out there any other questions about proactivity okay thanks so much oh oh I should tell at once of a story oh my gosh Oh Mike sorry I'll think about it I get back to you Lorelai verses are regularly yeah it's a great question so thankfully now we're mature enough to be if you have customers who will stand up and say roi with numbers can I show you the safer anecdotes for example yeah so we safe ways one of our customers and there's a woman who loses analytics team and I'll cut to the punch line just they got a 30x ROI and the cost of the software and it isn't just from the query tool right so so the crucial is helping you write a query better leveraging you know snippets Rosa the city of a query library so before you even start to write a query in the first place you can see Oh as somebody already written this or a variant that I could template eyes and revised slightly so I think it's interesting is that you know the more you zoom out to ask not just you know how do I get through the next few characters but like can I cut out major you know days weeks long parts of my process and application of effort that's where some of those big wins come from the other thing is you know when you get the right answer fast right you aren't only getting ROI from saving the individual end-user analyst time you get ROI for making better decisions as a business and avoiding in great decisions or avoiding this is a sad thing with people want to be data-driven but the vpu looks at two metrics there's be the same and one it's 3x to the other and says oh forget it I'm just gonna go with my gut right so I think there's some of these that are like very easy to quantify and some that are more intangible but it adds up to quite a big big benefit because what you wanna get lost I know to you before lunch if they probably can roll in from Salesforce keeping and they gave quite a blog to Malaysia ultimatum awesome yeah we were very happy they were able to speak here cool okay what's this funny story oh I have one so yeah I'm a huge geek NLP guy by background and have been playing catch up a little bit with learning the enterprise data space so i went to a customer call with an elation executive who asked as far as I could tell apropos of nothing this head of analytics how many of your analysts have Bo which is another way of abbreviating Bob J which is business objects but I was just a Paul like you can't ask that and of course stickers on me so there's a funny story with women on the clock ok I'll be here if you have any questions thanks so much