Bay Area AI: How I turned my PhD in NLP into a Y Combinator-backed Startup

Bay Area AI: How I turned my PhD in NLP into a Y Combinator-backed Startup

Recording: Bay Area AI: How I turned my PhD in NLP into a Y Combinator-backed Startup

the last in-person conference that that we attended so today I I'm happy to share my story of how I ended up becoming a CEO of a wisely fund funded startup thematic and how you know how we go to this stage we're where we are now but realizing that this is an AI Meetup I also have prepared to talk about my research it is a bit outdated so my I graduated in 2009 but my citations somehow are still growing like crazy I have two and a half thousand citations of my of my publications there are all on natural language processing and analyzing text to figure out what it is about what are the common themes and knowledge representation so the other part of my talk is really about commercialization so how can you turn your knowledge in this specialized field into into a company how can you make money with it and I'm also I'm also happy to show you a demo as well of our products so not sure like a round of introduction makes make sense but if people could just type in the zoom chat what would be more interesting to them is it more of the research or commercialization or the demo so that I can kind of tailor tailor my talk respectively just type you know is it or is there a poll function you guys can raise your hands I will see a show of little fans I don't you can ask for instance who was you know commercialization and I will kind of eye the hands raised and hopefully we'll get some idea oh wow so I can see pretty much everybody talks about commercialization other well let's do a raising hands I don't think we need to raise hands no they would have a lot of majority for commercialization perfect so I might skip a bunch of slides then but happy to talk about this and we can have a discussion as well to make sure that I answer any questions you might have but to start with I just want to tell a very quick story about how I ended up in this field I'm very passionate about natural language processing and I learned about the fact that this field even exists back in Ukraine so I'm originally from Ukraine from Crimea and I was in a room very similar to this one mostly women studying at a university and I was studying German and English linguistics and it was in a lecture of applied math when the that lecture is that by the way there is a new field called computational linguistics and this is where people use computers to understand language and most of us were on a track to become either translators or maybe tourist guides and when I heard that computers will be automating translation I knew that just I need to switch into this really cool field and then he gives some other examples well NOP is also when you use spelling errors in a document it could very much relate to this and ever since I'm trying to be active telling telling this story because because if I wouldn't have found out that this field exists I would never have ended up doing this so my past is kind of from linguistic rather than from computer science and I tried to speak at school schools as well because this is such a unusual field but at the same time so prevalent in our lives I moved to Germany and did my masters there but then I ended up in New Zealand and in New Zealand there is a university called Waikato and there they're famous for the one of the first data mining books and which came together was an open source software called Weka and most of you here would have heard of Weka if you were doing any machine learning back in early 2000 and even later so this is where I did my PhD and my supervisor was Ian Witten and my PhD was actually about keyword extraction so how how can you basically take an article and extract keywords that are similar to the ones that an author might have chosen because they understand this field well and ultimately I moved on from from this to this general area of basically and figuring out what the main topics in text which is very much related to what I do at somatic right now so somatic is a software as a service company and we analyze customer feedback and the main foundation of this analysis is being able to figure out what are the most common themes in surveys and this data is very different to the types of data I was I was doing during my PhD but the main ideas are the same and and the idea is that you basically have the problem is that there are small small errors a lot of variation about how people I'm just trying to find relevant relevant picture a lot of variation about how people might be say talking about the same thing for example third party it can be spelled out as the word it could be could be three and Rd as a third party so when people are leaving feedback they use very different words and phrases and the idea of a theme extraction algorithm is to figure out what what things mean the same thing and what is a theme that that is the most meaningful thing that you can assign to a piece of feedback so this is how it's all it's all related and how I did I did this in my PhD is by using controlled vocabularies so you can control vocabulary so agrovoc is one for agriculture and it's it actually came as a book about this thick and it listed all of the agricultural terminology and how people might be referring to the same thing so for example circle circulatory system is the same as cardiovascular system so it says if you see this if you see or say using this term then use this one to assign it as a topic and it also has some additional semantic relationships to the theme and so what I found is that Wikipedia that at that time was already huge and was growing it has actually the same information except it's crowd-sourced and it's richer and larger so you have less circulation as well as cardiovascular system mapping to circulatory system so I used Wikipedia in my PhD to normalize the terminology and but there were difference there were other difficulties for example heart could be the name of a song or hard maybe like the heart shape so wikipedia has articles on these and I'm like an agricultural vocabulary that has some sort of a theme or domain or vertical captured and has less ambiguity in Wikipedia it's incredibly ambiguous and sorry and so there is additional natural language processing that needs to happen to disambiguate the meaning of every single word and decide which article in Wikipedia would be the relevant one so this is this was part of my PhD and the interesting the interesting thing is how do you actually evaluate these things right because it's so subjective if you give two articles to different people they would come up with very different ideas of what this article is about and they would name them differently so the way you do it is by using consistency consistency is basically how two people how likely are they to agree with each other and professional indexers and libraries are more likely to agree with each other compared to students for example who might do the same task and so the the way I did it is by basically creating datasets where the same people are all gathering data sets the same people analyze the same article so I would analyze their consistency and then test the algorithms consistency with this with each person and then average this so if basically if the algorithm can't be distinguished from a from another person then we have achieved human competitive performance on tasks and indeed Maui the in some cases was more accurate than than people for example in the tagging case it was more accurate in the case of term assignment where it was tested against professionals it wasn't it was closed so and it was much better than the baseline which is using tf-idf to basically figure out what are the most common and what are the most prominent topics so commercialization so when I graduated I knew that academia wasn't for me I loved my kata University and but probably because of my upbringing I'm definitely have a kind of a sense of hustle and me and this is how I ended up in ez 1 in the first place and was able to leave Ukraine so I'm always trying to find cool opportunities and cool things to do and I was most of all I was frustrated in academia that people might be might be doing the research and using thematic and it was an open source 3 Maui was open source project that I was working on but I really wanted companies to use the research that I was producing and so I ended up joining a company called pinga arm and this was 2009 so it was financial crisis and this was literally the only company in New Zealand that was doing anything related to an LPN AR so this is their their slogan discover a new value from mr. unstructured data extremely generic and and it was an interesting and interesting experience they stay with finger for three years and what we ended up doing is basically create at first we were planning on figure out how to automatically create a report so if your search for something to straightaway create report from the first stop results and the use case was for any publishers who want to charge for the full articles but they wouldn't mind including a snippet in in the report and who created a prototype and we started demoing this but people weren't really interested in this and they said oh that's very cool but nobody actually wanted to put their money buy down and buy the software so then we pivoted into a another application of NLP that is more kind of API based so at that time at the eyes became more popular and there was a number of NLP api's where you can copy and paste any text and it would tell you what's the sentiment one of the most common keywords and we started showing this as a demo and people started straightaway asking what can you do can you read act all the named entities from it or can you detect profanities because that's what I care about when people post something so very quickly do this kind of language analysis but the application that ended up being the most successful for pinger was actually metadata so when you use something like SharePoint a document management system in order to find those documents you need to be able to search for them and people and the way SharePoint souls is you have to enter the main topics pretty much the core of my PhD and the nobody wants to do it is very tedious and there's inconsistencies as well it's different people do it when they upload their documents so we used our technology to basically summarize summarize these documents and it fingers still around and they're selling it as a document management solution and record management solution to different companies but the whole how the whole company developed I was employee number one and within about two months they hired a sales person and we didn't have any product at that time so I was just like cutting code creating demos and the sales person was sitting next to me and he basically would pick up the phone and call people and say hey we're this new company it's we're doing amazing things we're better than Google and you need to buy it and he was incredibly aggressive had no idea what we were selling didn't talk to customers to figure out what is it actually needed and he managed to do only one sale to police department because I had some money left over in the budget and wanted to have it used up so that next year they get the same amount of budget so this was kind of my experience in in sales and the company ended up hiring another five people in sales and I tried to start selling this technology that we were buying and they actually got into a lot of trouble they had to downsize massively and they pretty much lost 12 millions in in funding that they got so this was my first lesson of how not to do it and I thought well I joined this company the founders were like around 50 years old the head of chairman and a board and the business plan and investors and I thought well if they couldn't figure out how to do it and probably I should just do it myself there's no way I can do a worse job than them and so I ended up leaving the company to have a baby and within the first while I'm maturing to leave I decided not to come back and I knew that I wanted to start a company but I didn't know what the company should do so I decided to consult in this general area of natural language processing and machine learning and because of my because of my research the Kea and Maui the open source projects and I did some content while I was doing my PhD as well which I recommend everybody to do to do it people kept writing me writing me emails and one of the emails was from I come from NATO so NATO contacted me from Paris I was still an easy one at a time and they said well we'd like to use your open source software but we need a proper implementation and we need to use it to basically figure out what different scientists are publishing and turns out NATO employs 300 scientists sorry or 3,000 scientists or they're kind of maybe not employed but but connected to NATO and one of the ways they do is foster collaboration between them so they need to know who is actually working on the same thing so they need to analyze their research articles and and connect them from via via the the topics that the articles are about and sorry I just forgot that there is there is a chat and people have been asking questions I think we've covered what Maui does and can semantically rich knowledge graphs I think we also covered this talking about Wikipedia and yes how we how we did it but back to back to my name a story they I they didn't want to use the GPL open-source version and at the same time I convinced my my husband who is a software developer as well but he's very much interested in kind of high performance computing and so he ended up re-implementing Maui from scratch and created we created this website and started a company and kind of started market Maui in this way and what how the conversation was Nader developed is that they they said great you had now you have the solution but we cannot have do business with a New Zealand company because NATO is we can only work with NATO countries by that time I had another another consulting gig that also used Maui in England and so I tried to kind of partner with them to to have this deal done and then a company from California reached out to me and they did want to see a demo and a quote and it's been going on for probably a year and I haven't done a single sale I was selling my time my consulting hours but it wasn't so any technology any any software and I was talking to my friend who recently sold his company and spent three years in Silicon Valley working for them and he said well it's very simple the secret is very very simple you need to do an enterprise sales course and I trusted him and I did have a bad experience watching salespeople at think are and but I still decided well I'm just gonna do the sales core and he said you should do it with this guy he does this is still in New Zealand so don't ask me where you can take this course but you should do it with his guys like a weekend course and I did it and it changed my life coming from academia I had no idea that there is actually art and science in sales and it's not just having conversations there is actually a sales the idea of a sales pipeline that you're trying to take your prospect through and another insight was that you don't say yes to everybody you need to understand what is the problem that you're solving and say no to everybody who wants something different whereas I was like willing to talk to just anybody about my research who was interested in modeling whatever they use case was I was I was talking to them the third insight was that you need to ask the right types of questions so there is you you if you're doing anything if you're doing your first kind of sales meetings you need to basically write them down so you don't forget and these questions are very simple they're not sales see they're not like are you gonna buy this do you want to buy this you have money for this no they're actually um questions like when do you need this by what's your timeline how painful is this for you is this problem for you and so the turning point after I did this course I was invited again by a company to who were interested in Maui the completely contacted me and it was a media company called if their facts so Fairfax pretty much owned most of the media in New Zealand and and a lot of it in Australia as well and they invited me to show them Maui and I I went there and I showed them the demo and they said this is perfect we're just gonna feed news articles into it and when it tells us the topics will let people who let advertisers bid on the on the topics so this is how we gonna do our advertising bidding through your through my way and we'll make sense sounds like it solves a problem and so I said well when gene into spy and this was a meeting in June and they said probably next year January February February will reach out to you at this point I knew that this is like another situation will just waste my time but what's happened in parallel I was running I have been running this NLP meet up in in Oakland and some people knew me through that meter and there were three other companies who contacted me and they were interested in the analysis of customer feedback and one of them ended up even sending me like a sample actually several of them gave me like sample customer feedback datasets and asked me is it is it even possible to analyze this data and so I already wrote a couple of scripts and showed them and explain them how they could do it and so I was in this meeting at Fairfax when there was another lady who also came along to the to the meeting and when the original people who invited me said it's gonna be January February I turned around to her and I said well um why are you interested in Maui what is what's what's your interest in this topic and she said well we're running these NPS surveys and NPS is stands for Net Promoter Score and these are customer feedback surveys that companies used to measure how customer centric or customer friendly they are to measure their customer experience and NPS is a KPI it's something that's reported for most public companies in in board meetings and and it's it's a number that everybody is watching and how it's calculated is when you you get a survey from a company saying hey would you recommend us to your friends and family and you give them a rating and then they ask you why did you give us a score and so it's a way for them to collect feedback and then link it to the score as well so literally people are telling them how they can improve the score because if they give a low score they say things like real stuff is rude and if they give a high score they're highlighting the unique advantages of this company that they should be doubling down on like you are you have beautiful University and have a beautiful campus so they should be marketing this in their customer words since what matters to them so and then she said well I did reach out to this company called Clara bridge in the US and Clara which is one of the probably one of the bigger companies that do NLP in the US and she said well they wouldn't even talk to me because probably because I think we're a small company in New Zealand one of the largest companies and because by that time it was like fourth converstation I I knew there's something there so I came up with my enterprise sales questions when do you need this by and she said well ideally last month because we started this program and there's a lot of push on me to report the insights and none of those things were things I've tried work so this this is where I took this website and I dropped it completely I stopped I stopped using it and instead I tried to go back to those other companies and who asked for my help and I said look I I have this prototype and when I see if this is what you need and I started showing them the demo of of the of the prototypes that I built and the demo was basically a bunch of slides so I had a Python script that would analyze keywords and customer feedback and then I would build histograms and then put them into a slides and make them almost like a customer insights presentation this is what's driving your score down this is how things are correlated little did I know this is exactly how they present their findings to their senior leadership so I actually sold the whole problem and and he was really working this they were really interested and all three companies that I ended up talking to and one of them was a market research technology company that knew a lot about how this problem needs to be solved and they were the managing director was very motivated to have the stove and we probably met for an hour 10 times in the next kind of three months and ultimately I after the presentation I did my next enterprise stealth question would you'd like to see a proposal this is when they tell you whether they're ready to buy and it's a very soft question to ask and it's basically converts them to a sale if they're interested in and all three said that yes they would like to see a proposal and now I had to figure out how to charge for this thing and so I and I went back to the guy who equated this enterprise sales course with and asking for advice and he's like well what's the value of an insight you can put any number in it charge hi give say like seventy thousand dollars to one of those companies and I felt like I couldn't do it I didn't have the courage to put such a high number but I like mustard after put the highest number I could think of which was one and a half thousand dollars a month and two of them said yes and the other one negotiated which ended up more of like an a partner agreement but they still paid roughly the same and so this is how we signed our three customers and now I don't need to needed to build this end-to-end experience and at this point I convinced my husband to actually join me and help me build the product because while I I knew how to do the R&D and I was getting the hang of sales and a bit of marketing I didn't know how to do the product and I didn't know how to make a scalable product and and he joined me and the way I managed to convince him is at the time he works for one of the coolest companies in New Zealand called Serato they did software software for DJs if you go to any club in the u.s. they're most likely they use cerrado DJ and they he loved his job and it was very difficult to convince him to leave but he said I'm really I'm really interested in living in New York for three months then in New Zealand for three months so we ended up I ended up convincing him that if we do it together we're gonna have the freedom to do whatever we want and we could go to and live in Europe and so he put his job and we started working on this together and eventually we we started kind of building our website and we this is one of our recent website versions but we we from the very beginning we started writing blog articles and one of them was how to analyze your customer feedback manually where we explain how you do it manually and stripe picked it up and they ended up becoming a customer of ours and at the same time we apply to get into Y Combinator through complete coincidence as well and Y Combinator when they heard the stripe is using us they admitted us into the program and we moved to Silicon Valley and and started growing like crazy in this in those three months and raised our seed funding and then he started building our team and this is what I do full-time I'm I'm pretty much managing a team we're 15 people now we were about to raise our series a I'll we have more than tripled our revenue from our us customers and then kovat hit and we left our apartment in terment disco and and moved to in back to New Zealand and we have a team here as well so we're doing like a really remote thing I like most companies do at the moment I guess so that's that's our story so what sematic does they we collect or connect to different feedback sources and there are so many tools and review sites and and complaints as well and then we analyze this data we figure out the themes we built a way of adding human element into the into the data and then we create reports visualizations basically sorry this is tribute how somatic words so yeah this is this is the story so um I just say that I'm gonna show you a demo and it's gonna be a quick one and then we can do a discussion we go I can answer some questions let me just open it up so this is the same demo that I showed and at the Bay Area I know P Mita and it was a little embarrassing because Rob Monroe is one of the organizers and I completely forgot that he used to work at Amazon and did Amazon did a lot of work on Amazon comprehend so and what this demo does is actually it's it's taking customer feedback that we stand that that we scraped customer feedback reviews and we ran it through a lot of through a number of NLP api's that are still still very popular so what you're looking at here is a is our portal but instead of showing you somatic I'm showing you results from IBM Watson and a lot of these API have different different ways of solving the problem of analyzing feedback so this is actually one of the better one probably IBM Watson the classifier so it it basically comes with a predefined taxonomy so on the travel you have travel with kids for example and it automatically figures out that this piece of review is about traveling with kids and group sit here and and it's not bad but if you kind of if you look look look a bit deeper why this data set which is about air travel would be talking about roadside assistance assistance it doesn't so like the accuracy of the analysis is pretty poor and in fact all of the reviews are about travel not sixty percent and so the these results aren't very useful you can't really make any conclusions based on based on this data if you use the key phrase method and the so basically that figured out keywords and you get a laundry list of keywords so they picked up that flight attendants I mentioned mentioned a lot but you can see that it's again the the seams aren't organized into indoor taxonomy and you have a lot of redundancy so excellent customer service excellent service excellent flight and so the themes are relevant they do appear in customer feedback but they're not very useful and these are some of the better ones this is one one of the better api's as well for on this data set so a lot of the times when people say well how why people not just use NLP api's and that's because on this data they actually they cannot get any useful results so this is Amazon comprehend and it's basically a bunch of single words and topic modeling is what people think we do this is not what we do we two seem theme analysis and again it's not useful and the reason why and why these solutions solutions don't work on this data because they weren't designed to to work on this data they were designed to work on news articles just like Maui was and they will work fine and probably amazing on some of the data sets but on customer feedback they unfortunately they're gonna fail because they weren't designed for this data set and so when we built ematic we go to specifically for customer feedback for data that's shaped in this way and already our roar is out as they come out from the system look much more meaningful so there's some generic themes but you can see food under food we have great food then there is price customer service good some you notice there's some redundancy here something we're continuously working on but the core of our offering is actually the ability to edit these results yourself so see a human goes into the steams editor and refines the the results but basically dragging and dropping things just like you would do organizing files and folders and and you you are able to merge things delete things that aren't relevant to you so for example airplanes seeds all of these were merged so the algorithm tries to merge itself but if if it can't then this is the person might be able to do it and then we also we also have this idea of match phrases which are automatically discovered so nobody is typing them in they discover it automatically and there's a lot of them and you can go in and check every single phrase if you have this transparency so this is this element of explainable AI you can explain why the algorithm has tagged this piece of data was this particular theme and so you can make the results more accurate more relevant and with that the insights will be more likely to be to be heard and then once you have your refined without the the other part of somatic is visualizing visualizing this data in and allowing you to filter it showing you for example check-in and boarding water what's the sentiment around this theme how is it trending and so not only the there's roughly the same number of people who talk about this theme but it's its impact used to be negative and now it's actually trending towards positive so we can help people understand if they work on something did it actually make a difference did people notice that it's the one flying on the plane or there are security procedures how do they feel about them so that's that's a quick demo of automatic so stop writing right here we we have 13 minutes left and I would be very happy to answer any questions first question from Sachin about was where by YC was helpful and not helpful I get this question a lot for us coming from New Zealand YC was transformative life-changing in several ways first of all the whole mindset I say as you remember I convinced my husband to quit his job so we can travel the world and you know how this lifestyle company which is not a role and I don't have to tell many of you that this is true though the mindset in Silicon Valley you you go bigger you you don't bother somebody who has skills and NLP and AI can earn hundreds of thousands of dollars for a job at any any of the amazing companies and can be very well financially but when you start a start-up you don't do it to do a stock lifestyle business you do it to make a huge company and for us joining whiskey was the realization of wow we can actually do it this is actually a problem that were solving and has a huge market and we are the right people to do it they've chosen us they think that we can do it and recognize our progress so far and and the other the other thing that helped is of course the network and access to investors the validation and and they also helped with understand kind of what tools to use to grow faster and there was pretty much everything was helpful in terms of not helpful I can't to be honest I can't think of any of anything I'm such a fan of YC and their mindset and what they deliver and how they are constantly asking for feedback and just the nature of people they're very much very much a friend so I think everybody should try to get in and not next question how does a business model built an open-source compared to proprietary well I didn't managed to make my open-source model work so I'm not sure that there are different models so there is obviously the GPL license means that if somebody uses the software they they have to open-source the entire project so it doesn't lend itself to commercialization but there are other model other licenses in open-source right now there are much more suitable and companies that do well tend to provide consulting services so deep learning for J for example sky mines this is how they did their model and I think they did really well in this I would encourage you to check out how they did it it's also a YC company but they used deep learning open-source proprietary you just you just have full full power and I don't think there's open source SAS products so the whole idea of building a company and a business model is that you're solving somebody else's problem so whether it's something is open-source or not it's almost secondary it's a tool whatever you use is to help you solve a problem so it's different different model to and works differently next question from DVS the resources for learning about enterprise sales I think there are some amazing books on on Amazon and I is here if you like reading books then I highly recommend this if not sastra is a lot of these kind of stars come networks have articles on enterprise sale so where I learned is sales events so there are sales meetups that I would go to and just listen to how people talk how they try to obscure each other and share and share tips steven is asking sematic is for is thematic proprietary I didn't get the point when you abandon open source yes it is proprietary basically I had this open source software that I built during my PhD and it did not work on customer feedback so I needed to develop it from from scratch so I took some ideas some knowledge about how language works how the house themes can be created from text but I ended up writing every single line of code was written specifically for this problem and I did not open source that but there are some open source projects of mine on like similar topics that are floating around I just abandoned it because I didn't have time to raise children and do a start-up at the same time and help people in there in open source support them any other questions one once twice okay something is coming okay let me see competitive advantage of somatic our competitive advantage is that you don't have to train our system you give it any set of feedback and we figure out what the themes are so it's much much faster to get set up to get the first read out to the point where we even like launch the free version we only allow businesses to sign up for it but it is a competitive advantage to be able to be so agile and soso nimble and then human-in-the-loop demo that I showed you how you can edit the results this is also something unique not nobody else does have you seen great and I'll pee performance from newer models yes so we use it for sentiment analysis some of the latest models but I we don't use it for themes we just haven't we use it indirectly so for example if we see somebody talking positively about customer service so that's how we we know the sentiment using some of the newer models we could construct a cm great customer service of poor customer service where should new zealand start-ups incorporate in order to get into YC you need to incorporate in the u.s. so delaware companies so if you do it early it's fairly fairly simple and YC actually help you do it why there's the legal firm that does it for free but we basically started the company in new zealand and then flipped it so that now it's a US company that owns a new zealand company I'll ask Jeremy's question about the competitive advantage so so it's it's unstirred itself supervised it's or it's not a supervised system so it doesn't require training it basically the themes emerge from the data so no training is needed every single data stack gets its own themes so we don't have a like a pre trained Airlines model a free trained customer support you get model we we have a language model for customer feedback does how people talk in in the area of customer feedback sometimes we have this is how a company called waterphone how their customers talk about when they leave feedback and that model just captures and things like synonyms but the actual themes they they're not tied to any model and with that you get much higher specificity of themes so and with that we're like we're not using any labeling right because we use ladle labeling for testing but we don't have to Train custom models so I was trying trying to explain this Stephen Stephen is asking lessons learned would you do it again what would you do differently and other even enough enterprise sales in New Zealand in NLP to sustain a company well that that's the great thing from the very beginning we never sold only to New Zealand companies we signed up stripe and now we have most of our revenue is from outside of New Zealand probably 80% and we have lots of customers in the US Canada Australia some in Europe and the this is the advantage it doesn't matter where you are if you can have a website and you can jump on a zoom call you can sell anything to anybody and it's an enterprise is more of a like a stretchy title so enterprise I say that we do enterprise sales but really we do mid market so those types of amounts one and a half thousand dollars a month it's not it's not considered Enterprise enterprise true enterprise like $1,000,000 and this is not something we do our average deal size is now is 45,000 some company space and a couple of hundred thousand a year and um you you can still sell these types of deals via the internet so I assume and what I would do differently if I if I could do it all again I would I would try to so once I want to have signed those initial three customers what I should have done and I didn't do is actually open up the solution to more people we for many years we were very selective to only selling to people who approached us and more inbound and you could only request the demo you couldn't try the product out on our website I was always worried is it right so with the problem with this is that you don't get enough feedback back from your customers ironically right so what we're doing now by having this free version is opening it up more significantly more people sign up to try the software out and then they provide this feedback back to us and we can much we can get much better at figuring out what's valuable what's working and what doesn't and this is something that I should have done an accountant if you are starting a company I hope I hope you do this I think we're out of times the last question I answer may be general opinion of Spacey and other api's I am very grateful that people are building solutions like spacey I think these people are selfless for putting so much work into open-source and opie solutions that are enterprise grade scale and I'm very very pleased to see that a lot of the a lot of them find ways of making it worse in a while commercializing some of them and I think it's amazing we can all kind of benefit from it we don't have to build things from scratch and in a way this is what I did by making my solution open-source as well so maybe one day I will go back to that home all right well thank you Ilona for the great overview and thank everybody I'd see everybody's thinking here's well in the chat so thanks for joining us hope you stay well and productive and have fun in the beautiful New Zealand and come back and tell us about the progress hopefully you know things will get back to normal we'll see you back in the Bay Area and we'd love to see a doctor miss Kayla by the band remember regardless of where it is and thanks again guys submit motox and we hope to see you at the next Meetup cheers thanks for having me thanks for dialing in and so many questions I'm grateful for everybody to be able to share my story thank you guys thank you bye thanks good night