Grant Ingersoll, Q&A with Alexy Khrabrov of SF Text @VigLink 20150210

Grant Ingersoll, Q&A with Alexy Khrabrov of SF Text @VigLink 20150210

Recording: Grant Ingersoll, Q&A with Alexy Khrabrov of SF Text @VigLink 20150210

hello everybody this is SF text this is our first inaugural metop we're on location at viglink in San Francisco and uh one of our speakers tonight is gr anderol the CTO of lucid Works uh the company behind uh solar and Lucine and as uh many of you know Lucine is mostly widely used and deployed open- Source search engine uh almost every company which wants to have search on their site has Lucen somewhere uh so uh I want to ask uh first of all welcome Grant thank you glad to be here uh it's a I think search is a common theme in all kind of text applications and our general theme is text everybody needs to find some some text there looking for first I want to ask you uh how did you uh come to Lucine and and solar in the first place and how this interest evolved over several years to to become a company uh what's the trajectory yeah no it's a it's a great question I mean I think you're dead on that Lucine and solar have grown so much uh my my personal history is I started I used to live up in uh Syracuse New York and if anybody's familiar with that there they know Syracuse University which uh Dr Elizabeth lyy who who is a pretty big name in the the text in natural language processing field she had a company there called text wise which uh happened to have a job when I was looking for a job I had never done text in college or anything like that uh got hired to build uh a cross language search system between English French German Spanish and Japanese uh we used Oracle context search this is back in like 1999 I believe uh so that's kind of my first foray into search in NLP Flash Forward uh to 2004 I had left text wise went off and did some other stuff and then came back to work for Liz again at the University this time in a place called the center for natural language processing I was hired to build a English Arabic cross language search system this time you can guess who was paying for that right and uh I walked in the door my boss said hey we use this thing called Lucine I said great let me go uh look up what that is and uh started playing around with it as I as I started building out this system realized that I needed some things in Lucine that weren't there uh started submitting patches and kind of the rest is history if if you will uh Flash Forward to 2007 founded Lucid Works around the premise of commercializing the capabilities around solar and Lucine and providing product and all of that kind of on top of it and of course it's all exploded since then as well so a lot of luck uh right place right time and a bunch of hard work as well but that's great yeah I remember I think I first saw your talk at Hado Summit in 2012 and that was already in the business context and you basically outline big picture and big compon components of big data so it's clearly when people think big data solar Lucin comes up now and uh so I wonder how do you go from Lucine which runs on single Noe to solar and what is the landscape in terms of uh uh distributed systems and how do you see that area evolve yeah no it's uh it's really been interesting over the years because you I think with Lucine you know obviously it's just a set of java apis and that provides search functionality things like obviously matching queries and doing fuzzy matching and all that kind of stuff as well as like language analyzers and and and then a whole bunch of modules that kind of help you build you know a kit of parts to build a Search application Along Came solar and really kind of said all right well let's put this in a service layer and let's make it easy for people to interact with it in the very early days of solar the scaling primarily was in a replicated model for query volume because essentially you know nobody really had really large scale problems other than you know the the internet search people like Google and and being in the like because they were were and of course they were building their own uh so you know especially in the e-commerce environment you know it's just hey I've got a lot of queries but I don't have a big index so let's just replicate it and then you know Flash Forward a few more years as hadoop's taking off which you know of course Hadoop started as part as the Lucine project as well uh was okay well we now need to scale search to match those kinds of needs as well so not only do we need to scale for query volume but we also need to scale for a number of documents and so uh you know a lot of people had rolled their own using solar and Lucine over the years and so you know one of the things that we really focused on as a company was let's make sure that solar just has really good distributed search capabilities out of the box solar Cloud was born for whatever reason I don't know the naming of it but you know that was kind of the way we attacked it added in first distributed search capabilities you had to do the indexing yourself and then uh with solar Ford auto uh which was I don't know a year and a half two years ago now added the capability to do all the indexing and all of that as well such that you know I think I was at a meet up just recently up at Salesforce in Seattle I think they announced their their solar deployment somewhere 200 300 billion uh documents in it you know so people doing really large scale search you know across you know hundreds of billions of documents it's only a matter of time we before I'm sure we're doing trillions uh and so you know you're seeing just this massive uptake in and I think the other interesting thing is people are asking more interesting questions of their search engine as well it's not just hey find me keywords and get back results it's bring in you know all kinds of different data structures numbers spatial enumerated types how do I slice and dice that and bring me back what's important right ultimately you know shifting from it just being kind of traditional keyword search to being all about what's important in my data yes and then you know shifting through you know using that to then power your Downstream applications yes this this actually uh connects very well with one of the themes of our Meetup because we see text as emerging again as a window into human intent right the through this little box uh and this is how we are used to do search but now if you go to Google and you enter sf4 BOS it will give you tickets probably from uh San Francisco to Boston uh or more structured data and uh uh essentially search Eng ultimate goal is to figure out what was our original intent behind the query so here of course I want to ask how we open source people are going to compete with the big guys right because we have we have open source we have the community uh but the uh kind of structure data people want to ask uh may have semanti content and um how do we kind of map this uh more uh with structure data which you mentioned how do we kind of uncover more more behind you know more in our documents what do you think is going to happen here in the open source the next few years wow that's a lot of good questions in there I mean I think you know obviously I can't predict the future but you know what you're seeing in open source I think in the you know kind of the early days especially when I started with open source open source was primarily focused on commoditizing what people were seeing in the proprietary landscape or they just got tired of building the same thing every time they went to a different company so they said you know the Forward Thinking people were able to say let's open source this such that I can carry it with me whereas nowadays I think you're starting to see just a lot more Innovation happening right in open source out of the box as opposed to having you know being cooked up in some lab somewhere it's now being done in open source that being said I think that especially on the NLP side of things it's still a little bit trailing obviously there's a lot of good new projects out there the really the the biggest challenge for open source I think when it comes to n P not so much search but NLP is just having the sheer amount of data and being able as an open source Community to deal with the pedabytes of data that can feed into you know the the really large scale models that you're going to build around uh you know whether that's machine translation or question answering or whatever it is that takes a lot of effort and a lot of compute capabilities and as like uh open source Community you kind of have to beg and borrow to get that kind of stuff together right so those are I think some of the big challenges on the open source side that being said you know I mean there's just a lot of interesting projects out there with NLP such that you know for a long time I would always say oh well NLP is great and I love working on it but put it in practice is really challenging and and you know it demos very well but it doesn't always perform very well whereas I think in you know say the last I don't know 3 to 5 years maybe even less you're really starting to see really large scale practice iCal applications where people are leveraging NLP on a daily basis and it's it's becoming natural whether that's on your phone or on a website whatever you're able to interact with the machine in a natural way speak the language right um the other interesting thing I think is uh you know a lot of people talk your question about structured and unstructured um it's one of my pet peeve terms because I actually hate the word unstructured when it comes to text and the reason why is if you think about it right text is one of the most highly structured things we have right we're just not very good at telling the computer what that structure is and and letting the computer take advantage of those right the grammar and deal with the grammars and and the relationships and and all of those kinds of things the ambiguity um that's I think one of the cool things that's been happening as well is more and more capabilities are coming around that make that easier to deal with right and obviously you know more universities are teaching this kind of stuff more people are feeling comfortable dealing with probabilities and statistics as opposed to you know just kind of the pure relational models that you know probably you and I were trained on way back when we were undergrads uh and so I think you know as you get that Comfort level raised as the open source gets better that raises the bar for everybody and that's kind of you know in my book taming text to try to address that kind of crowd of people who are getting who are new to all this stuff how do I start to deal with the fact that I may not get the same answer every time I ask the question right because the DAT has changed or or because there's some fuzzy fuzziness in there that you know I just I I haven't accounted for uh by the way you know uh Tam is a great book actually Tom Morton and I were in the same PD program at pen and kind of he still is there uh uh working for Comcast so this is interesting connection and I actually enjoyed it uh I yeah I wonder uh it's it's a very useful book and it's a good primer uh on on on text processing uh so I wonder um um so in terms of scale right uh so I was at the uh 2007 machine translation Summit where Google uh translate talked for the first time and France o presented the model and he said it's very simple if you want to real like your translation let's just collect several trillion words and let's just do seven grams right and essentially that's how how you solve this problem so I wonder this a small this a small problem that's right so I wonder uh given the scale you mentioned at sales force deployment uh can I do a seven G model of several trillion words uh with solar Lucin uh uh we seven trillion words in um I mean I think you can well I mean it's obviously going to take a lot of Hardware there's a lot more involved in than just the pure like the pure uh programming side of that problem in solar right um you know there certainly people who you know you would have very very large indexes and and and you know we've worked really hard in the community to make sure we can scale to very large sizes can we do 7even trillion uh n seven grams uh or a trillion s war and um you know give me enough Hardware I'll get you there I don't know right off off hand I would think we could get in that range though that it would be interesting um you know and certainly when you bring in tools like Hadoop and Spark and and that kind of stuff you know very complimentary you know go off and do the really large scale stuff in in an approach like that and then you know you're reducing it down into something that is more manageable and then use search to kind of access that and and you know query around it and and slice and dice it in different ways in a more realtime approach right but yeah no I mean I don't know do you have a big enough budget for us to do I I'm just interested in kind of scale because you know here in the Salesforce size I I think that's very I think it's getting really there uh where you know uh big guys are playing uh so uh so I'm I'm curious um about your so I know that your technology side is very rigorous and I was really impressed by the fact that you guys are running your own Jepson tests right so japson is this distributed uh system uh testing uh Su which is basically integration test which it tests you know all kinds of failure scenarios so and and uh uh it was done uh as as as a series of of of studies uh but uh uh Lucid Works maintains its own jabson uh test Suite so I I wonder and and apparently it shows that you guys are really uh paying attention to to all kinds of fa scenaries I wonder how do you uh how do you make sure that the customers are not losing data and and you know how do you like what kind of tools do you have and and what kind of scenarios do you see around this right well this is you know I mean obviously the jeffson work happy to get into I mean I think this is one of the beauties of the open source is not just commod commoditization of functionality but commoditization of test across a large number of people when you're the most widely deployed search technology on the planet right you get a lot of people kicking the tires and coming up with error scenarios and all of those kinds of things such that it gets to be you know you get a lot of great feedback from the community at at the Quality Assurance level so you know you've got that layer then we as a company you know we look at where are areas that we can fund efforts around making sure that you know we can leverage the community stuff and then feedback our own capabilities the Jepson thing uh was definitely an area of Interest we wanted to we already knew like solar and solar Cloud was very robust and very mature and you know to some extent we needed to just simply prove it by going through you know what Jepson does is simulate uh Network partitions disc failure Hardware failure all of these kinds of things in a nice self-contained way such that you can test whether you have data loss because one of the big things you know we have companies who run their whole business off of solar right so data loss is a huge problem for them if it happens right and so in fact in where they're actually using solar as the authoritative store for their data right so we've got to make sure that it's rock solid under all conditions now obviously you know no software is perfect but so we want to make sure that we take a lot of you know put in a lot of effort to make sure that we're doing those things the jebson example is a is a great one of uh actually one of our Engineers uh you know one of the things that people often talk about with with solar cloud is oh I've got this extra thing that's called zookeeper and and that makes me have to do a little bit extra work one of the things we wanted to do is show why that was such an advantage because okay got this extra process but the thing that you get out of that now is you know okay it's a little bit harder to start but would you rather have harder to or easier to start or easier to finish I don't know about you I would rather have my weekends when the system is up and in production and all of that kind of stuff then you know a little extra work to set up zookeeper up front because well by the way it's not that hard anyway so we as a company wanted to Showcase why we chose zookeeper as the distributor coordination service for solar and show and demonstrate very cleanly especially compared to our competition of how you can avoid things like split bent brain scenarios Network partitions data loss by making a better decision about the robust you know about the way you coordinate your distributed activities so for people who don't care about data loss then you know you don't necessarily care about zookeeper and don't get me wrong there are use cases for that you know if it's not your authoritative store if you're just doing uh you know some data that you can throw away if it goes away or you can just reindex then you know maybe zookeeper isn't your thing but if you're going to run your business if you're doing you know $2 billion dollar a year in e-commerce and you need to handle massive scale at Christmas time or the holiday season you can't afford to have data loss you can't afford to have the kinds of problems that you know zookeeper helps you avoid so gladly pay that cost because it's going to save you you know in the long run on uh it's great to hear this I mean uh you know I uh come from a distributed systems Community where we use Scola and Spark and a and tools like that so and you know meses and Aurora of course and all these things and so it comes you know goes without saying what kind of benefits you derive from zuker but apparently uh you know it's the knowledge kind of you know spreading through through communities different speed uh so it's it's great to hear that you guys you know rely on that this is proven technology right which is designed to actually you know enable this um so uh so you mention spark and you know since I also uh organize SF scol mop and organize spark M up uh I got to ask about this uh so how do you guys interoperate with uh with Scala and Spark and given that spark now runs on top of almost any distributed system uh you know should I be able to run spark on top of solar for instance and open up you know a bunch of readers read a bunch of stuff in and kind of do something there have you been uh listening in on our meetings at work at Lucid works no I mean that's a great question now we just recently actually published uh solar spark integration it's up on our Lucid Works GitHub page so if you go just do Lucid Works GitHub you'll find uh a spark solar integration there all open source Apache License uh and so that's kind of one level that's essentially about how do I interact with getting data in and out of solar into spark and vice versa right um so that's kind of the one level of thing the other level is in in actually in our commercial uh product called Lucid Works Fusion we are in the process of bringing in so we you know with Fusion we do a lot of things about signal capture there things like click stream analysis all that and then turn that back into relevance Fe you know back into relevance gold if you will so if you take all the click information that you're getting then turn that back around into better search results right so we do a lot of that in solar right just out of the box and we can query solar in the right way such that we can do things like recommendation engines right out of solar collab of filtering all that kind of stuff where it then gets interesting is essentially those are all like aggregation problems right or building up models and they often require large scale batch you know style compute this is where Spark comes in as if in the truly large cases where we have you know uh you know you're doing millions and millions of events an hour or you know a day Etc and you get a really large essentially Matrix spark is a great place to come in and do that machine learning work where you're building a model in spark and then you you plug it back in Works seamlessly with fusion and then you can go and you know query that model so if you're building a classification model if you're building a recommender etc we can tie that all in together and so that's going to that's going to be part of our product here very soon as well so so yeah we love spark uh you know we did map ruce for a long time saw the shortcomings uh you know another project I'm involved with Mout uh that's all been switching over to spark and Scala as well I'm not as deeply involved there as I once was uh but you know a lot of great work there about bringing more machine learning capabilities into sculla and into spark uh you know around there's some really nice uh there's a really nice DSL in there for building generalized linear models now uh right you know leverages skull underneath the hood and and makes it super easy for people to build out uh those kinds of things build your models and that's all spark and scol underneath the hood as I understand it so uh you know I myself am not a scholar person but uh you know we we do see it in a lot of different places and have some of it within the company as well so that's great to hear you know because uh I can tell you you know kind of a as a developer who build systems uh we use both search and and NLP machine learning and extraction I think uh uh solar and Spark make an ideal combination because for instance if I round up of a database I essentially have to think how I'm representing my textual data in the database right and you guys spend a lot of time thinking how you do that so so essentially for a lot of applications we're going to see uh uh discuss SE text this makes actually a very good combination so maybe uh since you mentioned you guys are developing this maybe I can ask you to come back and and talk about it when you guys are developing it because that's certainly a great combination of Technologies so in the last question I want to ask you uh so uh the I think you really present interesting path in open source because you you you're the guy who interested in specific technology Masters it very deeply and specifically with Lucine and then you you turn it into a successful business right which is basically used by by a lot of people all over the world so uh what kind of advice do you have to open source people who who want to do the same there is a lot of uh folks especially in the uh kind of uh technology area who develop tools uh what is you know what are the important things you have to pay attention to if you want to you know take an important open source technology and make a business out of it wow uh that's a great question um you know I mean I think there's a lot of lessons learned across so many things you know I mean building a business is so much more than just the technology right uh and open source as a business model has a lot of great things and a lot of challenges as well right in terms of you know you've got uh you know for us it was interesting like uh very early on you know we were like uh let's just go we can just go put out our our welcome mat and people come to us for solar support and you know the brake fix hey I need insurance for instance off of my for my search engine right and we still do a lot of that business but what's interesting is that solar doesn't break and so people are like hey that's great we love solar but you know come renewal time hey I already know and it works and I haven't had any trouble and so one of the interesting things that we've evolved into is like you've got to have uh capabilities on top of that so when we think about how we engage with our customers we've shifted very much to what we call a customer success model which is hey you know we're going to be there for the break fix that's one thing but in reality what customers kept coming to us for is help me understand the difficult problems better you guys are expert experts in search you know machine learning you KN know natural language processing help me with product in terms of tools that make my life easier to do those things but then also hey how do I make this better how do I have better relevance for how do I scale this how do I grow with my business how do I ask more interesting questions of my data and so it it what the customer success model that we've put in place you know like I said obviously has The Brak fixed but is also really geared towards making sure customers can grow and be successful together with us so you know open source is one part of the business decision is should I put my technology out in the open and and you know everybody's going to be different there uh but you know I think one of the things that you have to think about is you know is this technology something that has a big enough base that I can actually build a company around obviously you know it's got to be more than just your pet project or you and your buddy's pet project uh and it's not enough just just show up and say okay well hey I've got uh you know kind of Open Source kind of goes through these phases of you know it's a couple of guys or or a couple of people together working on something that's of interesting to interest to them and then it usually evolves to where you can kind of support an ecosystem of one-off Consultants who can come and say hey I can help you with your hard problem and then you know it gets to the likes of the dupes and the Sparks and the the solar Etc where you can sustain multiple businesses around it and you know and and they all contribute back to the core but they all have you know presumably their unique differentiator and I think you know depending on where that project is and its lifespan will determine how you can go about uh commercializing uh open- Source technology um you know the other thing is like you know you just always have to keep investing in the open source and there's always this kind of delicate Balancing Act of you know at the end of the day you have to you know quote unquote put food on the table you have to pay salaries you got to keep the lights on Etc so there's something that you have to be uh you know that you have to be able to monetize but at the same time you know the community pressure is often how do I you know I want all of this for free they want your time for free they want your code for free Etc so like you have to kind of work together to come up with a win-win situation for everybody such that the community is benefiting as well as you know you're able to uh to pay salaries and all that kind good balance between contributions from the outside community and the company employees and so like for us you know as a company we look at how uh we can complement and extend solar we still invest a lot in core solar I think we're the single largest contributor or two solar in the community uh but then we also look at from a product standpoint is what things do we know customers want that the community doesn't do so for instance you know solar itself doesn't do connectors to you know crawl the web or connect to Twitter or you know SharePoint or whatever file system repository there is out there so that's a gap that we fill as a company right or solar is primarily made up of developers who are Focus on the back end you know algorithms and and core search functionality they don't really care about search UI so we come in and bring user interfaces to the customers that allow them to work with Solar without having to necessarily write Java code right they probably care about the U but not about building the US exactly and then we you know like and then like the the other part of how we are with solar really is how do we ask more interesting questions of solar so we want to drive more data into solar such that we can ask more interesting questions of it so again it's kind of that complement and extend model of Open Source if you will that I think uh you know has worked well for us and and you know we're continuing to invest a lot in in that way of doing things oh that's great I think