SF Scala: Q&A with Nadav Samet, CTO of TrueAccord, by Alexy Khrabrov
Recording: SF Scala: Q&A with Nadav Samet, CTO of TrueAccord, by Alexy Khrabrov
oh uh hello everybody i'm alexis kravrov the organizer of sf scala and here we are on location at radius intelligence for tonight's meetup devoted to protocol buffers it's a very exciting topic we all love our data we'll have civilization we have two talks with david you and nadav summit and here we have nadav summit with us hi love hi nice to meet you so now is the cto and co-founder of true accord a startup which is using scala and uh i let nadav tell us about it in his own words so what is to accord and uh how is it using scala torah code is fixing a broken industry the debt collection industry right now a lot of relationships between merchants and customers are being broken by the debt collection process if a customer owes you money and you send him today to a traditional collection agency you know that most chances that he will be treated badly and we're trying to transform the debt collection industry we've built using machine learning and behavioral analytics algorithms that would classify the customer we'll know what to what the customer is about infer the primary objection why the customer is not paying and based on that we'll try to engage him in communication with our system so we'll send him email and the content would be using the tone of voice the language that would resonate with that customer and get him to talk to us when he's in our platform we'll try to get him to negotiate with us on a settlement and eventually resolve the issue and all of that with low touch no call centers no no aggressive phone calls everything is a 21st century experience online very very advanced i mean that sounds really exciting i think a lot of folks you know our students and had dads and have experience with uh collectors uh and uh us is a lot uh driven by that so that seems like a very important subject i'm very actually interested in this because you mentioned machine learning and data mining and that's an area where scala is often used because of various uh data mining tools available in the gvm platforms maybe you can talk a little bit about why scholar and uh what kind of data mining is doing there yes so when we get an account into our system our system tries to classify it and figure out what this person is about is it an individual is it a business if it's a business what is the is the business up and running or is it a business that is about to fail and all of that is based on data that's online so for example if you start with an email address and you see mike77 gmail.com you'd guess it's an individual but if you see an email that looks like a firstname.lastname.company.com you can go to the company website can you go to to find that company and see who are the people who work for work on that company and you can tell this person is associated with the company or not how the company is doing and so on so we do a lot of data mining to to build to extract features about every account and use our base base our engagement strategy based on that account and we did it in a very reactive way so our system is always looking at those accounts it's looking always at our logs we're streaming the logs to a decision server and all of that is based on akka there's a actor system that is doing that it gets the it basically we send for to the to a decision making every action that happens on the system every page viewer every email can trigger um contribute decision making we make right now million decisions every day and we constantly look at the decisions the system making and monitor the results of them to make sure that we tune our algorithms and we improve the recovery rate for our customers and so this is basically what happens we are extracting data that we can find on public sources getting to analysis and decision making acting on that filtering through compliance checks because we are very regulated industry yes and then monitoring the decisions that we are making so you mentioned tone of voice do you analyze actually phone conversations and tone of voice or do you make recommendations in general how does that work so when we know about a person if he's a young american male we might choose we learn that we can speak to him maybe with a sport metaphors and that would resonate to him in in on a in contrast if it's a an older business guy we might use a different way to discuss the motherhood or with the state-owned mom will use different language in different words you'd use different triggers to talk them and this is where the behavioral analytics is coming in uh so when i say the tone of voice i refer to the way the emails look the colors the language and and the general look and feel of the way of our website changes based on the messaging and on the person this is very interesting so to a young guy you can speak like a parent right and so it's an older person you can speak like a friend and so the persuasion strategy would differ that's really really interesting so how much uh of uh natural language processing do you do do you plan to do because it sounds like you want to understand the conversations themselves so we don't do natural language posting yet i think it's going to come i'm looking at the emails we're getting and a lot of them look the same and i have thoughts about oh i can actually generalize this i can recognize these keywords they're repeating and i can infer something from them right now we have humans that actually read the emails we have right now one email per 15 000 customers sorry one customer a service person per 15 000 uh debtors or in the industry you usually find one per 800 so we're definitely doing something with more automation that's more scalable we're totally changing the cost structure of of this business um but your question we don't do email or language natural language processing yet but this is on our radar for next year yeah that's you know that resonates with the another conference we've been done recently called texts by the bay where uh multiple companies are coming to basically understand uh natural language spoken or written and uh i think uh there is a lot of uh parallels i see for instance zendesk right companies like that receive a lot of help uh support requests right that compel to understand some generality so that sounds like uh also another case where scholar has some good libraries uh but uh so why is color right it's uh it's uh not a typical choice for data mining uh i mean it's our favorite choice of course but i wonder how did you guys come to that uh technology so we started with python and python is a language that i used and for many years and i still love but at some point we went to realization as the system grew and got more complicated that it doesn't open when the team grows so for example you create a function that takes a phone number and now you need to know is it a phone number inside an object or is it a string that represents the phone number how do you know so you have to write a lot of test cases just to make sure problem program doesn't crash in production yes and but most important that is that people that read your code can't really tell without reading every line of the function to get some hints about what he's expecting to get and we figured out that we you know there is some cost when you use like a language like scala i've been using scala since 2011 and um there's some there's some way productivity costs when you have a type system it slows you down by 10 but you get the leverage out of that when the compiler can actually tell checks for things that common bugs and you have much stronger guarantees about the correctness of your program when you ship it that it pays for itself yes yes well that's a great uh that's like i think it's a great strength of scholar right that the database a lot of these cases i wonder about about your data mining which which libraries do you guys find find useful uh what do you use for storage so you mentioned use akka how do you persist all this volume of information okay so a lot of the data mining today is actually still in python in our system we use protocol buffers to represent our data and we store that data as a we use mysql in a very interesting way we use mysql as a key value store and where we serialize the protocol buffers into it okay it's a an approach that was inspired by an article that the friendfield was putting together back in 2009 so coming from google and bringing a lot of like you know a lot of like an out of fair of ideas and how to do things i really missed bigtable at the same time i didn't want to go through all the complicated or let's say adventurous tools to store my data we are working with financial institutions we work with banks right i didn't want to go on something that we might not understand well enough and will cause us you know problems down the road so i want something solid like my sequel and then it's working it's been working very very well for us this approach it's you basically get a very robust key value store yes my secret is way more than that of course but you it giving you a lot of advantages in there in the way you can get and like store and retrieve data very efficiently and then you know like a system around like replication right like there is a lot of uh tools around that it's actually very interesting because uh when i was at amazon we used oracle for for money open source for everything but oracle for money and i noticed that actually it's a pretty common pattern because oracle comes with all this compliance guarantees surrounding it and and so i wonder because you know why didn't you guys pick oracle because that's kind of what people use for transactions about money and you have my sequel there yes so that's actually an excellent question i came with a lot of years of using my sequel so i wanted to use something that i already know and i like at the same time um i wanted something that would help us you know it's more understood in the startup community there are more tooling for it and we wanted to use rds where we can actually have amazon run the server for us and amazon gives us a lot of things for free when we use myself so i don't need to have like our own team running the database we get the backups we get in the encryption we get a lot of things for free and a multiple availability zones fallbacks stand by servers all of that so it's a huge advantage okay nice no i see that's that's a very good use case for rds actually uh yeah and kind of uh i assume that every event is an individual collection event it's like it's not like if you really have big data yet right so so so that but you need to care about every event yes we cannot lose an event right in our system it's financial data it's a something that we know we work with banks and we have very you know we have very high expectations of how you treat your data how much you care about privacy where it is stored who can access it it's something that we really take seriously so you mentioned uh a google background maybe you can sort of outline your kind of technology trajectory from israel and another question i'm very interested in there is a lot of advanced israeli startups which kind of you know start in israel they move to the bay area very often like what is it about the different culture maybe if you can generalize about israeli startups like why why are they kind of focused on advanced technology is there any difference with what you see in general in bay area yeah you know i think it's an excellent question and i think nobody knows the answer for real so i mean i'll give you my take on it yeah and you know israel is a under very special circumstances all the time yes and i think people they learned how to deal with pressure that you usually find in startups you need to justify your existence all the time so you need to be very very innovative about the things you do and you need to be able to code under fire so that's i think why there's so many startups and there's a lot of things that attract startups here because the early startup is it's a magnet point for more for us a lot of the companies that we serve and are based here we will our favorite customers are companies who have like internet bone debt and then this would usually be companies that are around here with clients here around the us yeah and that's because it's already online they're looking for innovation right like evolutionary workers like you yes it doesn't make sense for those companies to move the experience back to offline to traditional collection agency deal with phone calls with checks right and all of that right and they already have the data in some format right so which brings us to to uh to the protocol buffers which i understand kind of uh i heard that you know folks from google are kind of uh bound to recreate google everywhere they go so they want to build everything uh so it sounds like you know regal buffers is one of those things but it's you know there are there are multiple options available now for something ever there is protocol buffers cryo and scala so there are different serializations and it's a big problem for jvm it's bad at serializing things like it's slow it's not c so i'm just wondering like uh why protocol buffers and you know you really did a lot of interesting work there you know using microsoft this is like really uh something whichever does not have you know scalably b has no equivalent than ever uh for you know forever so why practical buffers you know like why do you store all the data in them uh what what is it about that kind of media who writes color pv sure so when you leave google and start a company you suddenly figure out that you're on your own there's a lot of libraries that you used for years like protocol buffers yes and you and you're out of like all the tools and you're a little bit on your own to you know to build your development tools to choose the libraries and then you try to figure out what google already open sourced that they can actually use and leverage that i already know familia know all the ins and outs of it and that they can just leverage and be productive on day one protocol buffers was one of them for me and we landed on a we start with python like i said before and then we have protocol buffer bindings for python as we move to scala we use the java version that google provides for protocol buffers and on day one the java and python the java and python implementation could already talk to each other because it's the same binary format yes so we could actually get build a system and and and users would not feel the glue that we have between those things it would just like forward requests if we didn't implement requests in scala yet and we had a python implementation if you just send it to the python server as protocol buffers get it back everything just worked like magic at some point and this is a i'll touch in in the talk today the java protocol buffers were a little bit frustrating for us as scholar users a lot of builders a lot of mutable stuff yes and we wanted to make something that is more functional more idiomatic in scala and this is how scala pb came to be cool yeah that we also you know kind of a bit of background how we you know uh that's a scala filed about it so actually at night we're looking for a big data civilization format and we actually you know as many people were looking for ever and protocol buffers and so skull bp was one of the huge arguments in favor of protocol buffers uh you know but in the big data world alvarez actually uh gaining the momentum so we ended up uh using evra but now we have a task we need to write something like scala pb forever right because people who looked at it said this is so nice uh it even has lenses so they really kind of uh wanted to use it just for the ease of uh access from skull because skull people it's very important that there is an idiomatic way to access it right and never does not have it and then it just like have all this you know tons and tons of java and nobody wants to look at so but that's that's interesting so but uh i mean uh this is a serialization former sorry so so uh why not json for instance right like why did you have to use because google has big data right so it sends that all the time so you don't have this volume of data yet why did you need to serialize it anyways yes so there are a lot of reasons to use protocol buffers over json and json are great and they have their place but inside your company it's very good idea to use protocol buffers one of the one of the advantages is that you define your schema in a file and that file serves as documentation of your data format so you let's say you have a person a person has a first name the first name must be a string yes you send it to another person in a different team in your company and they can say this is what i expect this is the list i have of fields these are these are the types and you know that you can set up type safe communication between different teams in the company as well as the file that documents all the data structure and these very advantages at the same time those particle buffers uh support schema evolution so you can you can define fields as optional you can remove them add more optional fields later and the power cells and the serializers evolve and you can have different components of the system running different compiled versions of this of the same protocol buffers and they can still talk to each other and then in a very very efficient way so this is one of the advantages so this is data uh evolution strategy right you can have all the new data and still be compatible yeah uh cool and then we have of course i think for you know for legal reasons like you need to have data integrity right so uh so this is adds an extra layer there cool so you mentioned that your company is growing and i'm just wondering how does uh the engineering culture evolve around scholar what do you find you know when the team grows right you made some initial choices by the way how did you choose skull in the first place yes so i've been doing scala since 2011 and we look for something that can run on jv on the jvm and leverage the huge plethora of available open source libraries yes and i didn't want to do java at the same time yes for a lot of reasons we're in scala meetup today so there's obviously a preference for scala that's right and um and i decided to that you know it would be possible to build a team here of stellar engineers all of them doing scala and i think that you know in this industry in you'd find that there's some self-selection in people who actually get interested in scala so usually most people don't have a scholar job yet yes and but if they bothered to learn scala or they got interested or they you know opened the book or they watched the video uh that makes them very very good fit or or the type of engineers that we want to hire we want to hire the people that have like you know passion for technology and they want to learn new things and work on real world business problem that is really really ripe for a for a solution that's i mean that's i think it's a really good good uh situation for scala but how do you see your culture so you're a skull enthusiast and i'm a scientist you kind of know what we like about scholar but probably as you grow right you will probably get some people who are not necessarily uh as enthusiastic as us right like the maybe they will maybe they're not there yet maybe they heard about scholar right so i wonder how do you see the culture evolving uh as your team grows yes so i'd really would like to grow our team around scala with a lot of stuff that we are doing with scala we have another team in the in the company that does python for a lot of the analysis and the business intelligence part of of of our system but as we grow we really spend a lot of time thinking about like how we're using scala what libraries we're using and what parts of what we're doing is actually general enough we are building basically something like a marketing system that is focused on that collection right so a lot of things of the strategy we're using about marketing and a lot of things problems that we solve can actually be applied to different businesses in different industries so we always think how can we grow and also have like a more open source project that we can get out this is great i think you know this is a great uh this is a great model and and uh actually this is scala python interaction is something we're gonna address in august we have the conference called big data scholar and you know there are two themes end-to-end pipelines in in scala and data science on the jvm and we actually we're looking for folks who use both so this is great that we found a company like this right because uh our goal is to bring as many people as possible from python to the jvm and for that we kind of need to map out what are the missing pieces right like what are the gaps we have some linear algebra tools we have excellent data mining and machine learning right but some things are still missing some things python people do love like pandas and scikit and things like that right so it's so it's it's very interesting to see how you guys will evolve uh you know next year we should definitely check in again uh and so i will wrap up with uh kind of uh a future facing question right so where do you want to be in the year and how do you see your scholar stack evolve uh where do you see your technology going let's make a prediction and come back in the year see if it's true right so i'm taking a risk and making a prediction next year we want to serve the biggest financial institutions here in the in the in the country credit cards banks tech companies uh in terms of our stack we want to grow our scala stack get more you know more of our machine learning and more of our algorithms be implementing scala right now there's a lot of that happening in python and we are moving it slowly productionizing more of them to in scala we'll see more of that and as our system gets more and more complicated i'm really hopeful that we can release more open source projects out of the world so there's more visibility of what we're doing that sounds like a great plan so looking forward to this open source contributing collaborating and uh trialcore.com you can learn more about the company and the dove and the stack they're using and i'm looking forward to talk today thank you thanks you