scala bythebay io: Rajesh Muppala Interview

Recording: scala bythebay io: Rajesh Muppala Interview

hello everybody I'm Alexa Crum founder on the organizer of Scala by the way the yearly conference forskolin scale and here we are on location Twitter and I have a dish map along with me who is the founder and director of director feature of index a company out of Chennai yes and I met him at Pacific no rest call a couple of years ago and I'm very happy to finally have him here with us he was on a data panel welcome Rajesh thank you how can I do a little bit about index and what is the trajectory and how you guys are related to all this scala technologist we're machine yeah so so I tend X we are trying to build a Google of products I mean what do I mean by that is you know you have you know Facebook and LinkedIn having an index of people and Google having an index of locations so we want to be the index of products all things so we do that by you know crawling so we have a you know crawling engine that is built using akka cluster you know that's about three years old and that's when we actually started with Scala so over the course of time you know we've used started using a lot of color technologies we use carding for our machine learning data pipeline and more recently we are using spark for our end-user platform that allows end-users to you know build data pipelines and get data out out of our systems mm-hmm so yeah so we are we started as a Java shop but you know once you know we moved to Scala about three years back and we've not you know gone back since yes I remember vividly you showed me so you worked for parks before I see you using your OCD and you've got this amazing dev boards all the socket processes go away and all this stuff crawling and be recognized and so I mean it's a fascinating problem right basically to collect and resolve all the products in the world yes so what is what does call a platform give you why did you choose it yeah I think I mean see the collar it's already you know it called as a disputed system and you know I worked on go CD and you know I built I mean that goes into the disputed system and I we use Java there you know I didn't want to write the low-level you know code of handling concurrency and all because you know it's very difficult to reason out what's you know what happens when things go bad and with akka you know which you know which are just you know started it felt like a you know nice nice fit fit for our use case also at the same time you know we realized that for we were writing a lot of Java Java MapReduce and it was not the right kind of abstraction for writing complex data pipelines and scalding you know with its functional abstractions in appointed as the right kind of tool that we were looking at you know in people we also tried the pig but you know scalding again was was the right choice for us so so I think it was when not just one reason I mean there multiple reasons I mean these libraries and the ecosystem itself that you know help does I mean that made us choose color yes and I think there is a lot of interesting challenges there so how do you guys kind of connect data engineering data science right so you do machine learning and this products like what are the challenges of doing these two things together I mean when one challenge has been you know the data that we get is not structured mm-hmm I'd you know a lot of data is unstructured yes so so I mean bulk of a time is spent in actually structuring this data and making it you know useful for data science yes and you know to be frank you know we made a few mistakes in hiding itself you know where we went invented and and hired just data scientist but we're looking for people who can take do things end-to-end you know build models as well let's take them to production who can both be the data engineers rather that exactly yes this is great yeah we call this guys through social engineers in some of the companies I work for so so I this is a fascinating problem and I mean in San Francisco Bay Area there is nothing hotter than data science machine learning artificial intelligence right and obviously if you can do this with data platforms which scale right this is the Holy Grail right so and and you mentioned that kind of in not not everybody and you community understand this and so if you were to explain like obviously we're the epicenter here right so so how do you see kind of the evolution of let's say I know I take community in India right oh how do you see this kind of wave of machine learning big analytics taking a hold there and why is this important to really pay attention to this I mean in India I think I mean the startup wave is pretty good there right now but I don't think a lot of companies have are focusing on the data side of things yet I mean yes you know it's not their core business I mean it's you know they have apps or they have you know end-user applications or consumer apps but you know they use data science as a you know as something that adds to their their business but it's not part of their core business mm-hmm so I think I mean we'll we'll start seeing lot more data companies I think I think they're one of the earliest and I mean at least I don't know if any other company that deals you know as much data as as we deal with so I mean it's really hard to find you know people who have you know experience in some of the stuff that we do I mean we don't get Hadoop engineers but you know they've not worked at the scale that that we were coming therefore they want some courses or you know yeah it's yeah this is a tough problem so so I remember right Oh metal you basically mentioned that you know you went back to Chennai right and you want to work in India you want to employ Indian engineers and and I think I've seen kind of a wave with you know Modi government kind of technocratic focus I've seen a lot of young people actually and all people and all kind of people going back to India I saw Francis Pankaj Gupta was a Twitter guy right who did you know basically whom to follow recommendation system and now he went back there and so we see a lot of execs and developers so can you tell a little bit you know how this works what do you see you know from that other side yeah so I mean I think so so if you see right I mean there are there are a few unicorns so to say in India I mean there is there are the flip cards yes you know we're going against Amazon you have the Ola cabs that's going to going against uber you know then you have oh yeah so so there are a few few companies like that and and I think you know they're you know definitely you know having people from here in the Silicon Valley who have done these kind of things you know scaling engineering teams back in India would really help help is what what I would think yeah and a lot I think you know I think the the companies which hire kind of execs from this kind I think the exclusive to say we need guys who have experience with Big Data machine learning right so I think the court in the corporate level the pretty smart they they understand that they need this kind of talent infusion right because the scale here I think is leading will be capital big data capital of the world right and so from here kind of percolate through but so I'm curious about the community you mentioned the committee is not as intense as developed but community is the main way of learning about this technology so what you see here is completely organic right people understand because there is a demand for this technologists in the market and this is the highest paid jobs in the industry they basically folks learn oh this is this conference is learning vehicle right a lot of people come here they learn about the technologies and then the K basically go to companies who need them so I personally think this is inevitable in in India as well right so I'm curious you know how do you think this can be fostered and you know if somebody helps with the community what rewards they can reap by by being early kind of community organizers in India I mean I see I think gues there's definitely a need for building the community and you know I think um think companies like index has to you know be part of it because you know I think we are doing some some really interesting stuff in this area I mean unfortunately you know we've not been you know able to do in spend as much as as much time as we would like but it's one of the things that we would like to do I mean you know I'm probably when I go back you know we will start this community I mean the challenge is also you know the maturity of folks there is when people are just learning about big data is getting their hands dirty with Hadoop so how do we get them from mere learners to you know practitioners I think that's that that gap is what we need to bridge is what I think and and there is I mean and there's work to be done and you know hopefully you know we will do something about it yes and and there's also one more point I want to make I mean one you know this is about what the work that we're doing at index so the goal of goal for me to add I mean the reason I am here is to probably you know because there are a lot of speakers you know who who are actually building the stuff that we are using so I wanted to you know talk to them so probably you know 80% I wanted to you know learn from them and probably 20% share what we are we are doing endings but one thing I realized when I when I came here is that do you know actually you know it's 50% learning in 50% sharing in the sense that you know whatever stuff we build at index you know some of the stuff you are talking about using go CD as a big data pipeline yeah is something that you know folks from Apple from stripe I mean you know they've actually said okay can you can you show this I mean can you know can you send can you you know tell me more about it I want to yes I want to know more about it I think that's that's a little I mean it was surprising and it's also flattering yeah so yeah and yeah I mean I had the same reservation right I thought but basically it's kind of what really impressed me about in Dick's that you guys kind of quietly and you know built everything everybody's talking about and so we often have talked to people who build one half of this probably presenting how they put two things together you guys put like a million things together and they have a dashboard right and it has been going at scale right so I think it's really interesting whether you have something very vailable have a proven engine which is you know internet scale which right like comes and finds all the products and resolve them and it's a thing it's kind of the modern startup worlds it's an extremely valuable thing basically this is the final result of all the fundraising and all the promises people start with right so it's like two beats having said that you know we still have a long way to way in the sense that I mean one of the big challenges we're dealing with is the quality of data itself because you get data from multiple sources yeah so we actually and I think we figure out the problem of ingesting that data you know the scaling of the pipelines yeah now the problem that we're actually dealing with is actually how do we improve the quality of the data and make it more useful for our for our customers right in I mean again to me it's fascinating right because you know essentially data science is 80/20 it's 80% data plumbing there's veneering data wrangling right so so once you have the efficient structure to iterate upon then you can improve the quality because you can pump the data through it quickly and that's the phase we are in right now all right so I'm kind of interest like let's say you know I hope a bunch of folks in India or C see us and how would you describe your ideal candidate and you know the young people who want to of take courses and learn right what is your ideal research engineer and what would you suggest you know to them how can they build up the skills needed to do the job of research engineer now I think one is I mean then they should you know understand basic computer science concepts I mean they should you know they should understand algorithms you know they should understand yeah under the concept of computer science the second thing is I mean they should love data yes I mean they should you know they should you know they should not be afraid of you know taking the sleeves up and you're getting digging deeper into data and and understanding what what what it is all about and then and I think you know once they have these two things right I mean there is you know enough of let's say coaching and enough of mentoring that we can do in our company or or I mean you know they're enough resources that are available for them to you know be productive mm-hm and kind of let's say you know they're good enough and acquire them as junior musician engineers what will they learn on the job what can you give them back what is the advantages of being exposed to all these technologies you guys built for them no I think the the learning rate I mean the you know you get to do what you know you get to do in a year you know what you would get to do in another company in over five years right when you're I mean so I I have this I mean I talked about this concept of you know what is called an escape velocity right I mean once once the rocket reach an escape velocity you know it's it's on its orbits on its own right yes it's very important for you to figure out you know the right company in your in your start of your career so that you know you so that you're maximizing learning yes and then once that happens I mean you are you know you're all set in your career so I think you know in given that what given the kind of stuff that we are doing I mean there's hard problems you are solving I think we can draw a desk a velocity to developers is what I think nice so then you know my follow-up questions will be how do you make sure that your most valuable developers not escape all right because if you train all these guys and then satellite Twitter and Facebook and Google see that they use the same technologist and there's not fully what they're doing right and they all kind of come into India how how do you retain the top talent I mean I I don't know I mean I think we are we are struggling on that front you know I think it's very difficult for folks in you know in my company to appreciate the kind of work we are doing because I mean it's it's a hard problem in trying to create a new market I mean on a business side I mean we're not you know we're not having a trajectory that you would like because you know it's a hard problem to solve so so I don't know I mean I one think would be you know if they could come here and see for themselves you know the kind of impact that you know what I could see here right I mean people are coming and talking to me about you know what is about the scale about the idea pipe data pipelines mhm so so I don't know I mean yeah let's see yeah cool you know yeah we don't know that you sure but I think you know from what I can see I am really kind of happy about the trajectory of your garden right and I think it's definitely great to have index in our community it's great to finally have this conference will now hope you know next time you come and do a full talk right and kind of we collaborate on the meetups and community and we're help you know happy to help you guys build this community yeah thanks a lot and I think you're doing a great job with the community here and good luck thank you thank you yeah thanks