SF Scala: Rolando Manrique Interview

SF Scala: Rolando Manrique Interview

Recording: SF Scala: Rolando Manrique Interview

[Music] hello everybody I'm Alexey Krylov the organizer of Scala and for the first time I here at land up and we also have Rolando manrique who is a speaker tonight and he is the principal software engineer and turned up welcome Rwanda thank you very much thanks for hosting us yeah so tell us a little bit how you know did you get into the lab what you were doing before and what kind of technology are you guys using here sure I start at land up very recently just earlier this year in January I came recommended by a friend and as soon as I joined they mentioned that they needed a revamp of their or their data pipeline and and that's how that's how I kind of that's the project that can even get me introduced into a company we we are looking for scale now where we're making sure that that we can grow with their business and we have to have a data pipeline that can support the data growth that were expecting for for the next few years and be able to process all the records that come through our different external providers so that's kind of what that was the motivation to to build something like like lobster which is the type that we're gonna do today it's basically we had to have something that that would allow us to scale significantly in the in the in the years to come mm-hmm and obviously Scala integrated using Scala yeah so how did you come to use Scala you know what's your personal story of Scala well my personal story with Khalid started at Verizon when a lot of very famous guys in the scale community were working there I think I was lucky enough to to get introduced to a Scala world alongside a group of people that was really helpful at helping others I mean you could ask guys like Stu or team or any of these guys that have done talks in the past and and go ask them about questions on libraries that they've temperance yes Steve took honor and compared and Brunner definitely you know with the with the red book we we could actually ask questions directly to him and and it was a very good experience for me and and I was able to learn a lot from these guys from there I came here to lend up where they have the they have the motivation to move to a functional programming language to move to a kind of platform that would scale and and I think my experience a building pipelines at Verizon with very large scale data sets was a very good feet to to the level of needs so I think it was good moving and and I'm enjoying myself here so team and sue and ruin our rural speakers at our be tops and conference yeah about the bay which is in November and I guess should submit a talk to the CFP so and I mean ruler gives one of the most beautiful but hardest talks right so he sets the bar very very high so I'm just curious out of courtesy how does it work in that kind of day-to-day setting right obviously most folks probably will have questions yeah but all this stuff I think that was really interesting because I definitely can sit in one of those talks and I I say to myself I'm definitely not a functional guy it's not not I'm not at that level I'm definitely not a Bruno sliver or they didn't speed work or any of these guys but it was very satisfying that you could approach any of them with a very basic question and and they they feel motivated to teach people and they feel motivated to have more developers join the community and understand this concept and that's kind of what my talk is about it's like not you don't have to be at the level of the really famous category Theory guys you can you can produce very good code if you use the the right tools in in the functional tool bag and and yeah I think blobstore is a very good example of this okay and so I noticed it's called FS - Bob soar and you know like basically functional streaming it's it's it looks complicated so I'm curious how how is it to use it and maintain it in production because you probably need to understand thoroughly to debug it when something goes wrong so so how does this kind of beautiful theory translates to maintainability of this code yes it is definitely very high complex level of streaming programming that we're doing here but FS - I think it's a great tool that provides you with most of the abstractions abstractions that you need mm-hm we're going to see in the implementation samples that I'm going to show in the in the presentation that you you just need to use the tools correctly and and if you have an input stream use using reading for stream function that that FS 2 provides and and then you have very good tools that allow you to be very resource resource conscious and and be saving in the use of your resource and make sure that you're not leaking anything and and it allows you to just use in very abstract and very high level tools to actually make use of this very complicated set of code that they let they build for us okay and so you mentioned you have basically a variety of input sources right so you have to interact with water services I mean I used to while I go work in a payment company so I can understand like you felt like a crazy amount of services some of them probably legacy mentions FTP yeah so so I wonder how does call help you with this no it definitely helps a lot and we're talking legacy as in like batch COBOL files I mean from mainframes yes so what Scala is it's very useful and then you start combining all the tools you know you're streaming your data from the SFTP servers and and we need to make sure that if we say we if we tell the SFTP server we downloaded this file this how file has to be persisted properly otherwise the bank is going to just remove the file from the server for us so we combine the data capability of streaming the from the SFTP servers and streaming directly into s3 that gives you the atomic upload so we know that if there's no exceptions s3 is going to be an atomic operation so that's the kind of tools that we're using to make sure that our pipeline is robust that we don't think we transfer our file to a local file system we tell that to the SFTP server in the bank they delete it and the Box goes down then we lose the file we have to go do manual process restage the file with the bank and and it's it becomes a few hour process of an important communication with the external provider that by using the right tools we know that that we are persisting the file in an atomic fashion so if you buy Excel or whatever if like vowels Gong is the staging automatic or do you actually have to manually have to go and open a ticket in the band and send them an email and it takes a few hours for a little wallet no we definitely want to avoid that so by streaming directly from the SFTP from the bank to s3 we know that as soon as our stream is complete everything is closed we know that that s3 will will persist that atomic file so you know imagine data pipelines and I think it's kind of one of the key topics of our community because we noticed my skull is very good gluing things so skull was also used even you know in Kafka you know in in Sparks the skull is used for some of the pieces of this but it's also included kind of connecting things through microservices so I'm just curious you know obviously you guys wear Java shop right and so how do you see the kind of reduction of Scala what are the places where it's being used like how do you see the future was like what's the trajectory which parts of the legacy wanna live there like chipping off first yeah well when I joined I definitely started with the pipe line which is the back end of the system that has no direct user impact I mean the data it's gonna be served by our services but it there's no user impact so it was a very easy sell to a company like hey let's let's release in Scala so I can I can show you what we can do with it with a with a safe approach and not not risk our production systems now that we are going to like the process of splitting the monolith that that's what we're identifying different pieces of the system that we want to that we want to make sure we provide we start introducing SCADA and that's kind of my job is advocating for for Scylla in in and up and the other side it's great because it's definitely gonna have to get everybody excited here at land up but yeah I mean one of our big goals is scalability we need we are growing business and and then we have to be able to have systems that support that growth mm-hm and definitely going to to to micro services will help us with that but but we definitely think or I definitely think that Scott I would be the right tool for it and it's a matter of showing the rest of the team how how much stuff you can do by working on the pipeline we already started working with engineers in the team that are very excited about it I'm there they are becoming themselves kind of advocates for the for the change to Scala and and it's great to have like one of the the block story implementation was built by a person that had never seen Scala before this year and he was able to get it digested and bail it going back to the point that you don't have to be a category theory expert or a professor to build this very nice tools and very good very good coach this is great so I also heard from Ray our host here that you guys using graph QL so it got me immediately interested because I noticed that a lot of scholar companies Peak graph kill but as I said you guys are not yet using it it from Scala right there's kind of it's a front-end thing so I'm curious you know are you thinking about using graph gear what are your thoughts about it you know like would you guys try something in the backend yes that that's kind of where I have my eyes put for the next project I think once we get our pipeline stabilizing and working into and out there in production I would definitely start want to start looking at the services that we already have and graph QL and and all those things that we want to improve and grow in and see how I can combine Scala with Rafael I think that that's definitely a very good match and yeah it's it's it's in the roadmap and so I'm not just obviously like you guys have a JavaScript front-end yes right do you interact directly with it does it call skeletal rest or like what kind of a camp is the back like that's where that's where we have our graphical integration so basically our front is both of mobile front and our web dashboard they both call our graph Gale back in and that's that's where we want to start splitting apart the monolith and then we can we can start having less less dependency between all the systems that we have in the marlott and start having like this this dashboard and this runners can call directly to the graphical back end without having to go through a system that right now is doing way too much things in one place you know I'm also keeping track of graphical I'm very excited I think that's like it's a good way for scale of people to kind of sneak into the front end and make it less dynamic right so yeah I'm super anxious to see how that progress and we'll have folks doing this in the community you know Pinterest is doing graph QL critic are amazing graph QL on the front end and the Scala so I think if there is a very interesting evolution right like I think it's a good good opening for us well so thank you very much I think it was very good over you or flans up user chef I really you know thank you for food source is something and this really good way to integrate with the community so guys you know go to the length of this meetup and find the github repository for this project FS to blobstore if it's super I'm sorry write this on that will end up it's under the lendable and optional Oregon github and you know play with this see if it works for you I think a lot of companies to do this I remember like we did this you know a few floors now we do something like that in not in the streaming and I'm a cool fashion but like you need to get stuff institute in s3 yeah rest of like people are doing this so check it out and contribute it that's definitely the goal and and we hope to see more store employment agency soon okay thank you thank you very much thank you [Music]