scale.bythebay.io: Long Cao Interview
Recording: scale.bythebay.io: Long Cao Interview
[Music] so I am long cow I am a software engineer on the data science team at co2 management so specifically I work in data engineering and for us it's a combination of both the the depth of the work that we have to do and the breadth of the work at the scale that we that were working at so it's definitely like a truism of the Big Data and distributed systems are hard along with that is just the unspoken part about processes and you know how do you actually ship data and how you actually do data engineering in a way that can actually be shipped to production in a reliable way that you know your downstream clients can can expect at least for us it's you know not writing algorithms that that don't scale at you know with with large data sets it's really easy to write something that works in this small but then falls over in the large I think that's also very hard to test because you know when you have something that that can fail at the at a large scale it's it's really hard to to find that out until you're actually you know in production or in flight [Music] so I guess I'm just you know a little concern that everyone's getting into this you know big data game typically like the approach has been to throw more machines at a problem I think I do see some some people working on ok how do we make algorithms more efficient how do we make memory representations more efficient how do we do things in a way that's more efficient so that we can vertically do things better I think it's very easy for for us to be on elastic systems where it's just like toss more money on the on the fire and you know hope hope it goes faster I hope to see a bit more of you know optimization work [Music] I mean I'll be honest it's got me jobs so that's that's probably me benefiting the most along that I think it's just such a deep field in terms of what you can learn and so I know for myself growing up I was always afraid of you know getting into this like kind of career mode where I'm just working a really boring desk job and doing nine-to-five and just like waiting to get out but I can honestly say I don't have that it's a very interesting job I'm learning every day in and out that's all I'll never be bored my favorite part is um honestly I've been to CLI Bay three times in a row now and there's kind of like this same group of people who show up so more and more often scale by the vein and generally speaking Excel conferences are a time when I get to see a lot of my friends in the scala community