Transcript: scale.bythebay.io: Keita Broadwater Interview

quita Broadwater and i'm a co-founder of oxygen AI two types of problems I'd say one is a problem I think every data scientist and engineer faces it which is cleaning data having good input data for our models and for our analyses sometimes you get down a dead end you have to retrace your steps the other challenge is working with a lot of disparate systems and tools and the project that I spoke about at the conference I had to patch together a lot of different types of tools having to do with sensors having to do with messaging systems having to do with spark and the data data types within spark so sometimes piecing them together and configuring them so that you get out what you really want is a big challenge [Music] problems with scale cleaning data that is huge cleaning data that is in the Gigabyte scale can be challenging because sometimes there you sometimes that are corner cases sometimes when you have problems with data like that one way to solve it is to not just ignore the the corner cases but there's a desire to get as much data as possible because some of those corner cases to be important so it's to me it's cleaning data sets that are very large and very diverse [Music] so some issues that may come up we need to come to scale in the next year is we're transitioning a lot of our core technology from solely Python Python house to Scala so I think I can anticipate some challenges and making that transition and getting down a good set of procedures to systematically deal with new clients and new customers problems right now we're so used to Python and doing things in the pythonic way that it's like second nature sometimes and I can see it I can see us losing time on some projects just in doing that transition so functional programming how does it help this it's helped us in being able to use some tools more efficiently so the reason I got into functional programming is to take advantage of spark so I think Scala is a very good and efficient way to use spark right now I'm still transitioning to being full full scholarship developer and a lot of the projects that I think call for it and I'm using PI spark it's kind of a transition type of tool for me but I think if I could have everything in Scala I could have much cleaner code and much more efficient much more efficient algorithms [Music] I went to the one iteration of this conference last year so I like the diversity of speakers I like to hear what other people are doing I like to factor the projects that people talk about or not marketing presentations that are hands-on presentations and they talk about real problems that they have sometimes there there are things that aren't very relevant for me but sometimes something strikes a nerve and there's some insight that someone has a completely different project that could bear some relevance on what I'm doing [Music]