scale.bythebay.io: Feynman Liang Interview
Recording: scale.bythebay.io: Feynman Liang Interview
[Music] my name is Feynman Liang I'm the director of engineering at gigster a [Music] gigster we solve a lot of engineering problems over our hundred software projects one of the biggest challenges that we face is the ability to standardize across a variety of projects oftentimes developers want to use the tools they're most effective and efficient at but at the same time as a company we need to ensure consistent and reliable delivery standardization on one hand allows better gains across scale but limits developer productivity struggling to find the balance there has been one of the key technical challenges that gangsters had to face a gangster we're collecting data across hundreds of software engineering projects all running concurrently we ingest data from slack from Trello from github from Gmail and we aggregate all of that stuff together to provide real-time status updates and predictions about the outcomes of a project operating at this scale requires instrumentation of multiple systems in heterogeneous environments each system speaks its own language and a failure in one component can propagate two failures in numerous downstream components debugging and diagnosing these distributed systems has a really really long iteration speed and is often very non-trivial and requires expertise in numerous technologies so running such a large stack which processes so much data oftentimes results in a lot of engineering overhead [Music] some of the projects were working on right now we're able to run off of our simple little 4-node kubernetes cluster we don't really run into issues about synchronization and concurrency quite yet but as the business scales up and the data no longer starts fitting on single machines and you need parallelism on multiple parts of the stack you're gonna see these synchronization and locking issues resulting in more problems in your operations we really like functional programming functional programming gives us a lot of ability to reuse the the style of functional decomposition taking a problem and breaking it down into smaller sub functions that each and and having pure functions which do one thing well and then finally composing that to form your host solution it's kind of the way that we approach a lot of programming and it allows you to reuse individual chunks in other parts of your parts of your codebase one type of functional programming we really like a gigster is strongly taught or statically typed polymorphic functional programming languages Scala is an example of one haskell is another we're seeing this arise in other frameworks such as flow type and typescript and we've seen that utilization of these technologies results in developers writing safer code that fails in development rather than production and overall result in reduced downtime and higher developer productivity the community Aleksey the organizer of the conference and i we have his PhD advisor was my undergraduate advisor it's amazing coming here and you know connecting with alexei again as well as getting involved with the community that he's built around scaled by the bay you know i get to meet all of these people whose names i see on github and put a face to the name exchange contact information and get involved and discover new things that I didn't know about within the Scala community you