Scale By The Bay 2019: Rashmi Shamprasad & Ajit Koti, Finding Needles In Big Data Haystacks....

ai.bythebay.io Nov 2025, Oakland, full-stack AI conference Title: Finding Needles In Big Data Haystacks using Finite State Machine Working with data often means trying to locate data that fits patterns, akin to "Finding a needle in a haystack". When we add big data from non homogenous sources to the mix, this problem becomes exponentially complex. One of the use cases at Netflix, is about improving the Sign Up experience through experimentation. Being able to find user journeys across billions of events; that follow certain patterns, is a key insight into simplifying the sign up process. This gave us an idea to build a framework to express these user journey patterns that could be translated into a Non Deterministic Finite State Machine. One of the ideas that we adapted from Ken Thompson's 1968 CACM paper, was to create a Non Deterministic Finite Automaton around patterns defined using regex. The next step was applying the state machine across billions of events at scale using Spark. The final piece to the puzzle was to make it easily usable by Data Engineers, Scientists and Analysts alike. In this talk, we will cover how we built this framework (dubbed "Conduit") and the design decisions resulting from challenges along the way. We will also talk about how this can be adapted to real time applications in the future.