Scale By The Bay 2018: Evan Chan, FiloDB: Real-time, In-Memory Time Series at Massive SMACK Scale

Time series and event data is becoming huge for every business, and ingesting millions of series reliably while answering many concurrent queries from users is a huge challenge. In this talk I share the story of developing and productionizing FiloDB, an open source, in-memory time series solution built with the Scala, Akka, Kafka, Cassandra, Mesos (SMACK) stack. FiloDB is able to reliably ingest monitoring/time series data and answer tons of low latency queries at massive scale. * Why we developed our own solution after looking at Prometheus, OpenTSDB, Cassandra, etc. * Time series data model and low-latency distributed querying at scale * The benefits and challenges of off-heap, in-memory data processing at scale * Building a database for modern container environments * Challenges with scaling the Prometheus data model while remaining compatible * Persistent, recoverable data at scale with Kafka and Cassandra * Key lessons in building massively scalable, real-time, low-latency data systems

▶