At Zignal Labs we track brand health for major brands and organizations such as Nvidia, Airbnb and IBM. A critical aspect of this is understanding the polarity of conversations as they unfold in real time across social and traditional media sources. We will discuss how we pulled together AWS services including Mechanical Turk, Code Pipeline, Lambda, and Sagemaker to deliver a complete sentiment solution that better fits our customer use case and provides transparency into our sentiment quality. We will dive into how these services can work together to enable continuous delivery and continuous model retraining providing both system reliability and rapid inclusion of new labeled data to preserve model quality as conversations shift.