System and method for performing caller identity verification using multi-step voice analysis

Caller identity verification can be improved by employing a multi-step verification that leverages speech features that are obtained from multiple interactions with a caller. An enrollment is performed in which customer speech features and customer information are collected. When a caller calls into the call center, an attempt is made to verify the caller's identity by requesting the caller to speak a predefined phrase, extracting speech features from the spoken phrase, and comparing the phrase. If the match cannot be made with a high enough degree of confidence, the customer is asked to speak any phrase that is not predefined.