Agent-Centric Column-Aware Prompt Optimization for Structured Data Tasks with LLMs

Large language models (LLMs) are being used more commonly in applications that use structured data (tables and databases) in reasoning. Nonetheless, naive few-shot prompt selection tends to disregard the underlying schema, which results in poor performance and higher rates of hallucinations. The paper presents a column-aware few-shot prompt optimization model that uses schema knowledge and a multi-objective search based on Optuna to sample and rank examples to solve structured data problems. The proposed system has tunable parameters, which include the selection of columns, examples, and formatting, and are optimized based on the desired metrics, including the accuracy of the tasks and the rate of hallucinations. Table question answering, report generation and analytics assistant task experiments show that faster convergence and better performance are realized by column-aware optimization than by random or best-of-one/best-of-three selection. The framework is also incorporated into an agent optimization platform, which allows automatic and adaptive timely management of production environments.