ConversaETL and ConversaBench: Typed Planning and Deterministic Compilation for Verifiable Conversational ETL

Abstract

Large language model (LLM) data agents often expose transformation failures only at execution time. ConversaETL instead maps natural-language ETL requests to a typed intermediate representation and deterministic, contract-checked transformation operators, keeping the model's role bounded to planning, not execution. ConversaBench evaluates this design on ConversaBench-Full, a semantic-stress subset, and a cross-schema ETL extension. On ConversaBench-Full, the hybrid compiler (HC) improves over a compiler-only baseline (CO) by +0.274 correctness points (95% CI [0.218, 0.332]), rising to +0.356 on the semantic-stress subset and +0.407 on its hardest prompts. Against a repeat-matched direct LLM code-generation baseline, HC avoids common date, key, invalid-plan, and safety failures; the direct baseline reaches 0.308 mean correctness. A six-layer architecture ablation identifies exploratory-insight planning as the largest semantic contributor. These results support verifiable conversational ETL through schema-grounded planning, deterministic compilation, and a reproducible benchmark package. Streaming, dashboard, and local-provider checks are reported as supporting validations only.