The Use of Language Model Embeddings for Evaluation in Natural Language Generation Tasks

A good automatic evaluation metric for language generation ideally aligns strongly with human judgements of text quality. Yet, there is currently a dearth of such metrics, which inhibits the rapid and efficient progress of language generation systems. One exception to this is Mauve. In theory, Mauve measures an information-theoretic divergence between two probability distributions over strings: one representing the language generator under evaluation and the other representing the true natural language distribution. Mauve's authors argue that its success comes from the qualitative properties of their proposed divergence. Yet we show that both in theory and in practice, their proposed estimator of this divergence is quite a poor approximation. This begs the question: why does Mauve work so well? In this talk, I'll discuss our investigation of the empirical design choices behind Mauve that lead to its high correlation with human quality assessments. We find that its use of language model embeddings is critical for its success, and that while it is sensitive to syntactic- and coherence-level features of text, Mauve often ignores surface-level features. I'll discuss the implications of these findings for the trustworthiness of Mauve and for future directions of language generator evaluation metrics.