LAB42 Talk | CLS: Clara Meister - the Use of Language Model Embeddings for Evaluation in Natural Language Generation Tasks

LAB42, L3.36

Join us in this Computational Linguistics Seminar from Clara Meister from ETH Zürich.

The Use of Language Model Embeddings for Evaluation in Natural Language Generation Tasks

A good automatic evaluation metric for language generation ideally aligns strongly with human judgements of text quality. Yet, there is currently a dearth of such metrics, which inhibits the rapid and efficient progress of language generation systems. One exception to this is Mauve. In theory, Mauve measures an information-theoretic divergence between two probability distributions over strings: one representing the language generator under evaluation and the other representing the true natural language distribution. Mauve's authors argue that its success comes from the qualitative properties of their proposed divergence. Yet we show that both in theory and in practice, their proposed estimator of this divergence is quite a poor approximation. This begs the question: why does Mauve work so well? In this talk, I'll discuss our investigation of the empirical design choices behind Mauve that lead to its high correlation with human quality assessments. We find that its use of language model embeddings is critical for its success, and that while it is sensitive to syntactic- and coherence-level features of text, Mauve often ignores surface-level features. I'll discuss the implications of these findings for the trustworthiness of Mauve and for future directions of language generator evaluation metrics.

Computational Linguistics Seminar

The CLS is the Computational Linguistics Seminar of the University of Amsterdam. Seminars are open to all interested, researchers and students of all levels from UvA and elsewhere.


The seminar is organised by Jelke Bloem and Alina Leidinger.

To receive notifications about upcoming talks and the Zoom details, please join the CLS mailing list.

Subscribe to the announcement mailing list.


To make sure you do not miss any talk, you can add the CLS agenda to your calendar.

Link to the CLS calendar.


The CLS usually takes place on Tuesdays at 16:00 in room L3.36 at LAB42 in Amsterdam Science Park or via Zoom. Other days and locations are occasionally possible. See the details for each talk. To receive the details please subscribe to the CLS mailing list. The links to participate on Zoom will be distributed via the mailing list on the day of the seminars.