New methods offer fresh insights

The rapid development of large language models, such as ChatGPT and similar systems, in recent years has transformed computational linguistics as a research field. But what can these models really do—are they simply mimicking patterns like parrots, or do they represent artificial general intelligence? To answer this, the development of analytical methods is required.

A new doctoral thesis from the University of Gothenburg offers important insights into the ongoing debate about AI, its capabilities, and its future development.

“Large language models have impressed with their human-like linguistic abilities but have also raised questions about whether they truly understand what they are saying or are merely ‘stochastic parrots’ repeating patterns from their training data,” says Felix Morger, a research engineer at the University of Gothenburg.

“If we could truly confirm that they possess human-like linguistic capabilities, it would be a major step toward artificial general intelligence.”

AI models already have a significant impact on society, influencing an increasing number of users. This creates a need for transparent, systematic, and thorough evaluation, according to Morger. “To properly assess the benefits and risks of AI technology, we need to understand both the possibilities and the limitations of analytical methods.”

Collection of evaluation data

The development of analytical methods is essential to measuring and understanding the linguistic capabilities of large language models. This involves creating training and evaluation data to reliably measure their performance, as well as interpretive methods to understand the inner workings of language models.

In his thesis, Morger presents a range of approaches for analyzing large language models, from introducing SuperLim—a collection of training and evaluation data specifically designed for Swedish language comprehension—to examining large language models’ ability to predict linguistic variation.

The thesis explores both the possibilities and limitations of popular analytical methods, as well as their implications for the development of large language models.

“It’s not just about improving the technology but also about understanding the broader implications, including both the opportunities and risks of AI in society,” says Morger.

New insights into linguistics

Beyond contributing to the development of large language models, the thesis also has the potential to provide new insights into linguistics.

“An important conclusion is that methods derived from empirical linguistics play a crucial role in understanding the data and linguistic features used in the analysis of these models,” says Morger.

The thesis, “In the minds of stochastic parrots: Benchmarking, evaluating and interpreting large language models,” was defended at the Faculty of Humanities, University of Gothenburg, in December 2024.