© Ted Underwood
"Can Language Models Scale Down to Understand Local Context?"
Ted Underwood (Research Fellow am Center for Advanced Studies: https://www.cas.lmu.de/de/personen-am-cas/details/ted-underwood-6ee3cbc5.html)
Abstract:
Researchers have worked hard over the last quarter-century to stretch interpretive disciplines toward larger scales of analysis. Ironically, we are now confronted with a converse problem. The language models increasingly ubiquitous in our lives are typically trained on the largest possible context: almost everything their creators can find. This breadth has value, but models unfortunately turn out to be quite bad at ignoring things. So seeing a social panorama doesn't necessarily give them a human-like ability to focus on one part, understanding that situation from the inside, while tuning out other places, times, and social positions. When models try that, they often produce a caricature that is still seen from the outside. In this lunch talk, I'll discuss various strategies for scaling context down, and also describe a benchmark that aims to tell us whether models have scaled down successfully.