Despite widespread use of LLMs as conversational agents, evaluations of
performance fail to capture a crucial aspect of communication: interpreting
language in context. Humans interpret language using beliefs and prior
knowledge about the world. For example, we intuitively understand the response
"I wore gloves" to the question "Did you leave fingerprints?" as meaning "No".
To investigate whether LLMs have the ability to make this type of inference,
known as an implicature, we design a simple task and evaluate widely used
state-of-the-art models. We find that, despite only evaluating on utterances
that require a binary inference (yes or no), most perform close to random.
Models adapted to be äligned with human intent" perform much better, but still
show a significant gap with human performance. We present our findings as the
starting point for further research into evaluating how LLMs interpret language
in context and to drive the development of more pragmatic and useful models of
human discourse.
Description
Large language models are not zero-shot communicators
%0 Generic
%1 ruis2022large
%A Ruis, Laura
%A Khan, Akbir
%A Biderman, Stella
%A Hooker, Sara
%A Rocktäschel, Tim
%A Grefenstette, Edward
%D 2022
%K LLM machinelearning
%T Large language models are not zero-shot communicators
%U http://arxiv.org/abs/2210.14986
%X Despite widespread use of LLMs as conversational agents, evaluations of
performance fail to capture a crucial aspect of communication: interpreting
language in context. Humans interpret language using beliefs and prior
knowledge about the world. For example, we intuitively understand the response
"I wore gloves" to the question "Did you leave fingerprints?" as meaning "No".
To investigate whether LLMs have the ability to make this type of inference,
known as an implicature, we design a simple task and evaluate widely used
state-of-the-art models. We find that, despite only evaluating on utterances
that require a binary inference (yes or no), most perform close to random.
Models adapted to be äligned with human intent" perform much better, but still
show a significant gap with human performance. We present our findings as the
starting point for further research into evaluating how LLMs interpret language
in context and to drive the development of more pragmatic and useful models of
human discourse.
@misc{ruis2022large,
abstract = {Despite widespread use of LLMs as conversational agents, evaluations of
performance fail to capture a crucial aspect of communication: interpreting
language in context. Humans interpret language using beliefs and prior
knowledge about the world. For example, we intuitively understand the response
"I wore gloves" to the question "Did you leave fingerprints?" as meaning "No".
To investigate whether LLMs have the ability to make this type of inference,
known as an implicature, we design a simple task and evaluate widely used
state-of-the-art models. We find that, despite only evaluating on utterances
that require a binary inference (yes or no), most perform close to random.
Models adapted to be "aligned with human intent" perform much better, but still
show a significant gap with human performance. We present our findings as the
starting point for further research into evaluating how LLMs interpret language
in context and to drive the development of more pragmatic and useful models of
human discourse.},
added-at = {2022-11-29T05:59:08.000+0100},
author = {Ruis, Laura and Khan, Akbir and Biderman, Stella and Hooker, Sara and Rocktäschel, Tim and Grefenstette, Edward},
biburl = {https://www.bibsonomy.org/bibtex/20e765573863b7f00d062b009b385278d/sairahul},
description = {Large language models are not zero-shot communicators},
interhash = {d82ff6cd2214754193df1ac085a8e2dd},
intrahash = {0e765573863b7f00d062b009b385278d},
keywords = {LLM machinelearning},
note = {cite arxiv:2210.14986},
timestamp = {2022-11-29T05:59:08.000+0100},
title = {Large language models are not zero-shot communicators},
url = {http://arxiv.org/abs/2210.14986},
year = 2022
}