Seminar: Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry
- Datum: 20.10.2020
- Uhrzeit: 15:30
- Vortragender: Nils Köbis
- Ort: online
- Gastgeber: Center for Humans and Machines
- Kontakt: sekrahwan@mpib-berlin.mpg.de
Nils Köbis, Max Planck Institute for Human Development
Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry
The release of openly available, robust natural
language generation algorithms (NLG) has spurred
much public attention and debate. One reason lies in
the algorithms' purported ability to generate
humanlike text across various domains. Empirical
evidence using incentivized tasks to assess whether
people (a) can distinguish and (b) prefer algorithmgenerated versus human-written text is lacking. Nils Köbis and his team conducted two experiments assessing behavioral
reactions to the state-of-the-art Natural Language
Generation algorithm GPT-2 (N total = 830). Using the
identical starting lines of human poems, GPT-2
produced samples of poems. From these samples, either
a random poem was chosen (Human-out-of-the loop)
or the best one was selected (Human-in-the-loop)
and in turn matched with a human-written poem. In a new incentivized version of the
Turing Test, participants failed to reliably detect the algorithmically generated poems
in the Human-in-the-loop treatment, yet succeeded in the Human-out-of-the-loop
treatment. Further, people reveal a slight aversion to algorithm-generated poetry,
independent on whether participants were informed about the algorithmic origin of
the poem (Transparency) or not (Opacity). The scientists discuss what these results convey about
the performance of NLG algorithms to produce human-like text and propose
methodologies to study such learning algorithms in human-agent experimental
settings.