Who's a better writer, AI or me?
We took the NYT's viral AI writing quiz and (surprise!) have many, many thoughts
I first came across The New York Times’ “Who’s a Better Writer: A.I. or Humans?” quiz while scrolling through Instagram stories.
It arrived by way of someone I knew in college—a place where, for context, I could be found loafing about in a rickety old wooden clapboard building that belonged to the literary magazine, discussing poetry while my even-artier friends chainsmoked around me, joking, occasionally, that one day a cigarette butt would bring the end to a 150-year-old tradition that counts the likes of T. S. Eliot and E. E. Cummings in its membership, and leaving, invariably, my hair perfumed with the stale, acrid, stubborn fumes.
She had posted a screenshot of one of the questions, asking readers to choose the passage they like best, between the opening of Elizabeth Bishop’s “The Fish,” one of the great 20th-century poet’s more famous poems, and some bad poem, if you can call it that, by Anthropic’s Claude, Opus 4.5. She had, of course, answered correctly (it should be clear that by this I mean she chose Bishop), noting in the caption how obvious it was; besides the masterful construction and stunning emotional arc of the poem, Elizabeth Bishop is one of the most anthologized poets of her generation, and if you know her, you probably know “The Fish.”
I chuckled smugly at the absurdity of the exercise, paying hardly any attention to the AI-generated “poem” beside the masterpiece. Who needed to? “The Fish” is instantly recognizable.
I hadn’t noticed, in the fine print revealed underneath the passages, that only 52% of readers had chosen one of my favorite poems of all time as their champion in the human-vs-AI writing face-off. This, obviously, depressed me.
I have asserted before, in this newsletter, many times, that human creativity will prevail over generative AI. Our very first post, in fact, is titled “Nevertheless, Creativity Survives,” a nod to Marianne Moore’s poem “Nevertheless,” about the resilience of life, the victory of survival. I am an English major, the daughter of an English major, the granddaughter of an English professor, and it goes on. I quote poems as some do scripture. If I believe in one thing, it is human expression:
From our first post: “I suspect that those who try to erase and replace [human art] will find public disillusionment at the end of the road as people discover that AI is but another ‘mechanical trick’ [a borrowing from One Hundred Years of Solitude], no substitute for human expression. I also suspect that they will fail—that human art will persist, will wriggle its way through the concrete.”
And from The Whimsical Guide to Writing Well: “But our bots do not write, or speak, or mean. They mimic symbolic patterns, convert those patterns into phonemic ones, and produce something that ‘means’ something only in the sense that it has semantic meaning….To convey meaning is at once a practical act and a metaphysical one….No matter how advanced the models become, they will always remain just that: models. Representations. Replicas. Forgeries. Frauds.”
Here, however, was hard, crowdsourced data from the newspaper with the largest digital readership in the country, suggesting that readers could not distinguish between human writing and AI writing—and could even prefer the latter.
Indeed, Bishop’s was no exceptional case; other human writers faced a similar fate. In the genre of literary fiction, the results were split 50/50 between the human passage and the AI passage. In fantasy, human writing won by a razor-thin margin of two points. In science writing, readers overwhelmingly preferred the LLM’s writing, with only 35% of readers choosing the human-written text. Historical fiction fared the best, with the human excerpt winning 56% of votes. And then there was the narrow victory of Elizabeth Bishop, a disconcerting 52 to 48. While, on the whole, readers preferred human writing over AI writing, the odds were little better than a coin toss.
Now, it’s not as though a New York Times quiz is exactly the gold standard in research, so it’s important not to credit it unduly. Unfortunately, the results broadly mirror what Jeff Hancock, Harry and Norman Chandler Professor of Communication at Stanford, and his colleagues found in their study, “Human heuristics for AI-generated language are flawed,” where they had subjects distinguish between human and AI-generated self-presentations across different interpersonal contexts: professional, hospitality, and dating. The subjects’ accuracy rate was similar to that of the New York Times readers: 50-52%.
Why humans are unreliable AI detectors, they found, stems from our default tendency to trust others and our use of heuristics in determining whether or not to trust others, such as the use of first-person pronouns, informal language like contractions, and the topic of family. Basically, if it walks like a human and talks like a human, who’s to say, with any reasonable confidence, that it isn’t a human?
AI writing is optimized to blend in: to construct a sentence, as probabilistic machines trained on vast swaths of human communication over time, LLMs generate a string of words that are statistically likely to precede and follow one another. There’s something inevitable about AI-generated sentences, like water circling down a drain: the words go that way, converge to that point, read simply and smoothly if they don’t leap off the page, appearing there, on your screen, for your easy consumption; and we are not surprised, since it was the most likely thing to happen.
Nestled in the center of the bell curve, its deceptiveness hides in plain sight.
Were my paragraphs extolling human creation more prayers than claims, then? I reluctantly remember that my Instagram bio reads, quoting Charlotte Brontë’s Jane Eyre, “& what I believed in I wished to behold.” Have I just been speaking into existence what I wish to behold, clapping my hands and shouting “I do believe in human writing! I do! I do!” so that fucking Tinker Bell doesn’t die? Is the belief in the triumph of human writing over AI slop just another form of magical thinking? Another humanistic, faith-based tradition that’s about to be discredited by advancements in science and technology?
I wouldn’t have it. I do believe in human writing. I do, I do.
I decided to take the quiz.
(You should too, here, and we’ll compare notes in our next piece.)
TO BE CONTINUED…
weekly whimsy
This week, we’re back from our very whimsical business-school break. Céline grinded away on her top-secret job search (more to come to light on that when more has come to fruition on that) and still found time to sunset longboard in San Onofre because: touch grass (or ocean, or whatever your preferred ecological feature). Danielle drove up and down the California coast (edit: DJ’ed from the co-pilot seat) and had her first experience “roughing” it in an RV. She devoured arugula the size of her face in Mendocino and journaled by the moonlight in Bodega Bay.
what we’re watching/reading:
Céline: Paired watch/read – Train Dreams and “Amends” by Adrienne Rich
Danielle: Invincible, a superhero story with all the gore and moral wreckage, little of the glory
what we’re listening to:
Danielle: Syd Taylor’s After the Fact
Céline: If you want a good, juicy cry, Adele’s 30, particularly the opening and last two songs
what we’re wearing:
Written by: Céline Vendler
Edited by: Danielle Zhang
“To create anything … is to believe, if only momentarily, you are capable of magic. … That magic … is sometimes perilous, sometimes infectious, sometimes fragile, sometimes failed, sometimes infuriating, sometimes triumphant, and sometimes tragic.”
The problem with A.I. art is that it can only look backwards. It’s like an art forger who can make a painting that looks like a Cézanne, but who, no matter how skilled with a brush, will never be the next Cézanne. That said, in this first phase of the A.I. “revolution”, A.I. will look pretty good, because it’s mostly training (stealing) from real human work. But as the slop floods all outlets, the machines will end up just making more garbage derived from the garbage they already made. Will this be good enough to sell tooth-whitening strips on YouTube? Sure. But the real stuff: the artistic works we keep close to our hearts for generations, is under no threat.