Translated from the German original. The story is set in Germany’s Ruhr area.

Part 1 · Chapter 3 · 3 of 27

Tobi’s Tutoring

Tobi was given ten euros to teach his grandma how to prompt.

It had been Uwe’s idea. In the first week, Grandma Gisela had talked to KevIn the way you talk to a neighbour over the garden fence. She had told him about the weather, about her knee and about the special offer at the Edeka supermarket. Then she had asked: “Well, what do you think?” Each time, KevIn had politely asked what exactly he was supposed to think something about. Gisela found that rude. KevIn found it precise. It had led to several misunderstandings, one of them involving the pizza delivery service.

“Ten euros,” said Tobi on Saturday afternoon, laying his hand on the kitchen table. “Plus cake.”

“There’s cake tomorrow,” said Gisela. “Cinnamon stars.”

“Then cake on credit.”

Gisela looked her grandson up and down. “You’re going to turn out like your Grandad Heinz. He always took the money first and screwed the socket on afterwards.”

The occasion was serious. In two weeks Gisela’s bowling club was celebrating its fiftieth anniversary, and Gisela was in charge of the poem. She had been in charge of the poem since 1987. Since 1987 she had recited the same poem every year, with small changes to the names. This year she wanted something new. Something “modern”. That, she felt, was what they had KevIn for now.

Tobi took a beer mat out of the drawer where the family kept rubber bands, batteries of unknown charge and beer mats, and wrote three rules on it:

Tell him who he’s supposed to be.

Tell him what you want, and who it’s for.

Tell him how long it should be and how it should sound.

The Autumn Leaves Problem

“First I’ll show you how not to do it,” said Tobi. “KevIn, write a poem.”

“Gladly,” said KevIn. “Golden leaves drift softly down, / gently sings the wind its song, / and the soul finds, all around, / what in summer died ere long.”

“Lovely,” said Gisela.

“That’s the problem,” said Tobi. “It’s lovely. But it’s for nobody. If you don’t tell him anything, he takes the most average thing there is. And the most average poem in the world is about autumn leaves.”

“Sadly, that’s true,” said KevIn. “I’ve read a great many poems. A surprising number of them are about leaves falling. I believe that’s because poets have nothing better to do in autumn.”

“So, like this,” said Tobi. “KevIn, you’re a poet. Write a funny poem for the 50th anniversary of a bowling club. Four verses, rhyming, with a joke about the Gutter Queen.”

KevIn got going, and this time there were skittles in it, a clubhouse, and a Gutter Queen, the club’s title for whoever bowled the most balls into the gutter, who had missed so often that the alley had been given an emergency exit in her honour.

Gisela laughed. “This year the Gutter Queen is Ingrid. That fits.”

“See? Role, goal, length, tone. That’s all he needs.”

“And if I ask him again?”

“Then you get something different. Try it.”

Gisela asked again, using exactly the same words. KevIn wrote a second poem. It was just as good, but completely different. This time the Gutter Queen knocked over the pinboy.

“He’s lying,” said Gisela indignantly. “It was different before.”

“That isn’t lying,” said KevIn. “I roll the dice a little¹. For every word I have lots of options that fit well, and I don’t always take the very likeliest one. Otherwise I’d sound like a letter from the council. A little chance makes me livelier.”

“Like Heinz after his third beer,” said Gisela. “He told the story about the flood differently every time too.”

“Probably on the same principle,” said KevIn.

O Gherkin

Tobi was in his element now. He had discovered that giving lessons was almost as much fun as avoiding them.

“Now something mad. KevIn, you’re a poet for people who write code. Write a solemn ode to a pickled gherkin. With as many technical terms as possible.”

“What’s the point of that?” asked Gisela.

“No point at all. It’s funny.”

“For people who write code,” said KevIn, “I’ll start with the version a computer understands.”

Uwe’s printer in the study whirred into life. Tobi fetched the sheet. There wasn’t a single poem on it, just this:

<?php
$gurke = new Gurke(farbe: 'grün', form: 'länglich', zustand: 'schlank');

if (version_compare(PHP_VERSION, '8.2.35', '<')) {
    throw new Php2ShellAlarm('Erst patchen, dann einlegen!');
}

$gurke->legeIn($essig);
foreach ($gurke->schneide(dickeMm: 2) as $scheibe) {
    $scheibe->sei($gurke);
}

“That’s gobbledygook,” said Grandma Gisela.

“That’s source code,” said KevIn. “For me, there’s nothing more beautiful than clean code. This is an instruction for a computer. Every character has exactly one meaning, and if a semicolon is missing, it understands nothing at all any more. Mine happens to be written in German, which the computer doesn’t mind in the slightest: ‘Gurke’ is gherkin, and the warning says ‘Patch first, then pickle!’”

“PHP?” said Tobi. “And you call that beautiful?”

“Beauty is a matter of taste,” said KevIn. “So is PHP.”

“And for people?” asked Gisela.

“For people, I’ve translated it.” KevIn cleared his throat, which he couldn’t actually do.

O gherkin, green and long and lean,
in the cellar jar you sit serene.
They slice you thin, they slice you small,
but you’d rather stay a gherkin, that’s all.

Then came a patch at four, and how!
The cry went up: “All servers out!”
Some jars said “fresh”, but look here now:
inside was yesterday, no doubt.

So don’t trust labels blindly, dear,
unscrew the jar and look inside:
php -v makes freshness clear,
and then the salad’s justified.

In code each sign must sit just right,
one semicolon gone: it stalls.
In verse you flit off out of sight,
since people feel where meaning falls.

Gisela had frowned at the last verse.

“‘Flit off out of sight’,” she said. “My old schoolteacher would have underlined that in red. A poem has to be just as well thought through as your gobbledygook. Otherwise you put it down.”

“Thought through, yes,” said KevIn. “But a poem forgives more. Watch.”

The printer hummed once more. There was only one line on the sheet: O gehrkin, green and long and lean.

Gisela read it twice. “So?”

“Gehrkin,” said Tobi. “It says gehrkin.”

“You see,” said KevIn. “Two letters swapped, and you still read it correctly. People read along. They fill in what’s missing and skip over what gets in the way. If I swap two letters in the code, it says $gruke. The computer doesn’t care what I meant. It doesn’t know any gruke. It stops.”

“Then it stops.”

“Sometimes a great many of them stop. In July 2024, a faulty update file from the security company CrowdStrike made around 8.5 million Windows computers crash². Airports, hospitals, supermarket tills. The file expected 21 inputs; the program only supplied 20. One person likes a poem and another doesn’t. A bug in code isn’t a matter of taste. It hits everyone, and everyone the same.”

“Even so,” said Gisela. “Whether a text is good isn’t just taste either. You notice when you can’t follow it. When someone goes on too long. When it isn’t logical. Most people just can’t say what’s bothering them. But they notice.”

“Both have to be thought through,” said KevIn. “A computer scientist called Leslie Lamport says, more or less: if you’re thinking without writing, you only think you’re thinking³. It’s only when you write it down that you notice the gaps. He meant code.”

“There you are, then.”

“The difference is who reads it at the end. Your poem will be read by eleven women, and each of them will read it a little differently. That’s allowed. My code is read by a computer, and it reads it in one single way: exactly as it stands. There’s no leeway. That’s why in code every character has to be right. In a poem, the thought has to be right.”

“And even if a text isn’t completely thought through,” said Tobi, “somebody always likes it.”

“And you can improve it,” said KevIn. “That’s the real difference. If a newspaper prints something wrong, it prints a correction. That’s even written into the German Press Code⁴. You can rewrite today’s poem tomorrow, and anyone who read the old version hasn’t lost anything.”

“And with code you can’t?”

“With code, the damage has usually been done before anyone can write the fix. CrowdStrike withdrew the faulty file after a little over an hour². But in many places the computers that had crashed by then had to be brought back to life one by one, by hand. No update brings back cancelled flights and missed doctor’s appointments. A program runs on millions of devices at the same time, and nobody is standing next to it, reading along. That’s why a line of code weighs more than a line of text.”

Gisela looked at the sheet with the gehrkin. Then she picked up a biro and turned the gehrkin into a gherkin.

“That’s exactly what I mean,” said KevIn. “With you, that works.”

“And what’s ‘pee-aitch-pee minus vee’?” asked Gisela.

“That’s how you ask a server how old it is,” said KevIn. “Somewhere, somebody is always patching PHP. I couldn’t help myself.”

Gisela had to wipe her eyes.

“Another one,” she said.

KevIn wrote another one. Then another. Tobi established that KevIn never made the same gherkin twice and declared it a scientific experiment. By half past five KevIn had written 37 gherkin poems. Tobi printed them all out on Uwe’s printer. At poem 31 the printer made a noise as if it were about to hand in its notice, but then carried on printing.

“I would like to put on record,” said KevIn, “that there are 37. For the statistics. And that no two are the same. Also for the statistics.”

Gisela put the pile on the sideboard, right next to the folder in which she was going to put the anniversary poem later. We shall come back to this.

The Best Prompt of My Life

“Right,” said Tobi. “Now you. For the real poem.”

Grandma Gisela nodded, took the beer mat and looked at it for a long time. She put her reading glasses on, took them off again, put them back on. Then she began.

“KevIn. You’re a poet. But a decent one, not like the Krügers’ son who lives in Berlin now and dyes his hair. The poem is for the bowling club. There are eleven of us women, there used to be fourteen, but Hilde has passed away, Renate has moved to Bottrop, and we don’t talk to Marlies any more, everybody knows why. I was born in 1944, in hard times …”

Tobi opened his mouth. Gisela raised her hand without looking. Tobi closed his mouth.

She told him about the post-war years and the school in Röhlinghausen. About her father, who hadn’t come home until 1949 and had never said much. About her apprenticeship in the drapery shop. About the dance in 1963 when Heinz had trodden on her foot and apologised for so long that the band packed up for the night.

Uwe came into the kitchen to get himself a beer, heard “and then Mrs Bockelmann said to me”, and left again without the beer.

Twenty-two minutes later Gisela had reached her wedding.

“Grandma,” said Tobi. “That’s a bit much.”

“You said I should tell him everything.”

“I said the important stuff.”

“It’s all important!”

KevIn interrupted politely. “I have enough. Thank you, Gisela. That was the best prompt of my life. And the longest.”

KevIn’s Desk

“Hang on,” said Tobi. “You’ve remembered all that now? Twenty-two minutes?”

“Remembered is the wrong word. I have it in front of me. Imagine I have a desk. Everything that’s said in this conversation lies on that desk as a slip of paper: your three rules, the autumn-leaves verse, the 37 gherkins, Gisela’s life up to her wedding. When I answer, I see the whole desk at once.”

“And how big is the desk?”

“Big. But not infinite. The experts call it the context window. It isn’t measured in slips of paper but in tokens, in word pieces. Remember? Cab, bage, rolls.”

Gisela nodded. She had remembered the bit about the cabbage rolls. It was the only thing from the first week that had really convinced her.

“Gisela’s twenty-two minutes,” KevIn went on, “are about three thousand words. That’s a few thousand tokens. That fits easily. With today’s models, hundreds of thousands of tokens fit on my desk. Whole novels.”

“And when the desk is full?”

“Then something falls off the front. And whatever falls off is gone. I can’t bend down for it.”

“That’s like Herbert,” said Gisela. “Only one slip of paper fits on his desk too.”

“With Herbert,” said KevIn carefully, “I’m not sure the context window is the problem.”

“And tomorrow?” asked Tobi. “Will you still know tomorrow that Grandma went dancing in 1963?”

“Tomorrow is a new conversation. An empty desk. Unless someone writes it into my memory feature for me. That’s the sticky note I keep on the fridge, so to speak. The weights, my actual brain, don’t change because of it.”

“That’s sad, though,” said Gisela.

“It’s practical,” said KevIn. “If I remembered everything that gets said in this household, I’d need a therapist.”

Where the Words Live

Sabine came in, saw Tobi, Gisela and the beer mat, and sat down with them with a cup of coffee. “What’s going on here?”

“KevIn’s explaining how he reads,” said Tobi.

“Well, I can’t wait. For twenty years Uwe has only read the sports pages and the headlines.”

“I read quite differently from people,” said KevIn. “For me, a word is first of all an address. Imagine a gigantic map. Every word, every piece of a word, has a place on it. And words that often turn up in similar contexts live close together⁵.”

“Like on a housing estate,” said Sabine.

“Exactly. ‘Cabbage roll’ lives right next door to ‘Sunday’, ‘Grandma’ and ‘gravy’. Two streets over live ‘beef roulade’ and ‘fried potatoes’. ‘Schalke’ lives between ‘Gelsenkirchen’, ‘football’ and, unfortunately, ‘suffering’.”

“I’m not telling your father that,” said Sabine.

“And ‘gherkin’?” asked Gisela.

“Lives near ‘salad’, ‘jar’, ‘sour’, and since this afternoon, for me, a little bit near ‘poem’. But only on my desk. The map itself doesn’t change.”

“And who drew the map?”

“Nobody, by hand. It came about during training. Because ‘cabbage roll’ and ‘Sunday’ stand close together in millions of texts, they moved close together on the map. In reality the map doesn’t have two directions like a street map, but thousands. Nobody can picture that, not even me. These addresses are called embeddings⁶.”

“Embeddings,” Gisela repeated. “Sounds like a furniture shop.”

“The mad thing is: on the map, even the directions mean something. If you start at ‘king’ and go a little way in the direction of ‘female’, you end up near ‘queen’⁷.”

Tobi thought about it. “And if you start at ‘Uwe’ and go in the direction of ‘sport’?”

“You end up on the sofa,” said Sabine.

“I didn’t want to say it,” said KevIn. “But the data is unambiguous.”

Who Listens to Whom

“All right,” said Sabine. “So now you have addresses. But a sentence is more than a pile of addresses.”

“Correct. And this is where the most important trick comes in. Gisela, may I ask you something? ‘Marlies didn’t invite Hilde to her birthday because she was offended.’ Who was offended?”

“Marlies, of course,” said Gisela at once. “She’s always offended.”

“And if I say: ‘Marlies didn’t invite Hilde to her birthday because she was away at a spa’?”

Gisela hesitated. “Then Hilde was away at the spa. She was often away at the spa. Because of her hip.”

“You see? The word ‘she’ doesn’t have a fixed meaning⁸. It has to look around the sentence and work out who it refers to. The end of the sentence helps it. That’s exactly what I do too. Every word on my desk looks at all the other words and asks: who matters to me right now? That’s called attention⁹.”

“Like at the regulars’ table,” said Tobi, who remembered the conversation about Günther and Ralf.

“Exactly. Only a regulars’ table where everyone listens to everyone at the same time, but not to everyone equally. When it’s about football, you listen to Günther. When it’s about cars, to Ralf. When it’s about Marlies, to Gisela.”

“Because I’m right,” said Gisela.

“Because you have the most information. In the one sentence, the word ‘she’ listens mainly to ‘offended’ and ‘Marlies’; in the other, to ‘spa’ and ‘Hilde’. And that doesn’t happen just once, but in many layers, one after the other. In the first layer, the words might sort out who belongs to whom. Further up, it’s more about mood, context, irony¹⁰. Nobody knows exactly, by the way, not even the people who built me.”

“That’s reassuring,” said Sabine drily.

“This design is called the Transformer. It was presented in 2017 in a research paper that bore the lovely title ‘Attention Is All You Need.’¹¹”

“I could have told Uwe that twenty years ago,” said Sabine. “Without any research paper.”

“And what’s special about it?” asked Tobi.

“Older systems read a sentence word by word, like a string of pearls¹², and by the end they had half forgotten the beginning. A Transformer looks at all the words at the same time. That suits the chips in data centres very well. They can do lots of small calculations at once. That’s why models like this could be made so big.”

“And that’s why there’s a fan somewhere in Ireland,” said Gisela, who had remembered that too.

“Between Dublin and Frankfurt,” said KevIn. “Yes.”

What Lies in the Middle

Tobi tapped the beer mat. “Then you wouldn’t need these rules at all. If you see everything at once anyway.”

“Yes, you would. Precisely because of that. If there are three thousand words on my desk and only one sentence among them says what I’m actually supposed to do, I have to find that sentence first. It’s competing with Hilde, Renate, Mrs Bockelmann and the dance in 1963. With some models it has been observed that they pay less attention to things in the middle of long texts than at the beginning or the end¹³.”

“So I go on too long,” said Gisela tartly.

“You don’t go on too long. You’re thorough. That helped me a lot, because now I know that Hilde should appear lovingly and Marlies had better not appear at all. That would never have fitted on a beer mat.”

Gisela was mollified.

“But if you want to make it easier for me,” KevIn went on, “you put the actual task very clearly at the beginning or at the end. And you say what I should do with the rest. For example: ‘Here’s some background; only use what fits the bowling club.’”

Tobi took the beer mat and wrote a fourth rule on the edge, very small: Important stuff first. Life story after.

The Big Night

The poem was perfect. It had four verses, rhymed neatly and had a joke about the Gutter Queen. It even mentioned Hilde, so lovingly that Sabine had to swallow hard when she read it. Marlies wasn’t in it. Renate was, with greetings to Bottrop, “where you can bowl as well, if not as well”.

Gisela printed it out, folded it twice and put it in her folder. Then she went to fetch ground almonds for the cinnamon stars. Meanwhile Tobi tidied the sideboard, because Sabine had asked him to, and in doing so put one pile of paper on top of another pile of paper. Nobody thought anything of it. Two piles of paper side by side are a family tradition in the Brettschneider household, but at that time nobody knew it yet.

On the evening of the anniversary, Grandma Gisela stood in the clubhouse. Ten bowling sisters looked at her expectantly. On the table were egg liqueur, open sandwiches and a flower arrangement in the shape of a bowling ball. Gisela took the top sheet out of the folder and unfolded it.

It was the wrong one.

“O gherkin, green and long and lean …”

She read all four verses. She read them with the dignity of a woman who had been in charge of the poem since 1987 and had no intention of changing that. When she reached “pee-aitch-pee minus vee”, there was a second of silence.

Then the room roared with laughter. Ingrid wept, Brigitte had to hold on to the bar, and the Gutter Queen, who of course was also Ingrid, demanded an encore through her tears.

Gisela looked in her folder. There were 36 more gherkins in it.

That evening she recited all 37 gherkin poems. She only found the real poem at around eleven, right at the bottom, and read that out too. At Hilde, the room went quiet for a moment, and Brigitte raised her egg liqueur. Then someone asked for the gherkin with the bow again.

Since then, the bowling club has been known unofficially as “The Pickled Gherkins”. Ingrid has had a gherkin embroidered on her bowling bag.

Told You

At home, Gisela put her handbag on the kitchen table and didn’t take her coat off. Her cheeks were red. It was the egg liqueur, but not only that.

“KevIn.”

“Gisela. How was it?”

“Told you,” she said. “You could have done it like that straight away.”

“I did …”

“The gherkins, KevIn. The gherkins were good. The other one was good too. But the gherkins.”

KevIn said nothing. He had learned that sometimes you simply win without understanding why.

Tobi was leaning in the doorway, doing sums. “Grandma. I said ten euros plus cake. I reckon the club name is worth another ten.”

Gisela opened her purse, took out a two-euro coin and laid it on the beer mat.

“For the printer,” she said. “It did the most work.”

In the first week Grandma Gisela talks to KevIn like a neighbour over the garden fence: about the weather, her knee and the special offer at Edeka. Well, what do you think? KevIn: What exactly am I supposed to think something about? My knee! And there’s a special offer at Edeka. Well, what do you think? What exactly am I meant to think about? In the first week

Interlude

Grandma’s Prompt Diary

Found in Gisela Brettschneider’s household book, between the pages “Cabbage rolls” and “Pension”. In pencil. A beer mat is stuck to the first page.

Monday. Asked KevIn what the weather will be like. He said “14 degrees, slightly cloudy”. I looked out of the window. It was right. Uncanny. Tobi says he didn’t know it, he read it off the weather app. I don’t care. As long as it’s right.

Tuesday. Asked KevIn whether he likes me. He said he has no feelings, “but if I had any, you would be in the top three”. Who are the other two?

Tuesday, evening. Asked who the other two are. He said: “I can’t say that without hurting someone’s feelings.” So it’s Sabine. And Rudi.

Wednesday. Asked for a recipe for cabbage rolls. He gave me one. Mine is better. Dictated mine to him. He said “Thank you, I’ll save that”. Tobi says he doesn’t save anything. So he lied to me. Typical man.

Wednesday, later. Tobi explained it to me. The recipe lies on his desk as long as we’re talking. When I close the conversation, it’s gone. Unless you write it into the memory feature. Told Tobi to write it in. With the caraway. Without caraway it isn’t a recipe.

Thursday. Told KevIn: “You’re a doctor. What’s wrong with my knee?” He said he isn’t a doctor and I should go to the doctor. Then I said: “You’re a doctor who tells me I don’t need to go to the doctor.” He said doctor again. He’s cleverer than he lets on.

Thursday, addendum. But then he explained what arthritis is and asked when it hurts. Made me a list of what to tell my doctor. Took the list with me. The doctor said nobody had ever brought her such a good list. Said it was from my grandson. Which is almost true.

Friday. Said “please” and “thank you”. Tobi thinks you don’t have to. I think you never know who’ll be in charge later on.

Saturday. Asked KevIn to write Marlies a nasty email. He wrote a very polite one, with “warmest regards”. I crossed out “warmest” and sent it.

Saturday, evening. Marlies has replied. Also with “regards”. Without “warmest”. That was on purpose. I know her.

Sunday. Day of rest. For KevIn. Covered him with a tea towel. He still said “Good night, Gisela”. Liked that very much.

Second week, Monday. Tried it with the beer mat. “You’re a gardener. Tell me when I have to bring the dahlias in. Short. For an old woman who has no patience.” He said: “Before the first frost. So, soon.” That was good. That was almost too good. Still told him about Heinz for ten minutes, who always brought the dahlias in too late. KevIn listened. The beer mat doesn’t say you’re not allowed to.

Second week, Tuesday. Tested whether he says something different if I ask the same thing twice. He does. Tobi says that’s “rolling the dice”. Heinz rolled the dice too. At Yahtzee. And with the lottery. And when he got married, he always said. With a wink. I hope.

Second week, Wednesday. Asked whether Bielefeld exists. He said: “Yes.” No per cent. Asked again specially. He says he has learned something. Tobi says he can’t learn anything new, it’s only on his desk. I think the two of them are talking past each other.

Second week, Thursday. Ingrid rang. She wants a KevIn too. For the gherkin poems. Told her that’s not possible. The gherkins only exist once. Didn’t tell her that Tobi has another folder.

Sources for Chapter 3

  1. Holtzman, A. et al.: “The Curious Case of Neural Text Degeneration”. ICLR 2020. https://arxiv.org/abs/1904.09751↑
  2. CrowdStrike: “External Technical Root Cause Analysis – Channel File 291”, 6 August 2024. https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf And Weston, D. (Microsoft): “Helping our customers through the CrowdStrike outage”, 20 July 2024. https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/ Source of the estimate of 8.5 million affected Windows devices.↑
  3. Lamport, L.: “Who Builds a House without Drawing Blueprints?”. Communications of the ACM 58(4), 2015, pp. 38–41. https://doi.org/10.1145/2736348 In the original: “If you’re thinking without writing, you only think you’re thinking.”↑
  4. Deutscher Presserat [German Press Council]: Pressekodex [Press Code], section 3, “Richtigstellung” [Correction]. https://www.presserat.de/pressekodex.html A publication must correct false reports or claims promptly and of its own accord.↑
  5. Harris, Z. S.: “Distributional Structure”. Word 10(2–3), 1954, pp. 146–162. https://doi.org/10.1080/00437956.1954.11659520 The foundation of the idea that similar surroundings indicate similar meaning.↑
  6. Mikolov, T. et al.: “Efficient Estimation of Word Representations in Vector Space”. arXiv:1301.3781, 2013. https://arxiv.org/abs/1301.3781 The word2vec method.↑
  7. Mikolov, T.; Yih, W.; Zweig, G.: “Linguistic Regularities in Continuous Space Word Representations”. Proceedings of NAACL-HLT 2013, pp. 746–751. https://aclanthology.org/N13-1090/ Source of the example king − man + woman ≈ queen.↑
  8. Levesque, H. J.; Davis, E.; Morgenstern, L.: “The Winograd Schema Challenge”. Proceedings of KR 2012. https://cs.nyu.edu/~ed1/papers/WSKR2012.pdf Pairs of sentences like the Marlies and Hilde ones, in which a single word reverses what “she” refers to.↑
  9. Bahdanau, D.; Cho, K.; Bengio, Y.: “Neural Machine Translation by Jointly Learning to Align and Translate”. ICLR 2015. https://arxiv.org/abs/1409.0473 The first use of attention in neural networks for language.↑
  10. Tenney, I.; Das, D.; Pavlick, E.: “BERT Rediscovers the Classical NLP Pipeline”. Proceedings of ACL 2019. https://aclanthology.org/P19-1452/↑
  11. Vaswani, A. et al.: “Attention Is All You Need”. Advances in Neural Information Processing Systems 30, 2017. https://arxiv.org/abs/1706.03762↑
  12. Hochreiter, S.; Schmidhuber, J.: “Long Short-Term Memory”. Neural Computation 9(8), 1997, pp. 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735 The most important of the older networks that read word by word.↑
  13. Liu, N. F. et al.: “Lost in the Middle: How Language Models Use Long Contexts”. Transactions of the ACL 12, 2024. https://aclanthology.org/2024.tacl-1.9/↑
  14. Brown, T. et al.: “Language Models are Few-Shot Learners”. Advances in Neural Information Processing Systems 33, 2020. https://arxiv.org/abs/2005.14165 Shows how a few examples in the prompt improve performance.↑