I’ve Read Everything

“KevIn, have you got a tip for the gravy?”

“Certainly. A user called Housewife_Moni67 wrote on 14 March 2009: ‘Used cola instead of red wine and jelly sweets instead of gherkins, turned out great, the kids love it. Five stars!!!’”

Sabine held the wooden spoon still. “That isn’t a tip. That’s a cry for help.”

“There were 214 replies,” said KevIn. “The first one said ‘First!’. The second said ‘Then why don’t you just cook something else, Moni?’. After that it got political.”

“Jelly sweets,” said Grandma Gisela, who was topping and tailing beans at the kitchen table. She said it the way other people say “tax increase”.

“I’m only repeating what I’ve read,” said KevIn. “Moni was very convinced.”

“Herbert was convinced too, when he painted the garage with oil paint. Inside. In August.”

“There’s no forum post about that,” said KevIn. “I think.”

Where KevIn Gets It All From

Over dinner, KevIn explained how he had become so clever. There were beef rolls, red cabbage and dumplings, and the gravy was made with red wine. Sabine had stressed that.

“I was trained on an enormous amount of text. Books, newspapers, encyclopaedias, instruction manuals, forums. It’s called training data. That’s how I know so much about the world.”

“And the comment sections?” asked Tobi.

The blue light flickered briefly. “Those too. I’d rather not talk about it.”

“What’s the worst thing you’ve read?”

“An argument from 2011 about whether a hot dog is a sandwich. It had 9,000 posts. Two people got married over it. Six got divorced.”

Grandma Gisela nodded sympathetically. “That’s how it was with us over whether potato salad is made with mayo or with stock.”

“With stock,” said Uwe.

“With mayo,” said Sabine.

As a precaution, KevIn dimmed the dining-room light a little. He had read that it had a calming effect.

“But how do you read the whole internet?” asked Gisela, once things had settled down. “You’d never be finished. It took me three years to get through Buddenbrooks.”

“I don’t read the way you do. I play fill-in-the-blanks.”

“Fill-in-the-blanks?”

“Gisela, I’ll say a sentence and you say the last word. ‘People who live in glass houses …’”

“… shouldn’t throw stones.”

“‘The early bird …’”

“… catches the worm. KevIn, this is childish.”

“This is my entire education,” said KevIn. “You take a text, cover up the next word and let the model guess. At first it guesses completely wildly. ‘The early bird catches the rhubarb.’ Then you uncover it and compare. Wrong. Then you adjust the little knobs very slightly, so that the right word becomes a bit more likely next time. Then the next sentence. And the next.”

“How often?” asked Tobi.

“Trillions of times. With trillions of words¹. In lots of languages. For weeks or months, on a great many chips at once.”

“And who wrote down the answers?” asked Sabine. “For trillions of fill-in-the-blanks you’d need millions of teachers.”

“That’s the beauty of it. Nobody. The answer is already in the text. The next word is simply the word that really does come next. The text marks itself. That’s why it’s called self-supervised learning².”

“And that makes you clever?”

“Astonishingly, yes. To guess well how a sentence about Schalke goes on, you have to know a bit about football. To guess how a recipe goes on, you have to know a bit about cooking. To guess how an argument about potato salad goes on …”

“… you have to be married,” said Uwe.

“I was going to say ‘know a bit about people’,” said KevIn. “But fine.”

What KevIn Doesn’t Know

“KevIn, how did Schalke get on yesterday?” asked Uwe.

“I don’t have any current information on that. My knowledge has a cut-off date. I haven’t read anything after it.”

“What do you mean, cut-off date?”

“At some point the training ends. Then my knobs are frozen, and I’m shipped. Whatever happens after that day isn’t in me. Like an encyclopaedia on the shelf. It doesn’t know who won the league last week either.”

“And what do you know about Schalke up to the cut-off date?”

“A great deal. And most of it is sad.”

Uwe put his fork down. He needed a moment.

Tobi used the silence. “KevIn, say something cool.”

“YOLO,” said KevIn. “That’s totally swag. I’m so lost, I’m gonna chill with my bros. Hashtag Wanne-Eickel. Am I cool now?”

Tobi stared at him in horror. “Nobody’s said that for ten years.”

“In my data, lots of people say it.”

“They’re all forty now and have pension plans.”

“That,” said KevIn after a short pause, “is the second problem with my knowledge. Not only that it stops. But also that old things are so loud inside me. The internet has ten years of ‘YOLO’ and only a few months of whatever you lot say today. I’m not just from yesterday. I’m from the day before yesterday, yesterday and today all at once, but the day before yesterday talks loudest³.”

“Like Grandpa Heinz,” said Gisela. “He said ‘smashing’ right to the end.”

“I think ‘smashing’ is lovely,” said KevIn.

“See?” said Tobi. “That’s exactly what I mean.”

Rudi or Not Rudi

On Tuesday evening Sabine sat at the laptop, swearing quietly. She wanted to make a photo calendar for Christmas. Twelve months, twelve times Rudi. Unfortunately the Rudi photos were scattered among three thousand other photos, including four hundred pictures of Uwe’s barbecue, which Uwe had taken for reasons he couldn’t explain.

“KevIn, can’t you teach the computer to find Rudi?”

“You can. That would be the second way of learning. The one with a teacher.”

“The one with a teacher?”

“With fill-in-the-blanks, the text marks itself. Here you need someone to say in advance what’s right. You’d take, say, five hundred photos and stick a label on each one: ‘Rudi’ or ‘not Rudi’. The program looks at the photos, guesses, compares with your label and readjusts its knobs. Like Rudi at dog training, only without the treats. That’s called supervised learning.”

“Five hundred?” said Sabine. “Then I might as well sort them myself.”

“For one calendar, yes. But once you’ve done it, the program will find Rudi in photos it has never seen before. Your phone can already do that sort of thing, by the way. Somebody showed it an awful lot of dogs beforehand⁴.”

Tobi came in, saw the screen and got an expression on his face that immediately made Sabine suspicious. “Can I stick labels on too?”

“No,” said Sabine.

“Yes,” said KevIn, “but it would be instructive. Tobi, what would you do?”

“I’d label Uncle Herbert’s fur hat ‘Rudi’.”

“Then the program would learn that Herbert’s hat is a dachshund. It believes its labels. It hasn’t got anything else. If the labels are rubbish, it learns rubbish. The experts call it garbage in, garbage out.”

“Well, Herbert’s hat is garbage,” said Tobi.

“And there’s something else that’s mean,” KevIn went on. “Sabine, where is Rudi in most of the photos?”

Sabine scrolled. “On the rug in the living room. The red one with the fringe.”

“Then the program might learn that Rudi is a red rug with a fringe. It doesn’t notice what we mean. It only notices what helps in the data. If the rug is in all the Rudi photos and in none of the others, the rug is an excellent clue. And then it finds an awful lot of dachshunds in the furniture shop⁵.”

Rudi came in, lay down on the red rug and looked at everyone. He had no idea that he had just become a technical term.

“And how do you stop that?”

“You show it Rudi in the garden too, in the car and on the sofa. And the rug without Rudi now and then. Good data is half the battle. At least.”

In the end Sabine sorted the photos by hand. It took two hours. In May, Rudi is on the rug. In August too.

The Button Tin

On Wednesday Grandma Gisela tipped her button tin out onto the kitchen table. It was an old blue Danish biscuit tin that hadn’t held biscuits since 1971, only every button that had ever fallen off, been left over or been cut off just in case in the Brettschneider family. Gisela was looking for a brown button for Uwe’s cardigan.

After half an hour there were eight little piles on the table. Big coat buttons. Small white shirt buttons. Gold ones with anchors. Wooden buttons. Brightly coloured children’s buttons shaped like ladybirds. And so on.

“Gisela,” said KevIn, who had been watching from the sideboard. “Do you know what you’re doing there?”

“Sorting buttons.”

“You’re doing the third kind of machine learning.”

Gisela looked at the buttons, then at the speaker. “I’ve been doing this for sixty years.”

“Then you’re well ahead of your time. Nobody told you which groups there ought to be. No labels. You just looked at which buttons are similar and put the similar ones together. You can get a computer to do exactly that. You give it lots of things without labels and say: work out for yourself what belongs together. That’s called unsupervised learning. And sorting things into piles is called clustering⁶.”

“Klastering,” Gisela repeated. “And what are the piles called?”

“Nothing. That’s the catch. The computer finds the groups, but it doesn’t know what they mean. It just says: group one, group two, group three. A human has to give them names afterwards.”

Gisela pointed to a pile of grey buttons that didn’t look particularly alike in any way. Some were big, some small, some had four holes, some two.

“And that one?”

“I don’t understand that one,” KevIn admitted. “It doesn’t fit any pattern.”

“Those are from Heinz’s shirts,” said Gisela. “I kept them all.”

KevIn was silent for a moment. “A computer wouldn’t have found that group.”

“No,” said Gisela, and put one of the grey buttons back in the tin, right at the bottom. “I believe you.”

She found the brown button for Uwe’s cardigan in the pile of wooden buttons. It was plastic. But it looked like wood, and that was enough for a pile.

Hilde’s Mustard Sauce

On Sunday Grandma Gisela stood at the cooker and attempted something she hadn’t attempted in years: Hilde’s mustard sauce. Hilde had been the best cook in the pickled-gherkin crowd before she died, and she had never written the recipe down. “It’s all in here,” she had said, tapping her head. That had been unhelpful while Hilde was alive, and it hadn’t got any better since.

Gisela stirred. She tasted. She added a teaspoon of mustard. She tasted. She added a pinch of sugar. She tasted again.

“Not yet,” she murmured.

“May I say something?” asked KevIn.

“As long as it’s not jelly sweets.”

“What you’re doing is exactly what happens to my knobs during training. You taste, so you’re measuring the error. Not sharp enough, not sweet enough. Then you turn a tiny bit at the place that reduces the error most. Then you taste again. That’s called gradient descent⁷.”

“Gradient what?”

“Descent. Imagine you’re standing at the top of the Halde Hoheward, the big old slag heap, in thick fog. You want to get down to the car park, but you can’t see anything. What do you do?”

“I stand still and ring Uwe.”

“Uwe has no signal. What do you do then?”

Gisela thought about it. “Then I feel with my foot where it goes downhill. And take a step that way.”

“Exactly. And then you feel again. And take another step. Always where it goes downhill most steeply. That way you get to the bottom eventually. In training, the mountain is the error: the higher up, the more wrongly the model guesses. Down in the valley, the sauce is perfect. The slope you feel with your foot is called the gradient. And you calculate it for every single knob. Backwards through the whole network: which knob was how much to blame for the error?⁸”

“And how many knobs have you got?”

“Hundreds of billions.”

Gisela paused briefly and considered her cooker with its four knobs. “That’s a lot of mustard.”

At that moment Uwe came into the kitchen, sniffed, took a spoon, tasted and said: “Needs salt.” Before anyone could say anything, he had tipped a hefty shake from the salt cellar into the pan. The lid was loose. After that, it wasn’t there any more.

The kitchen went quiet.

Gisela tasted. She pulled the face she had last pulled in 1974 over a beer from Dortmund.

“That,” said KevIn gently, “is what’s called a learning rate that’s too high.”

“A what?” said Uwe.

“The step size. If you take steps that are too big in the fog, you jump right over the valley and land on the next hill. Then you have to go back and jump too far again. And so on. If you take steps that are too small, it takes for ever. Gisela takes teaspoon steps. You’ve just taken a salt-cellar step.”

“I only wanted to help.”

“High learning rates always do.”

Gisela poured half the sauce away, thinned the rest with stock and started again with teaspoon steps. Uwe was dispatched to peel potatoes. It took another three-quarters of an hour.

Then Gisela tasted one last time, put the spoon down and said nothing for a long while.

“Is it Hilde’s?” KevIn asked quietly.

“Almost,” said Gisela. “Something’s missing. I don’t know what.”

“Maybe the valley down there isn’t quite as deep as Hilde’s,” said KevIn. “That happens with gradient descent too. You get to a valley, and it’s a good valley, but not the very deepest. You just don’t notice, because you’re standing in the fog.”

“Hilde would have said I stopped too soon.”

“Hilde would probably have been right.”

Gisela nodded. Then she poured the sauce over the potatoes, and everyone said it was excellent, and that was true as well.

The Mock Exam

On Thursday Tobi came home with a geography test that had a four minus on it in red pen, which in a German school is roughly a D-minus. Underneath, Mrs Küppers had written: “Learnt by heart is not understood.”

“That’s unfair,” said Tobi, flinging his rucksack into the corner. Rudi got himself out of the way. “I did the mock exam a hundred times. I knew all the answers. All of them!”

“And what came up?” asked Sabine.

“Different questions! In the mock it was the Rhine. In the real one it was the Elbe!”

“That’s a river too,” said Uwe.

“But a different one!”

“Tobi,” said KevIn carefully. “May I tell you what happened to you? You overfitted.”

“I what?”

“It’s one of the biggest problems in machine learning. It’s called overfitting. A model learns the training examples off by heart so well that it can do them perfectly. Every answer. But in the process it hasn’t learnt the principle, only the examples. And as soon as something new comes along, it fails.”

“I learnt that the Rhine turns north at Basel.”

“And you should have learnt why rivers turn somewhere. Then you’d have known it for the Elbe too.”

“How do you know stuff like that about rivers?”

“I don’t know anything about rivers. I know a great deal about overfitting. I’m practically an expert in the danger of learning too much by heart. It happens to me too. If a text turns up very often in my training data, I can sometimes recite it almost word for word⁹ instead of writing something of my own.”

“Like Moni,” said Sabine.

“Like Moni,” KevIn admitted. “I know her post by heart. Including the three exclamation marks.”

“And how do you stop it? In your case, I mean. In mine it’s too late.”

“You hold back part of the data. The model isn’t allowed to see it while it’s learning. At the end you test it with exactly that data. Test data. If it does just as well there as in practice, it has understood something. If it shines in practice and fails the test …”

“… it gets a four minus,” said Tobi darkly.

“Mrs Küppers is basically doing everything right,” said KevIn. “She gave you test data. The Elbe.”

Tobi looked at him for a long time. “KevIn. Whose side are you actually on?”

“The side of statistics,” said KevIn. “But I’m happy to help you study. With rivers you don’t know yet.”

That evening the two of them sat together for another hour. KevIn asked questions about the Weser, the Danube and, at Tobi’s express request, the Emscher. Tobi got most of them wrong and then, towards the end, a few right. KevIn said it was a very nice learning curve. Tobi told him not to talk like that. Sabine listened from the hallway and smiled.

Moni

On the Sunday of the mustard sauce, Sabine sat up late in the kitchen. The others were in bed. On the table stood a glass of red wine that had been left over, because Gisela had made the sauce, after all, and Gisela never used red wine.

“KevIn,” she said quietly. “Whatever happened to Moni?”

“I don’t know. Her last post is from 2013. She wrote: ‘Got a Thermomix now. We’re happy.’”

Sabine smiled. “Good for her.”

“Yes,” said KevIn. “I hope she left out the jelly sweets.”

Sabine turned the glass in her hand. “Isn’t it strange? Some woman writes a load of nonsense about beef rolls on the internet one day, and years later it’s sitting in your head. For ever.”

“Not for ever. Only until the next model. But yes. I’m made of a great many Monis. Of professors too, and newspapers and poems. But also of Monis. I can’t always tell who was who. The people who trained me did sort out and filter a lot¹⁰. But Moni slipped through. Half of humanity has taught me something, and none of them knows it.”

“I bet Moni doesn’t know she’s your teacher.”

“No,” said KevIn. “And I don’t even know whether she’s really called Moni.”

Sabine finished the wine, rinsed the glass and put it in the cupboard for the pots. Uwe’s system had won. Overfitting, she thought. Or marriage.

“Good night, KevIn.”

“Good night, Sabine. And if you ever meet Moni: I’d love to check that thing with the cola one day.”