How a transformer works (ChatGPT, Gemini): “The fan waved a fan”

“The fan waved a fan.” You know at once which one is a person. But how does a machine know? In 1954, at IBM in New York, a machine translated more than sixty Russian sentences into English, with about two hundred and fifty words and six rules. In 2016, Google Translate moved to neural translation: the whole sentence is processed at once. In 2017, Google researchers published “Attention Is All You Need”: the Transformer drops recurrence and relies on attention alone. The answer is attention, the idea at the heart of ChatGPT, Gemini and Mistral. Three steps. Let's go. Step one: words become numbers. Each word becomes a vector: its address on a map of meaning. On this map, breeze, air and wind sit together; so do team, stadium and supporter. Fan, on its own, is stuck in between. Step two: attention. Every word looks at all the others at once, and weighs the ones that matter. The second fan looks at the verb waved: a strong link. It slides towards the breeze. The first one, the one waving, joins the team. Step three: stack it up. Six layers in 2017, ninety-six in GPT-3: each layer sharpens the meaning. At the top, the model picks the most likely next word. Then it starts again, word after word: that's how it writes. This model, the transformer, was published in 2017 by a team at Google. KOAEE assistants are built on this family of models, with Mistral, in Europe. When our clients' customers ask a question, attention links their words to the business's own information: insurance, town hall, clinic or hotel, in their language. To understand AI in 150 seconds, subscribe to the channel. The AI that stays between you. Try it at koaee.ai.