In June 2017, eight researchers published a paper with an unusually confident title:

The paper was not about chatbots.

It was not about artificial general intelligence.

It did not claim that machines could reason like humans.

Its primary problem was considerably narrower:

Can we build a better machine-translation system without relying on recurrent or convolutional neural networks?

The answer proposed by Ashish Vaswani and colleagues was the Transformer—an encoder-decoder architecture built around attention mechanisms rather than recurrence.

That architectural decision eventually became one of the foundations of modern language AI.

But to understand why the paper mattered, we need to separate three things:

what the researchers actually proposed, what their experiments actually demonstrated, and what the research community built afterward.

That is what this paper brief will do.

The Paper at a Glance

Title: Attention Is All You Need

Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin

Published: 2017

Venue: 31st Conference on Neural Information Processing Systems — NIPS 2017, now called NeurIPS

Primary task: Neural machine translation

Core contribution: The Transformer architecture

Central architectural idea: Replace sequence-aligned recurrence and convolution with attention-based computation.

The official NeurIPS paper states that the proposed Transformer relies on attention mechanisms while dispensing with recurrence and convolutions in its core sequence-processing architecture.

1. What Problem Were the Researchers Trying to Solve?

To understand the Transformer, first forget ChatGPT.

Imagine it is 2017 and you are trying to translate:

“The farmer went to the market because he needed seeds.”

into another language.

A machine must understand relationships across the sentence.

Who does “he” refer to?

What does “because” connect?

Which translated words should appear first?

For years, recurrent neural networks—including RNNs, LSTMs and GRUs—were major tools for sequence modelling.

They process a sequence progressively.

Conceptually:

Word 1 → Word 2 → Word 3 → Word 4 → …

At each step, information from earlier positions is carried forward through a hidden state.

The Transformer paper identifies a major computational disadvantage of this design: because the hidden state at position t depends on the previous state at t−1, computation across positions is inherently sequential. That restricts parallelization during training.

Think of it as a relay race.

Runner 4 cannot begin until runner 3 hands over the baton.

Modern hardware, however, is extremely good at performing many mathematical operations simultaneously.

The researchers asked:

What if sequence positions did not need to wait for one another during representation learning?

That question leads directly to attention.

2. Attention Already Existed

An important historical point is often lost in simplified accounts.

The Transformer paper did not invent attention itself.

Attention mechanisms were already being used with recurrent sequence-to-sequence models.

The authors explicitly acknowledge this background. Their novelty was much more specific:

Build the sequence model around attention rather than using attention merely as an addition to recurrence.

The paper describes the Transformer as, to the authors’ knowledge, the first transduction model relying entirely on self-attention for its input and output representations without sequence-aligned RNNs or convolution.

That distinction matters.

Scientific breakthroughs frequently come not from inventing every component, but from recombining existing ideas in a fundamentally better architecture.

3. The Central Idea: Self-Attention

Consider:

“The animal did not cross the road because it was tired.”

When humans read “it,” we naturally relate it to other words in the sentence.

The Transformer attempts something mathematically analogous.

For each position, self-attention asks:

Which other positions contain information useful for representing this position?

The original paper describes an attention function in terms of three objects:

Query

Key

Value

These names sound more mysterious than they are.

4. Query, Key and Value in Plain English

Imagine searching a library.

Your search request is the:

Query

Each book has information describing what it contains:

Key

And the useful content inside the matching book is the:

Value

The system compares the query with available keys.

Strong matches receive greater weight.

Their values contribute more strongly to the output.

The Transformer performs a learned mathematical version of this process.

The paper defines its scaled dot-product attention as:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V

The notation can look intimidating, but conceptually it says:

Compare → Score → Normalize → Combine

That is the heart of attention.

5. Why Divide by √dₖ?

This small piece of the equation is easy to ignore:

√dₖ

But it is an interesting example of research engineering.

When query and key vectors become higher-dimensional, their dot products can become large in magnitude.

Large values passed into softmax can push it into regions with very small gradients, making optimization harder.

The authors therefore scale the dot products by 1/√dₖ. They explain this motivation explicitly in the paper.

This illustrates an important lesson when reading research papers:

Major architectures are often built from many seemingly small engineering decisions that collectively matter.

6. What Makes It “Self”-Attention?

In self-attention, the queries, keys and values originate from representations within the same sequence.

Imagine:

The bank approved the loan.

versus:

The children sat on the bank of the river.

The word bank appears in both.

But its meaning depends on surrounding context.

Self-attention allows the representation of a token to incorporate information from other relevant positions in that sequence.

In the Transformer encoder, each position can attend across the encoder’s previous-layer representations.

That is one reason Transformer representations can become strongly contextual.

7. A Crucial Correction: Not Every Token Always Sees Every Future Token

You will often hear:

“In a Transformer, every word looks at every other word.”

That is a useful introductory intuition—but incomplete.

In the encoder self-attention, positions can attend across the input representations.

But the original Transformer’s decoder uses masking.

When predicting output position i, the decoder is prevented from seeing future output positions.

Why?

Because otherwise it could cheat.

If we are predicting:

“The cat sat on the ___”

the model should not be allowed to look at the answer appearing later in the training sequence.

The original paper implements this by masking illegal future connections before softmax.

This distinction eventually becomes extremely important for understanding autoregressive language models.

8. Why Multi-Head Attention?

The researchers did not use only one attention operation.

They used multi-head attention.

The original model employed eight attention heads in its standard configuration. Each head worked with learned projections of queries, keys and values, and their outputs were combined afterward.

Why?

Suppose we read:

“The student who submitted the assignment late said she had been ill.”

Different relationships may matter:

grammatical structure,
reference between student and she,
relationships involving assignment,
positional relationships,
contextual meaning.

Multiple heads allow the model to examine the sequence through multiple learned representation subspaces.

But we should avoid a popular oversimplification:

“One head learns grammar, one learns meaning, one learns pronouns.”

That is not guaranteed.

The original authors observed that different attention heads appeared to learn different behaviours, including patterns associated with syntactic and semantic structure. But attention heads are learned numerical mechanisms, not predefined linguistic modules.

9. If There Is No Recurrence, How Does the Model Know Order?

Consider:

Dog bites man.

and:

Man bites dog.

Same words.

Completely different meaning.

Self-attention alone has no inherent notion that one token came before another.

The Transformer therefore adds positional encodings to token embeddings.

The original paper used sine and cosine functions of different frequencies to generate positional information.

Conceptually, each input representation contains information about:

what the token represents

plus

where the token occurs

The authors also tested learned positional embeddings and reported nearly identical performance in their experiment. They selected sinusoidal encoding partly because they hypothesized that it might extrapolate better to sequence lengths beyond those encountered during training.

Notice the scientific wording:

they hypothesized.

The paper did not establish sinusoidal positional encoding as the only correct solution.

Later Transformer architectures adopted many different positional strategies.

10. “Attention Is All You Need” Is Not Literally True

The title is brilliant.

It is also deliberately provocative.

The Transformer contains much more than attention.

The original encoder layer includes:

Multi-head self-attention

↓

Residual connection + Layer Normalization

↓

Position-wise feed-forward network

↓

Residual connection + Layer Normalization

The decoder adds another attention mechanism over encoder outputs.

The architecture also uses:

embeddings,
positional encodings,
feed-forward networks,
residual connections,
normalization,
softmax,
masking,
dropout,
and an optimization schedule.

The original encoder and decoder each contained six stacked layers; the base architecture used a model dimension of 512, while its feed-forward sublayer expanded internally to 2,048 dimensions.

So a more literal title might have been:

“Attention Plus Feed-Forward Networks, Residual Connections, Layer Normalization, Positional Information and Several Other Things Is All You Need.”

It would not have been quite as memorable.

11. The Architecture in One Picture

At a high level:

Encoder

Input tokens

↓

Embeddings + Position

↓

Multi-Head Self-Attention

↓

Feed-Forward Network

↓

Repeated 6 times

↓

Encoded representations

Decoder

Previously generated tokens

↓

Masked Self-Attention

↓

Attention over Encoder Output

↓

Feed-Forward Network

↓

Repeated 6 times

↓

Linear + Softmax

↓

Next-token prediction

The original Transformer was therefore an encoder-decoder sequence-to-sequence model, specifically designed and tested primarily for translation.

This is important because modern models descended from the Transformer do not all use the complete original architecture.

12. The Most Important Engineering Advantage: Parallelization

This is arguably where the paper becomes transformational.

A recurrent network requires sequential operations across positions.

The paper compares this with self-attention.

For a sequence of length n and representation dimension d, the authors give:

Architecture	Sequential operations	Maximum path length
Self-attention	O(1)	O(1)
Recurrent	O(n)	O(n)

The paper’s comparison also gives self-attention per-layer complexity as O(n²d) and recurrent complexity as O(nd²).

What does this mean without the mathematics?

An RNN is like:

Token 1 → Token 2 → Token 3 → Token 4

Self-attention is closer to:

process relationships among positions using large parallel matrix operations

GPUs are exceptionally good at those operations.

That did not make Transformers computationally cheap.

It made their training computation far more parallelizable across sequence positions.

That distinction is crucial.

13. Long-Range Dependencies Became Easier to Connect

Suppose the first word of a sentence matters greatly to word 50.

In an RNN, the computational path connecting those positions spans many recurrent steps.

The Transformer creates much shorter paths.

The authors explicitly used maximum path length between positions as one of their arguments for self-attention, reasoning that shorter computational paths can make long-range dependencies easier to learn.

That does not mean Transformers possess unlimited memory.

It means the architecture provides a more direct route through which distant positions can interact.

14. What Was the Experiment?

This is where we move from architecture to evidence.

The primary experiments were machine translation.

The researchers trained on:

WMT 2014 English–German

approximately 4.5 million sentence pairs

and

WMT 2014 English–French

approximately 36 million sentence pairs.

The models were trained on a machine containing eight NVIDIA P100 GPUs.

The base models trained for approximately 12 hours, while the larger models trained for approximately 3.5 days.

The main evaluation metric was BLEU, a widely used machine-translation metric.

15. What Results Did the Paper Actually Report?

This is where the paper earned attention.

On WMT 2014 English-to-German, the large Transformer achieved:

28.4 BLEU

The paper reported that this exceeded the previously reported best results—including ensembles—by more than 2 BLEU points.

On WMT 2014 English-to-French, the large Transformer achieved:

41.0 BLEU

The authors reported that it surpassed previous published single models while using less than one-quarter of the estimated training cost of the previous state-of-the-art model used in their comparison.

The evidence therefore supported a strong but specific conclusion:

For the translation tasks studied, an attention-centered architecture without recurrence could achieve excellent translation quality while allowing substantially greater training parallelism.

That is a remarkable result.

It is also more precise than saying:

“Transformers understand language better than everything else.”

The paper did not test that proposition.

16. The Ablation Experiments Matter

Strong research does more than present the final model.

It asks:

Which design choices matter?

The authors varied components including:

number of attention heads,
attention dimensions,
model dimensions,
feed-forward dimensions,
dropout,
and positional representations.

One interesting finding was that single-head attention performed worse than their best multi-head setting, while simply increasing the number of heads indefinitely did not continuously improve quality.

This is an important research habit.

When a model performs well, we should not merely ask:

“What score did it get?”

We should ask:

“Which components contributed to that result?”

17. What Did the Paper NOT Demonstrate?

This section may be the most important part of reading the paper today.

It did not demonstrate a chatbot

There was no ChatGPT-style conversational assistant.

It did not demonstrate general reasoning

The principal evaluation was machine translation, with an additional constituency-parsing experiment.

It did not demonstrate few-shot learning

That became prominent in later large language models.

It did not demonstrate instruction following

Instruction-tuned systems came years later.

It did not demonstrate retrieval-augmented generation

RAG is a separate later architecture.

It did not prove that attention alone creates intelligence

The paper proposed and tested a sequence architecture.

That’s it.

These distinctions prevent us from rewriting history backward from today’s AI systems.

18. Then How Did We Get From the Transformer to GPT?

This is where follow-up research becomes critical.

In 2018, Radford and colleagues demonstrated that a Transformer-based language model could be generatively pretrained on large amounts of unlabeled text and then fine-tuned for downstream language-understanding tasks.

This introduced another major idea:

Transformer architecture + large-scale language-model pretraining

That combination was different from the original translation experiment.

19. Then Came BERT

BERT took another direction.

Instead of using a left-to-right generative objective, BERT was designed to learn deep bidirectional Transformer representations, conditioning on both left and right context.

The resulting model achieved new state-of-the-art results across eleven NLP tasks in the reported experiments, including question answering and language inference.

Again:

The Transformer supplied the architecture.

BERT supplied important new pretraining and adaptation ideas.

Those innovations should not be collapsed into a single 2017 breakthrough.

20. T5 Pushed the Framework Further

Raffel and colleagues later explored a unified text-to-text framework in T5, treating a wide range of NLP problems as input text mapped to output text.

Their work systematically studied transfer-learning choices around objectives, datasets, architectures and scaling.

This illustrates how research progresses.

The Transformer was not a finished destination.

It became a platform on which researchers could conduct new experiments.

21. The Original Transformer Had a Limitation Hiding in Plain Sight

Self-attention creates pairwise interactions among positions.

If a sequence has n tokens, standard full attention involves an n × n relationship structure.

The original paper gives its per-layer complexity as:

O(n²d)

For relatively short sentences, this was highly practical.

For extremely long sequences, the quadratic dependence on sequence length becomes expensive.

Later research explicitly targeted this limitation with sparse and efficient attention architectures. For example, BigBird identified quadratic sequence-length dependence as a core limitation of full attention and proposed sparse attention to reduce it.

So the same mechanism that enabled direct global interactions also introduced an important scaling challenge.

Research rarely gives us free advantages.

It gives us trade-offs.

22. Was Attention the Reason Transformers Won?

Partly—but this question deserves care.

The paper showed that the complete Transformer architecture performed extremely well.

It did not experimentally prove that attention alone was sufficient.

The model also contained feed-forward layers, residual connections, normalization, positional information and other design choices.

Later theoretical research has even examined what happens when self-attention is stripped of these surrounding components. Work on pure self-attention networks found that skip connections and multilayer perceptrons play important roles in preventing representational degeneration.

So the lasting lesson is not literally:

Attention is the only thing neural networks need.

A better interpretation is:

Recurrence was not necessary for achieving state-of-the-art sequence transduction. Attention could become the central computational mechanism.

That was revolutionary enough.

23. Why Did This Paper Matter So Much?

The significance can be understood through four contributions.

1. It challenged recurrence

Sequence modelling no longer had to mean processing representations sequentially through an RNN.

2. It made training far more parallelizable

This aligned the architecture extremely well with GPU-style matrix computation.

3. It shortened computational paths between distant positions

Self-attention directly connected positions that recurrent models would connect through many intermediate steps.

4. It created an adaptable architecture

Later researchers could use encoder-oriented, decoder-oriented and encoder-decoder Transformer variants for very different objectives.

BERT, GPT-style models and T5 illustrate these divergent paths.

24. The Researcher’s View: Strengths of the Paper

Looking at the paper as a research contribution rather than a historical monument, several strengths stand out.

Clear research hypothesis

Can attention replace recurrence for sequence transduction?

Architectural novelty

The work reorganized sequence processing around self-attention.

Strong empirical results

It reported state-of-the-art translation performance under the comparisons used.

Computational argument

The paper did not evaluate only accuracy; it explicitly examined complexity, sequential operations and path length.

Ablation studies

The authors investigated how architectural choices affected performance.

Reproducible architectural detail

The paper documents dimensions, number of layers, attention heads, optimizer, warm-up schedule, dropout and other training details.

This combination of idea + mechanism + empirical evidence + computational analysis is one reason the paper became so influential.

25. The Researcher’s View: Limitations

A landmark paper should still be read critically.

Limited primary task domain

The central experiments focused on machine translation.

Limited evidence for broader intelligence

Nothing in the experiments demonstrated the broad capabilities associated with today’s large language models.

Quadratic attention cost

Full self-attention becomes increasingly expensive as sequences grow.

Hardware-specific training comparisons

Training-cost comparisons depend partly on contemporary implementations and hardware.

Later capabilities required later research

Pretraining at scale, transfer learning, instruction tuning, reinforcement learning, retrieval and many other innovations were not part of the original Transformer paper.

None of these criticisms diminish the paper.

They simply describe what it actually established.

That is how research should be read.

26. The Most Important Figure to Understand

If you open the original paper, Figure 1 may initially look intimidating.

Do not attempt to understand every arrow.

Read it vertically.

For the encoder:

Embedding

↓

Position information

↓

Self-attention

↓

Feed-forward network

↓

repeat

For the decoder:

previous outputs

↓

masked self-attention

↓

attention over encoder information

↓

feed-forward network

↓

next-token probabilities

Once that structure is clear, the detailed diagram becomes much easier to read.

27. The Most Important Equation to Understand

You do not need to memorise:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V

Instead remember:

Q — What am I looking for?

K — What information do I contain?

V — What information should I contribute if I am relevant?

Then:

Q matches K → determines weight → combines V

That intuition will take you surprisingly far.

28. The Most Important Table to Understand

Table 1 may ultimately be more important than the attention formula.

It compares self-attention, recurrence and convolution in terms of:

computational complexity,
sequential operations,
and maximum path length.

The crucial observation was:

Self-attention: O(1) sequential operations

versus

Recurrent layer: O(n) sequential operations

That tells us why the paper was not merely proposing a different way to calculate relationships between words.

It was proposing a fundamentally different computational structure for sequence modelling.

29. If You Remember Only Five Things

First:

Attention existed before Transformers.

The breakthrough was making it central enough to remove sequence-aligned recurrence.

Second:

Self-attention lets representations interact directly across sequence positions.

Third:

Multi-head attention allows several learned attention projections to operate in parallel.

Fourth:

Removing recurrence made training much more parallelizable across sequence positions.

And fifth:

The original Transformer was a machine-translation architecture—not a chatbot, reasoning engine or claim of general intelligence.

Everything that followed should be understood as subsequent research built on that foundation.

The Bigger Lesson

It is tempting to view Attention Is All You Need through the lens of what came afterward.

GPT.

BERT.

T5.

Large language models.

Multimodal AI.

But doing so can make the original paper seem almost inevitable.

It wasn’t.

In 2017, recurrent networks were deeply established in sequence modelling.

The researchers made a striking architectural bet:

Remove recurrence. Let attention carry the sequence relationships.

Then they tested that proposition.

The results showed that the approach could outperform highly competitive translation systems while enabling substantially greater computational parallelism.

That is what made the paper important.

Not because it built ChatGPT.

Not because it solved intelligence.

But because it changed the answer to a fundamental engineering question:

How should a neural network process a sequence?

Before 2017, recurrence was one of the dominant answers.

After Attention Is All You Need, it no longer had to be.

And that architectural shift created the foundation on which much of modern language AI was subsequently built.

Paper Scorecard

Dimension	Assessment
Research question	Can sequence transduction work without recurrence or convolution?
Core innovation	Transformer architecture centered on self-attention
Primary experiments	English–German and English–French machine translation
Key result	State-of-the-art translation results under the paper’s comparisons
Engineering contribution	Much greater parallelization across sequence positions
Major trade-off	Quadratic full-attention cost with sequence length
What it did not show	Chatbots, instruction following, general reasoning or AGI
Long-term significance	Architectural foundation adapted by many later language-model families

References

[1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30. This is the primary source for the Transformer architecture, attention mechanism, complexity analysis, training procedure and translation experiments.

[2] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI. This work demonstrated generative Transformer pretraining followed by task-specific fine-tuning across language-understanding benchmarks.

[3] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019. BERT demonstrated deep bidirectional Transformer pretraining and achieved state-of-the-art results across multiple NLP benchmarks reported in the paper.

[4] Raffel, C., Shazeer, N., Roberts, A., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21. T5 systematically explored Transformer-based transfer learning under a unified text-to-text framework.

[5] Zaheer, M., Guruganesh, G., Dubey, A., et al. (2020). Big Bird: Transformers for Longer Sequences. NeurIPS 2020. This work addresses the quadratic sequence-length dependence of full attention using a sparse-attention architecture.

How to Read the Original Paper

If this is your first serious AI research paper, do not begin by trying to understand every equation.

Read it in this order:

1. Abstract
Understand the claim.

2. Introduction
Understand the problem.

3. Figure 1
Understand the architecture.

4. Sections 3.2 and 3.5
Understand attention and positional encoding.

5. Table 1
Understand why the computational structure matters.

6. Table 2
Understand the experimental evidence.

7. Model Variations
Understand what the researchers changed and what happened.

8. Return to the equations
Only now study the mathematics in detail.

This turns a difficult paper into a sequence of manageable questions.

Next in Papers Explained

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The next paper provides a natural continuation.

Attention Is All You Need gave us the architecture.

BERT asks a different question:

What happens if we pretrain a Transformer to build deep representations from context on both sides of a token, then adapt that model to many language-understanding tasks?

Following these papers chronologically helps us see modern AI not as one sudden breakthrough, but as a chain of research ideas—each solving a different part of the problem.