Python Transformers: Padding, Truncation, and Attention Masks

Understanding how python transformers padding truncation and attention masks work together is essential for batching text through Hugging Face models correctly. When you pass a batch of texts through a transformer model, every sequence must have the same length. Real text rarely does, so the tokenizer relies on three related settings: padding, truncation, and attention masks. These three features work together to produce a correctly shaped input tensor that the model can process without treating artificial tokens as real content.

Why Padding, Truncation, and Attention Masks Are Connected

Transformer models operate on fixed-shape tensors. A batch of sequences must be a single rectangular matrix, which means every sequence must have the same number of tokens. Natural language does not cooperate with that requirement, so the tokenizer has to normalize the input.

Padding appends special tokens to shorter sequences so they match the length of the longest sequence in the batch. Truncation removes tokens from sequences that exceed the model's maximum input length. Attention masks mark which positions are real tokens and which are padding tokens.

These three settings are not independent. Padding without truncation lets very long sequences exceed the model's limit. Truncation without padding leaves sequences at different lengths. And padding without an attention mask makes the model treat padding tokens as real content, which distorts attention weights and produces incorrect outputs.

How the Tokenizer Handles Padding and Truncation

The AutoTokenizer class exposes padding and truncation as arguments to its __call__ method. You pass padding and truncation along with max_length when tokenizing a batch of texts.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

texts = [
    "The quick brown fox jumps over the lazy dog.",
    "A much shorter sentence.",
]

encoded = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=128,
    return_tensors="pt",
)

padding=True pads all sequences in the batch to the same length. By default this is the length of the longest sequence in the batch, known as "longest" padding. truncation=True cuts any sequence longer than max_length down to that limit. The max_length value should match the model's maximum input length, which you can inspect with tokenizer.model_max_length.

The resulting encoded object contains input_ids, attention_mask, and, for models like BERT that use token type embeddings, token_type_ids.

How Attention Masks Work

The attention mask is a tensor with the same shape as input_ids. It contains 1s for real tokens and 0s for padding tokens. The model uses it during the self-attention computation to prevent padding positions from contributing to attention scores.

print(encoded["attention_mask"])

In the example above, the first sequence is longer than the second, so the second sequence receives padding tokens. Its attention mask has 1s for the real tokens and 0s for the padding tokens. The model's attention layers multiply the mask into the attention weight computation, so positions with a mask value of 0 are ignored when computing the weighted sum over the sequence.

This is why the mask is not optional. Without it, the model would attend to padding tokens, which would shift the hidden representations of the real tokens and degrade the output.

Choosing a Padding Mode

The padding argument accepts several values:

Use padding="longest" when sequence lengths vary within a batch and you want to minimize wasted computation. Use padding="max_length" when you need a fixed tensor shape, which is common in serving pipelines that preallocate memory or expect a consistent input shape.

Dynamic padding with "longest" is usually more efficient for training because it avoids padding short sequences up to a large fixed length. The tradeoff is that the batch tensor shape varies between batches, which is fine for most training loops but can complicate inference pipelines that expect a fixed shape.

Choosing a Truncation Strategy

The truncation argument accepts several values:

For single-sequence inputs, truncation=True is equivalent to "longest_first", which removes tokens from the end of the sequence. For paired inputs, such as question-answer pairs or sentence pairs, "only_first" and "only_second" let you control which sequence gets cut. This matters when one side of the pair is more important than the other.

encoded = tokenizer(
    question,
    context,
    padding=True,
    truncation="only_second",
    max_length=384,
)

This is a common pattern in extractive question answering, where the context is truncated to fit the model's limit while the question is preserved in full.

Dynamic Padding with DataCollatorWithPadding

In a training loop, you process batches of varying sizes. The DataCollatorWithPadding class dynamically pads each batch to the longest sequence in that batch, which avoids wasting tokens on fixed-length padding.

from transformers import DataCollatorWithPadding

data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")

When you pass this collator to a Trainer or use it in a PyTorch DataLoader, it pads each batch to the batch's longest sequence and generates the corresponding attention mask automatically. This is the recommended approach for training because it keeps the batch tensor as small as possible while still meeting the model's fixed-shape requirement.

Common Mistakes and Edge Cases

One common mistake is forgetting that some tokenizers do not define a padding token. If you try to pad without a pad_token, the tokenizer raises an error. You can set one explicitly:

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

For decoder-only models like GPT-2, padding is typically applied on the left side of the sequence because the model generates tokens autoregressively and needs padding to appear before the real tokens. The padding_side attribute controls this:

tokenizer.padding_side = "left"

If you pad on the right for a decoder model, the padding tokens can interfere with generation because the model continues generating from the padding positions.

Another edge case: setting both padding="max_length" and truncation=True does not mean every sequence is truncated. Truncation only removes tokens from sequences longer than max_length. Shorter sequences are padded up to max_length, not extended.

Performance Considerations

Padding wastes computation because the model processes padding tokens even though the attention mask zeroes out their contribution. The amount of wasted work depends on how much padding you add. Dynamic padding with "longest" minimizes waste within a batch, but across batches the waste depends on the variance of sequence lengths in your dataset.

For inference, you can sort requests by length and batch similar-length sequences together to reduce padding overhead. This is a common serving optimization, though it adds latency complexity.

The attention mask itself adds a small amount of memory per token, but it is negligible compared to the model weights and activations. The real cost is the extra forward-pass work on padding tokens, which is why dynamic padding and length-based batching matter in production.