Python Transformers for Summarization, Translation, and QA

Python transformers summarization translation and question answering tasks are commonly implemented with the Hugging Face transformers library. The pipeline abstraction reduces boilerplate, while the underlying model and tokenizer classes give you finer control for production workloads. This article covers both approaches, including tokenization details, batch processing, model selection, and memory management.

Using the Pipeline for All Three Tasks

The pipeline function is the fastest way to get started. It loads a pretrained model and its tokenizer, applies the necessary preprocessing, and returns human-readable results. For summarization, translation, and question answering, the API is nearly identical.

from transformers import pipeline

summarizer = pipeline("summarization")
translator = pipeline("translation", model="Helsinki-NLP/opus-mt-en-fr")
qa = pipeline("question-answering")

text = "The transformer architecture introduced in 2017 has become the foundation of modern NLP. It uses self-attention to weigh the importance of each token in a sequence."
summary = summarizer(text, max_length=30, min_length=10)[0]["summary_text"]
print("Summary:", summary)

translation = translator("Hello, world!")[0]["translation_text"]
print("Translation:", translation)

answer = qa(question="What architecture became the foundation of modern NLP?", context=text)["answer"]
print("Answer:", answer)

The pipeline handles tokenization, model inference, and decoding internally. For translation, you must specify the model because the default is English-to-German. For question answering, the pipeline expects a question and a context. Each task returns a dictionary with task-specific keys.

Choosing Between Pipeline and Explicit Model + Tokenizer

The pipeline is convenient for prototyping and small workloads, but it hides important details. When you need to control padding, truncation, or the exact tokenizer settings, or when you want to reuse a loaded model across multiple calls, you should use the model and tokenizer directly.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-large-cnn")

inputs = tokenizer(text, return_tensors="pt", max_length=1024, truncation=True)
summary_ids = model.generate(inputs["input_ids"], max_length=30, min_length=10)
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

This explicit approach makes it clear what parameters are passed to the tokenizer and the model. It also lets you move the model to a GPU with model.to("cuda") and manage batching manually. Use the pipeline when the default behavior is sufficient; use the explicit API when you need reproducibility or performance tuning.

Handling Tokenization and Model Inputs Correctly

Tokenization is the step where raw text becomes input IDs. Each model has a maximum sequence length, and inputs longer than that must be truncated or split. For summarization and translation, truncation is common because the model can only process a fixed window. For question answering, the context often exceeds the limit, so you must split it into overlapping chunks and aggregate answers.

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")
model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")

question = "What is the capital of France?"
context = "France is a country in Europe. Its capital is Paris."

inputs = tokenizer(question, context, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
start_scores = outputs.start_logits
end_scores = outputs.end_logits
start_idx = torch.argmax(start_scores)
end_idx = torch.argmax(end_scores)
answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(inputs["input_ids"][0][start_idx:end_idx+1]))

Notice that the tokenizer takes both question and context as a pair. The model returns start and end logits, and you select the span with the highest probability. The pipeline does this internally, but understanding the mechanics helps when you need to implement custom post-processing or handle multiple context chunks.

Batch Processing for Efficiency

When processing many documents or queries, batching inputs reduces overhead by parallelizing inference. The tokenizer can pad sequences to the same length, and the model processes them together. For summarization and translation, you must generate output sequences of varying lengths, so the generate method handles this internally.

from transformers import pipeline

summarizer = pipeline("summarization", device=0)  # use GPU if available

texts = [
    "The quick brown fox jumps over the lazy dog. This sentence contains every letter of the alphabet.",
    "Python is a high-level programming language. It emphasizes code readability and simplicity."
]

results = summarizer(texts, max_length=20, min_length=5, batch_size=2)
for r in results:
    print(r["summary_text"])

The batch_size parameter controls how many inputs are processed at once. Larger batches improve throughput but increase memory usage. For question answering, you can batch multiple question-context pairs, but each pair must have the same tokenized length after padding. The pipeline handles this automatically.

Model Selection and Task-Specific Considerations

Different models are pretrained for different tasks. For summarization, BART and T5 are common choices. For translation, MarianMT and T5 work well. For question answering, DistilBERT and BERT fine-tuned on SQuAD are standard. The model size directly affects inference speed and memory footprint.

Choose a smaller model when latency matters or when you deploy on CPU. For production, consider distillation or quantization to reduce memory without a major quality drop.

Memory and Inference Cost Management

Transformer models are memory-intensive. The default float32 weights for a 400 MB model occupy 400 MB of RAM or VRAM. You can reduce this by loading in float16 on a GPU or by using quantization. The from_pretrained method accepts a torch_dtype argument.

from transformers import AutoModelForSeq2SeqLM
import torch

model = AutoModelForSeq2SeqLM.from_pretrained(
    "facebook/bart-large-cnn",
    torch_dtype=torch.float16,
    device_map="auto"
)

On CPU, you can use bitsandbytes for 8-bit quantization, but this requires additional dependencies. Always monitor memory usage with tools like nvidia-smi or psutil. If you process long documents, the attention matrix grows quadratically with sequence length, so consider chunking or using models with sparse attention.

Error Handling and Edge Cases in Production

In production, inputs are unpredictable. For summarization, extremely short texts may produce empty summaries. For translation, unsupported language pairs will raise an error. For question answering, the model may return an answer that is not in the context if the context is too short or ambiguous.

def safe_summarize(text, max_length=30):
    if len(text.split()) < 10:
        return text  # fallback for short inputs
    try:
        return summarizer(text, max_length=max_length)[0]["summary_text"]
    except Exception as e:
        return text

For question answering, you should validate that the start and end indices are valid and that the answer is non-empty. When the context is longer than the model's maximum, split it into overlapping windows and choose the answer with the highest confidence score. The pipeline does not do this automatically, so you need a custom loop for long contexts.

def answer_long_context(question, context, max_len=512, stride=128):
    tokenized = tokenizer(question, context, return_offsets_mapping=True, truncation=True, max_length=max_len, stride=stride, return_overflowing_tokens=True)
    best_answer = ""
    best_score = -float("inf")
    for i, input_ids in enumerate(tokenized["input_ids"]):
        outputs = model(torch.tensor([input_ids]))
        start_scores = outputs.start_logits[0]
        end_scores = outputs.end_logits[0]
        start_idx = torch.argmax(start_scores)
        end_idx = torch.argmax(end_scores)
        if start_idx <= end_idx and start_scores[start_idx] + end_scores[end_idx] > best_score:
            best_score = start_scores[start_idx] + end_scores[end_idx]
            best_answer = tokenizer.decode(input_ids[start_idx:end_idx+1], skip_special_tokens=True)
    return best_answer

This sliding-window approach ensures that the answer is found even when the context exceeds the model's limit. It also demonstrates how to combine tokenizer options with model inference to handle a real production constraint.