Python Boto3 Pagination and Error Handling

python boto3 pagination and error handling requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you call a boto3 client method such as list_objects_v2, describe_instances, or scan, the AWS service returns a single page of results. Most services cap a page at 1,000 items, and some return fewer depending on the request. The response includes a continuation token when more data exists, but the client method does not follow it for you. A Paginator wraps the underlying API call, sends repeated requests with the continuation token, and yields each response as a page. Python boto3 pagination and error handling come down to understanding how paginators iterate and how botocore surfaces failures when a page request fails.

How Paginators Work in boto3

A paginator is created from a client with get_paginator, passing the name of the paginated operation. The resulting object exposes a paginate() method that accepts the same parameters as the underlying client call, plus an optional PaginationConfig dictionary.

import boto3

s3 = boto3.client("s3")
paginator = s3.get_paginator("list_objects_v2")

for page in paginator.paginate(Bucket="my-bucket"):
    for obj in page.get("Contents", []):
        print(obj["Key"])

The paginator keeps the continuation token internal. Each iteration of the outer loop is one HTTP request to S3, and the inner loop processes the objects in that page. If the bucket has 2,500 objects, the loop runs three times: two full pages of 1,000 and one page of 500.

Not every boto3 client method has a corresponding paginator. Paginators exist for operations that the AWS service model marks as paginated. You can check whether a client supports pagination for a given operation by calling client.can_paginate("operation_name"), or by calling get_paginator and catching botocore.exceptions.OperationNotPageableError.

Controlling Page Size and Result Limits with PaginationConfig

The paginate() method accepts a PaginationConfig dictionary with three commonly used keys.

for page in paginator.paginate(
    Bucket="my-bucket",
    PaginationConfig={"PageSize": 200, "MaxItems": 5000},
):
    for obj in page.get("Contents", []):
        print(obj["Key"])

PageSize does not change the service's hard limit; it only asks for a smaller page. A service that caps responses at 1,000 items will still return 1,000 if you request 2,000. MaxItems is useful when you need only a sample of results and want to avoid fetching the entire dataset.

StartingToken is returned in the paginate response under NextToken or a service-specific key. When you need to resume a paginated scan across separate invocations of your code, store that token and pass it back in PaginationConfig.

Filtering Results with search()

Paginator objects expose a search() method that applies a JMESPath expression to each page and yields only the matching elements. This avoids pulling every object into memory just to filter it in Python.

for obj in paginator.paginate(Bucket="my-bucket").search(
    "Contents[?Size > `1048576`]"
):
    print(obj["Key"], obj["Size"])

The expression selects objects from the Contents array where Size is greater than 1,048,576 bytes. The paginator still makes the same number of API calls, but the iterator yields only matching items. If the expression matches nothing, the loop simply does not run.

JMESPath is the same query language used by the AWS CLI's --query option. Keep expressions simple; complex projections can be harder to debug than a plain Python filter, especially when the response structure varies between pages.

Error Handling During Pagination

A paginated request can fail on any page. The paginator raises the same exceptions a direct client call would raise, most commonly botocore.exceptions.ClientError. When an exception is raised, the iterator stops; pages already yielded remain processed, but no further pages are fetched.

from botocore.exceptions import ClientError

try:
    for page in paginator.paginate(Bucket="my-bucket"):
        for obj in page.get("Contents", []):
            process(obj)
except ClientError as e:
    if e.response["Error"]["Code"] == "NoSuchBucket":
        print("Bucket does not exist")
    else:
        raise

ClientError carries the full service response in e.response, including Error, ResponseMetadata, and the HTTP status code. Match on e.response["Error"]["Code"] rather than parsing the message text; error codes are stable across services, while messages change.

A common mistake is to wrap only the first page in a try block. Because the paginator performs requests lazily, an error on the third page surfaces inside the loop, not at the paginate() call. The try block must wrap the entire iteration.

Retry Configuration and Throttling

AWS services throttle requests that exceed account-level or per-resource limits. Throttled calls return errors such as ThrottlingException, ProvisionedThroughputExceededException, or RequestLimitExceeded. botocore retries these automatically when you configure retries on the client.

from botocore.config import Config

config = Config(retries={"max_attempts": 5, "mode": "standard"})
s3 = boto3.client("s3", config=config)

max_attempts includes the initial request, so a value of 5 means one original call plus up to four retries. The standard mode applies the SDK's default retry rules, which cover throttling and transient transport errors. The legacy legacy mode retries fewer error types and is the default when you do not specify a config.

If retries are exhausted, botocore raises RetryError, not ClientError. Handle both when your code must distinguish between a throttled request that was retried and a permanent service error:

from botocore.exceptions import ClientError, RetryError

try:
    for page in paginator.paginate(Bucket="my-bucket"):
        process_page(page)
except RetryError:
    print("Request failed after retries")
except ClientError as e:
    print(f"Service error: {e.response['Error']['Code']}")

Retry behavior applies per request. A paginator making 50 requests can retry each one independently, so a slow or throttled service can extend the total runtime significantly. For long-running jobs, consider reducing PageSize to keep individual requests small, and set a reasonable max_attempts rather than relying on the default.

Memory and Runtime Considerations for Large Result Sets

The paginator yields pages as they arrive, but the way you consume them determines memory usage. Accumulating every object into a single list defeats the purpose of pagination:

all_keys = []
for page in paginator.paginate(Bucket="my-bucket"):
    all_keys.extend(obj["Key"] for obj in page.get("Contents", []))

This is fine for a few thousand objects but becomes wasteful for millions. Process each page as it arrives, or write results to a file or database incrementally. The search() method helps when you need only a subset, because it filters before your code sees the full page.

Each paginated request also has a fixed overhead: connection setup, authentication, and response parsing. Reducing PageSize increases the number of requests and the total latency. Increasing it reduces request count but can produce larger responses that take longer to parse. The right value depends on the service and the size of individual items; start with the service default and adjust only when you observe slow responses or throttling.

A Complete Example: Paginated S3 Listing with Error Handling

The following example combines pagination, retry configuration, and error handling into a single function that lists object keys from an S3 bucket and stops cleanly when the bucket does not exist or when the request fails after retries.

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError, RetryError

def list_object_keys(bucket: str, prefix: str = "") -> list[str]:
    config = Config(retries={"max_attempts": 4, "mode": "standard"})
    s3 = boto3.client("s3", config=config)
    paginator = s3.get_paginator("list_objects_v2")

    keys = []
    try:
        for page in paginator.paginate(
            Bucket=bucket,
            Prefix=prefix,
            PaginationConfig={"PageSize": 500},
        ):
            for obj in page.get("Contents", []):
                keys.append(obj["Key"])
    except RetryError:
        raise RuntimeError(f"Listing objects in {bucket} failed after retries")
    except ClientError as e:
        if e.response["Error"]["Code"] == "NoSuchBucket":
            raise ValueError(f"Bucket {bucket} does not exist") from e
        raise

    return keys

The function returns a list, which is acceptable when the caller needs random access. For streaming use cases, replace the list with a generator that yields each key as it is read. The PageSize of 500 keeps each response small enough to avoid timeouts on buckets with large objects, while still limiting the number of requests.

Note that list_objects_v2 returns Contents only when the bucket is not empty. Calling page["Contents"] directly would raise KeyError on an empty bucket, which is why the example uses page.get("Contents", []). This is a common edge case that only appears when a bucket happens to be empty.

The same pattern applies to other paginated AWS operations: describe_instances for EC2, scan for DynamoDB, list_tables, and most other services that expose a paginator. The paginator hides the continuation token, PaginationConfig controls the request shape, and botocore exceptions define how failures surface. Keep the try block around the entire loop, match on error codes rather than messages, and configure retries at the client level so throttled pages are retried before your code ever sees them.