Python Boto3 S3 Upload Download List and Delete Files

Working with files in AWS S3 from Python typically means using boto3. The core operations—upload, download, list, and delete—are straightforward once you understand the available methods and their parameters. This article covers python boto3 s3 upload download list and delete files with practical code examples, error handling, and production considerations.

Prerequisites and Setup

Before any S3 operation, you need boto3 installed and AWS credentials configured. Install the library with pip:

pip install boto3

Credentials can be provided via environment variables, an AWS credentials file, or an IAM role. For local development, the shared credentials file is common:

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
region = us-east-1

Boto3 offers two main interfaces: the low-level client and the high-level resource. The client mirrors the AWS API exactly, while the resource provides a more Pythonic object model. Most file operations work well with either, but the client is often preferred for explicit control and consistency across versions.

import boto3

s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3')

Both are valid. The examples below use the client unless noted, because it exposes the full set of parameters and is less surprising when you need fine-grained control.

Uploading Files to S3

Uploading a file from your local filesystem to an S3 bucket is done with upload_file. This method handles the underlying HTTP request and, for larger files, automatically uses multipart upload.

s3_client.upload_file(
    Filename='local/path/to/file.txt',
    Bucket='my-bucket',
    Key='remote/path/file.txt'
)

The Filename is the local path, Bucket is the bucket name, and Key is the destination object key. The key can include slashes to simulate folders, but S3 is flat—folders are just key prefixes.

If you already have a file-like object in memory, use upload_fileobj. This is useful when the data comes from a network stream or a BytesIO buffer.

import io

buffer = io.BytesIO(b'data to upload')
s3_client.upload_fileobj(buffer, 'my-bucket', 'data.txt')

For small payloads that are already in memory, put_object is simpler, but it does not support multipart upload and has a 5 GB limit.

s3_client.put_object(Bucket='my-bucket', Key='hello.txt', Body='Hello, S3!')

When uploading, you can set metadata, access control, and storage class via the ExtraArgs parameter. For example, to make an object publicly readable:

s3_client.upload_file(
    Filename='file.txt',
    Bucket='my-bucket',
    Key='file.txt',
    ExtraArgs={'ACL': 'public-read'}
)

Downloading Files from S3

The counterpart to upload_file is download_file, which saves an object directly to a local file.

s3_client.download_file(
    Bucket='my-bucket',
    Key='remote/path/file.txt',
    Filename='local/path/file.txt'
)

If you need the object content in memory, use get_object and read the Body stream.

response = s3_client.get_object(Bucket='my-bucket', Key='file.txt')
content = response['Body'].read().decode('utf-8')

For streaming downloads, download_fileobj writes to a file-like object.

with open('output.bin', 'wb') as f:
    s3_client.download_fileobj(Bucket='my-bucket', Key='data.bin', Fileobj=f)

When downloading, you can use VersionId to retrieve a specific version if versioning is enabled on the bucket.

Listing Objects in an S3 Bucket

Listing objects requires pagination because S3 returns at most 1,000 keys per request. The list_objects_v2 method is the current API and supports a ContinuationToken for pagination.

def list_all_objects(bucket_name, prefix=''):
    paginator = s3_client.get_paginator('list_objects_v2')
    for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix):
        for obj in page.get('Contents', []):
            yield obj['Key']

The Prefix parameter filters results to keys that start with a given string. This is how you simulate listing a "folder"—you use the folder path as the prefix.

If you need only object names, you can iterate directly over the Contents list. Each object also includes metadata like Size, LastModified, and ETag.

Deleting Files from S3

To delete a single object, use delete_object.

s3_client.delete_object(Bucket='my-bucket', Key='file.txt')

For multiple objects, delete_objects accepts a list of keys and performs the deletion in a single request. This is more efficient than looping over delete_object calls.

objects_to_delete = [{'Key': 'file1.txt'}, {'Key': 'file2.txt'}]
response = s3_client.delete_objects(
    Bucket='my-bucket',
    Delete={'Objects': objects_to_delete}
)

The response includes any errors that occurred, so you should check Errors in the response for partial failures.

Deleting an object that does not exist does not raise an error; S3 returns a 204 No Content. This is important for idempotent cleanup scripts.

Error Handling and Retries

S3 operations can fail for many reasons: network issues, invalid credentials, missing buckets, or permission errors. Boto3 raises ClientError for most service-side failures. The exception contains an error_code and error_message that you can inspect.

from botocore.exceptions import ClientError

try:
    s3_client.get_object(Bucket='my-bucket', Key='nonexistent.txt')
except ClientError as e:
    error_code = e.response['Error']['Code']
    if error_code == 'NoSuchKey':
        print('Object not found')
    elif error_code == 'AccessDenied':
        print('Permission denied')
    else:
        print(f'Unexpected error: {error_code}')

Boto3 automatically retries transient errors like throttling or network timeouts based on the configuration in your AWS SDK. You can adjust retry settings via the Config parameter when creating the client.

from botocore.config import Config

config = Config(retries={'max_attempts': 5, 'mode': 'standard'})
s3_client = boto3.client('s3', config=config)

Performance and Security Considerations

For large files, the default upload_file uses multipart upload, which parallelizes the transfer. The threshold is 8 MB by default, but you can tune it with MultipartThreshold and MaxConcurrency in TransferConfig.

from boto3.s3.transfer import TransferConfig

config = TransferConfig(multipart_threshold=16 * 1024 * 1024, max_concurrency=10)
s3_client.upload_file(
    Filename='large.bin',
    Bucket='my-bucket',
    Key='large.bin',
    Config=config
)

On the security side, always use IAM roles or least-privilege policies. For example, a policy that allows only s3:GetObject on a specific bucket prevents accidental deletion. For sensitive data, enable server-side encryption on the bucket or set the ServerSideEncryption parameter when uploading.

s3_client.put_object(
    Bucket='my-bucket',
    Key='secret.txt',
    Body='data',
    ServerSideEncryption='AES256'
)

Choosing Between Client and Resource for File Operations

The resource interface can be more readable for simple tasks. For example, uploading with the resource:

bucket = s3_resource.Bucket('my-bucket')
bucket.upload_file('local.txt', 'remote.txt')

But the resource does not expose every parameter. For instance, delete_objects is only available on the client. If you need to combine many operations in one script, using the client consistently avoids mixing two abstractions and reduces cognitive load.

For most production scripts, the client is the safer choice because it maps directly to the AWS API and gives you access to pagination, batch operations, and error details. The resource is fine for prototypes or when you prefer object-oriented syntax, but be prepared to drop down to the client when you hit its limitations.

A common pattern is to use the client for all data-plane operations and the resource only for bucket-level tasks like listing buckets or creating bucket objects. In a single script, stick to one interface to keep the code predictable.