Using Python Boto3 with AWS Secrets Manager

When you need to retrieve a secret from AWS Secrets Manager in Python, the boto3 SDK exposes the get_secret_value method on the Secrets Manager client. The standard flow is to create a client, call get_secret_value with the secret's name or ARN, and read the returned SecretString or SecretBinary field. This article covers the full lifecycle of working with secrets through python boto3 secrets manager: retrieving values, parsing JSON payloads, creating and updating secrets, handling API errors, and caching values to avoid unnecessary calls.

Setting Up the boto3 Secrets Manager Client

The first step is to create a Secrets Manager client. boto3 resolves credentials from the standard chain: environment variables, the shared credentials file, or an IAM role attached to the running instance.

import boto3

client = boto3.client("secretsmanager", region_name="us-east-1")

The region_name matters because secrets are regional. A secret stored in us-east-1 is not accessible from a client pointing at eu-west-1. If your application runs on EC2, ECS, or Lambda, prefer an IAM role over hard-coded credentials so that permissions are managed centrally and rotated automatically.

Retrieving a Secret Value with get_secret_value

The core operation is get_secret_value. Pass the secret's name or ARN as SecretId.

response = client.get_secret_value(SecretId="my-database-password")
secret_string = response["SecretString"]

Secrets Manager stores either a string or binary value. For string secrets, the response contains SecretString. For binary secrets, the response contains SecretBinary, which is a bytes object in Python.

response = client.get_secret_value(SecretId="my-tls-key")
binary_secret = response["SecretBinary"]

By default, get_secret_value returns the version staged as AWSCURRENT. If you need a previous version, pass the VersionStage or VersionId parameter. In most application code, the default is what you want because it always reflects the latest rotated value.

Working with JSON Secrets

Storing multiple fields in a single secret is common. For example, a database credential may contain a username, password, and host. Store the value as a JSON string and parse it after retrieval.

import json

response = client.get_secret_value(SecretId="my-db-credentials")
secret = json.loads(response["SecretString"])

username = secret["username"]
password = secret["password"]
host = secret["host"]

The json.loads call can raise JSONDecodeError if the secret is not valid JSON. If you control the secret's format, validate it at write time. If the secret may be plaintext, check whether the value parses as JSON before assuming a structure.

Creating and Updating Secrets Programmatically

Applications that provision their own infrastructure often create secrets at deployment time. create_secret stores a new secret and fails if the name already exists.

client.create_secret(
    Name="my-api-credentials",
    SecretString=json.dumps({"username": "admin", "password": "initial-password"})
)

To change an existing secret, use update_secret. This creates a new version and stages it as AWSCURRENT.

client.update_secret(
    SecretId="my-api-credentials",
    SecretString=json.dumps({"username": "admin", "password": "rotated-password"})
)

A common mistake is calling create_secret on every application start. If the secret already exists, the call raises ResourceExistsException. Decide whether your deployment should create or update, and branch accordingly.

Handling Common Errors

boto3 raises ClientError for API failures. The error code lives in the response metadata, and you should branch on it rather than catching the exception generically.

from botocore.exceptions import ClientError

try:
    response = client.get_secret_value(SecretId="my-secret")
except ClientError as e:
    code = e.response["Error"]["Code"]
    if code == "ResourceNotFoundException":
        print("Secret does not exist")
    elif code == "AccessDeniedException":
        print("Missing IAM permission for secretsmanager:GetSecretValue")
    elif code == "InvalidRequestException":
        print("Secret is scheduled for deletion or the request is malformed")
    else:
        raise

The AccessDeniedException case deserves attention. It usually means the IAM policy attached to the caller does not grant secretsmanager:GetSecretValue on that resource. Re-raising the exception after logging the code preserves the stack trace for debugging.

Caching Secrets to Reduce API Calls

Every get_secret_value call is an API request with latency and cost. In high-traffic services, fetching the secret on every request adds unnecessary overhead. A simple in-memory cache with a time-to-live avoids repeated calls while still picking up rotated values within the TTL window.

import time

class SecretCache:
    def __init__(self, client, ttl_seconds=300):
        self.client = client
        self.ttl_seconds = ttl_seconds
        self._cache = {}

    def get(self, secret_id):
        now = time.time()
        if secret_id in self._cache:
            value, expires_at = self._cache[secret_id]
            if now < expires_at:
                return value

        response = self.client.get_secret_value(SecretId=secret_id)
        value = response["SecretString"]
        self._cache[secret_id] = (value, now + self.ttl_seconds)
        return value

The cache stores the secret value and an expiry timestamp. On the first call, the value is fetched and cached. Subsequent calls within the TTL return the cached value without hitting the API. After expiry, the next call refreshes the value. A TTL of 300 seconds balances freshness against API cost for most applications. If your secrets rotate frequently, lower the TTL; if they change rarely, raise it.

IAM Permissions and Security Considerations

The boto3 client only works if the caller has the right IAM permissions. The minimum policy for reading a secret grants secretsmanager:GetSecretValue on the specific secret ARN.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret-*"
    }
  ]
}

Writing secrets requires additional actions such as secretsmanager:CreateSecret and secretsmanager:UpdateSecret. Scope the resource to the secrets your application actually manages rather than using a wildcard across the account.

Never log the value of a secret. Log the secret's name, the fact that retrieval succeeded, and the error code on failure, but never the SecretString content. If you use a structured logger, ensure the secret value is not included in the event payload. Also avoid embedding secrets in environment variables that get printed during deployment; the whole point of Secrets Manager is to keep values out of configuration files and logs.