Python PyMongo Indexes: Unique Indexes and Performance

python pymongo indexes unique indexes and performance requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When a MongoDB collection grows, queries without indexes force a collection scan. In PyMongo, creating indexes is straightforward, but unique indexes carry specific performance and behavioral tradeoffs. This article explains how to create unique indexes with PyMongo, what they guarantee, and how they affect read and write performance.

Creating Indexes with PyMongo

The create_index method on a pymongo.collection.Collection instance is the primary way to define an index. A simple single-field index is created by passing the field name and a direction constant:

from pymongo import MongoClient, ASCENDING

client = MongoClient("mongodb://localhost:27017")
db = client["example_db"]
collection = db["users"]

collection.create_index([("email", ASCENDING)])

This creates a non-unique index on the email field. The index accelerates queries that filter on email and sorts by it. Without it, MongoDB performs a collection scan, which becomes slower as the collection grows.

For compound indexes, pass multiple field-direction pairs:

collection.create_index([("last_name", ASCENDING), ("first_name", ASCENDING)])

The order of fields matters: the index supports queries that filter on last_name alone, but not on first_name alone unless last_name is also present.

Defining Unique Indexes

To enforce uniqueness on a field, set the unique parameter to True when calling create_index:

collection.create_index([("email", ASCENDING)], unique=True)

Once this index exists, MongoDB rejects any insert or update that would cause two documents to have the same value for email. The rejection surfaces as a pymongo.errors.DuplicateKeyError:

from pymongo.errors import DuplicateKeyError

try:
    collection.insert_one({"email": "alice@example.com", "name": "Alice"})
    collection.insert_one({"email": "alice@example.com", "name": "Alice Again"})
except DuplicateKeyError:
    print("Duplicate email rejected")

Unique indexes also apply to updates. If an update would create a duplicate, MongoDB aborts the entire update operation and raises the same error. This behavior is atomic at the document level, so you do not need to check for duplicates manually before writing.

Unique Indexes and Write Performance

Every index adds overhead to write operations because MongoDB must update the index structure for each inserted or modified document. A unique index adds an extra step: MongoDB must check the index for an existing key value before allowing the write. This check is a point lookup, which is fast, but it still consumes CPU and I/O.

For workloads with high insert or update rates, the uniqueness check can become a bottleneck, especially when the index does not fit entirely in RAM. The database may need to read index pages from disk to perform the lookup, adding latency. In contrast, a non-unique index simply appends the new key to the index structure without a preceding lookup.

The performance difference is usually negligible for moderate write volumes, but it becomes measurable when a collection has millions of documents and the index is large. If uniqueness is a hard requirement, the cost is unavoidable. If it is only a soft preference, consider using a non-unique index and handling duplicates in application logic instead.

Unique Indexes and Read Performance

For read operations, a unique index is often more efficient than a non-unique index when querying for a single value. Because the index guarantees that a key appears at most once, MongoDB can stop scanning the index as soon as it finds the first match. In a non-unique index, the query engine may need to examine multiple entries to confirm that no further matches exist, depending on the query shape.

For equality queries on a unique field, MongoDB can use the index to fetch the document directly. For range queries, a unique index does not provide a special advantage over a non-unique index; both allow an ordered scan. The primary read benefit of a unique index is the implicit guarantee of a single match, which simplifies query planning and can reduce the number of index entries examined.

Partial and Sparse Unique Indexes

Sometimes you need uniqueness only for a subset of documents. MongoDB supports partial indexes, which include only documents that match a filter expression. In PyMongo, use partialFilterExpression:

collection.create_index(
    [("email", ASCENDING)],
    unique=True,
    partialFilterExpression={"email": {"$exists": True}}
)

This index enforces uniqueness only on documents that have an email field. Documents without email are not indexed, so multiple documents can omit the field. This is useful when the field is optional but unique when present.

A sparse index is a simpler form that indexes only documents that contain the indexed field. In PyMongo, set sparse=True:

collection.create_index([("email", ASCENDING)], unique=True, sparse=True)

Sparse indexes and partial indexes differ in the types of expressions they support. Partial indexes can use arbitrary query filters, while sparse indexes only check for the existence of the field. For most cases, partial indexes are more flexible. Note that a unique index combined with sparse=True allows multiple documents to lack the field, which is often the desired behavior.

Managing Indexes and Monitoring Performance

To list all indexes on a collection, use list_indexes():

for index in collection.list_indexes():
    print(index)

Each index document includes the key pattern, options, and name. The default _id_ index is always present and cannot be dropped.

To remove an index, use drop_index with the index name or the key pattern:

collection.drop_index("email_1")

Index names are generated automatically from the key pattern unless you specify name in create_index. For complex indexes, giving an explicit name improves maintainability:

collection.create_index(
    [("email", ASCENDING)],
    unique=True,
    name="unique_email"
)

Monitoring index usage is important for performance. MongoDB provides the $indexStats aggregation stage, which returns the number of times each index was used and the time spent scanning it. You can run it from PyMongo:

collection.aggregate([{"$indexStats": {}}])

This output helps identify unused indexes, which consume write overhead and storage without benefiting reads. Dropping unused indexes can improve write performance and reduce memory pressure.

Common Pitfalls and Operational Considerations

Creating a unique index on a collection that already contains duplicate values fails with an error. The index creation is aborted, and no index is created. You must first remove or merge duplicate documents before applying the unique constraint.

Index creation in MongoDB can be either foreground or background. In PyMongo, the background parameter is deprecated in favor of the commitQuorum option in MongoDB 4.4 and later. For large collections, index creation can take time and affect database performance. Running index creation during low-traffic periods is advisable.

Another subtle issue is the interaction between unique indexes and sharded clusters. In a sharded collection, a unique index must include the shard key as a prefix unless the index is on the shard key itself. Attempting to create a unique index that violates this rule results in an error. If you plan to shard a collection, design the unique index with the shard key in mind.

Finally, remember that a unique index enforces uniqueness at the database level, but it does not prevent race conditions in application logic that checks for existence before inserting. The only reliable way to guarantee uniqueness is to rely on the index itself and handle DuplicateKeyError in the application. Checking first and then inserting still allows two concurrent requests to pass the check before either insert completes.

Understanding how unique indexes affect both read and write performance in PyMongo is essential for designing efficient MongoDB schemas. By choosing the right index type, using partial or sparse options when appropriate, and monitoring index usage, you can maintain both data integrity and application responsiveness.