Python PyMongo Find: Filters, Sorting, Pagination
python pymongo find filters sorting and pagination: Learn how to use PyMongo's find() with filters, sorting, and pagination. Covers query operators, skip/limit, cursor...
When you work with MongoDB from Python, the find() method in PyMongo is the primary way to retrieve documents. But find() alone only returns a cursor over all matching documents. To build a useful query, you need to combine filters, sorting, and pagination. This article covers the exact syntax and behavior of python pymongo find filters sorting and pagination, including practical examples and the performance tradeoffs you should know before using them in production.
The find() Method and Its Core Parameters
The find() method accepts several keyword arguments that shape the query. The most commonly used are filter, projection, sort, skip, and limit. Here is the basic signature:
from pymongo import MongoClient client = MongoClient("mongodb://localhost:27017") db = client["shop"] products = db["products"] cursor = products.find( filter={"category": "electronics"}, projection={"name": 1, "price": 1, "_id": 0}, sort=[("price", 1)], skip=10, limit=5 )
The filter parameter is a dictionary that uses MongoDB query operators to match documents. projection controls which fields are returned. sort is a list of (field, direction) tuples where direction is 1 for ascending and -1 for descending. skip and limit together implement offset-based pagination.
When you call find(), it returns a pymongo.cursor.Cursor object. The query is not actually executed until you iterate over the cursor or call a method like list(). This lazy behavior matters when you chain operations or need to reuse a cursor.
Building Filters with Query Operators
MongoDB provides a rich set of comparison and logical operators. In PyMongo, you use the same syntax as the MongoDB shell. For example, to find products with a price greater than 100 and in stock, you write:
cursor = products.find({ "price": {"$gt": 100}, "stock": {"$gt": 0} })
Common operators include $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $exists, and $regex. You can combine them with $and, $or, and $not. Here is a query that matches either a category or a tag:
cursor = products.find({ "$or": [ {"category": "electronics"}, {"tags": "sale"} ] })
Filters are not limited to top-level fields. You can use dot notation to query nested documents and arrays. For instance, {"specs.ram": {"$gte": 8}} matches documents where the specs.ram field is at least 8.
When building filters dynamically, be careful with types. MongoDB compares values by BSON type order. A string "100" does not match a numeric 100. If your data may contain mixed types, consider using $type or normalizing the data at write time.
Sorting Results with sort() and the sort Parameter
Sorting is essential for deterministic pagination. You can pass sort directly to find() or call the sort() method on the cursor. Both accept the same list-of-tuples format.
# Using the sort parameter cursor = products.find({}, sort=[("price", 1), ("name", -1)]) # Using the sort() method cursor = products.find({}).sort([("price", 1), ("name", -1)])
The order matters: the first tuple is the primary sort key, the second is secondary, and so on. For a single field, you can also pass a string like "price" or "-price" to the sort() method, but the tuple form is more explicit and works in both places.
Sorting uses the same collation as the collection unless you specify a different collation. If you need case-insensitive sorting, you must define a collation with a strength of 2 (or 1 for case and accent insensitive). For example:
from pymongo.collation import Collation cursor = products.find( {}, sort=[("name", 1)], collation=Collation(locale="en", strength=2) )
Without a collation, sorting is binary and case-sensitive, which often surprises developers expecting dictionary order.
Pagination Strategies: skip/limit vs. Cursor-Based
The simplest way to paginate is to use skip and limit. For page n with page size s, you set skip = n * s and limit = s. This works well for small offsets, but it becomes inefficient as skip grows because MongoDB must scan and discard all skipped documents.
page_size = 20 page_number = 3 cursor = products.find( filter={"category": "electronics"}, sort=[("price", 1)], skip=page_size * (page_number - 1), limit=page_size )
For large datasets, cursor-based pagination is often a better choice. Instead of using an offset, you filter on the last value of the sort key. For example, if you sort by price ascending, you can fetch the next page with a filter like {"price": {"$gt": last_price}}. This works well when the sort key is unique. If it is not unique, you need a secondary key to break ties, such as _id.
last_price = 129.99 last_id = ObjectId("665f1e9b...") cursor = products.find({ "$or": [ {"price": {"$gt": last_price}}, {"price": last_price, "_id": {"$gt": last_id}} ] }).sort([("price", 1), ("_id", 1)]).limit(page_size)
Cursor-based pagination avoids the performance penalty of large skip values and is stable even when new documents are inserted between page requests. However, it is more complex to implement and requires a unique sort key.
Combining Filters, Sorting, and Pagination in One Query
In practice, you will often need all three at once. The following example shows a complete function that returns a page of products matching a search term, sorted by price, with a cursor-based approach:
from bson.objectid import ObjectId def get_products_after(last_price=None, last_id=None, page_size=20): filter_condition = {} if last_price is not None and last_id is not None: filter_condition = { "$or": [ {"price": {"$gt": last_price}}, {"price": last_price, "_id": {"$gt": last_id}} ] } cursor = products.find( filter_condition, sort=[("price", 1), ("_id", 1)], limit=page_size ) return list(cursor)
Notice that skip is not used here. The query uses the filter to jump directly to the next page. This approach requires an index on (price, _id) to be efficient, which we will discuss next.
If you need to return total count for a UI, you can run a separate count_documents() query with the same filter. That operation scans the index or collection depending on the filter, so it is not free.
Indexing and Performance Considerations
Sorting and pagination performance depends heavily on indexes. MongoDB can only use an index for sorting if the sort fields are part of an index and the sort order matches the index order. For the cursor-based query above, an index on (price, 1) and (_id, 1) is ideal:
products.create_index([("price", 1), ("_id", 1)])
With this index, MongoDB can satisfy both the filter and the sort in a single index scan. Without it, MongoDB may have to sort documents in memory, which is limited to 100 MB by default. If the sort exceeds that limit, the query fails with an error.
For skip/limit pagination, a large skip forces MongoDB to scan and discard documents even if an index exists. The cost grows linearly with the offset. Cursor-based pagination avoids this by using the index to jump directly to the next key. If you must use offset pagination, keep the offset small or restrict it to admin-only views.
Another consideration is the limit value. A very large limit can cause high memory usage because the cursor batches documents. The default batch size is 101 documents or 16 MB, whichever comes first. You can adjust it with the batch_size() method on the cursor, but be aware that larger batches increase client memory usage.
Common Pitfalls and Edge Cases
Several issues commonly appear when working with filters, sorting, and pagination in PyMongo.
Case sensitivity: By default, string comparisons are case-sensitive. If you need case-insensitive matching, use a collation or a $regex with the i option. For sorting, a collation is required.
Missing sort keys: If a document does not have the sort field, MongoDB treats it as null and sorts it before numbers and strings in ascending order. This can cause unexpected ordering. You can filter out missing fields with {"field": {"$exists": True}} if needed.
ObjectId ordering: ObjectId values are sortable by creation time, but they are not guaranteed to be monotonically increasing across processes. For cursor-based pagination, using _id as a tiebreaker is safe only if you also sort by _id in the same direction.
Type mismatches in filters: As mentioned earlier, "100" is not equal to 100. Use the correct BSON type in your filters. If you are unsure, check the data with find_one() and inspect the types.
Cursor exhaustion: A cursor can be iterated only once. If you need to reuse the same query, create a new cursor or store the results in a list. Also, a cursor may time out on the server if it is left open for a long time, especially with the default no_cursor_timeout=False. Set no_cursor_timeout=True only if you know the cursor will be open for a long time, and always close it with cursor.close() when done.
Large skip values: If you must use offset pagination, consider capping the maximum page number. For example, reject requests where page_number * page_size exceeds a threshold like 10,000. This prevents expensive scans and protects your database from abusive queries.
Index usage verification: Use explain() on a cursor to see whether the query uses an index. In PyMongo, you can call cursor.explain() to get the execution plan. This is invaluable when debugging slow queries.
plan = products.find({"price": {"$gt": 100}}).sort("price").explain() print(plan["queryPlanner"]["winningPlan"])
The explain() output shows the index used and whether a sort stage is present. If you see SORT in the plan, your query is not fully index-backed and may need an index adjustment.
By understanding these behaviors, you can write PyMongo queries that are correct, efficient, and maintainable in production.