python packages: Practical Usage and Code Examples

A Python package is more than a directory with an __init__.py file. When you need to distribute code to other developers or deploy it across environments, you need a package that can be installed, versioned, and resolved as a dependency. This article covers the practical steps for creating and publishing Python packages, from project structure to PyPI release.

What a Python Package Is

A Python package is a directory that contains an __init__.py file, which signals to the interpreter that the directory should be treated as a package. This allows you to import submodules using dot notation:

# mypackage/__init__.py
from .core import helper

When you run import mypackage, Python executes __init__.py and makes the package object available. This is the fundamental unit of code organization in Python, but it is not enough for distribution. A package that lives only in your source tree cannot be installed into another environment or resolved by pip. To share code, you need to build a distribution that carries metadata about the package name, version, dependencies, and entry points.

Structuring a Package for Distribution

The modern way to define a Python package is with a pyproject.toml file. This file declares the build system and project metadata. A common layout is the src layout, which separates your package code from project-level files:

my-project/
├── pyproject.toml
├── README.md
└── src/
    └── mypackage/
        ├── __init__.py
        └── core.py

The src layout prevents accidental imports of the local package when running tests from the project root, because the package is not directly on sys.path. The pyproject.toml for this layout looks like:

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.1.0"
description = "A small example package"
readme = "README.md"
requires-python = ">=3.9"

[tool.setuptools.packages.find]
where = ["src"]

The [tool.setuptools.packages.find] section tells setuptools where to locate package directories. This is necessary when using the src layout. Without it, setuptools will look for packages in the project root and may fail to find your code.

Installing a Package Locally

During development, you want to install the package in editable mode so changes take effect without reinstalling. Use pip with the -e flag:

pip install -e .

This creates a link to the source directory instead of copying files into site-packages. Editable installs are essential when you are actively modifying the package and want to test changes in another project. They also respect the metadata in pyproject.toml, so dependencies are installed automatically.

To avoid polluting the global environment, create a virtual environment first:

python -m venv .venv
source .venv/bin/activate  # on Windows: .venv\Scripts\activate
pip install -e .

The virtual environment isolates the package and its dependencies from other projects. This is the standard workflow for Python development and becomes critical when you manage multiple packages with conflicting dependency versions.

Declaring Dependencies and Entry Points

The [project] table in pyproject.toml also declares dependencies. Use the dependencies key to list runtime requirements:

[project]
name = "mypackage"
version = "0.1.0"
dependencies = [
    "requests>=2.28",
    "click>=8.0",
]

Optional features can be defined with [project.optional-dependencies]. For example, a testing extra:

[project.optional-dependencies]
test = ["pytest>=7.0", "coverage>=6.0"]

Users can install the extra with pip install mypackage[test]. This is a clean way to keep development tools separate from the runtime dependency set.

Entry points let you expose command-line scripts. If your package provides a CLI, define a [project.scripts] table:

[project.scripts]
mytool = "mypackage.cli:main"

When the package is installed, pip creates an executable wrapper that calls the specified function. This is how tools like black and pytest are installed as commands.

Building and Publishing a Package

To distribute your package to other developers, you need to build distribution files: a source archive (sdist) and a wheel. The build module provides a simple interface:

python -m build

This generates dist/ with files like mypackage-0.1.0.tar.gz and mypackage-0.1.0-py3-none-any.whl. The wheel is a zip archive that can be installed directly without a build step, while the sdist contains the source and is used when building from source.

Before publishing, verify the metadata and content with twine check:

twine check dist/*

Then upload to PyPI:

twine upload dist/*

You will need a PyPI account and an API token. For testing, use TestPyPI first. Publishing is irreversible, so always check the package name, version, and file contents before uploading.

Versioning and Compatibility

Version numbers are not arbitrary. They communicate compatibility to pip and to other developers. Use semantic versioning: MAJOR.MINOR.PATCH. Increment MAJOR for breaking changes, MINOR for backward-compatible features, and PATCH for bug fixes. In pyproject.toml, the version field must match the version you intend to publish. If you use dynamic versioning from a VCS tag, you can set dynamic = ["version"] and configure a tool like setuptools-scm.

Python version compatibility is declared with requires-python. This is a hard constraint: pip will refuse to install the package on an incompatible interpreter. Be conservative and set a lower bound that matches the oldest Python version you test against. For example, requires-python = ">=3.9" means the package works on Python 3.9 and later, but not on 3.8.

Dependency resolution is a common source of operational issues. When two packages require conflicting versions of a third package, pip will raise an error. To avoid this, keep your dependency requirements as broad as possible while still being correct. Use >= with a lower bound and avoid upper bounds unless you know a future version will break. For libraries, upper bounds are often unnecessary and cause conflicts for downstream users.

Common Packaging Pitfalls

Several mistakes can break a package even when the code itself is correct.

Relative imports inside a package must use the full package path. For example, in mypackage/core.py, use from mypackage.utils import helper instead of from utils import helper. The latter only works if mypackage is on sys.path, which is not guaranteed when installed.

Forgetting to include data files such as templates or configuration files. Setuptools needs to be told about non-Python files via [tool.setuptools.package-data] or MANIFEST.in. Without this, the files will be missing from the wheel.

Using a flat layout where the package directory sits directly in the project root. This can cause the package to be accidentally imported when running tests from the root, leading to subtle bugs. The src layout avoids this.

Not testing the built artifact. After building, install the wheel in a fresh virtual environment and run a smoke test. This catches missing files and import errors that only appear when the package is installed, not when it is run from the source tree.

Dependency Resolution and Lock Files

For applications that depend on many packages, pip's default resolver can produce different versions across installations. To make builds reproducible, use a lock file. Tools like pip-tools or poetry generate a locked set of exact versions. With pip-tools, you maintain a requirements.in file and compile it:

pip-compile requirements.in

The output requirements.txt pins every transitive dependency. Install with pip install -r requirements.txt. This ensures that the same versions are installed in every environment. For libraries, you should not commit a lock file; instead, declare a broad range of dependencies and let the consumer resolve them. Lock files are for applications and deployments, not for published packages.