Python Create Process: Using subprocess

To python create process, the standard library provides the subprocess module. It is the recommended way to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This article walks through the core APIs—subprocess.run and subprocess.Popen—and shows how to handle output, errors, timeouts, and resource cleanup in real-world scripts.

The subprocess Module as the Primary API

Python's subprocess module replaces older functions like os.system and os.spawn. It gives you full control over how a child process is created, how its streams are connected, and how it terminates. The two most commonly used entry points are subprocess.run and subprocess.Popen. run is a high-level wrapper that waits for the process to complete and returns a CompletedProcess object. Popen is a lower-level constructor that gives you a handle to the running process, allowing you to interact with it while it executes.

Running a Command and Capturing Output

To run a command and capture its standard output and error, use subprocess.run with capture_output=True. The text=True argument makes the output strings instead of bytes. Here's a minimal example:

import subprocess

result = subprocess.run(["ls", "-l"], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)
print(result.returncode)

After the call, result.stdout and result.stderr contain the captured output, and result.returncode holds the exit status. If the command fails, the returncode is non-zero, but run does not raise an exception unless you pass check=True.

Passing Arguments Safely

One of the most common mistakes is passing a command as a single string and relying on shell=True. This can lead to injection vulnerabilities and makes quoting unpredictable. Instead, pass a list of arguments. subprocess will handle quoting and escaping correctly without invoking a shell. For example:

# Safe: list of arguments
subprocess.run(["grep", "pattern", "file.txt"])

# Unsafe: shell string
subprocess.run("grep pattern file.txt", shell=True)

The list form is always preferred unless you explicitly need shell features like globbing or environment variable expansion. If you must use shell=True, never pass user input directly into the string.

Controlling Process Input and Output Streams

When you need to send data to a process's stdin or read its output incrementally, use Popen. The communicate() method sends input and waits for the process to finish, returning a tuple of (stdout, stderr). Here's an example that feeds a line to a child process:

import subprocess

proc = subprocess.Popen(["cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
stdout, stderr = proc.communicate(input="hello\n")
print(stdout)

Popen is also useful when you need to start a process and continue doing other work while it runs. You can poll proc.poll() to check if it has exited, or read from proc.stdout line by line.

Handling Errors and Exit Codes

To make a failed command raise an exception, pass check=True to run. This raises subprocess.CalledProcessError if the return code is non-zero. The exception object contains the command, return code, and captured output:

import subprocess

try:
    subprocess.run(["false"], check=True)
except subprocess.CalledProcessError as e:
    print(f"Command failed with exit code {e.returncode}")

If you need to handle different failure modes, check the returncode manually and branch accordingly. For many scripts, check=True is the simplest way to fail fast.

Managing Long-Running Processes and Timeouts

Long-running processes can hang your script. Both run and Popen accept a timeout parameter. If the process does not finish within the given seconds, a TimeoutExpired exception is raised. The process is not automatically killed; you must terminate it explicitly:

import subprocess

try:
    subprocess.run(["sleep", "10"], timeout=2)
except subprocess.TimeoutExpired:
    print("Process timed out")

For Popen, you can call proc.kill() or proc.terminate() in the except block to stop the child. Always handle timeouts in production code to avoid orphaned processes.

Performance and Resource Considerations

Creating a process is expensive compared to a function call. Each subprocess spawns a separate OS process, which involves memory allocation, file descriptor setup, and scheduler overhead. If you need to run many short commands, consider batching them into a single shell command or using a persistent worker process. Also, always close or communicate with Popen objects to avoid resource leaks. The subprocess module does not automatically reap child processes; you must call wait() or communicate() to release the process table entry.