Python Pandas DataFrame Creation Series and Basic Operations

When working with pandas in Python, the DataFrame is the central two-dimensional data structure. Knowing how to construct a DataFrame from Series and apply basic operations is a foundational skill for data analysis. This article covers python pandas dataframe creation series and basic operations, including practical syntax, behavior, and common edge cases.

Creating a DataFrame from a Single Series

A pandas Series is a one-dimensional labeled array. To create a DataFrame from a single Series, pass the Series to the DataFrame constructor. By default, the Series becomes a single column with the Series name as the column label.

import pandas as pd

s = pd.Series([10, 20, 30], name="values")
df = pd.DataFrame(s)
print(df)

The output shows a single column named values with the index preserved from the Series. If the Series has no name, the column will be labeled 0. You can also pass a dictionary with a column name to the constructor for clarity:

df = pd.DataFrame({"values": s})

Both approaches produce the same structure, but the dictionary form is more explicit when you want to rename the column during creation.

Creating a DataFrame from Multiple Series

When you have multiple Series that share the same index, you can combine them into a DataFrame by passing a dictionary of Series. Each key becomes a column name, and the index is aligned across all Series.

s1 = pd.Series([1, 2, 3], name="a")
s2 = pd.Series([4, 5, 6], name="b")
df = pd.DataFrame({"a": s1, "b": s2})

If the Series have different lengths or indices, pandas aligns them by index and introduces NaN for missing positions. This behavior is important when working with data from different sources. For example, if s2 has index [0, 1, 2, 3] and s1 has index [0, 1, 2], the resulting DataFrame will have four rows, with NaN in column a for the row with index 3.

Accessing Columns and Rows

Once the DataFrame is created, you can access columns using bracket notation or attribute access. Rows are accessed using .loc for label-based selection and .iloc for integer position.

df["a"]          # returns a Series
df.loc[0]        # first row by label
df.iloc[0]       # first row by position

The difference between .loc and .iloc matters when the index is not a simple integer range. Using .loc with a label that does not exist raises a KeyError, while .iloc with an out-of-range position raises an IndexError. The table below summarizes their behavior.

For slicing, .loc is inclusive of the end label, while .iloc is exclusive of the end position. This is a common source of off-by-one errors.

Adding and Removing Columns

You can add a new column by assigning a value or a Series to a new key. To remove a column, use drop with axis=1 or the del statement.

df["c"] = [7, 8, 9]
df = df.drop("b", axis=1)

Assigning a scalar value broadcasts it across all rows. Assigning a Series aligns it by index, which can introduce NaN if the indices do not match. The drop method returns a new DataFrame by default; set inplace=True only if you understand the tradeoffs, as it can make code harder to follow.

Filtering Rows Based on Conditions

Filtering is a basic operation that selects rows meeting a condition. The condition returns a boolean Series, which you can pass to the DataFrame to filter.

filtered = df[df["a"] > 1]

You can combine multiple conditions with & and |, but you must wrap each condition in parentheses because of operator precedence. For example:

filtered = df[(df["a"] > 1) & (df["c"] < 9)]

Forgetting the parentheses leads to a ValueError because pandas interprets the expression incorrectly. This is one of the most frequent mistakes when filtering.

Handling Missing Data

Missing data appears as NaN in the DataFrame. You can detect missing values with isna() and drop or fill them using dropna() and fillna(). These methods return new DataFrames unless you set inplace=True.

df.isna()
df.dropna()
df.fillna(0)

The choice between dropping and filling depends on the analysis. Dropping rows with missing values reduces the dataset size, while filling preserves the shape and may require a strategy such as using the column mean. For time series, forward filling with method='ffill' is often appropriate.

Performance Considerations with Large DataFrames

When working with large DataFrames, certain operations are more efficient than others. Vectorized operations on columns are faster than iterating over rows. Using .loc and .iloc for selection is generally efficient, but repeated appending of rows with concat or append can be slow because it creates a new object each time. Building a list of rows and constructing a DataFrame once is often faster.

Also, be aware that inplace=True does not always improve performance and can be confusing. Many pandas methods return a new object, and reassigning is clearer and safer. For example, df = df.dropna() is preferred over df.dropna(inplace=True) because it avoids ambiguity about whether the original object is modified.

Common Pitfalls with Index Alignment

One of the most subtle issues in pandas is index alignment. When you assign a Series to a DataFrame column, pandas aligns by index, not by position. If the indices differ, you get NaN in unexpected places. Similarly, when combining DataFrames with concat, the default behavior is to align on the index, which can produce a larger DataFrame than intended. Understanding alignment is critical for correct data manipulation.

To avoid surprises, explicitly reset or set the index before operations when you need positional behavior. For example, df.reset_index(drop=True) replaces the current index with a default integer index, which can make alignment more predictable in certain workflows.