Python for data analysis: pandas basics

Most pandas tutorials start with a dataset that's already clean, which skips the part that takes the most time in practice. This one starts with a CSV that has the kind of problems real data actually has, and goes past the basics most tutorials stop at — filtering, combining tables, and reshaping — into the operations that come up in an actual first data task.

Loading data

import pandas as pd

df = pd.read_csv('signups.csv')
print(df.head())
print(df.shape)

df.head() shows the first five rows so you can eyeball the structure; df.shape gives you (rows, columns) so you know how much data you're working with before doing anything else.

Finding what's wrong with it

Real datasets have missing values, inconsistent types, and duplicate rows. Check for all three before analyzing anything:

df.isna().sum()
df.dtypes
df.duplicated().sum()

isna().sum() counts missing values per column — if signup_date shows 40 missing out of 2,000 rows, you now know exactly how much of your data would silently vanish from a date-based analysis.

Cleaning it up

df = df.drop_duplicates()
df['signup_date'] = pd.to_datetime(df['signup_date'], errors='coerce')
df = df.dropna(subset=['signup_date'])

errors='coerce' turns unparseable dates into NaT instead of crashing the whole conversion — which is exactly why the dropna step after it exists, to remove the rows that couldn't be parsed at all.

Filtering rows: boolean indexing, .loc, and .iloc

Once the data's clean, most real questions start by narrowing down to a subset of rows:

pro_signups = df[df['plan'] == 'pro']
recent_signups = df[df['signup_date'] > '2026-06-01']
enterprise_in_march = df[(df['plan'] == 'enterprise') & (df['signup_date'].dt.month == 3)]

df['plan'] == 'pro' produces a column of True/False values — one per row — and df[...] keeps only the rows where that's True. Combining conditions uses & (and) or | (or), not Python's and/or, and each condition needs its own parentheses.

.loc and .iloc are the other two ways to select data, and they answer different questions:

df.loc[df['plan'] == 'pro', 'plan'] = 'Pro'   # .loc: select AND assign, by label
first_three_rows = df.iloc[:3]                # .iloc: purely positional, by integer index

.loc selects (and can assign to) rows and columns by label — 'plan' is a column name, so this reads "wherever plan is 'pro', in the plan column, set the value to 'Pro'." .iloc ignores labels entirely and works by position, the same way list slicing does.

pro = df[df['plan'] == 'pro']
pro['flagged'] = True  # SettingWithCopyWarning

Combining tables with merge

Real data rarely lives in one table. A separate lookup table for plan pricing is the common case:

plans = pd.DataFrame({
    'plan': ['free', 'pro', 'enterprise'],
    'monthly_price': [0, 29, 199],
})
df = df.merge(plans, on='plan', how='left')

on='plan' tells pandas which column links the two tables. how='left' keeps every row from df even if a row's plan doesn't have a match in plans — filling in NaN for the price instead of dropping the row. how='inner' (the default) would drop any signup whose plan isn't in the lookup table entirely, which is rarely what you want for a customer dataset where the row itself is the important part.

Reshaping with pivot_table

groupby (from the cleaning step above) answers "one number per group." pivot_table answers the version of that question with two dimensions at once:

signups_by_month = (
    df.groupby(df['signup_date'].dt.to_period('M'))
      .size()
      .rename('signups')
)
print(signups_by_month)

monthly_by_plan = pd.pivot_table(
    df, index=df['signup_date'].dt.to_period('M'), columns='plan',
    values='customer_id', aggfunc='count', fill_value=0,
)
print(monthly_by_plan)

The groupby gives total signups per month. pivot_table gives the same monthly breakdown, but split into one column per plan — fill_value=0 means a month with zero enterprise signups shows 0 instead of a missing value, which matters if the next step charts this.

Counting categories with value_counts

For a quick read on how a categorical column is distributed, value_counts is faster to reach for than a full groupby:

df['plan'].value_counts()
df['plan'].value_counts(normalize=True)  # as proportions instead of raw counts

normalize=True turns the counts into proportions that sum to 1 — useful the moment the actual question is "what percent of signups are on the free plan" rather than the raw count.

Applying custom logic with apply

For logic that doesn't reduce to a comparison or a built-in aggregation, apply runs a function across every row or value:

df['is_high_value'] = df['monthly_price'].apply(lambda price: price >= 100)

This specific case is better written without apply — df['monthly_price'] >= 100 does the same thing directly and runs faster, since it operates on the whole column at once instead of calling a Python function once per row. Reach for apply when the logic genuinely can't be expressed as a vectorized comparison — parsing an inconsistent string format, or calling out to another function per row — not as the default way to add a computed column.

Exporting the cleaned data

df.to_csv('cleaned_signups.csv', index=False)

index=False skips writing pandas' own row-number index as a column in the output file — without it, the CSV gains an unlabeled first column nothing downstream expects. This is the same export step used when saving data pulled from a web scraper — once data is in a DataFrame, the path to a clean CSV is identical regardless of where the data came from.

FAQ

When should I use .loc instead of just df[condition]? df[condition] is shorter for simple row filtering. Reach for .loc when you need to filter rows and select specific columns in one step, or when you're assigning a new value to a filtered subset — df[condition]['col'] = value triggers the same SettingWithCopyWarning covered above, while df.loc[condition, 'col'] = value doesn't.

Why did my merge produce more rows than either original table? A merge key with duplicate values on either side produces one output row per matching pair — if a plan name appears twice in the lookup table, every signup on that plan gets duplicated in the result. Check plans['plan'].duplicated().sum() before merging if the row count comes out higher than expected.

Is pivot_table different from pivot? pivot (no _table) requires the index/column combination to be unique and has no aggfunc — it errors if it finds duplicates. pivot_table aggregates duplicates automatically, which is why it's the more common choice for real data that hasn't been pre-deduplicated to exactly one row per combination.

Do I need NumPy to do any of this? Not for anything above — pandas is built on NumPy internally, but everyday cleaning, filtering, and merging is standard pandas API. NumPy becomes directly useful once you're writing custom numeric operations pandas doesn't already provide a method for.

Where to go from here

Once filtering, merging, and grouping feel natural, df.plot() (backed by matplotlib) turns any of the grouped or pivoted tables above into a quick chart with almost no extra code — useful for a first look before reaching for a dedicated visualization library.