Inspect and transform tabular data with pandas

Most practical ML projects begin with a table whose meaning and quality are not yet understood. A disciplined data inspection pass catches problems before model code hides them.

implement
measure

You can now

Load a CSV into a DataFrame and inspect rows, columns, dtypes and memory-relevant cardinality.
Detect missing values, duplicate rows and suspicious category values.
Select, filter, group and aggregate records without mutating the source accidentally.
Create a derived column and join two tables while checking row counts.

Covers

Tabular data inspection and transformation

Study, in this order

2 steps

Pandas
Kaggle Learn — Pandas

Complete lessons 1–6: reading/writing, selection, summary functions, grouping, sorting, dtypes, missing values and combining tables.

Done
Getting started tutorials
pandas Getting Started Tutorials

Use the tutorials on reading/writing, subset selection, derived columns, summary statistics, combining tables and time-series data.

Done

Focus on

DataFrame rows, columns and dtypes
Filtering, sorting, groupby and aggregation
Missing values and duplicates
Joins, row-count checks and derived columns

Reproduce closed-book

Load the hourly bike-sharing CSV, report row count and dtypes, inspect missing values and duplicate rows, group demand by hour and weekday, and verify that grouping preserves the intended totals.

Traps

Do not assume a column's dtype reflects its conceptual meaning; integer codes can represent categories.
Do not drop missing values without stating why and measuring how many rows are removed.
Do not join tables without checking key uniqueness and resulting row counts.

Practice

Kaggle Pandas exercises

Redo one grouping exercise and one missing-value exercise from a fresh notebook.

Write down what each transformation is expected to do to the row count.

Self-test

Why should row counts be checked after a join?
What is the difference between a missing value and the numeric value zero?

Completion criterion

Stop when

You can load and inspect a table, identify missingness and duplicates, group and join data, and explain the row-count effect of each transformation.

Mark passed, then open 1.4 Only mark this when you can do it without notes.

Leads to

1.4 Visualize a dataset and write a data audit Follow the practical starter sequence; each unit uses the artifact from the previous step.

Comes after

1.2 NumPy arrays: shape, dtype and vectorized operationsFollow the practical starter sequence; each unit uses the artifact from the previous step.

Resources for this unit

Kaggle Learn — Pandas
Kaggle Learnprimarycourse
Complete the short exercises on reading data, selecting, summary functions, grouping, sorting, data types, missing values and combining tables.
pandas Getting Started Tutorials
pandas Development Teamrequireddocumentation
Use the official tutorials as a reference for reading and writing tables, selecting rows and columns, creating columns, summary statistics, joins and time-series handling.

Also in this section

Kaggle Learn — Python
Kaggle Learnprimarycourse
Complete the course lessons on variables, functions, conditionals, lists, loops, strings, dictionaries and external libraries. Treat it as the practical Python vocabulary needed for the next units.
The Python Tutorial
Python Software Foundationreferencereference
Use as a reference for control flow, data structures, modules, files and exceptions. It assumes some prior programming experience; use Kaggle Learn for the first pass if the syntax is new.
Google Colab FAQ — free resources and limits
Google Researchsupportnotebook-environment
Use a hosted notebook for the first exercises if local setup is a distraction. The free tier has variable, non-guaranteed compute and GPU access; keep every exercise small enough to run on CPU.
NumPy Quickstart Tutorial
NumPyprimarydocumentation
Study array creation, shape, indexing, universal functions, broadcasting and basic operations. Write down shapes and dtypes before changing an array.
Matplotlib Quick Start Guide
Matplotlibsupportdocumentation
Use the Figure/Axes workflow to make labeled line plots, histograms and scatter plots. Keep plots diagnostic rather than decorative.
Bike Sharing Dataset
UCI Machine Learning Repositoryrequireddataset
Use the hourly hour.csv data for the two linked starter projects. Cite Fanaee-T (2013), DOI 10.24432/C5W894, and the CC BY 4.0 licence. cnt is the target; casual and registered are components of that target and must never be used as input features.