Processing Large CSV Files with Pandas Chunksize Correctly

Checked on October 2, 2026. Examples use pandas 2.2.3 and six synthetic records. This article provides no speed benchmark or guarantee that a particular file size fits in RAM.

A CSV that was easy to open early in a project can become difficult as records accumulate. Loading everything creates a large DataFrame, while conversions and joins introduce more objects. chunksize helps read a subset at a time, but processing chunks does not automatically produce correct aggregates or bounded memory use.

Short answer: select necessary columns, define types, process chunks without retaining all of them, and combine statistics that can actually be merged. For means, keep a sum and valid-value count. Do not average chunk means without their weights.

What changes when using chunksize?

The pandas.read_csv documentation explains iterator-based chunk reading. low_memory=True affects internal parsing but still produces one DataFrame unless an iterator or chunks are requested. These options are not interchangeable.

A chunk contains records returned by the CSV parser, not an arbitrary byte slice. Quoted fields can contain newlines. Splitting raw text indiscriminately can corrupt record structure; use an appropriate parser.

Define the output before choosing chunk size. If the final output is every input record, concatenating all chunks still requires memory for a large result. If the output is a small summary, discard each chunk after calculating its contribution. The algorithm's goal matters more than the option name.

Why can averaging chunk means be wrong?

Suppose three chunks contain valid values [10, 20], [90], and [30]. Their means are 15, 90, and 30. Averaging those gives 45. The four observations' actual mean is (10 + 20 + 90 + 30) / 4 = 37.5.

The correct mean gives each observation equal weight. An unweighted mean of chunk means gives each chunk equal weight despite unequal counts. Combine sums and divide by combined counts. The numpy.average reference explains weighted averages.

Establish the data contract first

This exercise reads id, category, and value. IDs remain text to preserve leading zeros. Values are nonnegative measurements with a decimal point. Empty values are missing; failed conversions, infinities, and negative values are invalid.

That policy belongs to the exercise. A real project may allow negatives or use decimal commas. Do not reject negative refunds or temperatures without understanding the domain. Likewise, do not automatically remove thousands separators before establishing the source convention.

id,category,value
001,A,10
002,A,20
003,B,90
004,B,
005,A,broken
006,B,30

All fixture categories are present. The code rejects empty categories so groupby does not silently lose them. IDs are strings, but global uniqueness is not checked. That limitation matters because cross-chunk deduplication requires additional state.

Keep summaries, not every chunk

This standalone example uses StringIO and needs no external file. For real input, replace it with your path after confirming format and encoding. keep_default_na=False helps distinguish empty fields from other tokens under this exercise's policy.

from io import StringIO
from collections import defaultdict
import math
import pandas as pd

CSV = """id,category,value
001,A,10
002,A,20
003,B,90
004,B,
005,A,broken
006,B,30
"""

def summarize(chunksize):
    audit = {"read": 0, "missing": 0, "invalid": 0, "valid": 0}
    by_category = defaultdict(lambda: {"sum": 0.0, "count": 0})
    total = 0.0
    count = 0
    reader = pd.read_csv(
        StringIO(CSV), chunksize=chunksize,
        usecols=["id", "category", "value"],
        dtype="string", keep_default_na=False,
    )
    for chunk in reader:
        category = chunk["category"].str.strip()
        if category.eq("").any():
            raise ValueError("Empty category; define a policy")
        raw = chunk["value"].str.strip()
        missing = raw.eq("")
        numbers = pd.to_numeric(raw, errors="coerce")
        finite = numbers.map(lambda x: pd.notna(x) and math.isfinite(x))
        valid = finite & numbers.ge(0).fillna(False) & ~missing
        invalid = ~missing & ~valid

        audit["read"] += len(chunk)
        audit["missing"] += int(missing.sum())
        audit["invalid"] += int(invalid.sum())
        audit["valid"] += int(valid.sum())

        clean = pd.DataFrame({
            "category": category[valid], "value": numbers[valid]
        })
        total += float(clean["value"].sum())
        count += len(clean)
        for name, group in clean.groupby("category"):
            by_category[name]["sum"] += float(group["value"].sum())
            by_category[name]["count"] += len(group)

    assert audit["read"] == sum(audit[k] for k in ["missing", "invalid", "valid"])
    mean = total / count if count else None
    return audit, total, count, mean, dict(by_category)

result = summarize(2)
assert result[:4] == (
    {"read": 6, "missing": 1, "invalid": 1, "valid": 4},
    150.0, 4, 37.5,
)
assert result[4] == {"A": {"sum": 30.0, "count": 2},
                     "B": {"sum": 120.0, "count": 2}}
for size in [1, 3, 4, 10]:
    assert summarize(size) == result
print(result)

There is no list retaining every chunk and no full-input pd.concat. Retained state comprises audit counters, total, count, and category summaries. The pandas scaling guide emphasizes that chunking needs a suitable algorithm.

Preserve the audit through conversion

to_numeric(errors="coerce") produces missing values for failed conversions. That enables processing but does not distinguish originally empty data from malformed text. The code captures the raw missing mask first to report them separately.

Do not apply fillna(0) before calculating a mean without justification. A measured zero differs from an unknown value. Four valid observations here yield 37.5. Replacing every problem with zero and dividing 150 by all six records yields 25, answering another question.

The pandas missing-data guide describes operation-specific missing-value behavior. State the denominator in reports. With no valid observations, the example returns None for the mean rather than zero. Interpret a zero sum with its count, not as proof of no activity.

read = missing + invalid + valid holds because these categories are mutually exclusive under this contract. If several rejection reasons overlap, decide whether counters describe records or error occurrences. Do not sum overlapping categories as though each record appears once.

Aggregation state can still grow

Small chunks do not guarantee small memory use. If category is effectively a unique transaction ID, its dictionary can approach input size. Inspect cardinality first. Chunking reduces simultaneous input retention but does not eliminate output or state costs.

Inspect DataFrames with memory_usage(deep=True) and monitor process memory. They differ: the process also holds the interpreter, parser, temporary arrays, and other state. There is no fixed disk-CSV-to-RAM ratio.

Do not create independent categorical encodings per chunk and assume their dictionaries match. Define a shared domain or reconcile encodings when consistency matters. Choose types suited to values, not the smallest possible types simply to improve a memory number.

Which operations combine easily?

This is algorithm guidance, not a performance promise. The groupby guide explains split-apply-combine. For streamed input, combine summaries across chunks correctly as well. See mean, median, and outliers for distribution questions.

For large queries, global joins, or sorting, consider DuckDB and SQLite by workload. For repeated column scans, review CSV vs Parquet. Changing engines still requires result checks and does not repair an unclear data contract.

Verify before processing production files

Test several chunk sizes, including those that do not divide the record count evenly. Add all-missing input, infinity, malformed text, empty categories, and negatives according to policy. Compare against a simple reference on small data, using appropriate floating-point tolerances for real values.

Floating-point sums can differ slightly when aggregation order changes. This fixture returns identical results because its small values are represented exactly. For money or strict precision, choose a suitable representation and range; arbitrary decimal values should not rely on exact equality.

Retain raw files, dependency versions, parser options, record counts, and rejection summaries. If parsing skips malformed rows, subsequent counters no longer cover all source records. Do not enable on_bad_lines="skip" merely to finish without documenting its consequence.

Sources and test limits

Linked pandas and NumPy documentation supports the APIs and aggregation concepts. Fixture outcomes and chunk-size variation were tested locally. No production dataset, peak-RAM measurement, or engine benchmark was run. Explore small files with CSV to JSON and Data Explorer, then execute a chunked pipeline in an environment appropriate for your data.