Pandas Interpolation: Limit the Time Between Real Anchors, Not Just Missing Rows

A temperature chart looks clean after one call to interpolate(limit=2). Two missing readings have been filled, so the developer concludes that the program cannot bridge a long outage. That conclusion does not follow.

A row-count limit controls how many missing positions are filled in a direction. It does not mean “reject every gap whose real observations are more than three minutes apart.” It may fill the first part of a much longer gap.

This evergreen tutorial, checked on 30 October 2026, implements that second policy explicitly. The original fixture and code were executed with Python 3.12.14 and pandas 2.2.3. The three-minute threshold is illustrative, not a universal rule for sensor reliability.

Distinguish an absent row from an absent value

Our source contains a row at 00:01, but its reading is missing. It contains no row at 00:02. Those are different observations about the collection process even if both eventually appear as NaN on a regular grid.

The pandas reindex documentation explains that new index labels without an existing value receive missing values. Preserve the original row-membership mask before this operation so it remains possible to distinguish newly introduced grid positions from supplied-but-empty readings.

Neither condition proves why the measurement is missing. A lost packet, a failed sensor, a deliberate suppression and an out-of-service interval can require different treatment. If a source explicitly marks a reading invalid, do not automatically reinterpret it as an ordinary imputable gap.

Declare the grid and bracket policy separately

Assume one device, UTC timestamps, a one-minute expected schedule and unique source timestamps. The target display covers 00:00 through 00:09. Real observations occur at 00:00, 00:03 and 00:08.

Our policy accepts a missing point only when there are valid observed anchors on both sides and the complete elapsed time between those anchors is at most three minutes. The rule rejects the entire interior of a longer bracket rather than filling a few positions near its edge.

The anchor span differs from the age of the preceding observation. At 00:04 the preceding anchor is only one minute old, but the next anchor is four minutes away. This two-sided bracket rule still rejects the point.

Generate candidates and apply the acceptance mask

The pandas interpolate reference distinguishes method="time", which uses elapsed intervals, from ordinary linear interpolation's equal-position treatment. Its limit parameter counts missing entries. We generate candidates and then apply a separate elapsed-time mask.

import pandas as pd

source = pd.Series(
    [10.0, float("nan"), 16.0, 26.0],
    index=pd.to_datetime([
        "2026-10-30 00:00Z", "2026-10-30 00:01Z",
        "2026-10-30 00:03Z", "2026-10-30 00:08Z",
    ]),
    name="reading",
).sort_index()
grid = pd.date_range("2026-10-30 00:00Z", periods=10, freq="min")
assert source.index.is_unique
assert source.index.isin(grid).all()  # Reject off-grid inputs here.

raw = source.reindex(grid)
source_row = pd.Series(grid.isin(source.index), index=grid)
observed = raw.notna()
anchor_time = pd.Series(grid, index=grid).where(observed)
left = anchor_time.ffill()
right = anchor_time.bfill()
span = right - left
eligible = (
    raw.isna() & left.notna() & right.notna()
    & span.le(pd.Timedelta(minutes=3))
)
candidate = raw.interpolate(method="time", limit_area="inside")
result = raw.where(~eligible, candidate)
filled = raw.isna() & result.notna()

assert result.iloc[1] == 12.0
assert result.iloc[2] == 14.0
assert result.iloc[4:8].isna().all()
assert pd.isna(result.iloc[9])
assert result[observed].equals(raw[observed])
assert filled.sum() == 2
assert source_row.iloc[1] and not source_row.iloc[2]

partial = raw.interpolate(method="time", limit=2, limit_area="inside")
assert partial.iloc[4] == 18.0 and pd.isna(result.iloc[4])
print("Two estimates accepted; long and trailing gaps preserved")

Between 10 and 16 over three minutes, the assumed linear change is two units per minute. The program accepts 12 and 14. The final assertion demonstrates why limit=2 is a different contract: it fills 00:04 even though that point belongs to the rejected five-minute bracket.

Keep real anchors separate from earlier estimates

The bracket timestamps are calculated from raw.notna(), before any filling. That matters when the processing code runs more than once.

If an estimated point becomes a new “observed” anchor on the next pass, a sequence of short-looking fills can gradually cross a long outage. Retain raw measurements and fill provenance in separate columns or artifacts. Never overwrite the only copy of the source series.

Useful output fields include the original value, accepted estimate, missingness reason where known, interpolation method, left and right anchor times, bracket duration and policy version. These fields let a reviewer distinguish a smooth display from actual measurement coverage.

Handle irregular and off-grid data deliberately

The example rejects source timestamps outside the declared grid. That prevents a direct reindex from silently discarding an observation at, for example, 00:00:30. This is an explicit input restriction, not a claim that real devices always report on schedule.

For irregular data, either define a justified aggregation step or construct an interpolation index that retains the original anchor timestamps before selecting display positions. Decide how duplicates and conflicting readings are resolved. Group devices before calculating anchors so a reading from one machine cannot bracket a missing value from another.

A finer display grid creates more potential missing positions; it does not create more measured information. The resampling guide addresses bin membership and aggregation, which should be settled independently of interpolation acceptance.

Prevent future information from entering a prediction

Two-sided interpolation uses a later observation. That can be appropriate for a retrospective chart after the complete period is available. It is not automatically appropriate for a feature a model supposedly knew at 00:01.

At that decision time, the 00:03 observation may not exist yet or may not have arrived. Even the bracket's duration is then unknown. Splitting train and test rows after interpolating the full timeline can leak later information into earlier features.

Build preprocessing around the actual information cutoff, including arrival time when relevant. Validate a causal or one-sided alternative separately if the application requires live estimates. The time-series validation guide explains why time ordering alone does not repair a preprocessing step that already crossed the boundary.

Test absence, not just attractive filled values

The most informative checks confirm what remains missing. Test exactly the maximum span, a span just beyond it, no left anchor, no right anchor, an all-missing device and a single-observation device.

Also test that observed values remain unchanged and that rerunning the routine from the same raw data produces the same result. If the sensor can jump abruptly, compare interpolation against withheld valid readings under representative conditions before deciding that a short bracket is scientifically acceptable.

A smooth line is an output of a model assumption. Making its acceptance policy explicit gives the chart a defensible meaning without pretending that the missing measurements were observed.

Sources and scope

• pandas 2.2 Series.interpolate reference, checked 30 October 2026: method, count-limit and interior-fill semantics.
• pandas 2.2 Series.reindex reference, checked 30 October 2026: introducing missing values at new labels.

The complete bracket-duration acceptance rule, fixture and checks are original. They illustrate an auditable retrospective policy; no claim of sensor accuracy, uncertainty quantification or suitability for real-time control is established.