Streaming Watermarks: Why a Fast Dashboard Still Needs a Revision Policy

At 10:05, a sales dashboard reports $120 for the previous five minutes. At 10:07, a checkout terminal reconnects and sends a sale that happened at 10:03. Should the earlier total change, should today's total change, or should the event be discarded?

There is no universal answer independent of the business contract. A live operational view can reasonably revise an earlier period. A closed financial statement requires a controlled adjustment process. Streaming software needs an explicit policy connecting these two worlds.

The Apache Beam programming guide distinguishes event-time windows, watermarks, triggers and late-data handling. This article uses those concepts to build a small accounting example. It is not a Beam implementation or a claim that every runner has identical guarantees.

Separate the clocks

Event time is when the underlying activity occurred according to the timestamp assigned to the record. Processing time is when a pipeline processes it. Arrival time at your ingestion boundary is another useful operational timestamp, but it may differ from processing time downstream.

A store terminal can be offline; a message broker can retry delivery; a mobile device can have an incorrect clock. Preserve the source timestamp and ingestion timestamp separately. If you correct an invalid source clock, record that correction rather than silently rewriting history.

For exact boundaries, this guide uses intervals that include their start and exclude their end. A sale at precisely 10:05 belongs to the next window. Document the timezone and daylight-saving policy as carefully as the interval length.

Understand what a watermark tells you

Beam's model documentation describes a watermark as an estimate of completeness based on event timestamps. After it passes a window's end, additional records assigned to that window are late. It is not a wall-clock promise that no more relevant activity exists anywhere.

Sources and runners determine how this estimate advances. A partition with slow or missing records can complicate progress. A dashboard refresh every second says nothing by itself about the completeness of its input.

Keep these claims separate in user-facing labels: “updated just now,” “events through this event-time boundary,” and “subject to revision.” A recent refresh may contain older business events, while a stable displayed total may conceal missing source data.

Trace a late sale and a duplicate

Consider this invented stream for one store. Each sale has a unique immutable event identifier. The window is 10:00–10:05, and refunds are outside this simplified example.

The initial total is $120. After accepting C, the revised total is $150. Deduplication uses identity, not the amount and timestamp alone: two legitimate customers might spend the same amount at the same instant.

If an event is corrected, choose whether the source sends a replacement version or an explicit reversal and new event. A pipeline cannot reliably infer that contract from similar-looking rows. Preserve lineage so an analyst can reconstruct why a published value changed.

Define how outputs update the sink

An accumulating output reports the updated full total for the window. A delta output reports only the new change. The consumer must know which it receives.

If the publisher emits $120 and then $150 as replacement totals, adding both produces an incorrect $270. A suitable sink key might combine store, window start, window end and metric definition; a revision sequence then identifies the newer value. A delta stream instead requires reliable handling of each unique increment.

This is why an “exactly once” statement about one processing component does not automatically prove that the dashboard cannot double count. Inspect the entire path, including retries into the database, conflict resolution and consumer behavior after reconnecting.

Choose a lateness and closure policy

Beam supports allowed lateness and triggers for controlling late outputs. Window expiry relates to watermark progress plus the configured allowance. Do not translate that automatically into “wait exactly two days of wall-clock time.”

For your application, choose a provisional publication point, a normal correction window and a post-closure exception path. For example, an operational sales view could revise routinely during the trading day, while older corrections enter a separate reconciliation queue. That is an illustrative business policy, not a recommended accounting rule for every organization.

Longer retention can improve completeness but consumes resources and complicates state management. Shorter retention reduces that burden but makes the exception path more important. Estimate actual arrival-delay distributions before choosing a convenient round number.

Make revisions visible to readers

A useful dashboard can show a freshness timestamp, a provisional flag, the latest revision and a short explanation of included sources. If a store has not reported, display that fact; do not turn its missing input into zero sales without an explicit meaning.

Compare a period only after considering its maturity. A fully reconciled yesterday versus the first provisional five minutes of today is not a clean growth comparison. For recurring reports, save the vintage so colleagues can distinguish “what we knew then” from “what the database says now.”

Test the failures that change totals

Use fixtures containing out-of-order events, repeated IDs, late arrivals after closure, boundary timestamps and a correction to an existing event. Reprocess the same fixture and check whether the final sink state is stable. Test restart and partial-delivery behavior separately from pure aggregation arithmetic.

As a self-check, a dashboard receives cumulative outputs 100, 130 and 125 for the same window, with increasing revision numbers. The correct current value is 125 under a replacement contract. The decline may reflect a correction; it is not permission to discard the newest value because totals usually rise.

For the related batch-analysis problem, see economic data revisions. Both workflows need to preserve the difference between a timely estimate and a settled record.

Sources and scope

• Apache Beam programming guide: windows, triggers and late-data configuration.
• Basics of the Beam model: watermark estimates and window expiry.

Checked October 22, 2026. All sales, times and proposed dashboard policies are original teaching examples, independent of a particular financial reporting requirement.