Short answer: Select a stream processing platform after defining event owners, schemas, keys, timestamps, retention and compatibility; generating representative ordered, duplicate, missing, corrupt and late events; choosing event time versus processing time explicitly; testing watermark, lateness and window-finalization behavior; proving ordering within the exact partition or key boundary required; documenting at-most-once, at-least-once and exactly-once scope across producers, brokers, processors and external sinks; making sink writes idempotent or transactional; checkpointing state and validating restore after worker, coordinator and storage failure; replaying retained history without double-counting live traffic; measuring lag, backpressure, skew, hot keys and end-to-end percentile latency; failing zones, regions, networks and schema services; reconciling source events, committed offsets, state and final sinks; pricing ingress, storage, compute, partitions, egress and cross-region recovery; and exporting code or jobs, schemas, state contracts, checkpoints, offsets, policies and operational history. A platform's exactly-once label is not an end-to-end guarantee when an external payment or database sink cannot participate.
Apache Kafka's official design documentation separates publishing durability from consumption and warns that exactly-once claims require careful scope; processing into external systems generally needs destination cooperation. Apache Flink documentation distinguishes event time from processing time and describes watermarks as a latency-versus-completeness mechanism, with explicit handling for late events and state checkpoints. Buyers should test the complete topology and business sink, not inherit a component-level slogan.
Use the same schemas, keys, event distributions, ordering needs, late-data curve, state size, sink semantics, failures, replay horizon, latency objectives, regions and cost horizon. A throughput benchmark with tiny stateless records cannot be compared with the buyer's skewed, stateful and externally side-effecting workload.
Define Event Contracts, Keys And Time
Assign event owner, schema, key, unique ID, event timestamp, source sequence, retention, classification and compatibility policy. Test additions, removals, type changes, unknown enums, corrupt records and producers that deploy out of order.
Choose event time when results should reflect occurrence and processing time when wall-clock handling is acceptable. Document timestamp source, clock uncertainty, time zone, watermark generation, idle partitions, allowed lateness and when a result becomes final or correctable.
Test Ordering, State And Delivery Scope
State the required ordering boundary: one key, partition, source or global. Change partition counts, rebalance consumers and introduce retries. Verify that business keys remain stable and hot keys do not destroy latency.
Document guarantees separately for producer-to-log, log-to-processor, processor state, output log and external sink. Test duplicates and uncertain acknowledgments. Use stable event keys, sink idempotency or coordinated transactions for material side effects.
Exercise Checkpoints, Replay And Late Data
Build stateful joins, windows, timers and aggregations at expected scale. Fail workers during snapshots and rescale. Restore from checkpoints and reconcile state and outputs with an independently computed golden result.
Replay a retained interval while live events continue. Control offsets, versioned code, reference data and sink mode so history neither disappears nor double-counts. Route, update or quarantine late events according to an explicit business rule and preserve correction evidence.
Measure Backpressure, Failover And Observability
Throttle sinks, overload one partition, pause a source and grow state. Monitor ingress, throughput, event-time lag, processing lag, watermarks, queues, checkpoint duration and failures, state size, skew and sink acknowledgments.
Fail broker, worker, coordinator, schema registry, object storage, zone, region and network links. Measure recovery point and time, leader or task movement, duplicates, gaps and stale outputs. Test split-brain protections and manual regional return.
Reconcile Cost, Security And Exit
Reconcile unique source events, accepted records, rejected records, offsets, state updates, emitted results and final sink values. Restrict topic, job, schema, checkpoint, replay and administrative access; encrypt traffic and audit changes and data exports.
Model ingress, retained storage, partitions, compute, state backend, checkpoints, egress, cross-region replication, support and idle environments. Export jobs, schemas, keys, time rules, state contracts, offsets, checkpoints, access policies, monitoring and runbooks, then restore a priority pipeline elsewhere.
Normalize Stream Processing Evaluations
Normalize Events
Use One Event Set
Compare identical schemas, keys, duplicates, corruption, ordering and lateness.
Use One Time Contract
Apply the same timestamps, watermarks, idle sources, windows and correction rules.
Normalize Guarantees
Use One Topology
Include identical producer, broker, stateful processor and external sink behavior.
Use One Failure Set
Fail the same workers, brokers, coordinators, storage, zones and regions.
Normalize Proof
Use One Reconciliation
Compare source IDs, rejects, offsets, state, outputs and final sink values.
Use One Cost And Exit
Price the same load and restore the same priority pipeline elsewhere.
Stream Processing Platform Scorecard
| Buying area | What to confirm | Why it matters |
|---|---|---|
| Event contract | Owners, schemas, keys, IDs, time and compatibility | Makes producers and consumers evolve safely |
| Time | Event time, watermarks, lateness, windows and corrections | Balances complete and timely results |
| Ordering | Required boundary, partitions, rebalances and hot keys | Protects sequence-dependent logic |
| Delivery | Producer, log, state, output and sink guarantees | Exposes exactly-once scope |
| State | Checkpoints, restore, rescale, timers and golden result | Proves recovery correctness |
| Replay | Offsets, versions, live overlap, sinks and corrections | Makes history safe to recompute |
| Operations | Lag, backpressure, skew, failures and reconciliation | Detects silent real-time degradation |
| Commercial | Ingress, storage, compute, state, egress and exit | Reveals total cost and lock-in |
Questions To Ask Before Shortlisting
- Who owns each event contract and compatibility decision?
- Which key defines required ordering?
- Is event time or processing time correct for the use case?
- How do watermarks and allowed lateness affect final results?
- Where exactly does the delivery guarantee begin and end?
- Can the external sink handle duplicates or transactions?
- Does checkpoint restore match an independent golden result?
- Can replay coexist with live traffic without double counting?
- What happens to late and corrupt events?
- How are hot keys and backpressure detected?
- What are regional recovery point and time outcomes?
- Can the priority pipeline and state contract move elsewhere?
Buying Red Flags
Exactly-once is advertised without naming the external sink and failure boundary.
The benchmark excludes state, skew, late data and downstream throttling.
Watermarks and window-finalization rules are hidden in defaults.
Replay uses current code and reference data without version control or reconciliation.
Export omits offsets, checkpoints, state contracts, time rules or runbooks.
Source Links
- Apache Kafka: Design And Message Delivery Semantics
- Apache Kafka: Exactly Once Delivery And Transactional Messaging
- Apache Flink: Streaming Applications, Event Time And Watermarks
- Apache Flink: Generating Watermarks
FAQ
What is stream processing?
Stream processing continuously transforms, joins or analyzes events as they arrive, often maintaining state and emitting updated results.
What is event time?
Event time is when an event occurred at the source; processing time is when the platform handled it. The choice affects late and out-of-order results.
What is a watermark?
A watermark represents progress in event time and helps decide when windows can produce results, trading latency against completeness.
Does exactly once mean an external API runs once?
Not automatically. End-to-end behavior depends on producers, state, outputs and the external sink's transactional or idempotent cooperation.
Why test replay?
Replay is needed for recovery, corrections and new logic, but can duplicate live results or use inconsistent versions unless explicitly controlled.
What should a stream platform export?
Jobs, schemas, keys, time rules, state contracts, offsets, checkpoints, policies, monitoring, evidence and recovery runbooks should remain portable.
Related Software Buying Guides
- Data Integration Software Checklist
- Event Streaming Platform Checklist
- Observability Software Checklist
Real-time is not a latency number; it is a tested contract for time, state, ordering, delivery, correction and recovery across the whole topology.