Stream Processing: From Batch to Real Time

Picture a supermarket loyalty card being scanned at 18:45 on a Friday evening in Manchester. The transaction is recorded instantly, but the insight it contains — a shift in buying habits, a fraud signal, a stock warning — will not surface until the overnight batch job runs at 02:00. By the time anyone acts on it, the customer has gone home, the shelves have been restocked, and the moment has passed. Multiply that delay across millions of daily events in banking, healthcare and transport, and the cost of waiting becomes obvious. So why wait for batch when real-time is possible?

The Batch Era: Foundations and Limitations

Batch processing has been the workhorse of commercial computing since the mainframe era. Data is collected, stored, and then processed in scheduled windows — typically overnight — producing reports that inform the next day’s decisions. This model served enterprises well for decades because storage was expensive, compute was scarce, and business cycles moved slowly enough that a daily refresh was sufficient.

MapReduce and the Rise of Hadoop

The arrival of Hadoop and its MapReduce programming model in the mid-2000s changed the scale of batch processing. Suddenly, clusters of commodity servers could crunch petabytes of data that would have been unthinkable on a single machine. MapReduce split work into map and reduce phases, distributing computation across nodes and tolerating hardware failure gracefully. It was a genuine breakthrough — but it was also inherently high-latency. Jobs were designed to run over bounded datasets, often taking minutes or hours to complete. For analytical workloads like daily sales aggregation, that was fine. For anything requiring an immediate response, it was not.

Why Batch Persists in UK Enterprises

Many UK organisations still lean heavily on overnight batch jobs. Banks such as HSBC have historically processed end-of-day reconciliations and regulatory reporting in batch windows, while retailers like Tesco have relied on nightly jobs to consolidate stock, sales and supply chain data across thousands of stores. These pipelines are mature, well-understood and audited. They also create a structural lag: decisions are made on yesterday’s data, and anomalies are discovered hours after they occur. Batch persists not because it is optimal, but because it is familiar and dependable.

The Shift to Stream Processing: Key Technologies

Stream processing flips the batch model on its head. Instead of accumulating data and processing it later, events are handled continuously as they arrive. Records flow through a pipeline, are transformed and enriched in motion, and produce results within milliseconds. This shift has been enabled by a generation of purpose-built engines — Apache Kafka, Apache Flink and Apache Storm among them — that treat unbounded data as the default rather than the exception.

Apache Kafka: The Backbone of Real-Time Data

Kafka is a distributed event streaming platform that acts as a durable, high-throughput log. Producers write events to topics; consumers read them independently, at their own pace. This decoupling makes Kafka the connective tissue of modern real-time architectures. Monzo, a UK digital bank, uses Apache Kafka for real-time transaction processing, streaming every card payment and transfer through its platform so that balances, notifications and fraud checks update the instant a customer taps their card.

Apache Flink: Stateful Stream Processing

Where Kafka moves data, Flink computes on it. Flink is a stateful stream processor designed for low-latency, exactly-once processing over continuous event streams. It supports event-time semantics, windowing and sophisticated state management, which makes it suitable for complex operations like session analysis, pattern detection and continuous aggregation. Flink can also process bounded data, blurring the line between batch and stream.

Comparing Stream Processing Frameworks

Choosing a framework depends on latency requirements, state complexity and team expertise. A quick comparison:

• Apache Kafka (with Kafka Streams): Excellent for event transport and lightweight stream transformations; tightly coupled to the Kafka ecosystem.
• Apache Flink: Best for complex stateful computations, event-time processing and exactly-once guarantees.
• Apache Storm: A pioneer in low-latency processing, still used for simple, high-throughput pipelines but less common for stateful workloads.
• Apache Spark Structured Streaming: A micro-batch model that unifies batch and stream APIs, ideal for teams already invested in Spark.

SQL on Streams: Bridging Familiar and Real-Time

The biggest barrier to stream processing adoption has never been the technology — it has been the skills gap. Most analysts know SQL, not Java or Scala. Streaming SQL closes that gap by letting teams express continuous queries in a familiar declarative language, lowering the barrier to real-time analytics.

ksqlDB and Streaming SQL

ksqlDB, part of the Kafka ecosystem, allows developers and analysts to write persistent queries against Kafka topics using SQL-like syntax. Filters, joins, aggregations and windowed computations can be expressed in a few lines, with results written back to new topics. It turns Kafka from a plumbing layer into a queryable data platform.

Flink SQL for Unified Batch and Stream

Flink SQL extends the same idea across both bounded and unbounded data. A single query can run over a historical dataset or a live stream without rewriting logic. UK telecom Vodafone has used Flink SQL for network monitoring, querying continuous telemetry streams to detect anomalies and performance degradation as they happen rather than after the fact.

Real-World Applications in the UK

Stream processing is no longer theoretical. Across the UK, organisations in healthcare, finance and transport are deploying it to solve problems where latency directly affects outcomes.

Healthcare: NHS Patient Data Streams

The NHS has piloted stream processing for real-time patient monitoring in UK hospitals. Vital signs from bedside devices can be streamed and analysed continuously, allowing clinical teams to be alerted to deterioration far earlier than periodic checks would allow. The benefits are clear, though governance, data privacy and integration with legacy systems remain significant challenges.

Finance: Fraud Detection at Barclays

Barclays employs stream processing for fraud detection in the UK, scoring transactions as they occur and flagging suspicious patterns within milliseconds. In fraud prevention, latency is everything — a card payment approved in real time cannot be clawed back as easily as one blocked before settlement.

Transport: TfL Real-Time Analytics

Transport for London (TfL) uses real-time data analytics for traffic management, ingesting feeds from roads, buses and the Underground to monitor conditions and respond to disruptions. Streaming pipelines allow TfL to adjust signals, inform passengers and coordinate responses as events unfold.

Challenges and Best Practices

Stream processing introduces its own class of problems. Getting them wrong leads to incorrect results, duplicated records or systems that collapse under load.

Exactly-Once Semantics and State Management

Exactly-once processing ensures each event affects the result precisely once, even in the face of failures. Flink and Kafka provide primitives for this, but they require careful configuration of checkpoints, transactional producers and idempotent sinks. State must be managed deliberately — unbounded state growth will eventually exhaust memory.

Handling Late and Out-of-Order Data

In distributed systems, events rarely arrive in order. A payment recorded at 10:00 may reach the processor at 10:03. Event-time processing with watermarks allows systems to reason about lateness and decide when a window is complete. Best practice is to define allowed lateness explicitly and decide whether late events update results or are discarded.

Scalability and Fault Tolerance

Streaming pipelines must scale horizontally and recover from node failures without data loss. Common practices include:

1. Partitioning streams by key to distribute load evenly.
2. Using Kafka replication and Flink checkpointing for durability.
3. Monitoring with Prometheus and Grafana to catch backpressure and lag early.
4. Testing failure scenarios before production rollout.

Academic research continues to shape these practices. DASFAA (Database Systems for Advanced Applications) is a key conference for stream processing research, and its proceedings regularly inform how industry handles semantics, scheduling and query optimisation.

The Future: Unified Batch and Stream Processing

The direction of travel is clear: rather than maintaining separate batch and streaming stacks, organisations are moving toward unified engines that handle both with a single API and a single mental model.

Apache Spark Structured Streaming

Spark Structured Streaming treats a stream as an unbounded table, allowing the same DataFrame operations to run over batch and streaming sources. For teams already using Spark for analytics, this reduces architectural complexity and reuses existing skills.

The Lambda and Kappa Architectures Revisited

The Lambda architecture maintained parallel batch and speed layers, duplicating logic and creating maintenance overhead. The Kappa architecture argued for a single streaming pipeline that could reprocess historical data when needed. Unified engines like Flink and Spark Structured Streaming are making Kappa practical for more organisations, and UK research initiatives alongside DASFAA conferences continue to refine the trade-offs.

Embracing stream processing is no longer optional for UK organisations seeking real-time insights. From Monzo’s transaction pipelines to the NHS’s patient monitoring pilots, the organisations that act on data as it arrives are the ones that respond fastest. Our advice is simple: start small, experiment with Kafka and Flink, and let the batch jobs retire one by one.

FAQ

What is the difference between batch and stream processing?

Batch processing handles bounded datasets in scheduled windows, typically producing results after a delay. Stream processing handles unbounded data continuously, producing results within milliseconds of an event occurring.

Do I need to replace my existing batch systems?

Not necessarily. Many organisations run hybrid architectures, using streaming for time-sensitive use cases such as fraud detection while retaining batch for regulatory reporting and historical analysis.

Which tools should a UK team start with?

Apache Kafka is a sensible starting point for event transport, paired with either Kafka Streams for simple transformations or Apache Flink for stateful processing. ksqlDB and Flink SQL let analysts work in SQL rather than code.

How does DASFAA relate to stream processing?

DASFAA (Database Systems for Advanced Applications) is a key conference for stream processing research, publishing work on query semantics, state management and performance that often influences industry practice.

What is the biggest challenge in stream processing?

Handling exactly-once semantics, late-arriving data and state growth correctly. These require deliberate design, careful configuration and continuous monitoring with tools like Prometheus.