📘 How does streaming beat the nightly file?

What you’ll learn

1. Batch vs Stream ProcessingDistinguish bounded from unbounded data, and choose batch or streaming from the latency the result actually needs.The useful split is the shape of the input, not the logo on the engine. Batch waits for a closed set. Streaming starts while the set is still growing. Akidau reserves bounded/unbounded for data and streaming/batch for runtimes. You pick the tool from how late the answer can be.
2. Events and the Log AbstractionDescribe the append-only log and how immutable events let you replay, decouple, and derive state.A log is an append-only, totally ordered sequence with monotonic offsets. Events are facts; tables are projections. Because the tape is durable, a consumer can rewind to a retained offset and rebuild. Writers do not wait on readers. Kleppmann's move is to make that changelog the public interface.
3. Pub/Sub and Message BrokersExplain publish-subscribe and Kafka's topics, partitions, offsets, and classic consumer groups.Producers publish to a named topic. Kafka stores the topic as partitions, each an independent ordered log. Order is per partition. Offsets are bookmarks, not deletes. In a classic consumer group, each partition has one owner at a time; another group gets its own full copy. Partition count caps that group's parallelism.
4. Time, Windows, and StateSeparate event time from processing time, and use watermarks, windows, and checkpointed state without treating any of them as magic.Event time is when it happened. Processing time is when you saw it. A watermark is a heuristic lower bound on event time already seen, used to emit a window. Fixed/tumbling, sliding, and session windows cut the stream differently. Aggregations need durable keyed state, or a crash invents a new past.
5. Guided Project: Real-Time Order PipelineDesign a live per-minute revenue pipeline that keys for order, windows on event time, writes idempotently, and watches lag.Key by region for per-region order, then watch for skew. Tumbling minutes use event time; the watermark decides when to emit; late data needs an explicit path. Kafka transactions can be exactly-once Kafka-to-Kafka; an external dashboard needs an idempotent region+minute key. Backpressure and consumer lag keep the process from dying into its own buffer.

Questions this course answers

Grounded in trusted sources

• Streaming 101: The world beyond batch — Tyler Akidau, O'Reilly Radar, 5 August 2015 — unbounded vs bounded data; event time vs processing time; windowing — https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/
• The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing — Akidau et al., PVLDB 8(12), 2015 — §1.1 bounded/unbounded; §1.2 fixed/sliding/session; §1.3 watermarks as heuristic lower bounds — https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
• Turning the database inside-out with Apache Samza — Martin Kleppmann, 4 March 2015 — append-only log of immutable facts; materialized views as projections — https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html
• Apache Kafka Design — Message Delivery Semantics — Apache Kafka 4.3 documentation — partitions, consumer position, at-most/at-least/exactly-once, transactions, share groups — https://kafka.apache.org/43/design/design/
• Apache Kafka Introduction — Apache Kafka documentation — topics, partitions, consumer groups, per-partition order — https://kafka.apache.org/documentation/
• Timely Stream Processing — Apache Flink documentation — watermarks, late elements, event time vs processing time — https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses