Batch pipelines that run at midnight are not enough when the decision needs to happen in the next 60 seconds.

Apache Kafka cluster setup on AWS MSK or self-managed, AWS Kinesis Data Streams, or Google Pub/Sub -- platform selected based on your cloud environment and operational requirements. Topic design and partitioning strategy to match your event volume and consumer parallelism. Producer and consumer configuration for throughput and latency targets. Replication factor and retention policy set for the durability and replay window your use case requires. Infrastructure-as-code so the setup is reproducible and version-controlled from day one.

Apache Flink or Spark Streaming jobs for real-time transformation: event filtering by type or attribute, enrichment with reference data from a database or cache, aggregation over tumbling or sliding time windows, and anomaly detection logic running on the stream before events reach the destination. Flink's stateful processing handles complex event patterns -- session windows, late arrivals, and out-of-order events -- without requiring events to arrive in strict order. Processing jobs deployed and monitored alongside the streaming infrastructure.

Debezium-based CDC from MySQL, PostgreSQL, SQL Server, or Oracle. Row-level inserts, updates, and deletes captured from the transaction log and published to Kafka topics without polling the application tables. No full table scans, no added load on the production database beyond reading the binlog. Microsecond-level latency from database commit to Kafka topic. CDC is the right pattern when you need to replicate large tables, capture deletes, or avoid adding query load to a production database that is already under pressure.

Real-time feature computation for ML models that need current feature values at inference time. Streaming pipeline from events to feature store (Feast, Tecton, or a custom Redis-backed store) so fraud, recommendation, and personalisation models access features computed from the last seconds or minutes of activity rather than yesterday's batch. Low-latency feature retrieval path tested under the throughput your inference layer produces. Feature pipeline and batch feature pipeline running in parallel so the model always has a fallback if the streaming path has a temporary gap.

Delivery from the streaming layer to operational destinations: PostgreSQL or DynamoDB for application reads, Elasticsearch for search indexes and alerting, Redis for low-latency caching, and real-time dashboard systems like Apache Superset or Grafana. Consumer group management for multiple downstream systems consuming the same stream independently -- each consumer tracks its own offset so one slow consumer doesn't block others. Fan-out patterns where a single event stream feeds multiple destinations without duplication at the source.

Consumer lag monitoring per topic and partition with alerting when lag exceeds the threshold that would breach your latency SLA. Dead letter queue for events that fail processing after the configured retry count -- failed events land in a separate topic for inspection and reprocessing rather than being silently dropped. Backpressure handling to prevent pipeline failures when downstream systems slow down. Replay capability from Kafka retention for reprocessing events after a consumer bug is fixed. Operations runbook covering lag investigation, consumer restart, and offset reset procedures.