diff --git a/backpressure/README.md b/backpressure/README.md index 51a0307e1..d0734ad1f 100644 --- a/backpressure/README.md +++ b/backpressure/README.md @@ -1,65 +1,46 @@ --- -title: "Backpressure Pattern in Java: controlling data streams from producers to consumer inorder to prevent overwhelming the consumer" +title: "Backpressure Pattern in Java: Gracefully regulate producer-to-consumer data flow to prevent overload." shortTitle: Backpressure -description: "Explore the Backpressure design pattern in Java with detailed examples. Learn how it helps by preventing system overload, ensuring stability and optimal performance by matching data flow to the consumer’s processing capability." +description: "Dive into the Backpressure design pattern in Java through practical examples, discovering how it prevents overload while ensuring stability and peak performance by aligning data flow with consumer capacity." category: Concurrency language: en tag: - - Decoupling + - Asynchronous - Event-driven - Reactive + - Resilience --- +## Also known as + +* Flow Control +* Rate Limiting Mechanism + ## Intent of the Backpressure Design Pattern -The Backpressure Design Pattern is a strategy used in software systems (especially in data streaming, reactive programming, and distributed systems) -to handle situations where a fast producer overwhelms a slow consumer. The intent is to prevent system instability, resource exhaustion, or crashes by managing the flow of data between components. +Control the rate of data production so downstream consumers are not overwhelmed by excessive load. ## Detailed Explanation of Backpressure Pattern with Real-World Examples -### Real-world examples +Real-world example -#### 1. Real-Time Data Streaming (Reactive Systems) -- **Stock Market Data** - - High-frequency trading systems generate millions of price updates per second, but analytics engines can't process them all in real time. - - Backpressure mechanisms (e.g., in RxJava, Akka, Reactor) throttle or drop excess data to avoid overwhelming downstream systems. -- **IoT Sensor DataQ** - - Thousands of IoT devices (e.g., smart factories, wearables) send continuous telemetry, but cloud processing has limited capacity. - - Reactive frameworks apply backpressure to buffer, drop, or slow down data emission. +> Imagine a busy coffee shop where multiple baristas brew drinks (producers), and a single barista is responsible for carefully crafting latte art (consumer). If drinks are brewed faster than the latte-art barista can decorate them, they pile up, risking quality issues or discarded drinks. By introducing a pacing system—only sending new cups once the latte-art barista is ready—everyone stays synchronized, ensuring minimal waste and a consistently enjoyable customer experience. -#### 2. Message Queues (Kafka, RabbitMQ) -- **E-Commerce Order Processing** - - During flash sales (e.g., Black Friday), order requests spike, but payment and inventory systems can’t keep up. - - Message queues like Kafka and RabbitMQ use, Limited queue sizes to drop or reject messages when full or Consumer acknowledgments to slow producers if consumers lag. -- **Log Aggregation** - - Microservices generate massive logs, but centralized logging (E.g.: ELK Stack) can’t ingest them all at once. - - Kafka applies backpressure by pausing producers when consumers are slow. +In plain words -#### 3. Stream Processing (Apache Flink, Spark) -- **Social Media Trends (Twitter, TikTok)** - - Viral posts create sudden spikes in data, but trend analysis is computationally expensive. - - Backpressure in Spark Streaming prioritizes recent data and discards older, less relevant updates. -- **Fraud Detection in Banking** - - Millions of transactions flow in, but fraud detection models take time to analyze each one. - - slow down ingestion if processing lags (Throttling), save progress to recover from backpressure-induced delays (Checkpointing). +> The Backpressure design pattern is a flow control mechanism that prevents overwhelming a system by regulating data production based on the consumer’s processing capacity. -### In plain words +Wikipedia says -The Backpressure design pattern is a flow control mechanism that prevents overwhelming a system by regulating data production based on the consumer’s processing capacity. +> Back pressure (or backpressure) is the term for a resistance to the desired flow of fluid through pipes. Obstructions or tight bends create backpressure via friction loss and pressure drop. In distributed systems in particular event-driven architecture, back pressure is a technique to regulate flow of data, ensuring that components do not become overwhelmed. -### Wikipedia says +Sequence diagram -Back pressure (or backpressure) is the term for a resistance to the desired flow of fluid through pipes. Obstructions or tight bends create backpressure via friction loss and pressure drop. - -In distributed systems in particular event-driven architecture, back pressure is a technique to regulate flow of data, ensuring that components do not become overwhelmed. - -### Architectural Diagram -![backpressure](./etc/backpressure.png) +![Backpressure sequence diagram](./etc/backpressure-sequence-diagram.png) ## Programmatic Example of Backpressure Pattern in Java -First we will create a publisher that generates a data stream. -This publisher can generate a stream of integers. +This example demonstrates how backpressure can be implemented using Project Reactor. We begin by creating a simple publisher that emits a stream of integers, introducing a small delay to mimic a slower production rate: ```java public class Publisher { @@ -69,11 +50,7 @@ public class Publisher { } ``` -Then we can create a custom subscriber based on reactor BaseSubscriber. -It will take 500ms to process one item to simulate slow processing. -This subscriber will override following methods to apply backpressure on the publisher. -- hookOnSubscribe method and initially request for 10 items -- hookOnNext method which will process 5 items and request for 5 more items +Next, we define a custom subscriber by extending Reactor’s BaseSubscriber. It simulates slow processing by sleeping for 500ms per item. Initially, the subscriber requests ten items; for every five items processed, it requests five more: ```java public class Subscriber extends BaseSubscriber { @@ -111,7 +88,7 @@ public class Subscriber extends BaseSubscriber { } ``` -Then we can create the stream using the publisher and subscribe to that stream. +Finally, in the `main` method, we publish a range of integers and subscribe using the custom subscriber. A short sleep in the main thread allows the emission, backpressure requests, and processing to be fully observed: ```java public static void main(String[] args) throws InterruptedException { @@ -122,7 +99,7 @@ public static void main(String[] args) throws InterruptedException { } ``` -Program output: +Below is an example of the program’s output. It shows the subscriber’s log entries, including when it requests more data and when each integer is processed: ``` 23:09:55.746 [main] DEBUG reactor.util.Loggers -- Using Slf4j logging framework @@ -149,73 +126,31 @@ Program output: 23:10:01.437 [parallel-8] INFO reactor.Flux.ConcatMapNoPrefetch.1 -- onComplete() ``` -## When to Use the Backpressure Pattern +## When to Use the Backpressure Pattern In Java -- Producers Are Faster Than Consumers - - If a producer generates data at a much faster rate than the consumer can handle, backpressure prevents resource overload. - - Example: A server emitting events 10x faster than the client can process. - -- There’s Limited Memory or Resource Capacity - - Without flow control, queues or buffers can grow indefinitely, leading to out-of-memory errors or system crashes. - - Example: Streaming large datasets into a low-memory microservice. - -- Building Reactive or Event-Driven Architectures - - Reactive systems thrive on non-blocking, asynchronous flows—and backpressure is a core component of the Reactive Streams specification. - - Example: Using RxJava, Project Reactor, Akka Streams, or Node.js streams. - -- Unpredictable Workloads - - If the rate of data production or consumption can vary, backpressure helps adapt dynamically. - - Example: APIs receiving unpredictable spikes in traffic. - -- Need to Avoid Data Loss or Overflow - - Instead of dropping data arbitrarily, backpressure lets you control flow intentionally. - - Example: Video or audio processing pipelines where dropping frames is costly. - -## When to avoid the Backpressure Pattern - -- For batch processing or simple linear flows with well-matched speeds. -- If data loss is acceptable and simpler strategies like buffering or throttling are easier to manage. -- When using fire-and-forget patterns (e.g., log shipping with retries instead of slowing the producer). +* Use in Java systems where data is produced at high velocity and consumers risk overload. +* Applicable in reactive or event-driven architectures to maintain stability under varying load conditions. ## Benefits and Trade-offs of Backpressure Pattern -### Benefits: +Benefits: -- Improved System Stability - - Prevents overload by controlling data flow. - - Reduces chances of out-of-memory errors, thread exhaustion, or service crashes. -- Efficient Resource Usage - - Avoids excessive buffering and unnecessary computation. - - Enables systems to do only the work they can handle. -- Better Responsiveness - - Keeps queues short, which improves latency and throughput. - - More consistent performance under load. -- Graceful Degradation - - If the system can't keep up, it slows down cleanly rather than failing unpredictably. - - Consumers get a chance to control the pace, leading to predictable behavior. -- Fits Reactive Programming - - It's essential in Reactive Streams, RxJava, Project Reactor, and Akka Streams. - - Enables composing async streams safely and effectively. +* Protects consumers from saturation and resource exhaustion. -### Trade-offs: +Trade-offs: -- Complexity in Debugging - - Adds logic for flow control, demand signaling, and failure handling. - - More state to manage (e.g., request counts, pause/resume, buffer sizes). -- Harder Debugging & Testing - - Asynchronous flow + demand coordination = trickier to test and debug. - - Race conditions or deadlocks may occur if not handled carefully. -- Potential for Bottlenecks - - A slow consumer can throttle the entire system, even if other parts are fast. - - Needs smart handling (e.g., buffer + drop + retry strategies). +* Introduces possible delays if production must slow down to match consumer capacity. +* Requires careful orchestration in complex systems with multiple concurrent data sources. ## Related Java Design Patterns -* [Publish-Subscribe Pattern](https://github.com/sanurah/java-design-patterns/blob/master/publish-subscribe/): Pub-Sub pattern decouples producers from consumers so they can communicate without knowing about each other. Backpressure manages flow control between producer and consumer to avoid overwhelming the consumer. -* [Observer Pattern](https://github.com/sanurah/java-design-patterns/blob/master/observer/): Both involve a producer (subject/publisher) notifying consumers (observers/subscribers). Observer is synchronous & tightly coupled (observers know the subject). Pub-Sub is asynchronous & decoupled (via a message broker). -* [Mediator Pattern](https://github.com/sanurah/java-design-patterns/blob/master/mediator/): A mediator centralizes communication between components (like a message broker in Pub-Sub). Mediator focuses on reducing direct dependencies between objects. Pub-Sub focuses on broadcasting events to unknown subscribers. + +* [Observer Pattern](https://java-design-patterns.com/patterns/observer/): Both patterns involve a producer notifying consumers. Observer is synchronous and tightly coupled (observers know the subject). +* [Publish-Subscribe Pattern](https://java-design-patterns.com/patterns/publish-subscribe/): Both patterns deal with asynchronous data flow and can work together to manage message distribution and consumption effectively. ## References and Credits +* [Backpressure Explained (RedHat Developers Blog)](https://developers.redhat.com/articles/backpressure-explained) +* [Hands-On Reactive Programming in Spring 5](https://amzn.to/3YuYfyO) +* [Reactive Programming with RxJava: Creating Asynchronous, Event-Based Applications](https://amzn.to/42negbf) +* [Reactive Streams in Java](https://amzn.to/3RJjUzA) * [Reactive Streams Specification](https://www.reactive-streams.org/) -* [Reactive Programming with RxJava by Tomasz Nurkiewicz & Ben Christensen](https://www.oreilly.com/library/view/reactive-programming-with/9781491931646/) -* [RedHat Developers Blog](https://developers.redhat.com/articles/backpressure-explained) diff --git a/backpressure/etc/backpressure-sequence-diagram.png b/backpressure/etc/backpressure-sequence-diagram.png new file mode 100644 index 000000000..b74a5d2a5 Binary files /dev/null and b/backpressure/etc/backpressure-sequence-diagram.png differ diff --git a/backpressure/pom.xml b/backpressure/pom.xml index 8807f0faf..fcc15892f 100644 --- a/backpressure/pom.xml +++ b/backpressure/pom.xml @@ -1,4 +1,30 @@ + diff --git a/backpressure/src/main/java/com/iluwatar/backpressure/Publisher.java b/backpressure/src/main/java/com/iluwatar/backpressure/Publisher.java index 20fbd31f6..1d39a070a 100644 --- a/backpressure/src/main/java/com/iluwatar/backpressure/Publisher.java +++ b/backpressure/src/main/java/com/iluwatar/backpressure/Publisher.java @@ -30,6 +30,8 @@ import reactor.core.publisher.Flux; /** This class is the publisher that generates the data stream. */ public class Publisher { + private Publisher() {} + /** * On message method will trigger when the subscribed event is published. * diff --git a/bloc/README.md b/bloc/README.md index da494dba3..73e9da916 100644 --- a/bloc/README.md +++ b/bloc/README.md @@ -5,11 +5,7 @@ description: "Learn how the Bloc pattern helps manage state changes in Java appl category: Structural language: en tag: - - State Management - Event-driven - - Listener Management - - Object Composition - - Dynamic Behavior --- ## Also known as diff --git a/clean-architecture/README.md b/clean-architecture/README.md index 0e2e7f5da..a9a3e4964 100644 --- a/clean-architecture/README.md +++ b/clean-architecture/README.md @@ -6,7 +6,6 @@ category: Behavioral language: en tag: - Decoupling - - Architectural Style --- ## Also known as diff --git a/facade/README.md b/facade/README.md index b0dec75b8..e9f3daf0c 100644 --- a/facade/README.md +++ b/facade/README.md @@ -10,7 +10,7 @@ tag: - Code simplification - Decoupling - Encapsulation - - Gang Of Four + - Gang of Four - Interface - Object composition --- diff --git a/function-composition/README.md b/function-composition/README.md index 5bfef09d2..77f597164 100644 --- a/function-composition/README.md +++ b/function-composition/README.md @@ -5,12 +5,7 @@ description: "Learn about the Function Composition design pattern in Java. Disco category: Functional language: en tag: - - Code simplification - Composition - - Decoupling - - Functional decomposition - - Lambda - - Reusability --- ## Also known as diff --git a/map-reduce/README.md b/map-reduce/README.md index 2b0b15003..d73f513fc 100644 --- a/map-reduce/README.md +++ b/map-reduce/README.md @@ -2,13 +2,10 @@ title: "MapReduce Pattern in Java" shortTitle: MapReduce description: "Learn the MapReduce pattern in Java with real-world examples, class diagrams, and tutorials. Understand its intent, applicability, benefits, and known uses to enhance your design pattern knowledge." -category: Performance optimization +category: Structural language: en tag: - - Data processing - - Code simplification - Delegation - - Performance --- ## Also known as diff --git a/mediator/README.md b/mediator/README.md index f9650d3f1..bcf64f022 100644 --- a/mediator/README.md +++ b/mediator/README.md @@ -6,7 +6,7 @@ category: Behavioral language: en tag: - Decoupling - - Gang Of Four + - Gang of Four - Messaging - Object composition --- diff --git a/microservices-client-side-ui-composition/README.md b/microservices-client-side-ui-composition/README.md index c11b79903..f77bf70d8 100644 --- a/microservices-client-side-ui-composition/README.md +++ b/microservices-client-side-ui-composition/README.md @@ -2,15 +2,10 @@ title: "Client-Side UI Composition Pattern: Assembling Modular UIs in Microservices Architecture" shortTitle: Client-Side UI Composition Pattern description: "Learn how the Client-Side UI Composition pattern allows the assembly of modular UIs on the client side, enabling independent teams to develop, deploy, and scale UI components in a microservices architecture. Discover the benefits, implementation examples, and best practices." -category: User Interface +category: Structural language: en tag: - - Micro Frontends - - API Gateway - - Asynchronous Data Fetching - - UI Integration - - Microservices Architecture - - Scalability + - Asynchronous --- ## **Intent of Client-Side UI Composition Design Pattern** diff --git a/microservices-distributed-tracing/README.md b/microservices-distributed-tracing/README.md index 14b34726f..2ceb06bfb 100644 --- a/microservices-distributed-tracing/README.md +++ b/microservices-distributed-tracing/README.md @@ -2,15 +2,10 @@ title: "Microservices Distributed Tracing Pattern: Enhancing Visibility in Service Communication" shortTitle: Distributed Tracing in Microservices description: "Learn how the Distributed Tracing pattern enhances visibility into service communication across microservices. Discover its benefits, implementation examples, and best practices." -category: Integration +category: Structural language: en tag: - - Distributed tracing - - Microservices architecture - - Service communication - - Performance monitoring - Scalability - - Observability --- ## Intent of Microservices Distributed Tracing Design Pattern diff --git a/microservices-idempotent-consumer/README.md b/microservices-idempotent-consumer/README.md index 794eb6e64..a3065861a 100644 --- a/microservices-idempotent-consumer/README.md +++ b/microservices-idempotent-consumer/README.md @@ -2,13 +2,10 @@ title: "Idempotent Consumer Pattern in Java: Ensuring Reliable Message Processing" shortTitle: Idempotent Consumer description: "Learn about the Idempotent Consumer pattern in Java. Discover how it ensures reliable and consistent message processing, even in cases of duplicate messages." -category: Messaging +category: Structural language: en tag: - - Messaging - - Fault tolerance - Event-driven - - Reliability --- ## Also known as diff --git a/money/README.md b/money/README.md index ca64d68fe..55e6c2f69 100644 --- a/money/README.md +++ b/money/README.md @@ -6,14 +6,6 @@ category: Behavioral language: en tag: - Encapsulation - - Precision handling - - Currency safety - - Value Object - - Financial operations - - Currency - - Financial - - Immutable - - Value Object --- ## Also known as diff --git a/monolithic-architecture/README.md b/monolithic-architecture/README.md index fd288dab3..6704cee7c 100644 --- a/monolithic-architecture/README.md +++ b/monolithic-architecture/README.md @@ -6,10 +6,6 @@ category: Architectural language: en tag: - Cohesion - - Simplicity - - Scalability - - Deployment - - Maintainability --- ## Monolithic-Ecommerce App diff --git a/observer/README.md b/observer/README.md index f5402181a..6b8a70077 100644 --- a/observer/README.md +++ b/observer/README.md @@ -7,7 +7,7 @@ language: en tag: - Decoupling - Event-driven - - Gang Of Four + - Gang of Four - Publish/subscribe --- diff --git a/prototype/README.md b/prototype/README.md index 0293a588b..48f10868a 100644 --- a/prototype/README.md +++ b/prototype/README.md @@ -5,7 +5,7 @@ description: "Explore the Prototype design pattern in Java with a comprehensive category: Creational language: en tag: - - Gang Of Four + - Gang of Four - Instantiation - Object composition - Polymorphism diff --git a/proxy/README.md b/proxy/README.md index a63069bd0..5e5109348 100644 --- a/proxy/README.md +++ b/proxy/README.md @@ -7,7 +7,7 @@ language: en tag: - Decoupling - Encapsulation - - Gang Of Four + - Gang of Four - Lazy initialization - Proxy - Security diff --git a/publish-subscribe/README.md b/publish-subscribe/README.md index 4e88cf2e8..8ff29f2c6 100644 --- a/publish-subscribe/README.md +++ b/publish-subscribe/README.md @@ -6,9 +6,6 @@ category: Behavioral language: en tag: - Decoupling - - Event-driven - - Gang Of Four - - Publish/subscribe --- ## Intent of the Publish-Subscribe Design Pattern diff --git a/service-stub/README.md b/service-stub/README.md index 73b0be059..980e3561d 100644 --- a/service-stub/README.md +++ b/service-stub/README.md @@ -5,10 +5,7 @@ description: "Explore the Service Stub design pattern in Java using a Sentiment category: Structural language: en tag: - - Testing - Decoupling - - Dummy Services - - Dependency Injection --- ## Also known as diff --git a/session-facade/README.md b/session-facade/README.md index 2709e9fc6..4724afcfb 100644 --- a/session-facade/README.md +++ b/session-facade/README.md @@ -6,12 +6,6 @@ category: Structural language: en tag: - Abstraction - - API design - - Code simplification - - Decoupling - - Encapsulation - - Gang Of Four - - Interface --- ## Also known as diff --git a/table-inheritance/README.md b/table-inheritance/README.md index 3e3ad4f53..53e9b2ce0 100644 --- a/table-inheritance/README.md +++ b/table-inheritance/README.md @@ -2,19 +2,10 @@ title: "Table Inheritance Pattern in Java: Modeling Hierarchical Data in Relational Databases" shortTitle: Table Inheritance description: "Explore the Table Inheritance pattern in Java with real-world examples, database schema, and tutorials. Learn how to model class hierarchies elegantly in relational databases." -category: Data Access Pattern, Structural Pattern +category: Structural language: en tag: -- Decoupling -- Inheritance -- Polymorphism -- Object Mapping -- Persistence -- Data Transformation ---- - -## Also Known As -- Class Table Inheritance + - Decoupling --- ## Intent of Table Inheritance Pattern