Sending emails, firing off alerts or even committing resources, like telling a driver to start a delivery run, are all examples of side effects. Some of these external systems are designed in a way that don’t support continuous refinement, and it would be nice to support developers who have to work with these systems.Īlso, there are plenty of use cases that are not pure data processing but need to produce side effects. However, many data processing applications will eventually produce results into a system outside of Apache Kafka ®.
It eliminates irrelevant details from applications while providing complete semantics. The continuous-refinement-with-operational-parameters model is a very simple and powerful design for a data processing system. Because this is an operational problem, not a semantic one, Streams gives you operational parameters to tune a record cache rather than API operators.
#PHOTOMARKS COST UPDATE#
High-volume applications may not be able to process and transmit every update within the constraints of CPU, memory, network and disk. It is important for the operational characteristics, though. Whether Streams emits every single update or groups updates is irrelevant to the semantics of a data processing application. You get to focus on the logic of your data processing pipeline.
You don’t have to worry about telling each operator when it should emit results. This article explains how we are fundamentally sticking with this model, while also opening the door for use cases that are incompatible with continuous refinement.īy continuous refinement, I mean that Kafka Streams emits new results whenever records are updated. Back in May 2017, we laid out why we believe that Kafka Streams is better off without a concept of watermarks or triggers, and instead opts for a continuous refinement model.