Publish-Subscribe Channel
The main difference between a Point-to-Point and Publish-Subscribe messaging model is that a message is not sent to a specific receiver but to a specific topic, and all the subscribed receivers receive the message.
If you have a specific message producer (Apache Kafka, RabbitMQ, AWS Lambda, Google Pub/Sub), that delivers this message to a specific Datamin's pipeline by calling its API Endpoint, this pipeline acts as a topic that has a specific distribution logic responsible for forwarding the message to subscribed pipelines (receivers).
The pipeline itself needs to begin with the "External trigger" task and can contain a set of conditions that define what pipelines to forward this message to.
As we can see from this pipeline, there are two subscription condition checks. And how to define a condition itself is up to you. We recommend having a standard message type in your message schema that will always be present in all the messages and then the condition will check if that matches a certain value.
For example:
Or to make it even more advanced, you can create an environment variable for this message type, that can be reused for multiple tasks and pipelines:
Then after the condition is checked if the first one is true, it forwards the data to "pipeline 1", which is subscribed to that type of message.
If the second one is true, it forwards the message to "pipeline 2" and "pipeline 3", which is how you subscribe multiple pipelines to the same message.
As you can see this pipeline is not aware of what data the message contains and purely acts as an intermediary distributor to the subscribed pipelines that contain the logic itself. We recommend you always keep this way.
Last updated
