Streaming from Apache Kafka and messaging queues
Since Datamin is a real-time streaming platform, our goal is to provide you with a tool with which you can deliver your data from one place to another as fast as possible.
The fastest way of doing it is to stream data to Datamin directly from such software as Apache Kafka, RabbitMQ, Google Pub/Sub, Amazon SQS, AWS Lambda, etc.
Triggering pipeline from outside of Datamin
No matter which software you want to stream data from, you need to start your pipeline with the "External trigger" task.
This task doesn't require any additional configuration but allows a pipeline to be triggered through the API endpoint, which is used by all of the open-source libraries we developed for external integration.
After this task you can create any pipeline of any logic you want, using the input data that you are planning to send to Datamin from your messaging platforms.
With the following pipeline you can stream from the external messaging platforms to your API:
Or to your database:
Streaming from Apache Kafka
To stream from Apache Kafka, you can use our open-source library, which triggers pipelines when a new message arrives to a topic and forwards this message as input to the pipeline: https://github.com/datamin-io/kafka-trigger
Streaming from RabbitMQ
RabbitMQ allows you to listen to various events in your infrastructure. At the same time, Datamin can become one of the consumers of RabbitMQ messages.
For this integration, you can use our open-source RabbitMQ consumer.
Read the detailed documentation about how to configure it here.
Streaming from Google Pub/Sub
Streaming from Google Pub/Sub to Datamin's pipelines does not even require any additional library. It is as easy as just triggering pipelines via API.
The procedure of how to trigger pipelines via API is described here in our documentation where you can also get the Endpoint URL.
Streaming from AWS Lambda
AWS Lambda allows you to listen to various events in the AWS universe. Therefore these data can immediately be streamed to Datamin by triggering our API for pipelines.
We are building an open-source AWS Lambda function for you and it is coming live soon.
Streaming from other messaging platforms
If you want to stream from other messaging platforms, that are not mentioned here, you can either develop an integration yourself using our API, or contact us and request the new integration.
Last updated
