Connecting a data source
Types of Data Sources
SQL-based ones
On the data source page, you can configure your data sources to be used in workflows. It can be your production database replica, your data warehouse, staging or testing database, or any other storage.
We recommend having both production and testing data sources connected, so you can always properly test your workflows before using them on production.
At the moment we support a wide range of SQL-based data sources, such as
MySQL,
Snowflake,
Postgres,
Google Big Query,
AWS RDS,
Amazon Redshift
Immuta
Planetscale
ClickHouse
ElasticSearch
and others.
And this list is growing since we are adding new ones.
Connecting an SQL-based data source is easy. You just need to create us a read-only user, give it the necessary permissions, and don't forget to add an additional security layer such as SSH tunnel or/and enabling connections from our IP, which you can find in the data source form.
API-based data sources
On top of that, Datamin also supports JSON APIs as data sources. Any API which returns JSON data can be used as a data source, such as typically:
Salesforce
Hubspot
Your own backend APIs
etc.
In this case, instead of using a Query task start your workflow with an API_Call task and the rest stays the same.
Data streaming platforms as data sources
The unique feature of Datamin that makes us 100% real-time is an integration with Kafka as a data source. It allows you to trigger workflows in a real-time right after getting a new message from Kafka topic.
In order to do it, you need to make two steps:
Install and configure our open-source kafka-trigger library. A detailed description of how to do it is in its README.
Create a workflow you want to trigger, starting with the External_trigger task.
Last updated