Tasks
Workflows are built from tasks. Currently, Datamin supports the following 8 kinds of tasks:
Queries and API Calls are usually the first ones in the workflow.
Using Query you can write an SQL query you want to run against your Data Sources or use a constructor of queries if you are not familiar with SQL. Here is what the constructor of queries typically looks like:
Here we retrieve all customers, who created an account today and completed the onboarding process. Also, we sort the list from newest to oldest.
API Calls can be used at the beginning of the workflow to retrieve data and also at the end to send data to a certain external or internal API.
It supports all the attributes of a normal API Call when it is done from the command line using
curl or a software like Postman, such as body payload, headers, and query strings.
The result of the Query or API Call can be sent to Aggregators, Conditions, or Transformers.
Merge is typically used when you have two or more data sources and want to merge data sets coming from both.
Aggregators help you to implement basic aggregating formulas and pre-build functions such as COUNT(), AVG(), SUM(), and other ones. They are typically used for transforming and incoming datasets into one single value.
Condition is a block where you can compare input and expectations and continue with different workflow scenarios depending on if it is true or false.
For Each allows running the next task for each of the items in the dataset separately. Working with this block is pretty straightforward and doesn't even require any configuration, except the name.
Simple filters allow you to filter your dataset by a certain value of a certain key:
Transformers help you to transform data into different formats or extract values.
For example, convert the dataset to CSV before sending it as an attachment to an Email.
In the end, you can either call APIs with API Calls or send notifications with Notification tasks to the Destinations you've configured before.
When placed as a first task in a workflow, external_trigger allows you to trigger workflows from your data streaming platforms to make it 100% real-time.
The first data streaming platform we integrate with is Kafka. And the open-source library that can trigger workflows from it is hosted on our Github: https://github.com/datamin-io/kafka-trigger
This task allows you to connect workflows to each other by running one from another one.
The main benefit of it is in the DRY principle, instead of repeating the same logic in multiple workflows, it can be re-used by connecting workflows to each other as a chain.
As with each and every no-code solution, we still allow using some code if necessary:-)
Therefore, we developed a new special task called Python, which can be used for transforming data sets on the fly.
Here is an example, of how Python can help you iterate through input data, extracting from the database and transforming it on the fly. In this particular case, converting numbers into strings.
From now on you can utilize the power of GPT by OpenAI as a dedicated task in workflows for transforming, cleaning, or completing your data.
Last modified 30d ago