Built on shortcomings and possible improvement of other tooling in the market, Prefect is an open-source workflow management tool designed to orchestrate data stacks. With Prefect you can build, run, and monitor many different pipelines at a certain schedule.
- In the past, different tasks were carried out using cron jobs. The more tasks in a project, the more complicated it all became to keep track of.
- Dependencies between different tasks became more complicated to declare.
An orchestration tool, like Prefect, is used for automated configuration, management and coordination of computer systems, applications, and services. It will help to easily manage complex tasks and workflows.
Prefect Core & Prefect Cloud
Prefect Core can be used for building, testing, and executing workflows locally. In some cases, this offers enough functionality, but many users want some more features related to observability, persistence, monitoring, and multi-flow orchestration. This is where Prefect Cloud comes in.
Prefect Cloud is designed to monitor Prefect Core workflows. It comes with a database for states, a GraphQL API, a clear UI and a lot of other features. When Prefect schedules flows, it will create a ‘scheduled’ state for a flow and when the right time comes, the flow will be picked up by the Prefect agent and will be submitted for execution. Prefect doesn’t use a configuration file. You can pass configurations with a flow in the code, so you can have different configurations for each flow.
A useful feature of Prefect Cloud is that automations can be made based on a flow. This allows you to be notified immediately if a flow fails.
Prefect’s hybrid execution model
Prefect uses a hybrid model, which means that code created in the Prefect Core will never be visible to Prefect Cloud. It never receives your code or data. It orchestrates Prefect Core, running on your private infrastructure, by exchanging state information and metadata. This is designed to meet the strict standards of major financial institutions and companies that work with regulated data.
In addition, Prefect also works well with their featured partners Snowflake, Airbyte, and dbt. As Snowflake and dbt are also DataSense’s partners, and Airbyte is extensively used within DataSense, we can show use cases or offer personalised demos to match the business requirements of your organisation.
Want to learn more? Don’t hesitate to contact us!
