In data engineering, teams often receive daily full snapshots of data from legacy systems or third-party sources. Traditionally, ingesting these periodic snapshots and identifying what changed (inserts, updates, deletes) each day has been cumbersome. Engineers need to write and maintain source-specific Change Data Capture (CDC) logic for each data source. Databricks has introduced Lakeflow Spark Declarative […]
In a previous DataTalks,Wouter Pardon talked about how to implement CI/CD for Azure Data Factory and SQL Server, focusing on making deployments easier and improving the quality of data pipelines. In this post, he wants to dive deeper into unit testing for Azure Data Factory (ADF). It’ll explain how Azure DevOps can help you run […]
Managing infrastructure manually can be both time-consuming and error-prone. Setting up components like servers and databases through a graphical user interface often leads to inefficiencies, especially at the enterprise level. This becomes even more challenging when dealing with multiple environments for development, user acceptance testing, validation, and production. What is Infrastructure-as-Code? Lets introduce Infrastructure-as-Code, a […]
dbt, the Data Build Tool that has become the foundation for organizations managing data transformations. In our previous blog about dbt Mesh, we have unraveled the potential of dbt Mesh and its ability to converge different dbt functions, which enables scalability and security within and across projects. In this blog, we will discuss its evolution […]
Organizations are increasingly reliant on data warehouses to store, manage, and analyze vast amounts of information. However, the success of a data warehouse hinges not only on the quality of the data but also on the effectiveness of its underlying data model. Effective data modeling lays the foundation for a robust data warehouse architecture, enabling […]
What is dbt Power User? The dbt Power User extension accelerates dbt development within VS Code by seamlessly integrating dbt functionalities with the editor. After installation, only VS Code is required for dbt development. With the extension, users can run model SQL code or segments of it, compile Jinja into SQL and take advantage of […]
If you’re a developer, you know how challenging and time-consuming coding can be. You have to write code that follows best practices, is readable yet functional. Your code should be documented well where needed, and easily tested. Sometimes you would have to deal with complex and changing requirements, and possibly get stuck or need some […]
dbtLabs made several impressing new releases at the recent Coalesce 2023 conference. One of them is the dbt Explorer. This functionality allows you to have a clear and visually pleasing overview of your different dbt projects, all elements within them and the different access levels.
Notebooks are widely used in data science and machine learning to develop code and present the results. Databricks notebooks facilitate real-time collaboration with colleagues, creating data science and machine learning workflows in multiple languages with built-in data visualizations. // WHAT The various possibilities that Databricks notebooks can be used for: Developing code using Python, SQL, […]
In one of our previous blogposts we discussed one of the most vital tools in the modern data stack, dbt. Once you have chosen to use dbt there are two questions that remains, which version should I use and what is the difference between the two? In this blogpost we will dive into the differences […]
