// Agile
Multiple Speed Approach
Agile methodologies have known a serious uprising in the last few years. It is therefore that we have personalized an agile methodology to a “multiple speed approach”.
// Multiple Speed Approach
Agile methodologies have known a serious uprising in the last few years. It is therefore that we have personalized an agile methodology to a “multiple speed approach”.
This approach allows us to divide a Data Hub, as defined in our previous blog post, in two main parts. The first part is the integration of all data source systems, while the second part will be for the consumption of the data for analytics and applications.
The integration layer contains all integrated and historicized source data without applying any business rules (transformations, filters, functions). This means that the data captured in this layer represents the version of the facts. It is a more technical layer that reflects the data as it is stored within the data sources.
When moving further to the consumption layer, business rules will be applied for data analytics and applications purposes. The raw data captured in the integration layer will be transformed in the consumption layer and will represent the version of the truth. It is a functional / business layer where the data will be transformed into valuable and understandable data products.
How does this approach solve the challenge?
- Business requirements:
Business requirements will change over time because today’s use case differs from the one within a day, month, year,… That’s why the consumption layer should be resilient to change. By dividing the data hub into two layers, the consumption layer can change whenever the business wants because all data will be kept in the integration layer without changing it by applying business rules. - Divide and conquer:
- When implementing the data hub, we split up the team in two sub teams. One team will focus on integrating data entities coming from different data sources. All functional relevant data will be integrated, even if the end user does not need this data at that moment. This process is automated as much as possible by applying data warehouse automation, continuous integration and continuous delivery (CI/CD). This approach offers organizations scalability and flexibility in adding new data sources and new data entities of existing data sources.
- The other team will focus on the transformation of the raw data (stored in the integration layer) into valuable data products which will be used by business stakeholders. When requirements change and f.e. new data is needed in the consumption layer, it can be sourced from the integration layer because all functional relevant data is present in this layer.