Usually, business stakeholders are no data experts. They are primarily interested in getting valuable insights out of data, upon which they can take the appropriate actions. As a result, it is difficult for stakeholders to know what is possible until an analyst shows them. Furthermore, they have no idea about the effort it requires to deliver valuable insights.
In today’s fast changing market, it is rather impossible for business stakeholders to define clear and feasible goals upfront as they are not even aware what these goals might be. Moreover, when an opportunity presents itself, the company needs to explore and act on it faster than its competitors. Consequently, requirements often come in very late with tight deadlines. Even when the analytics team is finally able to deliver the required insights, the objectives might have already changed, or it is not entirely how business expected it to be.
The delivered analytics can also generate more questions than it provides answers because it has enabled stakeholders to understand the business in new ways which leads to a new series of questions, requirements and objectives.
Agile methodologies have known a serious uprising in the last few years. It is therefore that we have personalized an agile methodology to a “multiple speed approach”.
This approach allows us to divide a Data Hub
, as defined in our previous blog post, in two main parts. The first part is the integration of all data source systems, while the second part will be for the consumption of the data for analytics and applications.
The integration layer contains all integrated and historicized source data without applying any business rules (transformations, filters, functions). This means that the data captured in this layer represents the version of the facts. It is a more technical layer that reflects the data as it is stored within the data sources.
When moving further to the consumption layer, business rules will be applied for data analytics and applications purposes. The raw data captured in the integration layer will be transformed in the consumption layer and will represent the version of the truth. It is a functional / business layer where the data will be transformed into valuable and understandable data products.
How does this approach solve the challenge?
- Business requirements will change over time because today’s use case differs from the one within a day, month, year,… That’s why the consumption layer should be resilient to change. By dividing the data hub into two layers, the consumption layer can change whenever the business wants because all data will be kept in the integration layer without changing it by applying business rules.
- Divide and conquer: