// Automated Data Platform testing
Robot framework
An automated testing approach is preferable to a manual one, since it is much more time-efficient and cost-effective. Furthermore, this approach ensures a quick testing process, allowing for a timely detection of poor quality of the data warehouse. We have build an automated testing framework and approach based on the the open source Robot Framework.
// TEST AUTOMATION
Challenge
Companies dispose of enormous amounts of data, ranging from operational data over financial data, to benchmarks, social media data, … This data is then transformed into comprehensive reports to inform strategic decision-making processes. To ensure the accuracy and reliability that is expected of these reports, we must first ensure that the data itself is correct, complete, and in the right format.
However, in reality, the integrated data in many data warehouses is of bad quality. This is mainly due to release problems as a consequence of poor quality of delivery, corrupt data, …
The result is that the data becomes unusable. Consequently, decisions based on this poor quality data will likely be flawed and may even result in financial losses.
// TEST AUTOMATION
Solution
In order for the integrated data in data warehouses to be reliable and accurate, the data needs to be tested. You need to know whether the data itself is correct, complete and in the right format.
The testing process is usually largely manual. This has several disadvantages:
- The testing process is notoriously difficult due to the tremendous scale of ingested data. This makes comprehensive manual testing practically impossible, which leads to weakened data integrity and a higher risk of bugs slipping into production;
- Manual testing is error-prone, and a laborious and time-consuming process;
- Teams that rely primarily on manual testing ultimately end up deferring testing until dedicated testing periods, which allows bugs to accumulate;
- Manual testing is not sufficiently repeatable for regression testing;
- Manual tests may not be effective in finding certain classes of defects.