OVERVIEW

In order to achieve their goal of enhancing data accuracy, scalability, and faster availability, the leading global research and advisory company partnered with Bitwise to migrate their on-premise application to Azure. The client wanted to improve their reporting and analytics applications by ensuring that users have access to timely data and uninterrupted data availability for business intelligence functions. Bitwise provided a solution to migrate the client’s data to the cloud efficiently and reliably, using Python- based ETL and data validation accelerators.

WHY BITWISE?

Bitwise offers solutions to businesses facing challenges with migrating their data from on-premises systems to the cloud. With the increasing demand for improved flexibility, scalability, and cost-efficiency, many companies are moving their data to the cloud, but this process can be complex and time-consuming. Bitwise has aligned with the global market vision for efficient ETL and data migration and has developed Python-based ETL and data validation accelerators that are cloud-agnostic and can work with popular cloud platforms like Azure, GCP, AWS, and more. These accelerators can read on- prem databases and file sources, making it easier to migrate data to the cloud. The accelerators developed by Bitwise can generate meaningful validation reports for faster defect management, integrate with CI/CD tools like Jenkins and TeamCity, validate millions of records, and automatically generate validation SQLs. These features can significantly reduce the time and effort required for the migration process. In addition, Bitwise’s accelerators focus on data validation and code/script comparison and DDLs, ensuring that the migration process is comprehensive and reliable. As a result of our effective solutions, we have recorded an average of 45% savings across multiple migration projects. Our accelerators have proven to be successful in reducing effort and cost, making them a valuable asset to businesses looking to migrate their data to the cloud efficiently and reliably.

CLIENT CHALLENGES

Bitwise client faced several challenges in testing a large number of tables that were part of their ETL process. Some of the key challenges include:

• The client needed to test more than 1100 tables, which is a significant number. Manually testing each table is time-consuming, and it is not feasible to manually test all tables within the available time and resources.
• Manual testing had been limited to a small dataset, reducing its effectiveness and potentially missing undiscovered issues.
• Appropriate ETL testing automation tools were required for accurate and efficient testing of all tables.
• Data quality and referential integrity issues were complex and required attention to ensure accurate and reliable data. Data type differences between Oracle and Synapse were causing ETL errors, truncated data, and special character issues and had to be resolved to maintain consistency and accuracy.
• A flexible and intelligent testing strategy was required to test production data for incremental loads each day, considering that CDC loads run at different times for both DB systems.

BITWISE SOLUTION

To address the challenge of efficient database testing, our testing experts implemented a range of solutions.

• We created automation scripts to eliminate the need for manual data comparisons, saving considerable time and effort. Additionally, we developed a DDL mapping script to compare the structure of on-premises and cloud-based database systems, ensuring consistency and accuracy.
• To catch errors early, our team prioritized DDL validation over data validation. This addressed typical issues resulting from erroneous DDL mappings such as special character conversion, data accuracy, and truncation. This approach ensured accuracy and consistency while minimizing the risk of issues in production.
• Our team assisted the client in adopting an optimal data type testing strategy to achieve good performance and save space in the database.
• We grouped test cases in test plans and executed them in parallel, reducing testing time while ensuring thorough coverage of all necessary test cases.
• To minimize the risk of issues in production, testing was planned and coordinated in lower environments. We focused on base tables first, correcting errors caused by code issues so that they don’t spread to dependent tables. This approach helped to ensure a more stable and reliable database system, minimizing the risk of errors and issues in production.

KEY RESULTS

• Implementation of automation scripts and optimal testing strategies resulted in accelerating the testing process.
• Manual testing efforts were reduced by 60-70% due to the use of custom automation tools, and the UAT defects were reduced by 98-99%.
• The result was fully tested and defect-free tables were delivered with 100% test coverage.

These outcomes demonstrate the effectiveness of the measures taken by Bitwise in improving the quality and efficiency of the testing process.

TOOLS AND TECHNOLOGIES USED