Data Warehouse Modernization
Converting DataStage ETL to SSIS for Leading Insurance Company
Bitwise helped a large, US Insurance company successfully migrate their Data Integration (ETL) platform from DataStage to Microsoft SSIS. The first and primary step in this migration was to complete the inventory analysis, and then scope the exact ETL jobs to be migrated. The approach included a collaborative effort with client teams and the STIMA utility (an automated assessment tool put together by Bitwise), which helped in laying the foundation for the ETL Converter by analyzing the items below:
- Complexity definition of each ETL job
- Identification of common features across all the ETL jobs
The Bitwise ETL Converter is customized based on the information captured that makes the automation highly cost-effective in terms of reducing the overall development efforts. The converter is additionally bundled with Data Validation automation that enormously reduces
the turn-around time to productionalize the new solution.
The Bitwise ETL Converter enables the automation of the migration process, thereby minimizing the time-to-production and brings in better quality within the new Data Integration platform. Its support for various ETL Tools and easy customization, based on the client’s needs, enables it to lead the market in offering a dynamic and complete migration toolset.
The Client
A NASDAQ listed, $6 Billion company, providing affordable Insurance products across United States since 1925 with more than 5 million customer base.compatibility testing due to these changes
Client Challenges and Requirements
The client had to move their entire data integration platform from IBM DataStage to Microsoft SSIS, covering the following aspects:
- Development of the SSIS ETLs after analyzing the original DataStage jobs
- Unit testing, which involved the extensive field level comparison of DataStage outputs against those generated by newly developed SSIS ETLs
- System integration and performance testing
- Deployment to stage environment followed by Production environment
All of the above aspects had to be covered for about 700 Datastage jobs spread across 22 subject areas within the fixed timeline of 4 months to avoid Datastage license renewal. The jobs were of varied complexity with 42% jobs in the Medium-to-Complex category.
Bitwise Solution
Automated assessment with Dark Data Discovery framework
Phased migration approach
Consistency and standardization across the ETL Conversion jobs
Automated data validation
Repeatable process
Key Results
~22% Reduction during Assessment Phase and Inventory Cleanup
~55% Reduction in Development Effort
~25% Effort savings in Data Validation
~65% Effort savings in Defect cycle efficiencies brought automation consistencies
