Automated ETL Migration of DataStage to Azure Data Factory

A Fortune 500 multinational department store corporation needed to modernize its legacy ETL platform to a cloud-based solution using Azure as the preferred cloud service provider and Azure Data Factory (ADF) as the cloud-native ETL tool.

Client Challenges and Requirements

• Upgrade the underlying technology stack to Azure cloud, converting 124 Parallel jobs and 22 Sequence jobs from DataStage to ADF within fixed time and cost deadlines.
• Smooth migration of ETL jobs from DataStage to ADF, ensuring minimal disruption to client’s existing operations.
• Migrate the ETLs in stages to minimize sudden operational impacts.
• For a certain period, run both DataStage and ADF simultaneously to fix any abrupt issues and ensure a seamless transition.
• Employee training to manage converted ADF pipelines.

Bitwise Solution

• Bitwise used its migration services and proprietary automated ETL Conversion tool to expedite the timeline for migrating the DataStage ETLs to ADF while adhering to budgetary requirements.
• Orchestrate the converted ADF pipelines using ADF native features. This was one of the significant highlights of the ETL migration project.
• Develop generic functionality using Azure Functions for sending emails. This feature was crucial in achieving operational efficiency. Another generic functionality was created using Azure Functions for file validations. The objective was to streamline file validations and maintain stringent data quality checks.
• Carry out performance optimization of long running jobs in ADF to enhance the efficiency of the system.
• Conduct meticulous row-by-row and column-by-column data validation between DataStage and ADF output utilizing Bitwise's Python based data validation accelerator to ensure the consistency and conformity of data.

Tools & Technologies We Used

Key Results

Improved Code Efficiency: Reduced 180+ UNIX commands/scripts to 15 Azure Functions

Optimized performance of long-running jobs

Accelerated Time-to-Production: Conversion completed in 12 weeks using automated ETL migration

Rigorous Data Validation: Up to 50% effort savings using Bitwise data validation accelerator