Data Warehouse Modernization
Migrate On-Premise Control-M Jobs to Airflow
A leading worldwide provider of payment technology and software solutions needed to migrate its on-premise Control-M tool that was used to schedule ETL jobs to Google Cloud using Airflow service to reduce IT costs in accordance with its cloud adoption roadmap.
Client Challenges and Requirements
- Client chose to use Airflow scheduler on Google Cloud, so migration of older jobs was needed using Airflow
- Maintaining Control-M jobs was adding to the IT costs
- Jobs being migrated to Cloud using Airflow were required to meet all the scheduling requirements as defined on the existing on-premise
Bitwise Solution
- Analyze and document the list of on-premise Control-M jobs holding schedules of multiple on-premise ETL processes and its associated SLA.
- Took multiple runs of sample process with different environment configurations on Cloud Composer/Airflow to have best performance with minimal use of resources to ensure lower costs.
- Scheduled cloud ETL jobs using GCP Cloud Composer which is built on top of Apache Airflow to generate the reports as per SLA.
- Created customized tasks (operators) as per the requirement that could be used by multiple jobs for performance improvement. Generation of generic utilities for Airflow jobs to have code reusability.
- Implemented auditing of all the jobs at the backend to ensure data quality.
- Compare the completion of Control-M jobs and the new Airflow jobs/dags for validations. Retire the Control-M jobs and direct all triggers to Cloud Composer.
- Perform continuous monitoring of the Airflow jobs with the help of dashboard.
Tools & Technologies We Used
Python
Cloud Composer
Apache Airflow
Key Results
Setting up of environment is easier and no maintenance required at our end
Automatic scaling by GCP Composer improves performance of the scheduled Airflow jobs
40% saving in migration efforts due to common operators and automation
