On-premise to Cloud Data Replication with AWS Glue

A Global Research and Advisory Company intended to modernize its on-premises ‘Enterprise Data Quality’ application by migrating it to the AWS cloud. Bitwise spearheaded the initiative and replicated the on-premises Oracle data to PostgreSQL on AWS.

Client Challenges and Requirements

• Replicating data from on-premise to AWS cloud
• Scrapping Materialized View for data replication and creating a process that is Cloud native, scalable, secure, and open for modifications
• Build a metadata-driven, scalable data replication system using AWS services
• Trigger based on source system availability (online/offline)
• Orchestrate data replication processes

Bitwise Solution

• Developed a metadata-driven AWS-based data replication process that is re-runnable, fault-tolerant, and triggered by Rest API linked to the existing scheduler
• API triggers the step function that orchestrates all other services
• Lambda: Monitors database health. System up/down status. Reads Metadata table, returns the Staging table name, and metadata query to the Step function
• Map: Receives the details of all the tables in a particular workflow and creates ‘n’ number of virtual instances if there are ‘n’ number of tables present
• Virtual instances contain the glue job and a Lambda. Virtual instances run in parallel ensuring all the tables in the workflow are replicated into staging parallelly
• Glue: Connections are created and used for on-prem cloud databases and unloading takes place as truncate and load in the staging table in PostgreSQL
• Lambda: Loads the data from the staging table to the target table in PostgreSQL

Tools & Technologies We Used

Key Results

A cloud native application that is reliable, scalable and secure

Metadata driven process which has re-runabilty, fault tolerance and is open for modifications

Fully operational and unit-tested code delivered with ~40% overall time efficiency