Cloud Data Migration

On-premise to Cloud Data Replication with AWS Glue

case-study-feature-img

A Global Research and Advisory Company intended to modernize its on-premises ‘Enterprise Data Quality’ application by migrating it to the AWS cloud. Bitwise spearheaded the initiative and replicated the on-premises Oracle data to PostgreSQL on AWS.

Client Challenges and Requirements

  • Replicating data from on-premise to AWS cloud
  • Scrapping Materialized View for data replication and creating a process that is Cloud native, scalable, secure, and open for modifications
  • Build a metadata-driven, scalable data replication system using AWS services
  • Trigger based on source system availability (online/offline)
  • Orchestrate data replication processes

Bitwise Solution

  • Developed a metadata-driven AWS-based data replication process that is re-runnable, fault-tolerant, and triggered by Rest API linked to the existing scheduler
  • API triggers the step function that orchestrates all other services
  • Lambda: Monitors database health. System up/down status. Reads Metadata table, returns the Staging table name, and metadata query to the Step function
  • Map: Receives the details of all the tables in a particular workflow and creates ‘n’ number of virtual instances if there are ‘n’ number of tables present
  • Virtual instances contain the glue job and a Lambda. Virtual instances run in parallel ensuring all the tables in the workflow are replicated into staging parallelly
  • Glue: Connections are created and used for on-prem cloud databases and unloading takes place as truncate and load in the staging table in PostgreSQL
  • Lambda: Loads the data from the staging table to the target table in PostgreSQL

Tools & Technologies We Used

AWS Glue
AWS Lambda
PostgreSQL (AWS RDS)
AWS Step-Function
Rest API
SQL

Key Results

skill-icon

A cloud native application that is reliable, scalable and secure

skill-icon

Metadata driven process which has re-runabilty, fault tolerance and is open for modifications

skill-icon

Fully operational and unit-tested code delivered with ~40% overall time efficiency

Download Case Study

    To get our latest updates subscribe to our Newsletter.

    Ready to start a conversation?