Cloud Data Migration

Data Migration from On-Premise to GCP Cloud

Data Migration to GCP Cloud

A major payment processing firm based out of the United States wished to build a cloud-based big data strategy that the enterprise, as well as external users, could leverage to build better analytics and make better business decisions. One of the key components and challenges in the build process was migration of on premise data to the cloud.

Client Challenges and Requirements

  • Migration of 15+ TB of uncompressed data between geographically separated locations
  • Maintaining on-premise cluster performance so as to have a balance between data migration load and current user workload 
  • On-premise data in different file formats and compressions
  • Storage, data type and other compatibility changes between on premise and Cloud
  • Keeping migrated historical data in sync with on-premise cluster

Bitwise Solution

test (4)

Schema conversion, data extraction, ingestion and data load through GCP Migration agent

web (3)

Bitwise Cloud Data Migration Framework for Trigger based data load to ingest historical data once available in cloud storage bucket

Group 7

In-built DQ, validation checks and Bitwise Cloud Data Migration Framework’s full and sample validation utilities to validate migrated historical data to ensure accuracy

framework

Implemented throttling and elasticity for optimal load on source database and maximum extraction performance through Bitwise Cloud Data Migration Framework

synchronize (1)

CDC implementation through GCP Migration Agent to keep migrated historical data in sync with on premise cluster

Group 3

Dedicated 1gbps network connection between on premise Hadoop cluster and Google cloud to avoid network congestion

Tools & Technologies We Used

Unix

Google BigQuery Data Transfer Service

Shell Script

Gcloud SDK

BigQuery

Bitwise Cloud Data Migration Framework

Key Results
Group 3

Successful migration of historical data ensuring data integrity

trust

Automated Data validation resulted in 30% reduction in migration time

diagram (6)

Ease of scalability and enhancements

Share This Case Study

Share on linkedin
Share on twitter
Share on whatsapp
Share on telegram

Download Case Study

    Bitwise provides comprehensive solutions for all your cloud data migration projects

    Related Solution

    Cloud Data Migration

    Our ready-to-go Cloud Migration Roadmap helps guide you through your initiative for moving on-premise data to the cloud
    Ready to start a conversation?

    Share This Case Study

    Share on linkedin
    Share on twitter
    Share on whatsapp
    Share on telegram

    Download Case Study

    Bitwise provides comprehensive solution for all your data projects

    Related Solution

    Automated ETL Migration

    Risk-free conversion and optimization of source ETL jobs to a target ETL tool with maximum automation

    Ready to start a conversation?