Data Migration

Cloud Data Migration from Netezza to Google BigQuery

A major payment processing firm based out of USA wishes to build a cloud based big data strategy that the enterprise as well as external users can leverage to build better analytics and better business decisions. One of the key components and challenge in the build process was migration of on premise data to cloud.

Client Challenges and Requirements

  • Migration scope included 23TB of data, 1500 tables and 1200 plus jobs running on Netezza
  • Maintaining Netezza batch process performance (ensuring compliance with defined SLA) so as to have a balance between data migration load and regular batch process load

Bitwise Solution

  • Design and develop a generic process to extract data, generate delimited files and push the files to designated cloud buckets
  • Generic tokenization process to tokenize PCI data before pushing to cloud
  • Generic de-tokenization process to include actual value in output extracts
  • Auto generation of BigQuery YAML (for table creation) using Netezza metadata
  • Automatic validation of historical data migration
  • Build the cloud migration architecture and solution design
  • Compile the phased migration plan aligned with business priority and timeline
  • Adoption of PaaS model – managed service pipeline for processing and storage
  • Cloud native services for designing ETL workflows and scheduling
  • Integrated Data Quality Framework
  • Build a multi layered Data Lake for quick and controlled access to data layers (raw, transformed, aggregated)

Tools & Technologies We Used

Netezza

Ab Initio

Shell Script

Python

Control M

GCS

Dataflow

BigQuery

Cloud SQL

Airflow

Key Results
Group 3

Seamless migration of the DW on to GCP

code (1)

Successful migration of 23TB of historical data ensuring data integrity

test (4)

Automated Data Validation resulting in 25% effort saving

Group 3

Auto generation of BQ YAML from Netezza DDL resulted in 85% effort savings associated with DDL creation

Share This Case Study

Download Case Study

    To get our latest updates subscribe to our Newsletter.

    Bitwise provides comprehensive solution for all your data projects

    Related Solution

    Cloud Data Migration

    Our ready-to-go Cloud Migration Roadmap helps guide you through your initiative for moving on-premise data to the cloud
    Ready to start a conversation?

    Share This Case Study

    Download Case Study

    Bitwise provides comprehensive solution for all your data projects

    Related Solution

    Automated ETL Migration

    Risk-free conversion and optimization of source ETL jobs to a target ETL tool with maximum automation

    Ready to start a conversation?