Architecting for Change: How We Helped a Leading U.S. Insurer Cut Technical Debt by 97% and Modernize at Scale
Case Study
Payments
Cloud Data Migration from Netezza to Google BigQuery
A major payment processing firm based out of USA wishes to build a cloud based big data strategy that the enterprise as well as external users can leverage to build better analytics and better business decisions. One of the key components and challenge in the build process was migration of on premise data to cloud.
Client Challenges and Requirements
- Migration scope included 23TB of data, 1500 tables and 1200 plus jobs running on Netezza
- Maintaining Netezza batch process performance (ensuring compliance with defined SLA) so as to have a balance between data migration load and regular batch process load
Bitwise Solution
- Design and develop a generic process to extract data, generate delimited files and push the files to designated cloud buckets
- Generic tokenization process to tokenize PCI data before pushing to cloud
- Generic de-tokenization process to include actual value in output extracts
- Auto generation of BigQuery YAML (for table creation) using Netezza metadata
- Automatic validation of historical data migration
- Build the cloud migration architecture and solution design
- Compile the phased migration plan aligned with business priority and timeline
- Adoption of PaaS model – managed service pipeline for processing and storage
- Cloud native services for designing ETL workflows and scheduling
- Integrated Data Quality Framework
- Build a multi layered Data Lake for quick and controlled access to data layers (raw, transformed, aggregated)
Tools & Technologies We Used
Netezza
Ab Initio
Shell Script
Python
Control M
GCS
Dataflow
BigQuery
Cloud SQL
Airflow
Key Results
Seamless migration of the DW on to GCP
Successful migration of 23TB of historical data ensuring data integrity
Automated Data Validation resulting in 25% effort saving
Auto generation of BQ YAML from Netezza DDL resulted in 85% effort savings associated with DDL creation
Client Challenges and Requirements
- Migration scope included 23TB of data, 1500 tables and 1200 plus jobs running on Netezza
- Maintaining Netezza batch process performance (ensuring compliance with defined SLA) so as to have a balance between data migration load and regular batch process load
Bitwise Solution
- Design and develop a generic process to extract data, generate delimited files and push the files to designated cloud buckets
- Generic tokenization process to tokenize PCI data before pushing to cloud
- Generic de-tokenization process to include actual value in output extracts
- Auto generation of BigQuery YAML (for table creation) using Netezza metadata
- Automatic validation of historical data migration
- Build the cloud migration architecture and solution design
- Compile the phased migration plan aligned with business priority and timeline
- Adoption of PaaS model - managed service pipeline for processing and storage
- Cloud native services for designing ETL workflows and scheduling
- Integrated Data Quality Framework
- Build a multi layered Data Lake for quick and controlled access to data layers (raw, transformed, aggregated)
Tools & Technologies We Used
Netezza
Ab Initio
Shell Script
Python
Control M
GCS
Dataflow
BigQuery
Cloud SQL
Airflow
Key Results
Group 3
Seamless migration of the
DW on to GCP
code (1)
Successful migration of 23TB of historical data ensuring data integrity
test (4)
Automated Data Validation resulting in 25% effort saving
Group 3
Auto generation of BQ YAML from Netezza DDL resulted in 85% effort savings associated with DDL creation
Share This Case Study
Share on linkedin
Share on twitter
Share on whatsapp
Share on telegram
Download Case Study
[contact-form-7 id="39856" title="Case Study-Pardot: Cloud Data Migration from Netezza to Google BigQuery"]
Bitwise provides comprehensive solution for all your data projects
Related Solution
Cloud Data Migration
Our ready-to-go Cloud Migration Roadmap helps guide you through your initiative for moving on-premise data to the cloud
Learn More
Ready to start a conversation?
Contact Us
Related Insights
