ETL Migration
Application Modernization of Student Loan System
One of the largest student loan lenders in the U.S. needed to modernize its student loan system that involves the loan application, disbursement, collections and recovery process to introduce new functionality and improve accuracy to build better analytics and make better business decisions.
Client Challenges and Requirements
- Adapting and evolving to the technology needs for Spark, Scala, Hadoop and AWS
- Environmental and infrastructure issues for the existing system
- Need for 24x7 Support for the application
- Meeting SLA timeline for daily loads on PA and production environments
- Reduce dependency from product specialist
Bitwise Solution
- Develop a solution utilizing Ab Initio ETL technology and Teradata warehouse loads.
- Replace data marts with Finacle and load into the data warehouse using Hadoop.
- Create utilities using Spark / Scala to automate processes to load data into target tables.
- Implement a utility to automate the SLA monitoring process.
- Create new tables on PA environment to achieve 70-80% accuracy in testing and generate reports for SLA monitoring.
- As part of requirement change, decommission Hadoop and migrate all data through Ab Initio on AWS Snowflake / Redshift.
Tools & Technologies We Used
Ab Initio
Oracle
Hadoop
Spark / Scala
AWS (Snowflake / Redshift)
Unix/Linux
Datahub
IWS Scheduler
Tableau
Key Results
25% effort reduction using SLA monitoring utility
Enhanced security achieved using CRD data encryption with Ab Initio and Snowflake migration
40% overall run efficiency of migration with modern approach
