Data Warehouse Modernization
ETL Offload in a Hadoop Environment using Hydrograph
Bitwise helped a large Fortune 500 company save millions of dollars and an estimated 30-50% time in ETL development through utilization of the Bitwise proprietary ETL migration accelerator, offloading from a costly legacy platform to Hadoop.
The Client
Large, $25Bil financial company located in the United States.
Client Challenges and Requirements
- The company was looking to pivot from a traditional “closed” system for ETL development to a more modern, open-source environment. In addition, this company wanted to address its business strategy and become more technology aligned and nimble in order to continue to lead the market in innovation and consumer product offerings. The legacy technology did not address new use cases, scalability, increasing data size and corresponding cost, modern user experience, and managing performance with increased processing.
Bitwise Solution
Our proprietary ETL migration accelerator automates the migration of any ETL code into the XML format, which can be directly executed on Hadoop. The solution was designed to provide a unified developer experience for engineers throughout the company, whether they work on MapReduce, Spark, Storm, Flink or SQL, as well as accelerate data transformation on the data lake (or in other Big Data environments). This was achieved by implementing an easy-to-use developer productivity tool and GUI, which allows developers to interact with a drag-and-drop environment to create
new transformations or update/edit existing jobs, and visualize existing workloads to
facilitate ongoing management.
This approach enabled the company to realize their initial goals, as well as create a suite of extensions and tools to provide IDE-like functionality for ETL by utilizing Cascading so that it offers an abstraction layer over Hadoop, which is ideal for creating complex data transformations.
Key Results
Ensures future-proof business investment by retaining code and business logic at first-principal-abstraction layer using XML
70% Touch-Free Migration
50% Increase in Developer Productivity
75%-90% Effort Saved for Test Compliance Reports with QualiDI
