ETL Offload in a Hadoop Environment using Hydrograph

etl offlaod in a hadoop environment
Bitwise helped a large Fortune 500 company save millions of dollars and an estimated 30-50% time in ETL development through utilization of the Bitwise proprietary ETL migration accelerator, offloading from a costly legacy platform to Hadoop.

The Client

Large, $25Bil financial company located in the United States.

Client Challenges and Requirements

The company was looking to pivot from a traditional “closed” system for ETL development to a more modern, open-source environment. In addition, this company wanted to address its business strategy and become more technology aligned and nimble in order to continue to lead the market in innovation and consumer product offerings. The legacy technology did not address new use cases, scalability, increasing data size and corresponding cost, modern user experience, and managing performance with increased processing.

Bitwise Solution

Our proprietary ETL migration accelerator automates the migration of any ETL code into the XML format, which can be directly executed on Hadoop. The solution was designed to provide a unified developer experience for engineers throughout the company, whether they work on MapReduce, Spark, Storm, Flink or SQL, as well as accelerate data transformation on the data lake (or in other Big Data environments). This was achieved by implementing an easy-to-use developer productivity tool and GUI, which allows developers to interact with a drag-and-drop environment to create
new transformations or update/edit existing jobs, and visualize existing workloads to
facilitate ongoing management.

This approach enabled the company to realize their initial goals, as well as create a suite of extensions and tools to provide IDE-like functionality for ETL by utilizing Cascading so that it offers an abstraction layer over Hadoop, which is ideal for creating complex data transformations.

The Results

Ensures future-proof business investment by retaining code and business logic at first-principal-abstraction layer using XML

Creates a low learning curve for existing ETL developers

Provides all the benefits of underlying Hadoop platform including cost efficiency, fault tolerant, distributed processing, data integration, etc.

Eliminates the expensive lock-in costs from the legacy ETL platform

Speeds innovation

Creates a reusable framework that can be used to develop and deploy data transformations agnostic of platform

Enables fast, easy transitions between execution environments or the adoption of future environments

Provides a framework for ETL/ELT that is not tied to a single platform or development language

Key Benefits
Group 3

70% Touch-Free Migration

code (1)

50% Increase in Developer Productivity

test (4)

75%-90% Effort Saved for Test Compliance Reports with QualiDI

The Results

Share This Case Study

Download Case Study

    Bitwise provides comprehensive solution for all your data projects

    Related Solution

    Big Data Analytics & Data Science

    Uniquely positioned to help our clients utilize machine learning and artificial intelligence to achieve business results

    Ready to start a conversation?

    Share This Case Study

    Download Case Study

    Bitwise provides comprehensive solution for all your data projects

    Related Solution

    Automated ETL Migration

    Risk-free conversion and optimization of source ETL jobs to a target ETL tool with maximum automation

    Ready to start a conversation?