While looking at migrating your enterprise from one Data Integration/ETL platform to another, it becomes essential to ask yourself three fundamental questions. These questions are essential to any large scale program effort and make no mistake, your migration is one of those as well. These three questions will help define your migration strategy and help you avoid the common pitfalls in execution.
This article steers itself clear of the open source vs. proprietary debate or indulge in a comparison of platforms for Data Integration/ETL while looking at Migration strategies. That is a discussion for a later date.
When looking at migrating your Data Integration or ETL platform to another, you need to really look at the WHY behind it.
It’s not just the platform that adds or takes away value, it’s the environment it operates in that influences the platform in a large way.
It could be that the custom legacy code to update four different databases is insufficient to handle your increased DW & BI needs. Or you might be missing new platform feature sets like Data Profiling, Data Quality, Advanced Scheduling Features, etc. For example, the value for one Fortune 500 client was derived from being able to separate out the business rules from the operational batch which were tightly coupled and thereby less adaptable to changing the business strategy.
The vendor licensing strategy for platform and associated features it provides may be less than adequate over the course of time which forces you to reevaluate the impact on your departmental budget and productivity.
In any case, migrating your ETL system to a different platform is like running a marathon. In the short term, there is a distinct possibility that the migration itself costs more in terms of time and effort than the license agreement with the existing vendor for another year but over the long term would be more than worth cost borne.
The platform you pick needs careful consideration of not only the licensing costs but the Developer learning curve and other feature considerations like Data Profiling, Data Lineage, Data Quality, Scheduling, Source Control, connectors to Big Data platforms etc. From an administration stand point, vendor support, Ease of Maintainability, Ease of Administration and has the ability to scale up in terms of supporting the increasing demands in quantum of processing without any performance degrades. And having a good debate with your architect on open source and proprietary would come in handy.
There was a classic comment on the Windows XP migration blog, “If you are thinking about moving your enterprise off Windows XP NOW- you are already too late.” This was April 2013 when there was a year still left for MS to support XP.
Very often, licensing agreements run till the end of the calendar year which coincides with the reporting and real time information needs of business. If the operations or technology people cannot guarantee the up-time of the warehouse and the relevance of its data, the business folks cannot depend on its predictive analysis and risk assessment of marketing or sales strategies.
Timing your migration with a view and in cooperation with the business team is critical to ensuring the success of your migration.
Moving from one ETL platform to another is a time consuming process which needs thorough planning and execution. At BitWise, some of the critical migrations we have been involved with have ranged from 18 man months to 240 man months. What can help reduce the effort is building platform reassessment into life cycles of the applications using the tools and plan moving across vendors based on these reassessments.
Adding value by cutting costs (and not corners) is the way forward for today’s lean enterprises. This is challenging the set paradigms for design and architecture to offer solutions that don’t necessarily reinvent the wheel – but do away with it altogether.
It is forcing stakeholders like analysts, modelers, and design and development teams to leave the silos they work in and come up with an integrated approach. Like separation of critical business SLAs on proprietary products and leaving non critical functions on open source. The integration of the two then takes on its own criticality and needs to be treated with the same respect the SLA linked business functions get.
Before jumping on any new platform it is essential to run a scaled down version of your enterprise application in a Proof of Concept.
Furthermore, it would be a good idea to break down the application into segments that need to be migrated vs. elements that can easily or seamlessly be ported over to the new platform. Scripts that are used by the enterprise authorized batch scheduler and XML Metadata that can be parsed to generate Source to Target mappings are few of the examples of elements that are easily ported. However, custom scripts that are specific to administration or managing the execution of the product in your environment do not port easily and are better redone from scratch.
Metadata management and Data Lineage tracking is a key element of any enterprise Data Integration/ETL solution and the most appropriate platform for your enterprise should be including this as a focused offering. Besides the help it provides the Analysts, having effective data lineage in place reduces time required for setting up test data for the migration team.
As is with every critical system, budget for extensive testing and data validation while also allowing parallel execution of the old and the new systems with the same input data sets for Production validation.
To efficiently get this done, it may be a good idea to seek out tools/services that can automate such migration and associated testing from source to target platform. Having a data validation suite in place that can be used by both the migration team as well as the end business users helps reduce turnaround time for the UAT. This would save the efforts of manual conversion and bring in efficiencies both from cost and time to market perspective. BitWise’s AXES is a good candidate for such automated migration service offering which can serve as your starting point.
And, with a view of assessing the platform you are migrating to in a couple of years down the line, ensure the thought process behind a particular course of action is as well documented as the action itself.
Umesh has extensive experience in Data Integration on various platforms ranging from very high end proprietary platforms to open source tools. He has successfully consulted & delivered various development, re-engineering and migration projects in Data Integration for our Fortune 500 clients. The assignments Umesh has completed are considered as benchmark solutions used as reference implementations by teams across Bitwise