However, this migration is turning out to be harder than it sounds. Moving data and code must be well planned before executing.
First and foremost, you need to appreciate that migrating data and code from an Enterprise Data Warehouse (EDW) to a Big Data (Enterprise Data Lake) is not a simple “lift and shift” exercise; it truly is an architectural change.
EDW migration to Big Data is a migration from
Legacy Architecture to a Modern Big Data Architecture
An EDW migration (code and data) to a Big Data Environment (Cloud or otherwise) is an exercise in migrating from a legacy architecture to a modern architecture.
We recommend that you design your target architecture with the following considerations:
The conversion is of particular significance because it is complex in nature. This is especially true for code migrations where there will be multiple views, built-in functions, and thousands of tables.
You need to consider the structure of the statements; some will be simple statements, others moderate and some will be complex. The complexity of the statement drives the work effort to rewrite the language statement in another language (such as Hive, Spark). For example, a developer can convert 1 simple statement in 1 hour (minimum effort for a task) and a moderate statement about 2-4 hours and a complex statement can take even a couple of days.
Critical success factors of a migration effort include:
Performance tuning is the work effort to optimize both the code and the target environment.
Performance is often overlooked, probably because optimizing code requires highly technical staff and predicting the performance of code in the target environment is difficult.
Performance tuning involves several techniques, mainly gathered from years of experience and trial/error. There is no easy “one size fits all” method, you need to select the right approach and technology partner who can provide the appropriate experience. Otherwise, you risk migrating a lot of code and data, but it will run slow.
Next Pathway’s Migration Accelerator Toolkit contains
all the necessary components for a successful migration
Cornerstone is a fully automated, metadata-driven data platform automating the ingestion, technical standardization, security, metadata capture and lineage of data into Big Data environments or the Cloud.
Cornerstone removes the need to manually code or write “ETLs” to move data. Enabling users to ingest all types of structured and unstructured data in batch, streaming and direct-to-database methods; and land the data in various target formats – entirely driven by metadata.
Through its self-service model, Cornerstone greatly accelerates the time to market for data consumption, without a single line of code being written.
Fuse persists business data domains into a physical data model that is designed to provide historical storage of data coming in from multiple operational systems.
Fuse provides a solution that deals with requirements such as conforming data from various source systems and addresses auditing, tracing of data, loading speed and resilience to change issues.
Shift can accelerate the translation of up to 80% of the most complex code, and the output is always consistent, ensuring compliance with industry coding standards.
Shift allows organizations to minimize the cost and manual effort involved in many migration-oriented projects, including decommissioning and migrating an enterprise data warehouse to a Data Lake, source system rationalization, or modernizing legacy ETLs.
Our Performance Playbook is a collection of techniques to optimize both the migrated code and target environment. It is based on best practices from multiple migration initiatives and years of experience in Big Data tools and technologies.
To learn more about our Migration Accelerator Toolkit, and how we can assist with your migration efforts, visit us at www.nextpathway.com