PROJECT OVERVIEW
Team Emorphis engaged in the design, development, and execution of a data migration strategy. There are complex transformation rules, based on business scenarios to accurately migrate the data to the new schema.
With continuous migration of new data flowing into the source database, the new system and data run parallel to the old system in read-only mode to detect any glitches.
This involved creation of a staging database, which then was initialized with the most recent backed-up data, to carry out historical data migration.
ABOUT THE CLIENT
The National Livestock Identification System (NLIS) is a scheme for the identification and tracing of livestock and is crucial in protecting and enhancing its reputation as a producer of quality beef and sheep.
The country exports to over 100 markets and is one of the world’s largest animal exporters. The NLIS allows the country’s ongoing access to these valuable export markets.
THE REQUIREMENT
The client system is in place for tracking livestock through the complete lifecycle. This starts with bringing animals into the system by implanting RFID tags and then tracking location, health condition, and transfer through different properties, owners, and sale yards.
The existing system is in operation for close to two decades and provides a web-based interface that can be used to feed data into the system.
It has accumulated a huge amount of data, running into terabytes, over its lifetime. This data is stored in an RDBMS. The client is completely revamping the current system, to add new features, and services.
The database schema has also been altered to support new services, and also to record more information related to existing features. To completely migrate to the new system, a mechanism to migrate data from the old schema to the new one is needed.
THE SOLUTION
Emorphis engaged with the client to design, develop, and execute a data migration strategy. The strategy needed to take into account several things:
- Complex transformation rules, based on the business scenarios, to correctly migrate the data into the new schema.
- Simultaneous runs: The new system needed to be run in parallel with the old system in read-only mode for some time, in order to detect any problems with the migrated data. This required continuous migration of new data flowing into the source database.
- The downtime required for migration needed to be minimized.
With these aspects and strategy in consideration, we executed a one-time migration of historical data and migrated data on a continuous basis during parallel runs.
This involved creation of a staging database, initialized with the most recent backed-up data, to carry out historical data migration.
Replication is set up between the source database, and the staging database to allow the flow of new data into delta tables in staging, during the parallel run. Stored procedures and SQL server jobs are used to validate, transform, and load data from the staging database into the target. Migration statistics are gathered in separate tables in the staging database that included counts of migrated rows, along with details of data validation failures, due to missing/inaccurate data in the source. A notification mechanism is set up to alert the concerned person if the migration process encountered unexpected errors.
SALIENT FEATURES
APPLICATION BENEFITS
TECHNOLOGY
- Seamless migration of data and information from existing systems to the new schema.
- Enhanced error flag as a result of existing and new system parallel run.
- Minimized downtime requirement.
- One-time migration of historic data.
- The continuous migration of current data sets.
- A notification mechanism was set up to alert in case of migration process encountered unexpected errors.
- Staging database initialized with backed-up data, to carry out historic data migration.
- Replication between the staged source database and the staging database allows the flow of new data.
- Stored procedures and SQL server jobs to validation transformation, load data in staging database including migration of rows, data validation failures as a result of missing/inaccurate data sources.
- SQL Server Enterprise Edition
- SQL Server Integration Services
About Emorphis
Emorphis Technologies is a world-class software development and solutions company that truly believes in “Innovation in motion”. Delivery innovation on the go at an accelerated pace has been our success mantra to date. Over the years we have provided value to our clients in the field of enterprise mobility, cloud, IoT, backend development, Big Data Analytics, and Blockchain.
We serve industries ranging from unicorns, and startups to large multinationals in the healthcare, telecommunications, fintech, retail, and publishing industry. Our go-to-market software products – iStatement, iPublisher, and iBuggy have proved our metal with positive beneficial customer testimonials. We help our clients with successful product development, consulting services, and testing (manual & automated).
We have profound experience & expertise in various technologies like .Net, J2EE, PHP, iOS, Android, and Cloud Computing viz. Amazon Web services (AWS), Software QA & testing (Manual & Automation). Our designed products are cloud-ready and can be readily deployed on AWS/Azure cloud infrastructure.
Our pivot on engineering innovation and R&D helps quicken time-to-market, ensuring high quality at economies of scale, delivering cult competency for the global marketplace. We ensure that your ideas, concepts, and requirements are backed by brilliant execution at our end. Having said that we extend end-to-end ownership of product/application design, development, and deployment.
