Team Emorphis engaged in design, development, and execution of a data migration strategy. There are complex transformation rules, based on business scenario to accurately migrate the data to the new schema.
With continuous migration of new data flowing into the source database, the new system and data run parallel to old system in read only mode to detect any glitches.
This involved creation of a staging database, which then was initialized with the most recent backed up data, to carry out historical data migration.
ABOUT THE CLIENT
The National Livestock Identification System (NLIS) is a scheme for the identification and tracing of livestock and is crucial in protecting and enhancing reputation as a producer of quality beef and sheep.
The country exports to over 100 markets and is one of the world’s largest animal exporters. The NLIS allows the country’s ongoing access to these valuable export markets.
The client system in place for tracking livestock through the complete lifecycle. This starts with bringing animals into the system by implanting RFID tags, and then tracking location, health condition, transfer through different properties, owners, and saleyards.
The existing system is in operation for close to two decades, and provides a web based interface that can be used to feed data into the system.
It has accumulated a huge amount of data, running into terabytes, over its lifetime. This data is stored in an RDBMS. The client is completely revamping the current system, to add new features, and services.
The database schema has also been altered to support new services, and also to record more information related to existing features. To completely migrate to the new system, a mechanism to migrate data from the old schema to the new one is needed.
Emorphis engaged with the client to design, develop, and execute a data migration strategy. The strategy needed to take into account several things:
- Complex transformation rules, based on the business scenarios, to correctly migrate the data into the new schema.
- Simultaneous runs: The new system needed to be run in parallel with the old system in read only mode for some time, in order to detect any problems with the migrated data. This required continuous migration of new data flowing into the source database.
- The downtime required for migration needed to be minimized.
With these aspects and strategy in consideration, we executed a one time migration of historical data, and migrating data on a continuous basis during parallel runs.
This involved creation of a staging database, initialized with most recent backed up data, to carry out historical data migration.
Replication is set up between source database, and the staging database to allow flow of new data into delta tables in staging, during the parallel run. Stored procedures and SQL server jobs are used to validate, transform, and load data from the staging database into the target. Migration statistics is gathered in separate tables in the staging database that included counts of migrated rows, along with details of data validation failures, due to missing / inaccurate data in the source. Notification mechanism is set up to alert concerned person if the migration process encountered unexpected errors.
- Seamless migration of data and information from existing systems to the new schema.
- Enhanced error flag as a result of existing and new system parallel run.
- Minimized downtime requirement.
- One time migration of historic data.
- Continuous migration of current data sets.
- Notification mechanism set up to alert in case of migration process encountered unexpected errors.
- Staging database initialized with backed up data, to carry out historic data migration.
- Replication between staged source database and staging database allowing flow of new data.
- Stored procedures and SQL server jobs to validation transformation, load data in staging database including migration of rows, data validation failures as a result of missing/inaccurate data sources.
- SQL Server Enterprise Edition
- SQL Server Integration Services
Emorphis Technologies is a world class software development and solutions company which truly believes in “Innovation in motion”. Delivery innovation on the go at accelerated pace has been our success mantra till date. Over the years we have provided value to our clients in the field of enterprise mobility, cloud, IoT, backend development, Big Data Analytics, and Block Chain.
We serve industries ranging from unicorns, and startups to large multinationals in the healthcare, telecommunications, fintech, retail and publishing industry. Our go to market software products – iStatement, iPublisher and iBuggy have proved our metal with positive beneficial customer testimonials. We help our client in successful product development, consulting services and testing (manual & automated).
We have profound experience & expertise on various technologies like .Net, J2EE, PHP, iOS, Android and Cloud Computing viz. Amazon Web services (AWS), Software QA & testing (Manual & Automation). Our designed products are cloud ready and can be readily deployed on AWS/Azure cloud infrastructure.
Our pivot on engineering innovation and R&D helps quicken time-to-market, ensuring high quality at economies of scale, delivering cult competency for the global marketplace. We ensure that your ideas, concepts and requirements are backed by brilliant execution at our end. Having said that we extend end-to-end ownership of product/application design, development and deployment.