In Part 1 of this series of posts, we discussed how your existing business data could limit the benefits of a new information system and said there is likely to be work to do to mitigate those risks. No matter what system you buy (unless it is a ‘straight’ system upgrade), there will be data migration (DM) activity needed in order to populate the new system with your data in order for it to function and, therefore, realise its potential benefits. However, data migration projects are both high cost and high risk – figures vary, but suggest around 40% of data migration projects fail (A 2011 study by Bloor Research put the failure rate for data migration projects at 38%) and separate studies show that data migration costs can be 20-50% of the project budget. Therefore, data migration should be recognised as a high cost, high risk part of any project.
So, why are data migration projects so difficult?
- Data migration should be viewed as a business project supported by technology (see Practical Data Migration) and not as a technology activity, however, many organisations have in the past treated it as a technology activity
- Business data may need transforming/merging from multiple sources. Identifying the “right” source attribute can be time consuming and difficult. Different parts of the business may have a different view as to what is the “right” source of data
- Previously acceptable data may not be acceptable in new system, as the new system may have tighter data validation approaches which will highlight existing flaws that either you weren’t aware of, or must now be resolved
- Data may need to be combined from a number of existing data sources which were valid in isolation, but when combined, may highlight new data validation issues to resolved. For example, a ‘room’ may represent the walls, floor and ceiling that represent the room, but may also represent the space bounded by the walls – two subtly different concepts and if these concepts have to be combined, then some attributes may not be consistent
- Data migration can often be based upon the use of migration rules – developing rules for all valid variants of data can be time consuming and often no matter how many migration rules you develop, there may data that is correct but does not comply with the validation rules developed, so needs to be treated as an ‘exception’
- Data that may be valid and support operation of the new software may be inaccurate, which will for many users raise questions about the credibility of the new system. Users may incorrectly assume that a software project will correct all the data inaccuracies that were existing at the start of the project – new systems lose credibility rapidly if users see obviously wrong data
- Now if we consider the volume of data that needs to be migrated (thousands or perhaps millions of records), it’s easy to understand why carrying out investigative activities on such a large scale can mean data migration projects can be a high risk activity for any project and still risk having the “wrong” data at the end of it.
The above points illustrate why the challenge of migrating data successfully into a new system can be a high risk activity for a project.
In Part 3 of this series of blog posts we will look at steps that can be taken to reduce the risk of ‘surprises’ in project delivery and increase the likelihood of success, from a data perspective.