I am frequently surprised when I encounter data quality ostriches i.e. organisations that put their heads in the sands of data and hope that all will be well.
If you have spent any time looking at the different facets of data quality, you should be aware that some of the key quality metrics are:
- Accuracy – Does data match the thing it represents
- Validity – Does data match your system and process rules
- Consistency – Does the same entity recorded in different systems have the same identifier
- Completeness – Do you have a full data set?
Probably the most important of these is accuracy, however, this can also be one of the harder (i.e. more expensive) areas to assess, for example in the transport and utilities sectors assets are widely distributed, may be buried and may have safety risks associated with visiting them. Therefore accuracy checking can be expensive.
Some large organisations have small teams assessing the completeness and validity of data using data profiling tools, but are avoiding doing data accuracy checking as it is seen as difficult (i.e. expensive). Yet these same organisations can be planning and implementing major system and process changes based on assumptions about data accuracy. Will they achieve the benefits of these plans if data accuracy is not as expected? Will they make expensive mistakes? Are they being a data quality ostrich? Are you a DQ ostrich??