technologies > data validation

Data Validation

Public authorities and private business establishments collect large amounts of data in the course of their everyday activities. In recent years we have witnessed an explosion in the collection, storage and processing of information with computers. Powerful hardware and software enable organisations to perform any conceivable data analysis task for the extraction of information useful for monitoring and planning.

Data contain errors. Although modern data collection / data entry software can spot errors in real time, some types of complex errors can pass undetected. Examples are inconsistencies between the values of related variables or extreme differences between current and historical data.  Moreover, data which had originally been stored in paper form and have been converted in electronic format may have not been checked at all for errors. The existence of errors reduces the usefulness of data for analytical purposes.

Data validation is the process of checking data for errors and correcting them. Data are examined to see if they satisfy a number of logical conditions called validation rules. Data which fail to satisfy even one validation rule are marked as errors. The correction of an error involves the identification of the particular variables whose values cause the error, called error localisation, and the replacement of these values, called imputation, with correct one. With computers, data checking is fully automated while data correction is automated to a large extent: very important data records may be corrected ‘manually’ by human experts.

Agilis has long experience in the theory and practice of data validation. The services we offer comprise:
  • Design of data validation schemes
  • Evaluation of data validation schemes
  • Design and development of custom-made software for data validation
  • Implementation of data validation on customers’ data
  • Consulting in data validation
  • Training
 

Data Validation