Comparability

To ensure as much as possible that observed data from different countries or cultures are comparable (equivalent). 

Indicators:

Time:

• Any differences in concepts and methods of measurements between last and previous reference period
• A description of the differences, including an assessment of their effects on the estimates

Geographical:

• All differences between local practices and national standards (if such standards exist)
• An assessment of the effect of each reported difference on the estimates

Domains:

• A description of the differences in concepts and methods across study countries (e.g., in classifications, statistical methodology, statistical population, methods of data manipulation, etc.)
• An assessment of the magnitude of the effect of each difference

Minimize the amount of undue intrusion by ensuring comparable standards when appropriate (based on differences in local survey contexts) for informed consent and resistance aversion effort, as well as other potentially coercive measures such as large respondent incentives (see Ethical Considerations and Data Collection: General Considerations).

Define comparable target populations and verify that the sampling frames provide adequate coverage to enable the desired level of generalization (see Sample Design).

Minimize the amount of measurement error attributable to survey instrument design, including error resulting from context effects, as much as possible (see Questionnaire Design, Instrument Technical Design and Paradata and Other Auxiliary Data).

Minimize or account for the impact of language differences resulting from potential translations (see Questionnaire Design, Translation and Adaptation).

Minimize the effect interviewer attributes have on the data through appropriate recruitment, selection, and case assignment; minimize the effect that interviewer behavior has on the data through formal training (see Interviewer Recruitment, Selection, and Training and Paradata and Other Auxiliary Data).

Identify potential sources of unexpected error by implementing pretests of translated instruments or instruments fielded in different cultural contexts (see Pretesting and Paradata and Other Auxiliary Data).

Reduce the error associated with nonresponse as much as possible (see Data Collection: General Considerations for a discussion of nonresponse bias and see Paradata and Other Auxiliary Data, Section A for nonresponse error reduction).

Minimize the effect that coder error has on the data through appropriate coder training (see Data Processing and Statistical Adjustment).

Coherence

To ensure that the data can be combined with other statistical information for various secondary purposes.

Indicators:

• A description of every pair of statistics (statistical unit, indicator, domain, and breakdown) for the survey(s) that should be coherent
• A description of any of the differences that are not fully explained by the accuracy component
• A description of the reported lack of coherence for specific statistics

Create a clear, concise description of all survey implementation procedures to assist secondary users. Study Design and Organizational Structure lists topics which should be included in the study documentation; there are also documentation guidelines within each set of guidelines for each stage of the survey lifecycle.

Provide data files in all the major statistical software packages, and test all thoroughly before they are made available for dissemination (see Sample Design and Data Dissemination).

Designate resources to provide user support and training for secondary researchers (see Data Dissemination).

See Data Harmonization for a discussion of the creation of common measures of key economic, political, social, and health indicators.

Relevance

To ensure that the data meet the needs of the client or users.

Indicators:

• A description of clients and  users
• A description of users' needs (by main groups of users)
• An assessment of user satisfaction

To ensure that the data describe the phenomena they were designed to measure. This can be assessed in terms of the mean squared error (MSE).

Indicators:

Measurement error:

• A description of the methods used to assess measurement errors (any field tests, reinterviews, split sample experiments, or cognitive laboratory results, etc.)
• A description of the methods used to reduce measurement errors
• Average interview duration
• An assessment of the effect of measurement errors on accuracy

Processing error:

• A description of the methods used to reduce processing errors
• A description of the editing system
• The rate of failed edits for specific variables
• The error rate of data entry for specific variables and a description of estimation methodology
• The error rate of coding for specific variables and a description of the methodology followed for their estimation
• A description of confidentiality rules and the amount of data affected by confidentiality treatment

Coverage error:

• A description of the sampling frame
• Rates of over-coverage, under-coverage, and misclassification broken down according to the sampling stratification
• A description of the main misclassification and under- and over-coverage problems encountered in collecting the data
• A description of the methods used to process the coverage deficiencies
• Coefficients of variation
• An assessment of resulting bias due to the estimation method

Sampling error:

• Type of sample design (stratified, clustered, etc.)
• Sampling unit at each stage of sampling
• Stratification and sub-stratification criteria
• Selection schemes
• Sample distribution over time
• The effective sample size
• Coefficients of variation
• An assessment of resulting bias due to the estimation method

Nonresponse error:

• Unit nonresponse rate
• Identification and description of the main reasons for nonresponse (e.g., noncontact, refusal, unable to respond, ineligible, other nonresponse)
• A description of the methods used for minimizing nonresponse
• Item nonresponse rates for variables
• A description of the methods used for imputation and/or weighting for nonresponse
• Variance change due to imputation
• An assessment of resulting bias due to nonresponse

Model assumptions error:

• A description of the models used in the production of the survey’s statistics
• A description of assumptions used on which the model relies
• A description of any remaining (unaccounted for) bias and variability which could affect the statistics

Provide a report on each variable in the dataset of selected elements to check correctness of the overall and within-stratum sample size and the distribution of the sample elements by other specific groups such as census enumeration areas, extreme values, nonsensical values, and missing data (see Sample Design).

Timeliness and punctuality

To ensure that the data are available for analysis when they are needed.

Indicators:

• The legal deadline imposed on respondents
• The date the questionnaires were sent out
• Starting and finishing dates of fieldwork
• Dates of processing
• Dates of quality checks
• The dates the advance and detailed results were calculated and disseminated
• If data are transmitted later than required by regulation or contract, the average delay in days or months in the transmission of results with reference to the legal deadline
• If data are transmitted later than required by regulation or contract, the reasons for the late delivery and actions taken or planned for the improving timeliness

Time data collection activities appropriately (see Data Collection: General Considerations, Pretesting, and Paradata and Other Auxiliary Data).

Create a study timeline, production milestones, and deliverables with due dates (see Study Design and Organizational Structure).

Accessibility

To ensure that the data can easily be obtained and analyzed by users.

Indicators:

• A description of how to locate any publication(s) based on analysis of the data
• Information on what results are sent to reporting units included in the survey
• Information on the dissemination scheme for the results
• A list of variables required but not available for reporting
• Reasons why variables are not available

Save all data files and computer syntax from the preferred statistical software package needed during sample design process in safe, well-labeled folders for future reference and use (see Sample Design).

Document how paradata are collected and the steps used to construct the paradata-based indicators (see Paradata and Other Auxiliary Data).

Establish procedures early in the survey lifecycle to ensure that all important files are preserved (see Data Dissemination).

Test archived files periodically to verify user accessibility (see Data Dissemination).

Create digitized versions of all project materials whenever feasible (see Data Dissemination).

Produce and implement procedures to distribute restricted-use files, if applicable (see Data Dissemination).

Interpretability

To ensure that supplementary metadata and paradata are available to analysts.

Indicator:

• A copy of any methodological documents relating to the statistics provided

At the data processing stage of the study, create a codebook that provides question-level metadata matched to variables in the dataset.  Metadata include variable names, labels, and data types, as well as basic study documentation, question text, universes (the characteristics of respondents who were asked the question), the number of respondents who answered the question, and response frequencies or statistics (see Sample Design, Data Processing and Statistical Adjustment and Paradata and Other Auxiliary Data).

Collect and make available process data collected during data collection, such as timestamps, keystrokes, and mouse actions ('paradata') (see Instrument Technical Design and Paradata and Other Auxiliary Data).