• Home
• |
• Contact Us
• |
• Guestbook
• |
• User Guide
• |
• FAQ
• |
• Archive

Data Harmonization Guidelines

XIII. Data Harmonization

Webpage last modified: 2010-Aug 30

Peter Granda and Emily Blasczyk

Introduction

Harmonization refers to all efforts that standardize inputs and outputs in comparative surveys.

Harmonization is a generic term for procedures used predominantly in official statistics that aim at achieving, or at least improving, the comparability of different surveys and measures collected. The term is closely related to that of standardization (see Sample Design and Questionnaire Design). Harmonizing procedures may be applied in any part of the survey life cycle, such as study design, choice of indicators, question wording, translation, adaptation, questionnaire designs, sampling, field work, data coding, data editing, or documentation. The need to harmonize arises for all comparative surveys. This is particularly true if the goal is to combine the data into a single integrated dataset.

Two general approaches for harmonizing can be identified: input harmonization and output harmonization:

1. Input harmonization aims to achieve standardized measurement processes and methods in all national or regional populations. Comparability is realized through standardization of definitions, indicators, classifications, and technical requirements.
2. Output harmonization uses different national or regional measurements possibly derived from non-standardized measurement processes. These measurements are "mapped" into a unified measurement scheme. Thus, only the statistical outputs are specified, leaving it to the individual countries/regions to decide how to collect and process the data necessary to achieve the desired outputs [8] [16] [18].

Figure 1 shows data collection within the survey production process lifecycle survey lifecycle) as represented in these guidelines. The lifecycle begins with establishing study structure (Study, Organizational, and Operational Structure) and ends with data dissemination (Data Dissemination). In some study designs, the lifecycle may be completely or partially repeated. There might also be iteration within a production process. The order in which survey production processes are shown in the lifecycle does not represent a strict order to their actual implementation, and some processes may be simultaneous and interlocked (e.g., sample design and contractual work). Quality and ethical considerations are relevant to all processes throughout the survey production lifecycle. Survey quality can be assessed in terms of fitness for intended use (also known as fitness for purpose), total survey error, and the monitoring of survey production process quality, which may be affected by survey infrastructure, costs, respondent and interviewer burden, and study design specifications (see Survey Quality).

Figure 1. The Survey Lifecycle

Guidelines

Goal: To ensure that survey and statistical research teams follow accepted standards when creating harmonized data and documentation files, and use a harmonization strategy that best fits their basic source materials and the objectives they wish to achieve.

1. Decide what type of harmonization strategy to employ, taking into account that all harmonization efforts will require some combination of strategies.
   1. Consider "input" harmonization when the survey process is centrally coordinated.
      Rationale

      "Input" harmonization, usually applied in a multi-national context, seeks to impose strict standards and protocols from the beginning for the whole survey process (ex-ante) by which each national survey applies the same survey procedures and a common questionnaire (see Sample Design and Translation) This strategy is meant to assure a high degree of comparability.

      Procedural steps

      • Provide detailed specifications, protocols, and procedures for all aspects of the survey process. The different specifications (Data Protocol, Sampling, Translation, etc.) of the European Social Survey (ESS) are a good example [4].
      • Decide what to standardize.
      • Create an overall monitoring team that coordinates data collection, merging national data files, and archiving.
      • Publish the details of the plan and provide a schedule for the release of public-use files to the user community.
      Lessons learned

      • This approach involves adherence to appropriately standardized methodologies throughout the survey life cycle (see Sample Design). For example, the ESS seeks to collect data every other year, uses face-to-face interviews, aims at high-precision data, applies detailed sampling and fieldwork protocols, uses standardized translation protocols in all participating countries, aims to achieve standardized response rates, adopts consistent coding procedures, and creates and distributes well-documented datasets in a timely fashion. All of these procedures require greater organizational capabilities and resources throughout the planning and data collection stages. The results are transparent, high quality, and can produce more valuable public-use data files at the end.
      • Not all comparative research will be able to follow the same procedures, so it is important to decide which methods are best, given the actual resources, survey process structure, and the intended level of precision. In addition, the creation of such common standards and their implementation at the local level requires considerable expertise. This also may not available in all multinational, multiregional, and multicultural contexts.
      • It is difficult to have common standards applied once the survey is in the field. The World Mental Health Survey Initiative created such standards in the planning stage, but was unable to implement all of them, as some were not relevant in each survey country, and some countries' current survey collection practices could not support the recommended standards.
   2. Consider "output" harmonization when the survey collection process is largely determined at the level of individual countries or cultures and there is agreement on standardization.
      Rationale

      This type of harmonization is implemented through two main strategies, one "ex-ante" and the other "ex-post."

      Ex-ante refers to a) all measurements, such as education or employment information, which cannot be harmonized before the data collection; and b) a situation where surveys in different countries or cultures are planned to be comparable but not with the strict protocols used in input harmonization. When harmonization has already been considered during survey planning with regard to the development of common goals, measurements, and understanding of concepts, the ex-ante strategy ensures that specific targets are established for the collection of data on key variables. However, the questions used to collect these data may vary from country to country (see Questionnaire Design and Adaptation of Survey Instruments)

      The second variant is an ex-post strategy, by which national statistical or survey data from archives are made comparable after the fact through a conversion procedure. ex-post strategies can be used in situations where existing repositories will be exploited for comparative research or where intensive early planning is not possible because of financial or policy constraints.

      Procedural steps

      • Use an ex-ante strategy whenever possible. This enhances comparability since harmonization is addressed at the planning stage of each national data collection, as well as at the end of the process when creating harmonized data files.
      • Implement an appropriate planning process.
      • Use an ex-post strategy only if no consideration regarding harmonization has been given by data collectors at the start of data collection(s), but researchers later believe (e.g., because of common concepts or similar questions across surveys) that a harmonized data file can be produced through a conversion process to create comparable variables or statistics. The Integrated Fertility Survey Series is one such example [13].
      • For both ex-ante and ex-post harmonization, adopt a detailed "data processing plan" that includes descriptions of how the producer(s) of the harmonized data deal with the following:
        • Differences across studies with regard to what is to be measured (e.g., definitions of population, concepts, variables).
        • Differences in how to measure (e.g., scale of measurement, wording and routing of questions, respondents).
        • Data editing.
        • Procedures used to create and define harmonized variables.
        • Construction of recoded variables (e.g., creating a common time format).
        • Sample weights and sample design variables.
        • Imputation.
        • Differences in how estimates are generated (weighting, nonresponse adjustments).
      • Record all decisions about the "conversion" process systematically. One option is to use two separate databases to record all work: a production database which stores the original and harmonized materials, and a user's database which provides the analysts access to the overall process.
      • Make provisions that all data conversions can be traced back to the original data.
      Lessons learned

      • In a working paper, Roland Gunther describes in detail the harmonization efforts surrounding the European Community Household Panel (ECHP) [9]. This survey began as a major example of input harmonization, with its design of uniform questionnaires as well as detailed definitions, rules, procedures, and models to make comparability across nations easier. After the first phase of the project, a few countries decided to cease collecting national samples for the ECHP, and instead to conduct their own national surveys, resulting in the need to do ex-post harmonization. Those doing the harmonization work learned that this kind of ex-post harmonization was resource-intensive and required staff experienced in both the original source and target formats of the ECHP framework. They also had to know in detail how their national questionnaires differed. Common problems included concepts heavily affected by national contexts, as well as differences in scales of measurement, variable coding schemes, and definitions of these concepts. Solutions to such problems were often found through ad hoc decisions about recoding, combining, or collapsing variables, and almost never through estimation techniques.
      • These harmonization strategies are almost never applied exclusively on any single statistical or survey data collection. Depending on specific cultural and national characteristics, data producers should consider strategies that will enable them to collect their data in the most efficient manner. In some situations, they may want to combine strategies. For example, data producers may start with an input harmonization plan, but should be prepared to do some ex-post output harmonization to account for differences across cultures.
2. When deciding which variables to harmonize, create an initial plan and define clear objectives about what you want to achieve. The plan should include making all data conversions reversible.
   Rationale

   Creating a harmonization plan from the beginning of the project allows data producers to document all of their decisions at the time they are made. In case errors occur or are identified by users at a later time, all data conversions should be reversible.

   Procedural steps

   • Form an advisory committee of researchers knowledgeable about the subject matter at the beginning of the harmonization process, if possible, and consult with them regularly.
   • Before fieldwork, consult with experts or an advisory committee on a systematic design process, and with methodology groups to investigate comparability issues.
   • Show the advisory group results of the harmonization process at different points in the process to allow for possible changes in rules used to create new variables.
   • Realize that not all concepts measured in the survey process are equally susceptible to harmonization efforts. For example, cross-national harmonization of the number of births and marriages is a far easier task than comparisons of divorce rates where local laws, customs, and data collection methods may differ substantially. Other concepts, such as international population migration, may not, due to a lack of precise definition and great variety in measurement criteria, lend themselves to harmonization at all, or only at the most basic level.
   • Consider establishing a testing group of users knowledgeable about the subject matter but not about the harmonization process, who provide feedback on the analytic usefulness of the data before they are released publicly.
   • Implement a systematic conversion creation process with appropriate quality controls.
   • Identify and become familiar with software tools that facilitate a comparison of variables from different surveys, in order to determine if and how these could be harmonized. Such tools often work from a common database that stores all the information about each variable.
   • Establish partnerships with producers of harmonization tools. This may be more beneficial than creating new tools, which often requires costly programming efforts.
   • Where software tools are unavailable or impractical, use manual comparisons in making harmonization decisions and consult with substantive and methodological experts in doing so.
   • Identify and become familiar with interactive documentation tools that allow for proper and transparent documentation.
   Lessons learned

   • Good decision-making about the harmonization process will benefit from the use of software tools, as well as input from a diverse group of survey researchers who can offer advice on various procedures and techniques to use when producing harmonized files. The ISSP Data Wizard [6] is used by the International Social Survey Programme (ISSP). The Data Wizard supports procedures that were previously performed manually to harmonize data at the cross-national level. The tool offers rule-based checks, automation of partial steps, and the visualization of certain conditions, to make the harmonization process more efficient, easier, and less susceptible to mistakes.
   • The European Values Study (EVS) formed a number of work groups, both before and after fieldwork. The aim on the one hand was to set standards at an early stage, and on the other to consolidate and merge data which had been cleaned by participating national survey teams. This project produced an integrated source questionnaire and a set of equivalency tables to assist secondary researchers. The project web site makes all of this information easily accessible [4]. These processes and products provide critical information to secondary users of these data.
3. Focus on both the variable and survey levels in the harmonization process.
   Rationale

   Harmonization efforts usually concentrate on comparing and integrating information involving specific variables across data files. However, it is equally important to consider the overall characteristics of the surveys that make them good candidates for harmonization, and to report the decisions involving this process to end users.

   Procedural steps

   • Recognize the different aspects involved in converting source variables into target variables, such as "differences in the definitions of underlying concepts or in the definitions of the variables, deviations in the scales of measurements and so on." The concept of citizenship, for example, presents significant challenges to researchers who want to investigate this topic [17].
   • Describe similarities and differences between the source variables and the target variables, including discussion of universe statements, question wording, coding schemes, and missing data definitions.
   • Consider file-level attributes when creating the harmonized data file, including how survey weights, imputation procedures, variance estimation, and key substantive and demographic concepts will change in the process.
   • Pay particular attention to sampling designs and data collection methods in making assessments about the degree of comparability between different surveys.
   Lessons learned

   • Data producers must recognize the degrees of individual item or variable persistency when creating questionnaires and collecting data. Item persistency over time is very important in generating harmonized data files. There are considerable differences, for example, between an "absolute" persistent variable, such as "country of birth," and a less persistent variable, such as "country of citizenship." The concept might mean different things in different countries, is subject to change, and could be reported validly for multiple countries by some respondents [17].
   • Quota sampling destroys comparability. Mixing surveys using quota sampling with those using probability sampling is not advisable [10] [12]. The ISSP is an example, where a comparative survey program abolished quota sampling (search for ISSP monitoring reports on the web).
   • The European Social Survey (ESS) provides detailed insight into weighting issues and makes this information available. (See the ESS data site for each survey round to get the latest version).
   • The Collaborative Psychiatric Epidemiology Surveys (CPES) [1] created a harmonized data file from three comparable surveys on mental health. Data producers created a pooled weight for the harmonized file, based on race/ancestry groupings and on the geographic domains of the sampling frames of each individual survey. Understanding the specific characteristics of each input file was an essential part of creating a harmonized output file [11]. All of this information was provided to users in a comprehensive explanation of the original and harmonized weights.
4. Develop criteria for measuring the quality of the harmonization process. This includes testing it with users knowledgeable about the characteristics of the underlying surveys, the meaning of source variables, and the transformation of source variables into target variables.
   Rationale

   Researchers may analyze harmonized files in new and unexpected ways. It is crucial to provide them sufficient information about the concepts and definitions presented, and the assumptions underlying the decisions made in their construction.

   Procedural steps

   • Devise procedures to judge the quality of the harmonized outputs based on such quality criteria as consistency, completeness, and comparability.
     • Consistency can be judged by comparing the results from multiple independent efforts of harmonizing a variable; completeness is assessed based on the degree to which the original information is preserved in the harmonized data; and comparability is the degree to which the harmonized outputs can accurately report important social or economic concepts over time or between countries or cultures.
     • The Statistical Office of the European Communities (EUROSTAT) proposed the following set of quality criteria when reporting statistics which also apply to harmonization outputs (2003) [2] [15]:
       • Relevance of the statistical concepts.
       • Accuracy of the estimates.
       • Topicality and timeliness of the dissemination of results.
       • Accessibility and clarity of the information.
       • Comparability of the statistical data.
       • Coherence.
       • Completeness.
     • Strictly speaking, these traits apply to official statistical data. However, many of them would apply equally to academically produced survey data, particularly those regarding the comparability of social, economic, and demographic concepts cross-nationally or cross-culturally, and the accuracy of estimates (Total Survey Error).
   • Be prepared to modify and update harmonized datasets after public release, based on comments from the research community if errors are uncovered or if certain variables need further explanation.
   • Prepare presentations at social science research conferences that describe the harmonization process to potential users.
   Lessons learned

   • The usefulness of well-harmonized data is clearly recognized by many international organizations. For example, the United Nations Economic and Social Council indicated in a recent report that it "was working towards the harmonization of relevant environmental data-collection activities with concepts and definitions of environmental accounts. Such harmonization would result in substantial benefits in the quality of the data because it would introduce consistency checks to the environmental data and would also provide additional analytical value. The dissemination of national accounts, complemented with environment statistics information, was a very powerful analytical tool for the derivation of consistent and coherent indicators, such as resource efficiency indicators and resource use as percentage of value added. It would also allow for more in-depth analysis through scenario-modeling using input-output techniques" [19].
5. Provide the widest range of data and documentation products about the complete process.
   Rationale

   Regardless of whether researchers adopt input or output harmonization as a strategy, all aspects of the survey planning, collection, and dissemination process should be considered when producing harmonized data files or creating accompanying documentation. Users should have access not only to the harmonized end result, but also to detailed information about all steps taken by the producers, in order for them to fully understand what decisions were made during the entire process.

   Procedural steps

   • Define the elements of the harmonization process and start documenting it at the very beginning, in order to ensure that all decisions are captured even before a definite plan to produce a public-use data file exists.
   • Document each target variable with information from all source variables, transformation algorithms, and any deviations from the intended harmonized approach, if known.
   • If possible, provide users with access to the original data files used in producing the harmonized file. If direct access to original data is not permissible due to confidentiality concerns, implement procedures to assist users in proper check-backs or re-transformations. Also consider implementing some form of restricted-use data agreement to allow access under controlled conditions.
   • Provide users with the code or syntax used in creating new variables for the harmonized file.
   • Provide users with complete documentation, including crosswalks, which describe all the relationships between variables in individual data files with their counterparts in the harmonized file. An interactive, web-based documentation tool is often the best way to present such documentation.
     • Include original questionnaires and information about the data collection process whenever possible.
   • Report on as many of the following elements of the data life cycle as apply to the particular harmonization process:
     • Project planning.
     • Sampling frame.
     • Sample size.
     • Sample design (See Instrument Technical Design, Questionnaire Design, Sample Design).
     • Extent of the field period.
     • Instrument construction and design.
     • Translation and adaptation (See Translation).
     • Mode(s) of interview.
     • Respondent follow-up if panel survey.
     • Data collection (See Data Collection, Survey Quality).
     • Editing.
     • Item nonresponse.
     • Unit nonresponse.
     • Any special treatment given to demographic and country-specific variables.
     • Sample weights.
     • Variance estimation.
     • Data production, including both long-planned and ad-hoc decisions implemented during variable conversion.
     • Documentation production.
     • Dissemination (See Data Dissemination).
     This list is based on documentation provided in the Integrated Health Interview Series (IHIS). The IHIS is an effort to provide an assortment of variables from the core household and person level files from the National Center for Health Statistics' seminal data collection effort on the health conditions for the US population from 1969 to the present. It provides extensive user notes and FAQ pages to describe how their harmonization project coped with several of these components [14].
   • Consider archiving the harmonized data with a data archive to ensure continued availability of all data and documentation files and long-term preservation.
   Lessons learned

   • The Eurobarometer Survey Series, in operation since 1970, now includes several dozen cross-sectional surveys, all of which have been harmonized into single cross-national files before being made available to researchers. These surveys are released initially with basic information about each study and the characteristics of all variables, and are then further processed by the social science data archives, led by GESIS (German Social Sciences Infrastructure Services), to include variable frequencies, more complete documentation, and online analysis services for researchers [15]. Such partnerships between data producer and social science data archives encourage long-term preservation, enhance access, and make it possible to continually improve services to the research community.
   • Some harmonization projects have gone to great lengths to describe their procedures in specific detail. For example, the Multinational Time Use Study (MTUS) has a User Guide and a comprehensive description of its coding procedures used in creating its harmonized data file [18]. Similarly, the Generations and Gender Programme (GGP) of the United Nations Economic Commission for Europe Population Activities Unit (UNECE-PAU) provides reports and guidelines about how the organization implements its harmonization decisions [7]. These projects provide transparency to both creators and users of these data and serve as an example for others to follow.

Return to top

Previous chapter | Next chapter | Home

© 2008 The authors of the Guidelines hold the copyright. Please contact us if you wish to publish any of this material in any form.

View Chapters

• Introduction
• Study Structure
• Survey Quality
• Ethical Considerations
• Tenders, Bids, & Contracts
• Sample Design
• Questionnaire Design
• Adaptation
• Translation
•      Translation Assessment
•      Language Harmonization
•      Translators
•      Translation Budgeting
•      Translation Scheduling
•      Translation Tools
• Instrument Technical Design
• Interviewer Recruitment & Training
• Pretesting
• Data Collection
• Data Harmonization
• Data Processing
• Data Dissemination
• Glossary
• Bibliography