PDF 
Frost Hubbard, Yu-chieh (Jay) Lin, Dan Zahs, and Mengyao Hu, 2016
 
Appendices:  A | B  |  C |  D

Introduction

Although one could employ a census to measure the entire population, it is more common to take a sample of the population.  A properly designed probability sample (see probability sampling) can be used to make estimates for not only the sample itself, but also for the underlying population from which it was selected. A probability sample is one in which each element of the (underlying) population has a known and non-zero chance of being selected. That is, every person has a chance to be included in the study and have his or her characteristics, opinions, etc., become part of the data. It should be noted that everyone does not have to have an equal chance of being selected – just a known non-zero chance of being selected.

Probability samples have several desirable characteristics. They enable us to put a margin of error or confidence interval on our estimates – essentially a measure of how accurate the estimate is compared to the same estimate calculated on the full population. Probability samples make it possible to not only compare the sample to the population, but also to compare a sample from one population to a sample from another population, as occurs in multinational, multicultural, or multiregional surveys, which we refer to as “3MC” surveys.

An optimal sample design is one that maximizes the amount of information obtained per monetary unit spent within the allotted time and meets the specified level of precision (Heeringa & O’Muircheartaigh, 2010). One important prerequisite for 3MC surveys is that all samples are full probability samples from comparable target populations (Kish, 1994).

Different nations have different sampling resources and conditions. For a multinational survey, this means that the optimal sample design for one country may not be the optimal design for another. (Please note this chapter uses the term “participating country” to encompass any participating country, culture, region or organization in a 3MC study.) Therefore, allowing each participating country some flexibility in its choice of sample design is highly recommended, so long as all sample designs use probability methods at each stage of selection (Häder & Gabler, 2003; Kish, 1994).

This chapter outlines the decisions that need to be made when designing a 3MC probability survey sample. It encourages survey organizers to allow sample designs to differ among participating countries while, at the same time, ensuring standardization on the principles of probability sampling. 

Although flexibility will usually be necessary in sample design and implementation, the guidelines and recommendations discussed in this chapter are only directly applicable if a probability framework is followed. Probability designs can be expensive. To reduce costs, some survey organizations select nonprobability samples, including convenience samples and quota samples. While not all nonprobability samples are biased, the risk of bias is extremely high and, most importantly, cannot be measured—a survey that uses nonprobability sampling cannot estimate the error in the sample estimates (Heeringa & O’Muircheartaigh, 2010).

Please note that this chapter assumes that the reader has a basic understanding of statistics and terms such as “varianceand “standard deviation.”  Please refer to Further Reading or an introductory statistics textbook if a statistics refresher is needed.

 

⇡ Back to top

Guidelines

Goal: To select an optimal, cost-efficient probability sample in each participating country that is representative of the target population and allows researchers to make inferences to the target population, and to standardize sample designs without hampering optimal designs in each participating country.

 

1.   Define the target population for the study across all countries and the survey population within each participating country.

Rationale

The survey planners of any 3MC survey need to develop a detailed, concise definition of the target population in order to ensure that each participating country collects data from the same population. Without a precise definition, one country may collect data that include a certain subgroup, such as noncitizens, while another country excludes this subgroup. This difference in sample composition may influence the estimates of key statistics across countries. In addition, a precise definition will let future users of the survey data know to which exact population the survey data refer. The data users can then make a more informed decision about whether to include the survey data in their analyses.

Procedural steps

1.1   Define the target population across all participating countries as clearly as possible, including the elements of the population and the time frame of the group (Groves et al., 2009). For example, a target population might be defined as, “All persons above the age of eighteen, who usually slept most nights in housing units in South Africa, Zimbabwe, Lesotho, and Swaziland during April, 2007.” Note that this definition would, in turn, require definitions of the terms “usually,” “most,” and “housing unit.”

1.2   To ensure a clear description of the target population, think about all potential inclusion/exclusion criteria. For example, the target population might exclude:

1.2.1   Persons outside a defined age range.

1.2.2   Persons in institutions, such as hospitals, nursing homes, prisons, group quarters, colleges, monasteries, or military bases.

1.2.3   Persons living in certain sparsely populated or remote geographic regions.

1.2.4   Non-citizens, ethnic minorities, homeless or nomadic populations, language groups.

1.3   Define the survey population within each participating country by refining the target population based on cost, security, or access restrictions to all target population elements (Groves et al., 2009).

1.3.1   In some cases the target and final survey populations are the same. However, there can be differences. The target population can be thought of as “Who we want to include in our survey” while the survey population is “Who we can include in our survey”. Differences can occur due to frame availability, screening ability, cost, etc. For example, the survey population may exclude those residing in war-torn areas, or the data collection period may be narrowed in areas with civil disturbances that are threatening to escalate.

1.3.2   Ideally, the resulting survey populations are comparable across all countries. To help analysts determine comparability, the decisions involving the definition and implementation of the target and survey populations should be well documented.

Lessons learned

1.1   Large established 3MC surveys have defined their target and survey populations differently, depending upon the goals and topics of the study.

1.1.1   The Afrobarometer is an independent, nonpartisan research project that measures the social, political, and economic atmosphere in Africa. Afrobarometer surveys are conducted in more than a dozen African countries and are repeated on a regular cycle. Participants in Round 6 of the Afrobarometer Survey had to be citizens of their country and of voting age on the day of the survey.  They had to complete the interview in their country’s national language or in an official local language for which a translation was available. People living in areas of armed conflict or natural disasters, national parks and game reserves, and in institutionalized settings were excluded. Special cases, like areas of political unrest, were reviewed on a case-by-case basis (AfroBarometer, 2014).

1.1.2   The Asian Barometer is an applied research program studying public opinion on political values, democracy, and governance in thirteen East Asian political systems (Japan, Mongolia, South Koreas, Taiwan, Hong Kong, China, the Philippines, Thailand, Vietnam, Cambodia, Singapore, Indonesia, and Malaysia) and five South Asian countries (India, Pakistan, Bangladesh, Sri Lanka, and Nepal). The target population of the Asian Barometer was defined as citizens who were at least 20 years of age and were eligible to vote (i.e., were not disenfranchised due to mental illness or incarceration) (see http://www.asianbarometer.org/survey/survey-methods).

1.1.3   The European Social Survey (ESS) is an academically-driven social survey designed to chart and explain the interaction between Europe's changing institutions and the attitudes, beliefs and behavior patterns of its diverse populations. Round 7 of the ESS covers more than 22 nations and includes persons 15 years or older who are resident within private households, regardless of nationality, citizenship, or language; homeless and institutional populations are excluded from the sample (European Social Survey, 2014).

1.1.4   The Living Standards Measurement Study (LSMS) was established by the World Bank in 1980 to explore ways of improving the type and quality of household data collected by statistical offices in developing countries. Its goal is to foster increased use of household data as a basis for policy decision making. Respondent requirements and exclusions vary across participating countries (LSMS, 1996).

1.1.5   The Survey of Health, Ageing and Retirement in Europe (SHARE) studies the health, socio-economic status and social and family networks of individuals, aged 50 or over, in nearly 20 European countries and Israel. In addition to the age requirement, respondents had to be residents of the country and speak (one of) the official language(s) of the country. The study excludes seasonal or vacationing residents, persons physically or mentally unable to participate, those who died before the start of the field period, or who are unable to speak the specific language of the national questionnaire. It also excludes residents of institutions, except facilities for the elderly (De Luca, Rossetti, & Malter, 2015).

1.1.6   The World Value Survey is conducted by a non-profit association seated in Stockholm, Sweden, to help social scientists and policy makers better understand worldviews and changes that are taking place in the beliefs, values, and motivations of people throughout the world. Respondents are adults, 18 years and older; some countries also place upper limits on age (http://www.worldvaluessurvey.com/).

1.1.7   The World Mental Health (WMH) Survey studies mental illness in selected countries in Europe, Asia, Africa, and North and South America.  One of the major goals of the WMH Study was to compare the age of onset of disease across countries. Best practice might suggest strictly defining the age of majority (e.g., 18 years old). However, the WMH study organizers recognized that strictly defining this inclusion criterion would be difficult, given that age of majority varies by country (and even within a country). Also, a strict definition would affect study protocols such as ethics reviews and informed consent (seeking permission to interview minors). Therefore, the WMH Study had to make a difficult decision to allow the age eligibility criterion to vary across countries. In the end, the WMH Study allowed the age range to vary, with 16 years of age being the youngest lower age limit; some countries also set upper age limits. This was taken into consideration in the analysis stage (http://www.hcp.med.harvard.edu/wmh/).Participating countries were also allowed to vary in whether or not respondents must be citizens or be fluent in specific languages (Kessler, Ustun, & World Health Organization, 2008).

1.2   An increasingly common form of housing seen in international studies is workers’ quarters. Survey designers may want to explicitly state in the definition of the target population whether workers’ quarters should be included or excluded.

⇡ Back to top

2.   Identify and evaluate potential sampling frames. Select or create the sampling frame that best covers the target population given the country’s survey budget.

Rationale

An ideal sampling frame contains all of the elements of the target population. However, very few sampling frames exist that allow access to every element in the target population. The goal, then, is to choose a sampling frame or a set of sampling frames that allows access to the largest numberof elements in the target population and contains the fewest number of ineligible elements, given the constraints of the survey budget.

Procedural steps

Note: Although this chapter focuses heavily on the method and practice of in-person interviews, telephone interviews using Random-Digit-Dialing (RDD) frames (Tucker, Lepkowski, & Piekarski, 2002) or other lists are also widely used. In cross-national surveys, one country may conduct interviews over the telephone while another country conducts face-to-face interviews (see, for example, the Gallup World Poll: http://www.gallup.com/178667/gallup-world-poll-work.aspx).  This difference in the mode of data collection, driven by the available sampling frames, might lead to differences in the results. (See Data Collection: General Considerations for more information about mixed-mode design and mode effects.)

2.1   Have each participating country identify a pre-existing list (or lists) of desired elements or clusters of elements of the target population to create a sampling frame. Examples include:

2.1.1   Official population registries.

2.1.2   Health registries.

2.1.3   Lists of schools.

2.1.4   Postal registries.

2.1.5   Electoral rolls.

2.1.6   Utility customer lists.

2.1.7   Pre-existing sampling frames used by other surveys.

2.1.8   Telephone directories.

2.1.9    Random-Digit-Dialing (RDD) telephone frames.

  • The phone system in many nations makes it possible to generate telephone numbers for sampling purposes without first generating all possible telephone number combinations.
  • In some places there might be separate lists or frames for landline telephone numbers and cellular telephone

2.1.10  Lists of email addresses.

  • These are only appropriate in special circumstances.
  • In some populations, many people may not have an email address; at the same time, some people may have multiple email addresses.

2.1.11  Other list(s) of addresses, phone numbers or names.

2.2   Create a sampling frame via area probability sampling methods if there are no appropriate pre-existing lists of elements (or if the lists are not made available for use by the survey project) of the target population. Even if such lists do exist, it is wise to assess the cost and coverage errors associated with creating an area probability sampling frame and how this might compare to the use of a pre-existing list. Many texts and documents provide detailed guidance regarding the development of area probability samples (Kish, 1965; Üstun, Chatterji, Mechbal, & Murray, 2005). Below, we outline a simple two-stage area probability sample of households,including the following steps used in many 3MC surveys.  (Additional information can be found in Appendices A and C).

2.2.1   Create a list of primary sampling units (PSUs) based on geographic clusters. In the United States, for example, these clusters are typically census enumeration areas.

2.2.2   Using a probability sampling method, select a sample of PSUs.

2.2.3   Determine the appropriate method for listing the housing units (secondary samplingunits (SSUs)) within selected PSUs.

2.2.4   Send staff to list the housing units in selected PSUs, maintaining a uniform definition of what constitutes a “housing unit.”

2.2.5   Once the housing units in a PSU have been enumerated, select a randomsample of housing units from the list.

2.2.6   During data collection, ask the selected housing units within the PSUs to participate. Once the housing unit has agreed to participate, complete a list of all eligible members within the housing unit. (See Appendix B for more detailed instructions on enumerating eligible members of the housing unit.)

2.2.7   Using a probability method, select one or more eligible members within the housing unit (Koch, forthcoming).

  • Train the interviewer or, where possible, program the computer, to select an eligible respondent based on the selection method specified.
  • While some “quasi-probability” and “non-probability” or “quota” within-household selection methods can be used, be aware that such procedures produce a non-probability sample.
  • Some studies may want to survey the most knowledgeable adult, the one with primary child care responsibilities,

2.3   Evaluate how well each potential sampling frame covers the target population (Groves, 1989).

2.3.1   Examine the sampling frame(s) for a one-to-one mapping between the elements on the sampling frame and the target population. There are four potential problems:

  • Undercoverage(missing elements): elements in the target population do not appear on sampling frame.
  • Ineligible elements: elements on the sampling frame do not exist in the target population.
  • Duplication: several sampling frame elements match one target population element.
  • Other mismatches: for example,

2.3.2    Area frames generally have better coverage properties than pre-existing lists of addresses, names, or telephone numbers because area frames have fewer missing eligible elements, fewer duplications, and fewer ineligible elements. For more information on the creation of area probability frames, see Appendices A and C.

2.4   Consider combining multiple sampling frames which cover the same population if the union of the different frames would cover the target population better than any one of the frames on its own (Hartley, 1962). When combining multiple lists to create a sampling frame, the following steps should be considered (Groves & Lepkowski, 1985):

2.4.1   First, determine for each element on the combined frame whether it is a member of Frame A only, Frame B only, or both Frame A and B, and calculate probabilities (Lohr & Rao, 2006).

  • If the membership of each element can be determined before sampling, duplicates can be removed from the sampling frame.
  • A variation on this is to use a rule that can be applied to just the sample, rather than to the entire frame. Frame A might be designated as the controlling frame, in the sense that a unit that is in both frames is allowed to be sampled only from A. After the sample is selected, determine whether each unit from B is on the A frame, and retain the unit only if it is not on frame A. This method extends to more than two frames by assigning a priority order to the frames.
  • If the membership cannot be determined prior to sampling, then elements belonging to both frames can be weighted for unequal probabilities of selection after data collection (see Data Processing and Statistical Adjustment for best practices for weighting and nonresponse

2.5   Assess the cost of obtaining or creating each potential sampling frame.

2.5.1   In most circumstances, it is less expensive to purchase pre-existing lists than to create area probability frames.

2.5.2   While three stage area probability samples are more costly to develop than pre-existing lists, they facilitate cost-effective clustering for interviews.

2.5.3   If pre-existing lists are not up-to-date, potential respondents may no longer live at the address on the list or may have changed phone numbers; tracking these individuals can be very expensive.

2.5.4   Pre-existing lists for household surveys often contain more ineligible elements than area probability frames, increasing survey costs. 

2.6   Update an already existing frame, if necessary. For example, World Health Survey (WHS) administrators have suggested that frames that are two years old or more require updating (Üstun et al., 2005). However, that is only a rough rule of thumb. In mobile societies, a frame that is one year old may need to be updated, while in other  societies, even older frames might still be accurate.

2.6.1   If the frame is a pre-existing list, contact the provider of the list for the newest version and its quality documentation.

2.6.2   If the frame is an area probability sample and the target population has undergone extensive movement or substantial housing growth since the creation of the frame, then updating the PSUs and SSUs will be required. However, what is most important is the quality of the enumerative listing.

2.6.3   Select the sampling frame based on the undercoverage error vs. cost tradeoff.

Lessons learned

2.1   Most countries do not have complete registers of the resident population and, therefore, construct area frames for sample selection.

2.1.1   Some surveys in countries with limited resources have found that it can be difficult to enumerate the rural, impoverished areas (Bergsten, 1980Dunzhu et al., 2003; Kalton, 1983) and, consequently, surveys in these countries may under-represent poorer or more rural residents. However, not all survey methodologists agree with the opinions expressed by these authors regarding enumeration in rural, poor areas. Those who disagree argue that the poor enumerations are mainly due to low expectations and insufficient training and supervision. Regardless, if the statistic of interest is correlated with income and/or urbanicity, the sample estimate will be biased. For example, the Tibet Eye Care Assessment, a study on blindness and eye diseases in the Tibet Autonomous Region of China, used an area sampling frame (Dunzhu et al., 2003). One of the PSUs was the township of Nakchu, an area of high elevation that is primarily populated by nomadic herders. Because of the elevation and rough terrain, Nakchu proved difficult to enumerate accurately. As a result, the survey sample underrepresented the residents of the roughest terrain of Nakchu. This was potentially important, as ophthalmologists believe that Tibetans who live in the most inaccessible regions and the highest elevation have the highest prevalence of eye disease and visual impairment.

2.1.2    Even when available, the quality and recency_of census data and/or administrative lists can vary across countries. Data from other sources might be available. For example IPUMS-International is an “effort to inventory, preserve, harmonize, and disseminate census microdata from around the world. The project has collected the world's largest archive of publicly available census samples. The data are coded and documented consistently across countries and over time to facilitate comparative research. IPUMS-International makes these data available to qualified researchers free of charge through a web dissemination system” (Minnesota Population Center, 2015).

2.2   Local residents can help produce maps for an area probability sample. When measuring the size of the rural population in Malawi, researchers used statistical methods to determine the sample size and selection of villages. Then they asked members of the selected communities to help draw maps, including exact village boundaries, key landmarks, and each individual household (Barahona & Levy, 2006).

2.3   Technology can also permit the associated use of GPS to facilitate sample selection. Vanden-Eng et al. (2007) used PDAs equipped with GPS units in household surveys to rapidly map all households in selected areas, choose a random sample, and navigate back to the sampled households to conduct an interview in Togo and Niger. Incorporating GPS into the sampling method allows researchers to select a random sample of households for interviewing from a complete and up-to-date listing of the households and, as a result, the probability of selection is known. The GPS data collected also provides geospatial information for reports and analyses. Vanden-Eng et al. (2007) were able to generate a preliminary report of survey findings from the aggregated data (including maps of the districts sampled showing all of the households and their sample inclusion status) within a few days of completion of data collection. This can be particularly useful in situations where data are needed quickly to guide public health action, such as in routine monitoring and evaluation and rapid needs assessments.

2.4   Eckman, Himelein, and Dever (forthcoming) discuss several sampling approaches using geographic information system (GIS) tools for face-to-face household surveys. Examples include the use of Google Maps, Google Earth, satellite photos, handheld global positioning system (GPS) devices and location-enabled applications on mobile phones. These technologies can be used to form and select clusters in early stages of selection and to select households in a later stage of selection.

2.5   Koch (forthcoming) provides an overview on within-household selection methods for face-to-face household surveys. Various methods exist, and the two commonly used ones are Kish (see Appendix B) and birthday methods. The Kish method is believed to be the gold standard for within-household selection. The technique requires all eligible persons in a household to be listed. A person is sampled with equal probability from all eligible persons. Birthday methods use household members’ birthday information to select the respondent. Both within-household selection methods are used in European Social Survey (ESS) countries.

⇡ Back to top

3.   Choose a selection procedure that will randomly select elements from the sampling frame and ensure that important subgroups in the population will be represented.

Rationale

Sample selection is a crucial part of the survey lifecycle. Since we cannot survey every possible element from the target population, we must rely on probability theory to make inferences from the sample back to the target population.

Procedural steps

3.1   Consider only selection methods that will provide a probability sample.

3.1.1   Statisticians have developed procedures for estimating sampling errors in probability samples which apply to any type of population.

3.1.2   Random sample selection protects the researcher against accusations that his or her bias, whether conscious or unconscious, affected the selection.

 3.2   Identify the optimal sampling method available in each country. Below are summaries of each selection method.See Appendix C for additional information about each selection method.

3.3   Consider Simple Random Sampling (SRS) without replacement. In SRS, each element on the frame has an equal probability of selection, and each combination of n elements has the same probability of being selected. Due to the benefits of stratification, this technique is seldom used in practice.

3.3.1   Advantages of SRS:

  • The procedure is easy to understand and implement.

3.3.2   Disadvantages of SRS:

  • The costs in attempting to interview a simple random sample of persons can be quite high.
  • SRS provides no assurance that important subpopulations will be included in the sample.

3.4   Consider Stratified Sampling (see Appendix C for a detailed description). Stratified sampling uses auxiliary information on the sampling frame to ensure that specified subgroups are represented in the sample and to improve survey precision. Virtually all sample implementations use some form of stratification.

3.4.1   Some examples of commonly used stratification variables are:

  • Region of the country.
  • State/province.
  • City/town, community, municipality.
  • Postal code.
  • Metropolitan status/urbanicity.
  • Size of sampling unit (e.g., population of city).
  • Race/ethnicity.
  • National origin.

3.4.2   Advantages of stratified sampling:

  • Stratified samples can be ether proportionate (which is general used to improve the precision of estimates of the total population) or  disproportionate (to help improve sub-group estimates)
  • Depending

3.4.3   Disadvantages of stratified sampling:

  • For any given frame stratification variables may be limited.
  • No gains in precision will be seen if the stratification variables are not correlated with the statistic(s) of interest. In some cases, the precision may even decrease.

3.5   Consider Systematic Sampling to reduce the operational effort needed to select the sample. In systematic sampling, every kth element on the sampling frame is selected after a random start.

3.5.1   Advantages of systematic sampling:

  • The operational time necessary to select the sample can be reduced substantially.
  • If the sampling frame is sorted into groups or ordered in some other way prior to selection, the systematic sampling method will select a proportionately allocated sample (see description below of stratified sampling). This is often referred to as “implicitly stratified sampling.”

3.5.2   Disadvantages of systematic sampling:

  • If the key selection variables on the sampling frame are sorted in a periodic pattern (e.g., 2, 4, 6, 2, 4, 6…) and the selection interval coincides with periodic pattern, the systematic sampling method will not perform well (Kalton, 1983). If periodicity is a problem, several systematic samples can be selected and concatenated to form the total survey sample.
  • If the list is sorted in a specific order before selection, the repeated sampling variance

3.6   Consider Cluster Sampling (see Appendix C for a detailed description). With cluster sampling, clusters of frame elements are selected, rather than selecting individual elements one at a time. Within the selected clusters, we can interview all or a sample of households. We can even do multi-stage cluster sampling where we select an additional sample of clusters inside the larger selected clusters.

3.6.1   Advantages of cluster sampling:

  • When survey populations are spread over a wide geographic area and interviews are to be done face-to-face, it can be very costly to create an element frame and visit n elements randomly selected over the entire area.
  • A full frame of all elements in the entire population is not required only the elements within selected clusters are needed. This will reduce listing costs.

3.6.2   Disadvantages of cluster sampling:

  • Estimates are not as precise as with SRS, necessitating a larger sample size in order to get the same level of precision. See Appendix D for more information about effective sample size.

3.7   Consider Two-Phase (or Double) Sampling (see Appendix C for a further description). The concept of two-phase sampling is to sample elements, measure one or more variables on these 1st-phase elements, and use that information to select a 2nd-phase subsample.

3.7.1   A common application is to collect 1st-phase data that is used to stratify elements for the 2nd-phase subsample.

3.7.2   Survey samplers use two-phase sampling to help reduce nonresponse, with the stratifying variable from phase one being whether the person responded to the initial survey request. For example, samplers might select a subsample of nonrespondents and try to entice the nonrespondents to participate by offering incentives.

3.8   Consider Replicated (or interpenetrated) Sampling. Replicated sampling is a method in which “the total sample is made up of a set of replicate subsamples, each of the identical sample design (Kalton, 1983).”

3.8.1   Advantages of replicated sampling:

  • It allows the study of variable nonsampling errors, such as interviewer variance.
  • It allows for simple and general sampling variance estimation (see Data Processing and Statistical Adjustment for further explanation, especially regarding the methods Balanced Repeated Replication and Jackknife Repeated Replication).

3.8.2   Disadvantages of replicated sampling:

  • There is a loss in the precision of sampling variance estimators; a small number of replicates leads to a decrease in the number of degrees of freedom when calculating confidence intervals.

3.9   Consider using a combination of techniques such as a stratified multistage cluster design.

3.9.1   Most surveys in countries with limited resources are based on stratified multistage cluster designs (Yansaneh, 2005). The combination of these techniques reduces data collection costs by clustering while striving to increase or maintain precision through stratification.

Lessons learned

3.1   Probability sampling at every stage generally requires more labor and funding than other methods. Therefore, some 3MC studies have used probability sampling in the first stage of selection and then allowed quota sampling or substitution to occur at later stages (Chikwanha, 2005; Heeringa & O’Muircheartaigh, 2010). However, a survey that uses a nonprobability sampling method at any stage of selection cannot estimate the sampling error of the estimates (Heeringa & O’Muircheartaigh, 2010). Therefore, the coordinating center should make every effort to promote the use of a full probability sample and remove any obstacles that would prevent participating countries from using probability methods at each stage of selection. For the first few waves of data collection, the International Social Survey Programme (ISSP) allowed countries to use nonprobability methods at the household level.  After recognizing the problem this caused in variance estimation, the ISSP has required countries to use full probability samples since 2000 (Häder & Gabler, 2003).

3.2   Existing 3MC surveys have employed various strategies for selecting a probability sample.

3.2.1   Round 6 of the Afrobarometer Survey uses a clustered, stratified, multi-stage, area probability sample.  The sampling design has four stages in urban areas: (1) stratify and randomly select primary sampling units, (2) randomly select sampling start-points, (3) randomly choose households, and (4) randomly select individual respondents within households ; and five stages in rural areas: (1) randomly select secondary sampling units (SSU), (2) randomly select two primary sampling units (PSU) from each SSU, (3) randomly select sampling start-points from each PSU, (4) randomly choose households, and (5) randomly select individual respondents within households (AfroBarometer, 2014).

3.2.2   Sample designs for Round 5 of the European Social Survey (ESS) must use random probability sampling at every stage.  Samples are designed by a sampling expert or panel and may include clustering and stratification. Quota sampling and substitutions are not allowed although subgroups may be over-sampled. Sample designs and frames must be documented in full and be pre-approved by a sampling expert or panel. The target minimum response rate is 70% (ESS, 2014).

3.2.3   Sampling frames and designs for the Living Standard Measurement Study Survey (LSMS) vary across participating countries but generally consist of two stages. In the first stage, the sample frame is developed from census files and Primary Sampling Units are randomly selected with probability proportionate to size; in the second stage, households (usually 16) are randomly selected from each of the designated Primary Sampling Units.  Clustering and stratifying are permitted, but all sampling procedures must be documented and made available to data analysts (LSMS, 1996).

3.2.4   Survey of Health, Aging and Retirement in Europe (SHARE) sampling designs vary by country but all are required to be probability samples. Three sampling designs may be used: (1) stratified simple random sampling from national population registers, (2) multi-stage sampling using regional or local population registers, or (3) single or multi-stage sampling using telephone directories followed by screening in the field (De Luca et al., 2015).

3.2.5   Sampling frames for the World Mental Health Survey vary across participating countries, but generally consist of three types of sampling frames: (1) individual contact information databases such as national population registries, voter registration lists, or household telephone directories, (2) multistage area probability sample frames, or; (3) hybrid multistage frames that combine area probability methods and a individual contact database in the final stages. Sampling designs vary across participating countries, including stratification and clustering, but probability sampling is required at all stages. The target minimum response rate is 65% (Kessler et al., 2008).

3.2.6   Probability sampling is strongly recommended, but not required, in the World Value Survey; any deviations from probability sampling are to be reported in the Methodology Questionnaire report (http://www.worldvaluessurvey.org/wvs).

3.3.  When there is very little information on the population, surveys (including Europe-wide surveys, surveys in developing countries and worldwide international surveys) sometimes use random route (random walk) as part of the multistage face-to-face interview sampling method.

3.3.1   For each randomly-chosen sampling points (e.g., urban units, small cities, or voting districts), interviewers are assigned with a starting location and provided with instructions on the random walking rules – e.g., which direction to start, on which side of the streets to walk and which crossroads to take. Households are selected by interviewers following the instructions. The routes end when the predefined number of respondents (or households) is achieved (Bauer, 2016). Since the probability of the selected household is unknown, this method is categorized as non-probability sampling methods (Bauer, 2016).  See Table 1 in Bauer (2016) fora list of surveys which used this method in data collection.

3.3.2    The advantage of random route sampling is that it is usually cheaper and easy to conduct. Most importantly, this sampling method makes it possible when there is no list of respondents available (Bauer, 2016).

3.3.3    This method assumes that each household in this sampling point has equal probability to be selected. However, a recent study evaluated the bias in random route sampling using registration office data (to verify the impact of selection errors on survey results), and found that “all tested routes strongly violate the equal probability assumption and lead to biased expected values in multiple variables” (Bauer, 2016). Additional error source also includes incorrect interviewer behaviors, which can lead to coverage and sampling error (also see Paradata and Other Auxiliary Data). 

3.3.4   In a review of a survey of blindness in war-torn southern Sudan, Kuper and Gilbert (2006) note that the estimate of trachoma prevalence is higher than previously reported for Africa, including other studies in conflict-affected places. Kuper and Gilbert suggest that methodological problems, including an element of subjectivity resulting from the random route method employed for household selection, may have resulted in an over estimate of the disease.

⇡ Back to top

4.   Determine the sample size necessary to meet the desired level of precision for the statistics of interest at population or subgroup levels for the different potential sample selection procedures.

Rationale

After choosing a sample design, and before selecting the sample from the sampling frame, the sample size must be determined. The sample size takes into account the desired level of precision for the statistic(s) of interest, estimates of the statistic of interest from previous surveys, the design effect, and estimated outcome rates of the survey. (See Lynn, Häder, Gabler, & Laaksonen (2007) for a detailed treatment of the approach used in the European Social Survey. For a more extensive example of sample size calculation, see Appendix D).

Procedural steps

4.1   Specify the desired level of precision, both overall and within key subgroups. Practical experience has determined that often it is easiest for sponsors to conceptualize desired levels of precision in terms of 95% confidence intervals.

4.2   Convert these 95% confidence intervals into a sampling variance of the mean or proportion.

4.3   Obtain an estimate of S2 (population element  variance).

4.3.1   If the statistic of interest is not a proportion, find an estimate of S2 from a previous survey on the same target population or from a small pilot study.

4.3.2   If the statistic of interest is a proportion, the sampler can use the expected value of the proportion (p), even if it is a guess, to estimate S2 by using the formula s2= p(1-p).

4.3.3   If no information about p is available, a researcher can assume that it is 50%, which will yield a conservative estimate of the sample size.

4.4   Estimate the required number of completed interviews for an SRS by dividing the estimate of S2by the desired sampling variance of the mean. See Cochran (1977) for more on sample size computation for SRS.

4.5   Multiply the number of completed interviews by the design effect to account for a non-SRS design.

4.6   Calculate the necessary sample size by dividing the number of completed interviews by the expected response rate, eligibility rateand coverage rate.

4.6.1   The sampler can estimate these three rates by looking at the rates obtained in previous surveys with the same or similar survey population and survey design.

Lessons learned

4.1   Prior to the first implementation of the European Social Survey (ESS), many of the participating survey organizations had never encountered the concepts of sample size determination and calculating design effects (Lynn et al., 2007). Therefore, the ESS expert sampling panel spent considerable time explaining these. In return, the organizations that were new to these methods were very enthusiastic to learn about them, and eager to meet the standards of the coordinating center. In fact, after completing Round 1 of the study, many nations commented that designing the sample was one of the most educational aspects of the entire survey process, and had significantly improved the survey methods within their country.

4.2   Sample size frequently varies among countries participating in cross-cultural surveys. In Round 6 of the Afrobarometer Survey, sample size ranges from a minimum of 1,200 respondents to 2,400 or more in extremely heterogeneous areas (Afrobarometer, 2014); sample size ranges from 800 to 3,200 respondents in the Asian Barometer study (Asian Barometer, 2016); Round 5 of the European Social Survey (ESS) requires a minimum of 800 respondents for participating countries that have a population of less than two million, 1,500 from larger countries (ESS, 2015); the International Social Survey Programme (ISSP) requires a minimum of 1,000 respondents, with a goal of 1,400 respondents (ISSP, 2015); sample size ranges from 1,600 to 5000 households in the Living Standard Measurement Study Survey (LSMS) (LSMS, 1996); the Survey of Health, Ageing and Retirement in Europe (SHARE) requires 1,500 respondents from each participating country (De Luca et al., 2015); samples in the World Mental Health Survey range from 2,357 (Romania) to 12,992 (New Zealand) (Kessler et al., 2008); and the World Value Survey requires a minimum of 1,000 respondents (World Values Survey, 2016).

⇡ Back to top

5.   Institute and follow appropriate quality control procedures at each step of the sample design process.

Rationale

Development and implementation of quality control procedures for the sample design are necessary to ensure the highest level of coverage possible and to maintain a probability sample that meets the desired level of precision for key survey statistics.  If a failure to meet those standards is detected, protocols should be in place to remedy the failure. In addition, monitoring of procedures related to the sample design of the study should inform efforts to improve the quality and cost-effectiveness of the study over time.

Procedural steps

5.1   Define the target population for the study across all participating countries/cultures as well as the target population within each country/culture. If the study design does not change over time, strive to keep each target population, both overall and within each participating country, consistent over time.

5.2   Prior to selecting sample elements or sampling units, provide the data collection staff with a list of all of the variables on the sampling frame and ask which variables they would like and the format in which they would like these variables delivered once sampling is complete for data collection purposes. 

5.2.1   After sample selection, check that each selected sampling unit or element contains this information and is in the specified format.

5.3   If possible, use a responsive survey design (Groves & Heeringa, 2006; Groves et al., 2008to help achieve an optimal sampling design (see Survey Quality, Paradata and Other Auxiliary Data, and Data Collection: General Considerations for more information about responsive survey designs). A responsive survey design uses prespecified paradata (quantitative indicators of the data collection process such as “contact attempts” or “interviewer success rate”) for intervention during data collection.

5.3.1   Advantages of responsive survey designs are the prespecification of interventions instead of ad hoc decisions and the possibility to target efforts on hard to interview groups.

5.3.2   A disadvantage is that the survey designers walk a thin line between full probability and quota samples if they deviate from carefully predefined paradata-driven interventions.

5.4   After each stage of selection, generate frequency tables for key variables from the frame of sampling units to check for the following:

5.4.1   Overall sample size and within stratum sample size.

5.4.2   Distribution of the sample units by other specific groups such as census enumeration areas.  

5.4.3   Extreme values.

5.4.4   Nonsensical values.

5.4.5    Missing data.

5.5   Create a unique, sample identification code for each selected sampling unit. This code will allow identifying information to be easily removed after completing data collection. The code value can then be used in place of other values to preserve confidentiality when the original values may contain information that could lead to identifying respondents.

5.6   Whether the participating country or the coordinating center is selecting the sample, assign a second sampling statistician within that organization to check the sample design methodology and the statistical software syntax of the survey’s primary sampling statistician.

5.7   Save all data files and computer syntax from the statistical software package used during the sample design process in safe and well-labeled folders for future reference and use.

Lessons learned

5.1   The construction and maintenance of sampling frames constitute an expensive and time-consuming exercise. If a participating country determines that no sampling frame meeting the specified coverage level of the target population exists, they can create a frame from sources such as census data collected by national statistics offices. However, one should be aware that official statistics differ greatly in accuracy, as well as availability, from country to country.

5.2   As discussed in the Lessons Learned section of Guideline 2, the decision to stray from full probability sampling reflects the conflict between standardization and flexibility in 3MC surveys. However, it bears repeating that without probability sampling, one cannot make justifiable inferences about the target population from the sample estimates.

⇡ Back to top

6.   Document each step of the sample selection procedure.

Rationale

Over the course of many years, various researchers will analyze the same survey data set. In order to provide these different users with a clear sense of how and why the data were collected, it is critical that all properties of the data set be documented. In terms of the sample design and selection, the best time to document is generally shortly after sample selection, when the information regarding sample selection is fresh in one’s mind.

Procedural steps

6.1   Have participating countries document the sample selection procedure while selection is occurring or shortly thereafter. Ideally, set a deadline that specifies the number of days after sample selection by which each participating country must send sampling selection documentation to the host survey organization. Be sure to allow for appropriate time to review and revise documentation when setting the deadline (see Tenders, Bids and Contracts and Study Management.)

6.2   Include the following:

6.2.1   A clear definition of the survey population, as well as the differences between the target population and survey population.

6.2.2   The sampling frame:

  • Both the sampling frame used and the date the frame was last updated.
  • A description of the development of the sampling frame and the frame elements.
  • A description of the how well the sampling frame is thought to cover to target population and the potential for coverage error.

6.2.3   The data file of selected elements:

  • A descriptive and distinct variable name and label.
  • Unique variables that contain the selection probabilities at each stage of selection as well as the overall selection probabilities.  If a participating country used a nonprobability method in at least one stage of selection and therefore the selection probabilities are unknown, ensure that this is clearly documented.   
  • A clear description of all variables in the selected element data file, with all variable names, an accompanying description and a codebook, which provides question-level metadata that are matched to variables in a dataset.
  • The statistical software syntax used for checking the dataset of selectedsampling unitsor elements.

6.3   For each sample, indicate how many stages were involved in selecting the sample (include the final stage in which the respondent was selected within the household, if applicable), and a description of each stage, including how many sampling units were selected at each stage.

6.3.1   Examples of different stages include:

  • State/province.
  • County or group of counties.
  • City/town, community, municipality.
  • Census/election district.
  • Area segment/group of neighborhood blocks.
  • Housing unit/physical address (not necessarily the postal address).
  • Postal delivery point/address.
  • Block of telephone numbers (e.g., by regional prefix).
  • Telephone number.
  • Person selected from a household listing or a non household listing (a list, registry or other source)

6.3.2   Examples of how sampling units were selected:

  • All selected with equal probability.
  • All selected with probability proportional to size; specify the measure of size used. (See Appendix C for more on probability proportional to size sampling methods.)
  • Some units selected with certainty, others selected with probability proportional to size; describe the criteria used for certainty selection.
  • Census/enumeration (all units selected with certainty).
  • Units selected using a nonprobability method (e.g., convenience sample, quota sample).

6.3.3   At each stage of selection, describe the stratification variables and reasons for choosing these variables

6.3.4   At each stage of selection, explain the allocation method used and the sample size for each stratum at each stage of selection. (See Appendix C for more on allocation methods in stratified sampling.)

6.4   If systematic sampling was used at any stage of selection, indicate whether the frame was sorted by any variables prior to systematic selection in order to achieve implicit stratification. If this is the case, describe the variable(s).

6.5   Describe the time dimension of the design ( i.e., one-time cross-sectional, fixed panel, rotating panel design).

6.5.1   If a panel study:

  • State how many previous waves or rounds of data collection there have been for this panel study.
  • Describe the initial sample design for the panel study and any subsequent modifications to the design that are important in documenting this study.

6.5.2   If a rotating panel design:

  • Fully describe the rotating panel design for the study (e.g., fresh cross-section is drawn each month and respondents are interviewed once that month, and then reinterviewed once six months later).
  • State the anticipated precision
  • Explain any problems encountered during the sampling process and any deviations from the sampling plan during implementation.

6.5.3   Additional sampling documentation:

  • Report any (additional) subsampling of eligible respondents, carried out in order to control the number of interviews completed by respondents with particular characteristics (e.g., one in two eligible males was interviewed, one in four eligible persons with no previous history of depression was interviewed (describe protocol)).
  • Describe any use of replicates (see Data Processing and Statistical Adjustment).
  • Explain if releases (nonrandom subsets of total sample) were used or the entire sample was released to data collection staff at the start of the study.
  • Recount in detail any substitution

Lessons learned

6.1   As the procedural steps outlined above show, selecting a sample can involve many detailed steps that may be hard to recall after the fact. For example, the coordinating center for the World Mental Health Survey began gathering sampling documentation for weighting and other purposes after many of the participating countries had finished data collection. They found that some countries had a difficult time recalling all the necessary details, such as the sample size for each stratum at each stage of selection. It is wise to document sampling procedures in detail shortly after sample selection(see Data Processing and Statistical Adjustment for further explanation of the weighting process).

⇡ Back to top