Cognitive Interviewing

 
Kristen Cibelli Hibben & Julie de Jong 

Introduction

Cognitive interviewing (CI) is a method for evaluating survey questions to determine whether the true meaning of the question, as intended by the researcher(s), is conveyed to respondents, and more generally whether the question is functioning as intended. It is one out of several methods available to pretest survey questionnaires (see Table 1 in Pretesting for a detailed list of different pretesting methods). While perhaps best known as a method for evaluating individual survey questions, CI can also be used to pretest other materials developed to convey or request information from survey respondents such as recruitment letters or scripts, instructions, and consent forms .

CI initially derived from the application of cognitive psychology to survey methodology and the cognitive aspects of survey methodology (CASM) movement in the 1980s. The CASM movement brought attention to the issue of measurement error (and more specifically, response error) and established the idea that respondents’ individual thought processes must be understood to assess validity and potential sources of error ; see for further discussion. The underlying idea is that, as explains, “the respondent’s cognitive processes drive the survey response, and an understanding of cognition is central to designing questions and to understanding and reducing sources of response error” (p. 23).

Traditionally, CI has been used as a pretesting method to identify question-response problems before fielding the main survey . CI can be practiced in different ways, but it typically consists of in-depth interviews with a small, purposive sample of respondents. Respondents are first asked to answer a survey question, and are then asked to describe how and why they answered the way they did. The interviewer typically follows up by probing for other specific information relevant to the question or to the answer given. CI in this way permits an in-depth exploration of the response process (which is comprised of the following processes: comprehension, retrieval, judgment, and response ) and the detection of potential difficulties respondents may have with forming an answer or providing answers that are consistent with a question’s intent. Questions found to have problems can then be modified to reduce errors in the response process.

However, ideas of what constitutes CI are shifting as notions of the question response process broaden (see for discussion). The use of CI has expanded recently beyond identifying problems with questions or whether respondents are interpreting the questions as intended to investigating the constructs captured by questions and whether or not those same constructs are captured comparably across culturally diverse respondents . As notes, this is beneficial for pretesting questions as well as determining how existing questions operate, providing a better understanding of how to interpret and use resulting survey data.

CI is an important step in questionnaire development to reduce measurement error (see Survey Quality for further discussion of measurement error and its relationship to other elements in the Total Survey Error (TSE) paradigm). Measurement error represents the difference between a theoretical ‘true’ response and that which is obtained by the survey. If respondents consistently, and in the same direction, report answers deviating from the ‘true’ response, then this can produce bias in the overall statistic.

Researchers are increasingly recognizing the benefit of CI when applied in multinational, multicultural, or multiregional surveys, which we refer to as ‘3MC’ surveys. Some refer to cognitive interviewing in the context of 3MC surveys as ‘cross-cultural cognitive interviewing’ (CCCI). In a research synthesis, reviews 32 CCCI studies located in peer-reviewed journals and books, along with key unpublished sources.

CI is important (some would argue essential) in a 3MC context because a well-designed question in the source language and associated culture could have a different meaning or be inappropriate for a different target culture. Cognitive interviews can help uncover problems with translated and adapted survey items to a target language and culture that is different from the original language for which it was designed (for a review, see ; see also ). Furthermore, if also used to investigate the constructs captured by the questions, CI can help ensure validity and cross-cultural comparability, which is particularly essential for studies seeking to make comparative estimates .

The application of CI in 3MC studies or in countries without a strong survey research tradition poses a number of methodological and practical challenges. When testing translated versions of a questionnaire, for example, there are additional issues to investigate in cognitive testing beyond those encompassed by the standard cognitive model. For example, it is important to consider how questions may function pragmatically across languages, given differences in social desirability bias and how answer scales and response options are used (see and Guideline 1 for more details). Research has also shown that different cultural and linguistic groups may respond differently to particular CI methods. Special attention should be paid to orienting and preparing respondents for the cognitive interviewing task when carrying out cognitive interviews with non-English speakers or in contexts with little survey research tradition (see further discussion in Guideline 4).

Practically, carrying out cognitive interviewing in 3MC studies typically involves coordination with multiple partners in various locations, across different time zones, and speaking a variety of languages . The availability of trained staff and respondents’ familiarity with this research approach also often varies widely. As notes, in some cases, those who are conducting the interviews may have never before conducted a cognitive interview, or participated in qualitative research of any kind. When working with multiple languages, it may be necessary to transcribe and translate interviews (or detailed summary notes, at a minimum) into a common language so that they can be reviewed to ensure quality and comparability of interviewing approaches. Furthermore, CI analysis and results must themselves be translated into a common language, if these are to be discussed by the larger group . Even when a pretest is carefully carried out, interpreting and integrating results may prove difficult. Results may highlight problems with survey questions, but solutions may be less obviousor identified solutions may solve a problem in one context, but create harmonization problems with the measure in other contexts, particularly for items that may have been used widely and are needed for comparability across time . Furthermore, changes made to a questionnaire based on cognitive interviewing or other pretest outcomes may necessitate a subsequent round of testing. Where possible, we highlight possible solutions and lessons learned to address these and other challenges. We also discuss alternative or complementary approaches to standard cognitive interviewing such as online probing and crowdsourced cognitive interviewing .

Cognitive interviewing should be integrated into the overall survey design process and is a part of the questionnaire development process. The method is valuable for identifying those questions that present a challenge to respondents before the field period, providing an opportunity for revision before the main study is fielded. If a question is identified as ambiguous or difficult by the respondent, it may indicate that it will not suffice for the proposed analyses. Survey questions that are confusing for the respondent may also lend themselves to increased interviewer error; that is, if a question is confusing to the respondents, interviewers may offer differing clarifications, or attempt to reword the questions themselves, which can lead to further measurement error.

In the survey lifecycle, cognitive interviewing typically precedes a field pilot study. The procedural steps in this chapter are written with the assumption that if a field pilot study is planned for a project, it will be subsequent to the cognitive interview process.

It is important to keep in mind some limitations of cognitive interviewing. First, cognitive laboratory subjects are not survey respondents. Second, the cognitive laboratory environment is different from the field, and the motivation of cognitive interviewing respondents is likely to be different from respondents in the field; for example, laboratory respondents are likely to think harder about the questions and are very unlikely to break off, particularly if they are receiving an incentive of some kind to participate. It may be easy, therefore, to underestimate potential problems with respondent willingness to answer a question or apply effort in the field. Furthermore, because cognitive interviewers tend to focus on the respondent side of the interaction, they are likely to overlook features in the questionnaire that might cause difficulties for interviewers in the field. Finally, sample sizes for cognitive interviews are generally small, and therefore unlikely to include examples of all types of people or uncover every potential problem that may occur among the eventual survey population, particularly in a 3MC survey, for which it is rarely possible to carry out cognitive interviewing in every language or participating country. The resulting data is also not typically suitable for statistical estimation. Rather, the focus of cognitive interviews is on producing in-depth data of a qualitative nature that, nonetheless, frequently yield valuable insights and improvements to the quality of survey questionnaires.

⇡ Back to top

Guidelines

Goal: To use cognitive interviewing as a step in the questionnaire design process to evaluate whether the true meaning of questions or other materials developed to convey or request information from survey respondents is conveyed to respondents as intended by the researcher(s). More recently, some researchers have also discussed the use of cognitive interviewing as a possible approach to quality assessment, which can be used during or even after data collection to assess item functioning .

⇡ Back to top

1. Identify objectives and design the cognitive interviewing study.

Rationale

The first step in cognitive interviewing is to identify the objectives of the cognitive interviews. Subsequently, it is important to develop a study design that considers the timing of the cognitive interviewing in the survey lifecycle, the number of rounds of cognitive interviews, the mode of both the cognitive interviews and the eventual survey, the number of interviews needed, the expected length of the interview, and other logistical considerations.

Procedural steps

1.1     Identify the overall testing objectives (what the CI is attempting to accomplish) and any research questions, both general and specific to individual survey items, as well as the question objectives for each item being evaluated. Note that the overall testing objectives and item-level objectives, while related, are different, and equally important to define.

1.1.1    The overall testing objective of any cognitive interviewing study is typically to determine whether the survey questions included in the study capture the intended construct and to identify any difficulties that respondents experience when formulating a response. However, the level of emphasis on question function (i.e., what the questions are capturing) vs. question defects can vary. See Guideline 2 for a discussion of different overall approaches to cognitive interviewing, which differ mainly in terms of the testing objective, or what is trying to be accomplished through the cognitive testing.

1.1.2    It is important to articulate the specific objective(s) for each cognitive interviewing study and each round (if there are multiple rounds).

1.1.3    Objectives can also relate to other non-questionnaire materials to be tested, such as consent forms and confidentiality agreements, recruitment materials, advance letters, brochures to mitigate refusal rates, etc.

1.1.4    For each evaluated item, consider whether the focus is on the construct captured by the question in general, or to investigate suspected problems with a particular stage of the response process (e.g., retrieval) in a particular survey question.

1.1.5    For each item, consider whether the objective is comparativethat is, whether the objective is to identify the various constructs captured by individual questions and to determine whether they are consistently captured across groups of respondents.

1.1.6    If testing a translated questionnaire, objectives should include investigating issues that can arise when translating survey questions, including differences in social desirability bias, level of diction, naturalness of language, and how answer scales and response options are used .

1.1.7    Experimental designs can also be used with cognitive interviewing studies. For example, respondents can be randomly assigned to different versions of the instrument in order to examine the effect on respondent comprehension of different question wordings, different sets of instructions, or a different question order. However, it is important to keep in mind that sample sizes for cognitive interviews are usually relatively small; although quantitative results may be computed, statistical power may be limited to detect differences. Nonetheless, a combination of qualitative and quantitative results may offer useful insights regarding different patterns of response.

1.2     Consider the timing of cognitive interviews in the survey lifecycle.

1.2.1    Cognitive interviewing is one of several activities in the developmental phase of the survey lifecycle. Ideally, cognitive interviews are conducted with survey questions that have passed appropriate ‘first-stage’ tests , such as focus groups to develop key concepts and expert review, but early enough in the process so that changes can be made, and possibly re-tested, based on the results.

1.2.2    It is important to consider the timing when cognitively testing translated questionnaires. In addition to uncovering translation problems and problems related to cultural differences indicating the need for adaptation, cognitive testing of a translated questionnaire may also reveal problems with the source questionnaire. In fact, results from some previous multilingual cognitive interviewing projects suggest that problems with the source questionnaire may be more common than translation or cultural portability errors . Therefore, it is beneficial to organize and time the questionnaire development process such that results from cognitive interviewing of translated versions can inform modifications to the source questionnaire.

1.2.3    has also discussed CI as a ‘quality assessment’ tool, which can be used during or even after data collection to assess item functioning, even after there is no opportunity to change the items. For discussion of a descriptive approach to CI as an approach to quality assessment, see Guideline 2.

1.3     If working in the context of a 3MC survey, consider the selection of countries and languages in which to do cognitive interviews.

1.3.1    Ideally, cognitive interviews would be carried out in every country or language in which the survey will eventually be fielded. However, the financial and human resources required to include all countries or languages in cognitive testing often makes this impossible. When it is not possible to test in all languages/countries, a potential alternative approach is to select languages/countries that cover the main language families, regional subgroups, or countries that will be included in the main study, as resources permit. For example, for a cross-national survey among EU countries, one might attempt to conduct cognitive testing in one of each of the following language families: Slavic (e.g., Czech, Polish), Romance (e.g., French, Italian), Germanic (e.g., Dutch, Swedish), as well as the different geographical regions of Eastern Europe (e.g., Czech, Polish), Northern Europe (e.g., Swedish), Southern Europe (e.g., Italian). and Western Europe (e.g., Dutch, French).

1.4     Consider how many rounds of cognitive interviews will be administered, as well as how many interviews will be administered in each round.

1.4.1    It is helpful to carry out at least two rounds of cognitive interviews. The second round makes it possible to adjust the protocol or to test possible refinements made to the questions based on results from the initial round. It is important to reserve at least a week for each round .

1.5     Consider the mode of the eventual survey.

1.5.1    Face-to-face cognitive interviews are typically used for survey interviews that will be carried out face-to-face. Interviews intended for administration via telephone, however, often are also tested using face-to-face cognitive interviews . This allows for interviewers to observe any nonverbal cues. However, telephone surveys vary in other ways from face-to face interviews (see Data Collection: Telephone Surveys), so there are advantages to doing cognitive interviews via telephone as well. If possible, do multiple testing rounds including both face-to-face and telephone interviews.

1.5.2    It is possible for respondents to self-administer cognitive interviews. However, a self-administered approach may introduce substantial respondent burden because it relies on the respondent to follow skip patterns, and it may be difficult to train respondents adequately for the task and burdensome for the respondents to adequately record reactions to items. An alternative is to have respondents complete the questionnaire themselves (i.e. self-administered) and then have an interviewer debrief them afterward using scripted retrospective probes. Another method is to ask the respondents to self-administer the questionnaire item-by-item in real-time with the interviewer who then follows completion of each item with relevant probes, as exemplified in ‘s Three-Step Test Interview (TSTI), which involves pre-testing a self-completion questionnaire by observing actual instances of interaction between the instrument and respondents (the response process). Because this process mainly consists of cognitive processing (‘thinking’) and is therefore hidden from the observer, (concurrent) think-aloud is used as a technique for making the thought process observable. The TSTI consists of the following three steps: 1) concurrent thinking aloud aimed at collecting observational data; 2) focused interview aimed at remedying gaps in observational data; and 3) semi-structured interview aimed at eliciting experiences and opinions.

1.5.3    As notes, for surveys using computer-assisted personal interviewing (CAPI) for face-to-face interviews or computer-assisted telephone interviewing (CATI) for telephone interviews, it is most efficient to conduct cognitive testing prior to beginning programming for the instrument. Testing the cognitive aspects of computer-based instruments or Web surveys to be self-administered by respondents generally falls under usability testing (see Instrument Technical Design).

1.6     Consider the number of completed cognitive interviews that are needed for each round of testing.

1.6.1    The specific number of completed interviews in each round is dependent on the extent of the potential issues in the questionnaire, and may need to be evaluated after the first few interviews. In the very early design stage, two interviews may be sufficient; an initial cognitive interview followed by an additional interview after revisions subsequent to the first. In later stages, research is mixed regarding the ideal number of cognitive interviews, although “general guidance calls for…between 5 and 15 interviews” per round .

1.6.2    When determining the number of interviews per round, there should be an emphasis of quality over quantity. As notes, it is generally better to conduct a smaller number of careful interviews than to go through the motions to achieve a result that may appear impressive (e.g., 50 completions) but is lacking in information of value.

1.7     Consider the length of the cognitive interview.

1.7.1    It is not always possible to test an entire survey instrument in a cognitive interview because it would be too burdensome for the respondent, especially when cognitive probes are included. In surveys of significant length, consider including in the cognitive interview only those items with greatest potential issue. More effort is likely to be put into designing the cognitive test and analyzing the results when focusing on fewer questions .

1.7.2    Cognitive interviews should ideally last approximately 60 minutes, and should not exceed 90 minutes, as the cognitive approach can be demanding to the respondent. Some suggest that a ratio of between 2:1 and 3:1 for survey administration time to cognitive interviewing time is a good estimatethat is, a survey that would take 20 minutes to complete under normal circumstances should take 40 to 60 minutes in a cognitive interview . As it is also draining to the interviewer, no more than three cognitive interviews a day is ideal.

1.8     Determine how the results from the cognitive interview will be captured.

1.8.1    note that the respondent-interviewer interaction generated from the interview process is the raw data for a cognitive interviewing study. Thus, interviews should ideally be documented by audio or video recordings. As discuss, transcriptions may also be produced, but maintaining the actual interview recording (audio or video) is critical for two reasons: 1) it allows the investigator to go back to the original data source to verify potential inconsistencies or investigate themes that may not have been identified in interview summaries and 2) the recordings form an essential part of the audit trail that allows all findings to be traced back to the original data source.

1.8.2    When recording interviews, it is important to obtain consent from respondents. The consent form should mention that the interview will be recorded, but should also make it clear to respondents that the purpose is to collect information about the performance of the survey questions. If study clients/sponsor or anyone else will be given access to recordings, this should also be made clear in the consent form. For more information and example consent language see Ethical Considerations.

1.9     Determine the location where cognitive interviews will take place. Large organizations often have laboratories with one-way mirrors and permanent recording equipment. However, this is not necessaryany quiet and private room is all that is required, although it is also very helpful to have even a basic mode of audio recordingprovided that privacy and confidentiality requirements are met and that the respondent has consented.

1.10   Consider how the client/sponsor will be integrated into the cognitive interviewing process. It can be extremely helpful to share examples with clients/sponsors from audio or video recordings (after having obtained consent; see above), particularly if the discussion on an issue was particularly complex.

1.11   Consider using online or Web probing or crowdsourced cognitive interviewing ; essentially, cognitive interviewing without a cognitive interviewerstandardized probe questions are programmed to appear at strategic points throughout the questionnaire and are self-administered. These approaches can be used to complement more traditional lab-based or face-to-face cognitive interviews by confirming results found in an initial round of interviews. They may also offer a cost-effective way to cover additional language and national subgroups of respondents (see also for discussion).

Lessons learned

1.1     Use of cognitive interviewing to assess construct validity and comparability offers a particular advantage in the 3MC context. describes in detail the analytic goal of comparative cognitive interviewing studies; the relevant components of such a study, including data collection (e.g., the structure of the interview and data quality) and data analysis techniques; and strategies and tools for conducting such studies.

1.2     CI may also be used to investigate potential item-level contextual effects. A cognitive interviewing study conducted by the Survey Research Center at the University of Michigan examining select questions from the World Health Organization’s (WHO) Model Disability Survey (MDS) and the Washington Group Extended Question Set on Functioning (WG ES-F) in 2014 included an experimental design to determine whether or not the constructs captured were affected by the order in which the questions were asked, and if so, in what ways. Respondents in the study were randomly assigned to two versions of the questionnaire that differed in question order, and results clearly showed different patterns of interpretation based on the order in which the questions were asked, with one version of the questionnaire showing less of a question order effect. The results informed a number of recommendations for further improvements and testing of the questionnaire.

1.3     Organizations employ many different strategies when the eventual survey mode is not face-to-face. For example, one technique that has been used at the National Center for Health Statistics (NCHS) is to invite respondents into the lab but conduct the interviews over the phone, with the interviewer in one room and the respondent in another and the interviewers monitoring the respondents by video during the interview. This method allows for the testing of the telephone as a mode of administration, but also provides the interviewer with important nonverbal cues through the video monitoring .

1.4     A team at NCHS employed a mixed-methods approach to identify measurement error and examine comparability. Cognitive interviewing was done initially to identify the constructs captured by the questions and whether they were captured consistently across salient respondent groups. A Web survey was then conducted with closed-ended probes that were developed based on the preceding cognitive interviews to identify and quantify the phenomenon captured, identify measurement error, and examine comparability across different groups of respondents and different questions .

1.5     EvalAnswer, a tool that detects and provides tailored feedback to responses to open-ended questions, can help address the challenge of potentially poorer response quality in online self-administered forms of cognitive interviewing such as online probing, Web probing, and crowdsourced cognitive interviewing . The source code for EvalAnswer can be found here.

⇡ Back to top

2. Consider different approaches to cognitive interviewing.

Rationale

identifies two main approaches to cognitive interviewingthe reparative and the descriptivewhich differ mainly in terms of the testing objective, or what the cognitive testing is trying to accomplish. These approaches are not mutually exclusive, and can be considered different endpoints on a continuum; indeed, many studies contain elements of both , and selection of an approach is likely to depend on the stage of questionnaire development.

Procedural steps

2.1     Consider the reparative approach.

2.1.1    The reparative approach, which also refers to as “inspect and repair,” is devoted to “finding and fixing” problems (pp.18), and is the approach that has dominated the operations of cognitive laboratories that focus on improvements to survey questions and the reduction of response error . The reparative approach is typically applied when survey materials (e.g., survey questions, survey instructions, advanced letter, etc.) exist to be inspected.

2.2     Consider the descriptive approach.

2.2.1    The focus of the descriptive approach is on assessing differences in what questions capture, as opposed to detecting problems to be ‘fixed.’ The descriptive approach is applicable at multiple stages of the questionnaire development process. For example, has suggested conducting early interviews using expansive/ethnographic probes to help inform the formulation of more standard cognitive formulations for subsequent cognitive interviews. An example of an expansive probe is “Tell me more about that.” This type of expansive probing fits within the descriptive approach to cognitive interviewing, and can be particularly useful when studying unfamiliar populations or a range of cultural or ethnic groups. Such expansive probing, typically used in ethnographic interviewing, provides information that can also be used as a check on the accuracy of the answers to targeted survey questions and can enhance cognitive interviewing studies. See , Chapter 7 for more extensive discussion.

2.3     While not a new concept, what identify as the “interpretivist” approach has brought increasing attention to the more expansive or descriptive approach to cognitive interviewing. The interpretivist approach focuses on respondents’ interpretations, and specifically on how their own experiences inform their answers to survey questions . The goal of the interpretivist approach is to collect a storyor ‘narrative’that illuminates the respondent’s circumstances that informed how and why the respondent answered a question the way they did. Verbal probes (see Guideline 3 for more on verbal probing) are used to fill in gaps in respondents’ narrative or to help address apparent contradictions, rather than to understand the respondents’ thought processes. Rather than focusing on trying to understand the cognitive processes of respondents, the interpretivist approach aims to understand the meanings respondents bring to survey questions, or “…how their own lived experiences inform their answers to survey questions” , as well as the constructs captured by the question.

Lessons learned

2.1     When organized as a comparative cognitive interviewing study, cognitive interviewing using the descriptive or interpretivist approach can help determine whether or not those same constructs are captured across various groups of respondents, which is particularly valuable in a 3MC study . As discusses further, well-documented comparative cognitive interviewing studies can help inform decisions about question wording or translation and determine whether specific questions should even be asked within a given sociocultural context and how questions should be used and the resulting data analyzed and interpreted.

⇡ Back to top

3. Select a cognitive interviewing procedure or combination of procedures.

Rationale

There are two main types of procedures used in cognitive interviewing: think-aloud and verbal probing. In practice, a combination of these procedures is typically used. A detailed discussion of these procedures is available in , and the main advantages and disadvantages of each are outlined below. A few additional cognitive interviewing techniques that may complement think-aloud and verbal probing are also discussed.

Procedural steps

3.1     Consider using think-aloud interviewing. Subjects in the think-aloud approach are asked to verbalize their thought process as they think through and come up with answers to the survey questions. Interviewers may use general probes to keep the subject thinking and sharing their thoughts (e.g. “Tell me more about what you’re thinking”).

3.1.1    Think-aloud advantages: the think-aloud approach is fully open-ended, and may allow subjects to spontaneously volunteer information pointing to potential problems with the questions that are otherwise unanticipated. If interviewers use general verbal probes in conjunction with the think-aloud approach in order to keep the subject thinking aloud (e.g. “Tell me more about that”), relatively less interviewer training is required, and there is less opportunity for interviewer bias and variance.

3.1.2    Think-aloud disadvantages: think-aloud can be more burdensome for respondents. In order to provide detailed and useful information, respondents need to be motivated enough to cognitively process different aspects of the question and provide the needed details . Furthermore, interviewers need to be well-trained in the CI objectives and experienced in order to come up with additional general and non-leading probes to encourage respondents to think about the response process (which is not a natural process for many respondents) . There is some evidence that thinking aloud is particularly unfamiliar or challenging for some subgroupssee ‘Lessons learned‘ for examples.

3.2     Consider using verbal probing. Interviewers in the verbal probing approach take a more active role than in the think-aloud approach by probing for specific information relevant to the survey questions.

3.2.1    Verbal probing advantages: verbal probing can be beneficial when researchers anticipate particular problems with questions, because interviewers can ask scripted ‘anticipated’ probes or unscripted ‘spontaneous’ probes to address pre-identified concerns . Using verbal probes, the interviewer is able to collect specific information regarding the different stages of the four-stage response process (comprehension, recall, judgement, and response) . identifies typical probes to capture information about each stage. For example, “What does that term mean to you?” captures information associated with the comprehension. The interviewer maintains more control over the interview and the process may be easier and place less burden on respondents. Verbal probing is likely to be easier for novice cognitive interviewers, particularly if probes are largely scripted (see below for a discussion of approaches to probing).

3.2.2    Verbal probing disadvantages: a possible downside to verbal probing, and the use of anticipated probes in particular (especially if there is is overreliance on the script by interviewers), is that searching for particular problems may create the appearance of problems that do not really exist . Interviewers require more careful training and need to take care to choose non-leading probing techniques to minimize bias.

3.3     Consider other types of cognitive interviewing techniques that can be used to complement think aloud and probing :

3.3.1    Vignettes are short stories or descriptions of a hypothetical respondent that are used to investigate a subject’s cognitive processing with respect to survey-relevant decisions. A respondent may first be presented with a vignette, and then asked a series of questions related to it to investigate specific cognitive processes. See ‘Lessons learned’ 3.2 for an example.

3.3.2    A card sort is an exercise wherein a respondent is presented with a number of categories on cards and asked to sort them. For example, each card may have a potential reason for maintaining current employment, and a respondent may be asked to sort them into three piles: those that are big factors, those that are smaller factors, and those that are not factors at all for maintaining employment. This task provides a systematic means to determine the ways in which subjects think about a key topic. More specifically, it is used to determine how individuals organize concepts, and in particular, what they believe a concept includes or excludes. This type of task could also be used to investigate how individuals define complex concepts, such as what ‘having any kind of work’ means.

Lessons learned

3.1     Note that attitudinal and behavioral questions may require different cognitive interviewing approaches. found that people do not have a great deal of access to the basis for their attitudes, and asking questions about why a person has a belief or attitude can be unreliable and produce reactivity effects. Therefore, such probing might not result in accurate data. More effective are probes that judge whether the question effectively represents the belief.

3.2    studied comprehension of rules for assignment of individuals to households within a census form, using vignettes such as the following: “Maria is a live-in housekeeper for the Smiths during the week, but spends weekends with her husband and children at their apartment. Where should Maria be listed on a census form?” According to the survey instructions, the correct answer is that Maria should be included with the Smiths, but respondents were correct only 36% of time, indicating potentially serious problems with these instructions .

⇡ Back to top

4. Determine the timing and nature of the probes that will be used.

Rationale

As noted above, in practice, verbal probing is also often used in conjunction with the think-aloud approach. Probing with the verbal probing approach is used to try to understand the respondents’ thought process while answering survey questions. Probes are typically designed to capture each stage of the response process, based on the four-stage cognitive model: comprehension, retrieval, judgment, and response , and tend to be used in a specific and prescribed way. The nature and timing of the probes relative to each survey item need to be determined ahead of time, as detailed below. The specific type of verbal probing used also depends in part on whether the objective is reparative or descriptive.

Procedural steps

4.1     Consider the timing of probes. Probes can be asked concurrently, where the interviewer asks the target question, the subject answers it, and the interviewer follows up immediately by probing for specific information related to the question or the subject’s answer. Probes can also be asked retrospectively; that is, at the end of a section of questions or at the end of the interview.

4.1.1    Retrospective probing advantages: the possible advantage of retrospective probing is that it avoids disrupting the interview and potential reactivity effects due to the cumulative effect of probing.

4.1.2    Retrospective probing disadvantages: the argument for concurrent probing is that waiting until the end of the interview risks respondents forgetting what they were thinking when they were answering the questions; rather, probing should occur when the information is fresh in respondents’ minds; indeed, that is more frequently done in practice .

4.2     Consider using proactive vs. reactive probes. With proactive probing, probes are administered proactively; that is, initiated by the researcher/interviewer based on anticipated problems.

4.2.1    Proactive probing advantages: proactive probing can be beneficial when particular problems with questions are anticipated, because interviewers can ask scripted ‘anticipated’ probes or unscripted ‘spontaneous’ probes to address pre-identified concerns. argues that proactive probing can be considered more systematic, because it is based on hypotheses about suspected flaws in the questions being evaluated. Furthermore, using anticipated or scripted probes can help minimize the variance between interviewers, since they use the same probes. Proactive probing using scripted probes can be particularly advantageous when interviewers are relatively inexperienced . Table 1 provides examples of common proactive probes used to address anticipated problems in the four stages of the question response process.

Table 1: Proactive probe examples.

Stages of the response process Proactive probe examples
Comprehension What does the term ‘outpatient’ mean to you?
How would you put that question in your own words?
What would you say this question is asking you?
Recall/Retrieval How did you arrive at your answer?
How did you remember that?
How did you work that out?
Judgment How easy or difficult was it for you to remember this?
How sure are you of your answer?
How easy or difficult was it for you to come up with your answer?
How easy or difficult is it for you to talk about this issue?
(See Guideline 5 below for more discussion about exploring potential question sensitivity)
Response How difficult was it for you to pick a response?

 

4.3     Proactive probing disadvantages: as have argued, a possible downside to probing and the use of proactive probes in particular, especially if used intensively by interviewers, is that searching for particular problems may create the appearance of problems that do not really exist.

4.4     If using proactive probes, consider whether anticipated or scripted probes, spontaneous probes, or a combination thereof will be most effective:

4.4.1    Scripted (otherwise called ‘anticipated,’ ‘planned,’ or ‘structured’) probes address pre-identified, concrete concerns. Scripting probes is a good idea when there is sufficient time to prepare for interviews and to design the cognitive testing protocol, when interviewers are relatively inexperienced, and when the questionnaire has a complex or hard-to-follow format. Scripted probes, along with conditional probes (noted in Guideline 4.5.4), reflect a ‘professional’ interviewing style and, as some argue, may be less potentially biasing . Scripting at least some probes could also be beneficial when conducting CIs in a 3MC study context to help standardize the CI process cross-culturally or cross-nationally.

4.4.2    Spontaneous probes are developed during the interview by the interviewer and are unstandardized. Spontaneous probes are sometimes used when time does not permit a thorough review of the questionnaire and the identification of potential problems and scripting of probes. Spontaneous probes, along with emergent probes (noted in Guideline 4.5.5), also permit the interviewer a more “personal” style . As notes, free-form probes (spontaneous or emergent) are particularly useful when an interviewer develops insight about a potential source of question failure, and can then follow up to learn more about the underlying cause of the problem illuminated by the particular test subject. In these situations, it is important that the interviewers are experienced and can generate appropriate probes on the fly during the interview.

4.5     Consider using reactive probes. Reactive probes are administered reactively based on the behavior of the respondent.

4.5.1    Reactive probing advantages: reactive probes allow interviewers to respond flexibly based on an unexpected response or behavior from the respondent that may signal a problem. It can also limit the identification of illusory problems.

4.5.2    Reactive probing disadvantages: reactive probing may fail to uncover covert problems such as misunderstandings by the respondent that are not obvious to the interviewer. Reactive probing also requires more extensive interviewer training, because interviewers must use more judgment to determine how and at what point to probe. And, in a 3MC study, the need for more extensive training across multiple study sites adds complexity.

4.5.3    If using reactive probes, consider whether conditional probes, emergent probes, or a combination thereof will be most effective.

4.5.4    Conditional probes are triggered by the respondent’s behavior and selected from a set of prepared and standardized items.

4.5.5    Emergent probes are developed flexibly during the interview based on the behavior of the respondent.

4.5.6    With the interpretivist approach, where the goal is to explore the respondents’ interpretation of the question and the construct it measures, the focus of probing is to fill in gaps in the respondents’ narrative and address any possible contradictions to explain how the survey question was understood and why a particular response was selected. Probes are thus developed and applied quite flexibly and reactively by the interviewer during interviews using the interpretivist approach.

4.6     There can be tension between proactive and reactive probing. The overuse or reliance on proactive probing risks fishing for problems that do not actually exist. This can cause local reactivity effectswhen the act of probing a target question produces evidence of spurious problems in that questionor extended reactivity effectswhen the cumulative effect of probing is to induce problems that do not exist in subsequent questions in the absence of probing. However, as argues, if only reactive probes are used, there is a danger of missing a misunderstanding or “silent misinterpretation.” While there is little empirical evidence to resolve the debate between proactive and reactive probing, advocates proactive probing as good practice but also offers the following tips to minimize the potentially negative effects discussed above:

4.6.1    Probes should be balanced and avoid bias (e.g., “Was that question difficult or easy to answer?”).

4.6.2    Caution should be used with paraphrasing probes, which ask respondents to repeat the question in their own words. Some respondents may have difficulty with this task, yet adequately understand the question.

4.6.3    Since it is not possible to pretest probes, it is important to be flexible and rework a probe if it seems unclear to the respondent or if it doesn’t elicit the desired information.

4.6.4    Limit speculative probing or probes that may cause respondents to focus on issues that are not central to the mental processing of the question.

4.6.5    Be conservative in the number of probes asked; it may not be necessary to probe every question if there is no evidence of any problem and there is no clear issue to investigate further.

4.6.6    Avoid viewing the subject as a questionnaire design critic. As notes, “a fundamental reason for subjects’ lack of awareness of questionnaire problems seems to be that they are not explicitly focused on possible sources of response error and have no inkling of our measurement objectives” (p. 123). Instead, they treat the exercise as a conversation, and tend to answer the questions we ask. Therefore, when potential pitfalls are revealed through a respondents’ answer or responses to probes, it is important that the interviewer not engage with respondents on a full critique of any given question or point out respondents’ mistakes. It is complicated and unnecessary to get into questionnaire-design related discussions with respondents.

4.6.7    Accept that expectations held prior to the cognitive interviewing study may be wrong. If there is a failure to find problems, researchers should be prepared to accept this as an outcome.

Lessons learned

4.1     In a 3MC cognitive interviewing study, found that the effectiveness of different probes varied for Chinese, Russian, Korean, and Spanish respondents due to differences in communication styles and cultural norms. For example, they found that probes that aimed to elicit direct responses and opinion-oriented comments (e.g., “Do the words, terms, and ideas used in this letter sound right or appropriate in your language or to your culture? Was there anything that caused you concern (about the letter)?”) were effective for Spanish and Russian respondents but were less effective for Chinese and Korean respondents. Their findings suggest that alternative probes may be needed to encourage respondents and to collect high-quality, comparable data across different language groups in cognitive interviews.

4.2     Reflecting on ‘s results and her own extensive experience conducting cognitive interviews with different language and cultural groups, observes that probes that ask the respondent to evaluate the question (e.g., “Do the words, terms, and ideas used in this letter sound right or appropriate in your language or to your culture?”), or comment on an aspect of the question such as possible sensitivity, tend to be problematic. She has found that probes that ask about the respondent’s answer, (e.g., that ask the respondent to elaborate further on their answer, why they gave the answer they did, how confident they are in their answer, etc.), on the other hand, tend to elicit more useful information and are more culturally portable.

⇡ Back to top

5. Develop the protocol for the cognitive interviewing study

Rationale

Once the overall design and approach to cognitive interviewing has been determined, in addition to the timing and nature of any specific probes that will be used, the next step is to develop the protocol or interview guide for the cognitive interviewing study.

Procedural steps

5.1     The protocol for the cognitive interviewing study should contain the survey questions to be evaluated along with interviewer instructions, such as any scripted or planned follow-up probes, to ensure that the information needed to meet the study objectives is gathered from each interview.

5.1.1    Other non-questionnaire materials such as consent forms and confidentiality agreements, recruitment materials, advance letters, brochures to mitigate refusal rates, etc., as well as corresponding interviewer instructions, should also be considered for inclusion in the testing protocol.

5.2     The protocol should include an introduction that briefly communicates the study purpose and procedures, seeks consent (see Guideline 8.3.1), and an introduction to the cognitive interview process in general.

5.2.1    The introduction should highlight the “focus on questions, not answers” .

5.2.2    If using the think-aloud approach, consider doing a practice task, such as the classic and often-used example of asking the respondent to visualize the place where they live and think about how many windows there are in that place . The interviewer asks the respondent to tell them what they are seeing and thinking about as they count up the windows. If the respondent is having difficulty, the interviewer may ask “What are you thinking?” or “What thoughts are going through your mind right now?”. If the respondent is thinking aloud with no difficulty, the interviewer may say “That’s great. Thinking aloud like this is just what we need” or “Good. Your comments help me understand what you are thinking about.”

5.3     It is important to adequately orient and prepare respondents for the cognitive interviewing task. This is particularly important when carrying out cognitive testing among linguistic or cultural groups who may have different communication styles or who may be less familiar with survey norms (see for discussion). For example, minors or respondents from different language or cultural backgrounds with little previous exposure to surveys may require a more detailed introduction and clear examples to help them understand the task and put them at ease. It can be helpful to provide subjects with an example and some practice questions. Alternative probes and communication styles may be needed depending on the specific culture, conversational norms, and level of experience with surveys of study subjects. See more in ‘Lessons learned.’

5.4     Translation of the cognitive interviewing probes and protocol should follow translation best practice methods: the TRAPD (Translation, Review, Adjudication, Pretesting, and Documentation) team translation model. See Translation: Overview for more information.

Lessons learned

5.1     Some studies have examined ways of improving the cognitive interviewing experience for Spanish and Asian (Chinese and Korean) monolingual speakers in the United States and for respondents outside of the United States and Europe . Results from these studies suggest that tailoring the introduction and communication style of interviewers for the specific culture, conversational norms, and level of experience with surveys can improve the experience for the respondent and the quality of the data collected in cognitive interviews.

⇡ Back to top

6. Develop specific protocols to test potentially sensitive questions.

Rationale

define sensitive questions as a broad category of questions including those which trigger social desirability concerns, those which are seen as intrusive by respondents, and those which raise concerns about the repercussions of disclosing the requested information. Responses to sensitive questions are vulnerable to measurement error stemming from social disability bias, wherein responses conform to prevailing cultural norms rather than accurate reports of sensitive behavior or attitudes . Exploring the issue of topic sensitivity is particularly relevant when adapting and translating a questionnaire for use in a 3MC survey, especially when the target population differs significantly from the one for which it was originally designed. This is important because what is considered socially desirable or sensitive varies across cultures . Insights from cognitive interviewing related to item sensitivity can be used to inform possible changes to question wording or the survey design (e.g., using a self-administered mode) to help reduce socially desirable reporting and measurement error. However, specific protocols may need to be developed to cognitively test questions that may be considered sensitive.

Procedural steps

6.1     Determine whether any questions may be considered sensitive by the target population or particular subgroups in the target population for the survey.

6.2     Prepare respondents adequately. It is useful to let respondents know ahead of time (at the time of recruitment and consent) the nature of the questions they will be asked (e.g., questions about drug and alcohol use). It is also important to communicate that respondents do not have to answer any questions that they are uncomfortable with answering. Furthermore, respondents should be advised that no personal information or responses to questions will be shared outside the research team, and that personally identifying information will be kept separate from their answers. Note that this type of information should be included in the informed consent process. See Ethical Considerations.

6.3     Consider preparing scripted probes to explore question sensitivity if it is anticipated (see Guideline 4.2). As discuss, these probes may be direct or indirect. Direct probes ask about how the respondents themselves react to the question, such as “How did you react to being asked this question?” or “How easy or difficult was it for you to come up with an answer?” . Indirect probes ask the respondent how they think other people would react to the survey question, such as “In general, do you feel that people might purposely say…or would they try to answer accurately?” or “How do you think people will react to being asked about …?” .

6.4     Note that it is not always necessary in cognitive interviews to know the respondent’s answer to a question; rather, probes can be used to ask respondents about their interpretation of specific terms and of the question overall, or about how they react or think others would react to the question. For self-administered questionnaires (SAQs), it is possible to have respondents read through the questionnaire and privately answer the questions, but to then review the questions with them to talk about their thought processes (and not their answers).

6.5    also suggests maintaining a neutral interviewer approach, by which he includes helping put respondents at ease by asking some non-sensitive questions first, announcing a switch to questions that some people find personal and reminding the respondent that they are not required to answer any questions that they might find uncomfortable, and proceeding with the questions in a direct but personable manner. He notes also that interviewers should avoid providing any social or behavioral cues (e.g., nervousness, hesitancy, embarrassment) that may be projected onto respondents or displaying any overt reactions to respondents’ answers.

6.6     Consider not recording sections pertaining to sensitive items to maintain respondent’s perception of confidentiality.

6.7     Consider interviewer-respondent matching on demographic variables such as gender or race, depending on the topic and population. Conversely, in certain situations, respondents may be more comfortable in divulging sensitive information to an interviewer belonging to an out-group with whom the respondent is not likely to cross paths.

Lessons learned

6.1    report on a cognitive interviewing study carried out in the Kingdom of Saudi Arabia on the Arabic version of the World Mental Health survey instrument (CIDI 3.0) in preparation for the Saudi National Mental Health Survey. Different types of cognitive probesproactive direct, proactive indirect, and general probeswere randomly assigned to survey questions to investigate differences in the feedback elicited by each type of probe. The study found that an indirect cognitive probe identified a topic to be sensitive in more instances than a direct probe or a general probe. A general probe, on the other hand, elicited more non-codable feedback, especially when paired with a survey question that asked about a more abstract concept such as a respondent’s feelings.

⇡ Back to top

7. Recruit and train interviewers to complete the cognitive interviews.

Rationale

Cognitive interviewing requires a discursive interviewing style, which is different from traditional standardized survey interviewing and requires particular training. Sufficient time and effort should be allowed to recruit and adequately train staff members to carry out the cognitive interviewing approach selected for the study.

Procedural steps

7.1     Consider those characteristics which are best suited to cognitive interviewing when recruiting interviewers.

7.1.1    Cognitive interviewers need strong interpersonal skills as well as technical ability , and to be able to put subjects at ease and have “the ability to be flexible, spontaneous, and cool under duress” (p. 130). In terms of technical and training background, notes that good cognitive interviewers have the following:

      • Some type of social science background, and exposure to measurement phenomena such as bias, context effects, acquiescence effects, and so on;
      • Basic knowledge and experience in questionnaire design;
      • Exposure to the type of subject matter of the questionnaire (e.g., employment and labor force issues for U.S. Bureau of Labor Statistics surveys); and
      • Experiencethat is, simply to have done a lot of cognitive interviews.

7.1.2    note that there are at least two distinct schools of thought about the optimal background and expertise for cognitive interviewers: 1) social scientists with advanced degrees and 2) veteran production survey interviewers. Both have their strengths, and particularly strong field interviewers can make good cognitive interviewers, although it is important that they be retrained specifically for cognitive interviewing, because the interviewing task differs fairly substantially from the standardized interviewing methods typically used in field interviews. In particular, notes that they must be trained to:

      • Find problems rather than work around them. The emphasis in cognitive interviewing is to identify issues so that they can be fixed before the field period.
      • Slow down. Interviewers often rush to finish a field interview out of fear that the respondent will not want to complete it. However, this is not such a concern in the lab.
      • Be flexible and forgiving of formatting problems or layout and skip pattern errors. It is important for cognitive interviewers to be flexible in ways that would not be desirable when using standardized interviewing methods in the field.

7.1.3    When conducting cognitive interviews on a translated questionnaire in a multicultural or multilingual context, interviewers need both linguistic and qualitative research skills. Cognitive interviewers of a translated version should be native speakers of the target language, or if not native speakers, the interviewers should have lived immersed in the culture of the target population for some extended period of time and have near-native speaker proficiency and ‘intuition’ so that they are sensitive to subtle nuances that other fluent speakers of the language might not detect .

7.1.4    When implementing cognitive interviewing among different cultural and linguistic groups and in settings where survey research is new, it can be challenging to find interviewers with experience conducting cognitive interviews. In these situations, recommend prioritizing “inherent” skills such as fluency in both the source and target language, an understanding of cultural nuances, and above-average interpersonal skills (i.e., the ability to recognize and respond to both verbal and nonverbal communication cues) rather than “teachable” skills (i.e., neutral, professional, nonthreatening behavior and cognitive interviewing techniques). Important also are good listening skills and strong analytic skills to know when to probe further (Schoua-Glusberg, personal communication, October 11, 2018).

7.2     Conduct comprehensive interviewer training and certification prior to commencing cognitive interviewing.

7.2.1    The amount of training needed will vary depending on the approach to cognitive interviewing and to the type(s) of probing selected for the study, as well as the educational background and level of experience of the interviewers. However, in general, training should cover the following:

      • Background on surveys and survey questions. Some general background on surveys, the importance of good questions, question response theory (e.g., ‘s four stage model), and examples of question and questionnaire problems are important topics to include in training. The extent of this background will depend on the background and experience of the interviewers participating in the training. Willis suggests allowing cognitive interviewing trainees to familiarize themselves with typical question design issues by having them appraise several sample questionnaires with the help of a checklist of common problems (see , Chapter 2).
      • Background on cognitive interviewing. Giving a background on cognitive interviewing should help trainees understand what cognitive interviewing is, from where it stemmed (e.g., its roots in cognitive psychology and the cognitive aspects of survey methodology (CASM), etc., as discussed above), different approaches to carrying out cognitive interviews, and the importance of it and what can be learned from the interviews.
      • Training in probing techniques. Assuming verbal probing rather than pure think-aloud is to be applied, interviewers should be taught specific probing procedures. As notes, this training should emphasize specific examples of the way that probing is used to detect problems in survey questions (see Chapter 5), as opposed to abstract or theoretical discussion. Audio and video recordings of previous cognitive interviews or examples from real interviews can be particularly effective in helping trainees understand how, when, and what types of probes can be used.
      • Study- or project-specific briefing. The study-specific training should include an introduction to the survey for which the questionnaire is being cognitively tested and its main research objectives. It should also include an overview of the overall study design (e.g., the survey mode, respondents, etc.) and questionnaire and a detailed review of the questions and question constructs to be tested. If possible, provide examples from previous cognitive interviewing rounds or testing in other languages/contexts.
      • Exposure to cognitive interviews and, if possible, field interviews. recommends that trainees be given the opportunity to observe experienced interviewers conducting laboratory-based cognitive interviews, as well as field interviews. As Willis notes, observing the challenges that can arise in the field (observed through monitoring telephone interviews or accompanying a household interviewer) and learning to appreciate the conditions that field interviewers must work under (e.g., television blaring, loud argument in adjacent room) can be a very illuminating experience.
      • Note-taking and detailed summaries. Detailed notes and summaries from each cognitive interview form a key stage in the analysis process (see Guideline 9 below), and are thus often required from interviewers for each question included in the cognitive interviewing protocol. Note that in some cases, interviewers just carry out and record the interviews while the person writing the report listens and takes notes on all the interviews. If interviewers are expected to take notes and provide detailed summaries, they must be trained on the type of notes and summaries that are expected. Summaries should include a detailed description of how and why the respondent answered the question the way they did, the themes mentioned, any problems that came up at any stage of the response process (e.g., comprehension, retrieval, etc.), and any other unexpected issues or problems. Since interviewers need to be able to focus on the respondent and not on taking notes during the interview, interviewers should rely on an audio or video-recording or a transcript of the interview produced by a note-taker (if recording is not possible for some reason) to produce their notes and detailed summaries. It is important that interviewers produce their notes as soon as possible after an interview, and before conducting more interviews, so that their memories of the interview are as fresh as possible and not confounded with those from other interviews.
      • Practice. It is essential, particularly for novice cognitive interviewers, to have ample opportunity to practice conducting interviews. As suggests, an especially useful practice is the mock interview, in which the new interviewer conducts a cognitive interview with an experienced interviewer who can assume the role of subject in a number of challenging guises. Trainees may also be paired up to take turns practicing mock interviews or may be instructed to interview friends or family members. If part of their expected duties, trainees should also be asked to produce notes from their practice interviews. Trainees should be observed and provided with detailed feedback on their performance carrying out interviews (and note-taking, if relevant) until it is determined that they are ready to perform actual interviews. Multiple rounds of practice and feedback is particularly important when training cognitive testing interviewers in different settings and languages .

Lessons learned

7.1     For a cognitive interviewing study carried out in the Kingdom of Saudi Arabia (KSA) for the Arabic version of the World Mental Health survey instrument (CIDI 3.0), staff at the University of Michigan provided in-depth remote training for a supervisor of the team in KSA who then trained additional interviewers. This remote train-the-trainer approach worked reasonably well. However, the interviewers would have benefited from more practice sessions and more opportunities for review and feedback of their work .

7.2     Based on their experience training interviewers for multi-lingual cognitive interviewing at the U.S. Census Bureau, recommend providing general cognitive interview training, project-specific training, and a segment on issues of linguistic or cultural sensitivity that should be considered. They also suggest a brainstorming activity with the interviewers, particularly when the lead researchers are not familiar with the cultures and languages included in the study, to try to anticipate issues that could arise in interviews and ideas for how to address them, spending at least half a day on practice interviews and conducting interviewer training in the survey language when possible.

7.3    has had success in the United States training language-savvy translators who live and work within the language community and have had experience translating the protocol to serve as cognitive interviewers, and believes that this approach could also be useful in other countries.

7.4     Instead of recruiting and training interviewers with the necessary language skills to conduct the cognitive interviews, used online video interpreters for a multilingual cognitive testing study among asylum-seekers and refugees in Arabic, Serbian, Farsi, Russian, and English. The use of online video interpreters avoided the time and expense of recruiting and training cognitive interviewers; however, noted several challenges: 1) it was difficult to build rapport with respondents; 2) the interpreters tended to dominate the discussions; and 3) and the researchers had to rely on the summaries provided by the interpreters, as opposed to the respondents’ narratives themselves. They found that it was important to use one interpreter per language, brief the interpreters before interviews on the aim of the cognitive interviewing and the task of the interpreter during the interview, and debrief with the interpreters after each interview. It was important to allow more time per interview for communication with the interpreter before and after each interview, to build rapport with the respondent, and to allow for translation. See for more detail.

⇡ Back to top

8. Identify and recruit cognitive interviewing subjects and conduct the interviews.

Rationale

As noted above, cognitive interviews are generally conducted among a small, purposive sample of respondents or study subjects. Suitable subjects must be located who are appropriate, willing, and available to be interviewed for the study. Ideally, the subjects recruited for cognitive testing should reflect the key socio-demographic and other characteristics of the intended target population for the survey.

Procedural steps

8.1     Recruit cognitive interview subjects.

8.1.1    Determine the types of respondents needed for the study. As noted above, the cognitive interviewing study sample should mirror the intended target population for the survey in terms of key socio-demographic and other characteristics. Overall, the focus should be on the variation between subjects, as opposed to statistical representativeness . The goal is to obtain a sample with variation across the range of socio-demographic and other characteristics, and which may be related to linguistic or cultural representation and/or to the topic of the survey. For example, if the survey relates to disability, it would be important to have a sample of cognitive interviewing subjects who represent a range of experience in the area of disability.

8.2     As notes, if there are gateway or filter questions in the questionnairethat is, questions to identify a specific sub-population of individuals of interest (e.g., drug-users, smokers, people with a particular medical condition, etc.)it is important to conduct cognitive interviews with both subjects who do and do not exhibit the key characteristic(s) of interest.

8.2.1    Develop a recruitment strategy, using flyers, advertisements, Internet ads, social networking sites (e.g., Facebook), word of mouth, etc. If the survey relates to a particular health issue, it may be possible to obtain help recruiting subjects who are affected by it from a related support group.

8.2.2    Screen respondents to ensure that they match the sampling criteria and any other criteria set for the study (e.g., age, disability status, etc.).

8.2.3    Schedule respondents, considering any additional time which might be involved in traveling to the site of the cognitive interview.

8.2.4    Consider providing incentives. In the U.S., monetary incentives are typically provided to cognitive interviewing study subjects where, as notes, the amount paid tends to depend on how difficult it is to induce people to participate, how difficult it may be to find people with the required profile, and subjects’ travel time (if they are requested to come to a lab or central location for the interview). As with survey respondents, the nature or amount of incentives should be selected appropriately given the local context. See Data Collection: General Considerations for further discussion.

8.3     Conduct the interviews.

8.3.1    Obtain and document voluntary informed consent. Consent information should be conveyed in language that is easy for subjects to understand. Obtaining informed consent often involves having subjects sign a written consent form that has been approved by an Institutional Review Board (IRB). See Ethical Considerations, Guideline 3 for more details.

8.3.2    Consider the needs of particular types of respondents. For example:

      • If interviewing children, interviewers may need to probe more and adjust for attention lapses . Willis also notes that children develop quickly; a 13-year-old and 19-year-old may respond very differently in a study of teenagers.
      • With the elderly, there may be more potential for getting off topic .

8.3.4    Adopt an iterative process and assess the process and results before too many interviews have been completed. This makes it possible to make adjustments to ensure that the study protocol and procedures are yielding the type of information needed from the study. Possible adjustments include adding scripted probes to explore particular issues or themes that have emerged, or further adapting the introduction or interviewing style to the subjects’ culture or language or to other features of the local context. This can be particularly important in a 3MC survey, where an issue with a specific question may arise during the cognitive interview process in one study country, and should subsequently be explored in the cognitive interviews in other study countries.

8.3.5    Consider follow-up interviews or a subsequent round of testing to examine changes to the questionnaire.

      • Resources permitting, follow-up interviews or a subsequent round (or even multiple rounds) of testing can be valuable to further understand the functioning of questions or to further explore issues that may have come up. It is also important to test changes to questions that have been introduced to address problems identified in cognitive interviews to assess whether the changes have 1) effectively addressed the problem and 2) introduced other, unanticipated problems.

The first step is to assess the results based on the interviews that have been conducted and identify whether there are open questions regarding the functioning of items, response problems, or other issues identified with the items. Next, determine whether the questions that have been identified could be effectively addressed through follow-up interviews or a subsequent round of testing. Consider also any changes that have been proposed based on the results thus far that should be tested. Repeat the steps outlined above to carry out the follow-up or subsequent round of testing.

Lessons learned

8.1     Based on their experience with multi-lingual cognitive interviewing at the U.S. Census Bureau, recommend that respondents be monolingual in the target language version of the questionnaire being tested, as bilingual respondents may compensate based on their knowledge of English, and thus problems may not be illuminated. They also note the importance of spending sufficient time orienting and ‘teaching’ respondents the task.

⇡ Back to top

9. Develop an analysis plan and carry out the analysis.

Rationale

Cognitive interviews do not provide specific solutions to question design issues. While individual interviews can reveal discrete problems such as an ambiguous term or recall difficulty, systematic analysis of the entire set of interviews is necessary to identify interpretive themes and explanatory patterns . A systematic analysis means that the full range of responses are examined within and across respondents (by question) and across subgroups (as appropriate given the purpose of the study) to identify patterns of interpretation, response process errors, and other potential problems. It is important to note that the analysis of CI interviews is complex and intensive. A significant amount of judgment and interpretation at the analysis stage is required to arrive at the findings of a cognitive interviewing study. Please see the resources in Further Reading for additional detail and guidance for the analysis stage.

Procedural steps

9.1     Develop an analysis plan to prepare for and guide the analysis process. Consider how the analysis steps outlined below will need to be carried out to address the research questions set out for the study (see Guideline 1), whether the cognitive interviewing approach will provide the requisite data for the proposed analysis, who will be involved in the various stages of the analysis (interviewers and/or other staff), and the software tools that may be needed (see Guideline 9.3).

9.2     As explain, the general process for analyzing cognitive interviewing data involves going from a large amount of textual data and ending with a set of conclusions that are meaningful and help address the purpose of the studythat is, a process of synthesis and reduction. The following five incremental steps for the synthesis and reduction of cognitive interviewing data are based on those outlined by :

9.2.1    Conduct interviews and capture the raw data. As argue, the process of carrying out the cognitive interview itselfa process that reveals how the respondent made sense of and went about answering a survey question and requires the interpretation and judgement on the part of the interviewerconstitutes the first stage of analysis. As noted above, it is essential that the text of each interview is captured either by audio or video recording, which may be converted into transcripts, as this is constitutes the raw data for the cognitive interviewing study. Be sure to think through how the interview recordings will be labeled and organized, electronically stored, and backed up.

9.2.2    Prepare detailed summaries. As noted in Guideline 7.2.1, summaries, often prepared by the interviewers, should include a detailed description of how and why the respondent answered each survey question the way they did, the themes mentioned, any problems that came up during any stage of the response process (e.g., comprehension, retrieval, etc.), and any other unexpected issues or problems.

9.2.3    Compare detailed summaries across respondents. At this stage, common themes and patterns of interpretation are identified across respondents. This process often involves the development of a coding scheme. Coding is the process of transforming qualitative information into a countable set of data categories. Categories may be developed to classify emergent themes in what respondents thought about when interpreting a question, how and why they formulated an answer, particular problems during a stage of the response process (e.g., comprehension, difficulty in recall), or other issues (e.g., translation problem, sensitivity, burden, etc.). Coding schemes generally need to be developed for each new study, but may draw upon existing codes or taxonomies, such as that developed by . Particular categories may be needed in cognitive testing of translated questions. As discussed by , three broad categories are frequently used to classify the types of problems found with translated questions: 1) problems related to translation choices, 2) problems related to the source question, and 3) problems related to culturally-specific issues or the cultural portability of the concept.

             The application of a coding scheme can make it possible to carry out limited quantitative analyses. Quantitative results based on a convenience sample of respondents, which is the type of sample typically used for cognitive interviewing studies, cannot be extrapolated to a larger population. However, they can be useful in providing a sense of the magnitude and patterns in the phenomenon under study as a complement to specific and detailed qualitative results.

             The development of the classification scheme or codes is an iterative process; in the process of synthesizing interviews and comparing results across respondents, it often becomes apparent that similar categories can be combined, or that existing categories may need to be broken down or refined.

             It is important to note that if more than one person is identifying, classifying, and counting the elements reported in the detailed summaries, the results of what they find may differ slightly based on each individual’s interpretation of the coding categories and the underlying data. Measures should be taken to assess the level of agreement among coders. This can be done by assigning the same source material to several coders and comparing their coded resultsthe level of agreement between the results indicates the reliability of the coding process (See Data Processing and Statistical Adjustment, Guideline 2.9.3 for further discussion).

9.2.4    Compare themes and issues across subgroups. Once the key themes and issues have been identified across respondents, these can be compared across important subgroups such as males vs. females, highly-educated vs. less-educated, and older vs. younger respondents. This part of the process helps to determine whether different types of respondents process questions differently.

9.2.5    Draw conclusions. Having completed all of the previous steps, it is possible to draw final study conclusions about the extent to which survey questions are functioning as intended by the researchers, as well as a number of other outcomes such as the following, as identified by :

      • Item-specific recommendations for changes to wording (cognitive, logical/structural, culturally-oriented defects);
      • Need for further specification of objectives or the matter in which the questions satisfy them;
      • Problems related to ordering (of items, sections, and so on) and other interactions between survey questions;
      • Issues related to reduction in overall survey length or burden; and
      • Limitations on what can be asked of survey respondents using the intended procedures.

9.3     Consider which software tools, if any, to use to manage the data and carry out the coding (if applicable) and the analysis. Possible tools include qualitative software packages such as Nvivo. Spreadsheet programs such as Microsoft Excel can also be used effectively for coding. Also available is Q-Notes, an online qualitative software program developed specifically for cognitive interviewing data and analysis by the National Center for Health Statistics (NCHS). A number of Q-Notes features lend it to comparative cognitive interviewing studies and collaboration across different sites. In particular, with Q-Notes it is possible for interviewers to enter summary notes from any location and in a consistent format, and translated versions of questions can be entered and displayed alongside the original source version. Q-Notes also offers analytic features based on five steps for synthesis and reduction outlined in and above. See and the CDC for more information.

Lessons learned

9.1     It is helpful to begin the analysis while interviews are being collected or, at a minimum, to examine interviewers’ notes to ensure that they are complete and to check for emerging themes that may need to be explored in upcoming interviews. has found it helpful to include interviewers in this process as she did in a study in Mumbai, India, where interviewers were shown the set of notes for a specific question. By seeing that there were three ways in which respondents were approaching the question, the interviewers could understand how the question was performing and could also better appreciate how their own interview contributed to a better understanding. As notes, “this level of involvement fully engaged interviewers, providing them with a keener understanding as to the information that they should collect and document in upcoming interviews” (pp. 219).

⇡ Back to top

10. Document and communicate findings.

Rationale

A report containing detailed documentation of the protocol and procedures, respondent materials, themes identified and problems encountered in the cognitive interviews, and any subsequent changes made to the questionnaire will aid the staff and researchers working on later rounds of the same study or on similar future studies.

Procedural steps

10.1   Prepare a final cognitive testing report in a manner consistent across countries or test sites, which may be organized into the following sections, as suggested by :

10.1.1   An introductory section describing the survey for which the questionnaire/questions were evaluated, including the intended mode of administration, the purpose of the cognitive testing, and the research objectives.

10.1.2   A methods section detailing the following:

      • A description of the subject population and a description of the recruitment methods and any incentives offered;
      • The number of subjects testedusually presented in a table with the sample’s demographics (e.g., age, gender, education);
      • The number of cognitive interviewers, and their level of experience;
      • Information about the cognitive interviewing techniques used, such as the overall approach (e.g., think-aloud, verbal probing, interpretivist), the type of probing (concurrent, retrospective, proactive vs. reactive, degree of probe standardization, etc.);
      • The location of the interviews (e.g., in a lab, subjects’ homes, etc.) and how the interviews were documented (e.g., audio recordings, video recordings, transcripts); and
      • The methods used for analysis:
        • A summary-of-findings section. This section highlights the main findings of the study synthesized across all of the questions. For example, the findings presented in this section could include a term used in more than one question that was not consistently understood, or a pattern of interpretation among a particular subgroup of respondents .
        • A question-by-question section summarizing the results and recommendations, including proposed changes to the tested questions and to the questionnaire overall (e.g., question order, instructions, etc.).
        • A conclusion section listing the potential limitations of the testing and anticipated next steps.
        • An appendix with the final testing protocol and probes.

Lessons learned

10.1   The National Center for Health Statistics (NCHS) has developed an online resource called Q-Bank for the dissemination of cognitive interviewing and question evaluation findings. Q-Bank provides access to reports from a number of statistical agencies and research organizations. Users can search for evaluated questions, which are linked with the associated testing report, and enter their own reports upon completing a question evaluation project. More information is available here

⇡ Back to top

References

⇡ Back to top