Appendix A (Interviewer Design Effect)

[zotpressInText item="{2265844:DDYLNLET},{2265844:5C3CI2N4},{2265844:GDGL2VTC},{2265844:ZTID3A9P}" format="%a% (%d%)"]

Research indicates that the interviewer design effect may be even larger than the design effect attributable to geographic clustering [zotpressInText item="{2265844:YH8HNSGL}"]. This is especially true in some 3MC studies where cultural and other factors contribute to large interviewer variances (this variance can differ between countries as well). Interviewer variance occurs when response errors of persons interviewed by the same interviewer are correlated; therefore, interviewer variance is part of the correlated variance component of the total variance (other correlated variances stem from coders, editors, supervisors, and crew leaders).

The intraclass coefficient [latex]\rho_{int}[/latex] is a measure of the ratio of interviewer variance to the total variance, and is defined as: [latex]\rho_{int}=\frac{\text{between-interviewer variance}}{(\text{between-interviewer variance})+(\text{within-interviewer variance})}[/latex].

The value of [latex]\rho_{int}[/latex] is theoretically always between 0 and 1, although calculated estimates of [latex]\rho_{int}[/latex] may sometimes be negative. In this case, they are usually treated as zeros. When [latex]\rho_{int}[/latex] for a particular variable is 0 or is negative, we interpret this to mean that the interviewers had no effect on the variance of responses to that variable; the larger the value of [latex]\rho_{int}[/latex], the larger the effect interviewers had on the variance of that particular variable.

The interviewer design effect (deffint) is a measure of the effect of interviewers carrying out multiple interviews, compared to what you would get if there was a different interviewer for each respondent, all else being equal (if the addition of more interviewers increases costs such that supervision or training must be reduced to compensate, interviewer variance may actually increase): [latex]\text{deff}_{int}=1+\rho_{int}(m-1)[/latex], where [latex]m[/latex] is the average number of interviews per interviewer.

Thus, even a small interviewer variance ([latex]\rho_{int}[/latex]) can have a significant effect on the variance of a survey estimate if [latex]m[/latex] is large. The interviewer variance contribution is usually not included in textbook variance estimation formulas. Interviewer variance leads to a loss of sample information when the effective sample size [latex]\text{neff}[/latex], defined as [latex]n/\text{deff}_{int}[/latex], is smaller than the actual sample size [latex]n[/latex].

Standardized interviewing aims to reduce interviewer variance.

For specification of a mathematical model of response errors when interviewers are used, see [zotpressInText item="{2265844:7XL3V2I6}" format="%a% (%d%)"]; for further discussion of interviewer variance see [zotpressInText item="{2265844:ALXV3VKD}" format="%a% (%d%)"] and [zotpressInText item="{2265844:YHPXI3LA}" format="%a% (%d%)"].

See Statistical Analysis for a discussion on incorporating interviewer effects into multi-level models.

References [zotpressInTextBib style="apa" sortby="author"]