Webpage last modified: 2011-Nov-28
Following terminology used in the translation sciences, this chapter distinguishes between "source languages" used in
"source questionnaires" and "target languages" used in "target questionnaires." The language translated out of is the source language; the language translated into is the target language.
Translation procedures play a central and important role in multilingual survey projects. Although good translation products do not assure the success of a survey, badly translated questionnaires can ensure that an otherwise sound project fails because the poor quality of translation prevents researchers from collecting comparable data.
The guidelines presented below envisage a team translation approach for survey instrument production. Evidence is growing that such procedures are effective for survey translation     . The guidelines address, at a general level, the steps and protocols recommended for survey translation efforts conducted using a team approach.
Before discussing team translation procedures, the chapter briefly outlines other approaches sometimes followed to produce or check survey translations and indicates why these are not recommended here. For discussion see    and .
Survey questions are a complex text type with multiple functions and components    . As a result, any reduction of human involvement in the decision-making process of survey translation is ill-advised .
One of the main goals of machine translation, however, is to greatly reduce human involvement in translation production.
Do-it-yourself ad hoc translation
It is a mistake to think that because someone can speak and write two languages he or she will also be a good translator for these languages. Translation is a profession with training and qualifications. Translatology (under various names) is a discipline taught at the university level. Students of the translation sciences learn an array of skills and procedures and become versed in translation approaches and theories which they employ in their work. At the same time, as explained in the description of team translation following here, survey translation calls for not only a good understanding of translation but also of the business of survey measurement and how to write good questions. Under normal circumstances, a trained translator should not be expected to have a strong understanding of survey practices and needs, hence the need for a team of people with different skills      .
Sometimes bilingual interviewers translate for respondents as they conduct the interview. In other words, there is a written source questionnaire that the interviewers look at but there is never a written translation, only what they produce orally on the spot. This is sometimes called "on sight" translation, "on the fly translation," or "oral translation."
Another context in which survey translation is oral is when interpreters are used to mediate between an interviewer speaking language A and a respondent speaking language B. The interviewer reads aloud the interview script in language A and the interpreter is expected translate this into language B for the respondent. The interpreter is also expected to translate everything the respondent says in language B into language A for the interviewer. Research directly on the process of oral translation in surveys and how this affects interpretation, understanding, and data is quite sparse. Evidence available from recent investigations suggests that these modes of translation must be avoided whenever possible and that extensive training and briefing should take place if they must be used   .
Translation and back translation
Even today, many projects rely on procedures variously called "back translation" to check that their survey translations are adequate. In its simplest form, this means that the translation which has been produced for a target language population is re-(or back-) translated into the source language. The two source language versions are compared to try to find out if there are problems in the target language text. As argued elsewhere, instead of looking at two source language texts, it is much better in practical and theoretical terms to focus attention on first producing the best possible translation and then directly evaluating the translation produced in the target language, rather than indirectly through a back translation. Comparisons of an original source text and a backtranslated source text provide only limited and potentially misleading insight into the quality of the target language text     .
Introduction to team translation
In a team approach to survey translation, a group of people work together. Translators produce draft translation, reviewers review translations with the translators, one (or more) adjudicator decides whether the translation is ready to move to detailed pretesting and also decides when the translation can be considered to be finalized and ready for fielding.
Figure 1 below presents the TRAPD (Translation, Review, Adjudication, Pretesting, and Documentation) team translation model. In TRAPD, translators provide the draft materials for the first discussion and review with an expanded team. Pretesting is an integral part of the TRAPD translation development. Documentation of each step is used as a quality assurance and monitoring tool and each step of the translation effort includes assurance and monitoring elements    .
Figure 1. The TRAPD Team Translation Model
Procedures are partially iterative in team translation. The review stage reviews and refines draft translations. Adjudication, often a separate step from review, can lead to further modifications of the translation before it is signed off for pretesting (see Pretesting). Pretesting may again result in modifications before the adjudicator signs off on the version for final fielding.
Team approaches to survey translation and translation assessment have been found to be particularly useful in dealing with the fairly unique challenges of survey translation. The team can be thought of as a group with different talents and functions, bringing together the mix of skills and discipline expertise needed to produce an optimal version in the survey context where translation skill alone is not sufficient. Other approaches include having a single translator deliver a translation to the researchers or survey organization or having a translation agency deliver a translation to researchers. Such procedures are not specifically designed to bring together translators with other relevant experts in reviewing the translation in the way a team translation does. Further consideration of advantages that team efforts have over other approaches can be found in     and .
Each stage of the team translation process builds on the foregoing steps and uses the documentation required for the previous step to inform the next. In addition, each phase of translation engages the appropriate personnel for that particular activity and provides them with relevant tools for the work at hand. These tools (e.g., documentation templates; see Appendix A) increase process efficiency and make it easier to monitor output. For example, translators producing the draft translations are required to keep notes about any queries they have on their translations or the source text. These notes are considered along with the translation output during the next review stage in which reviewers work together with the translators   .
Team translation efforts work with more than one translator. Translators produce translation material and attend review meetings. Either each translator produces a draft version of the source questionnaire (double or full translation) or each translator gets parts of the source questionnaire to translate (split translation)   . The double translations or the sections of the split translation are refined in the review stage and possibly again after subsequent steps, as just described.
Whenever possible, translation efforts that follow a team approach work with more than one draft version of the translated text. A sharing of these draft versions and discussion of their merits is a central part of the review process. Two draft translations, for example, can dispel the idea of there only being one "good" or "right" translation. They also ensure that more than one translation is offered for consideration, thus enriching the review discussion. This encourages a balanced critique of versions    . Contributions from more than one translator also make it easier to deal with regional variance, idiosyncratic interpretations, and translator oversight   .
Survey translations also often call for sensitivity for words people speak rather than words people write. Apart from ensuring the needed spread of survey expertise and language expertise, the discussion that is part of team approaches is more likely to reveal vocabulary or vocabulary level/style (register) problems which might be overlooked in a review made without vocalization. Pretesting may, of course, reveal further respondent needs that "experts" missed.
As noted, team-based approaches aim to include the translators in the review process. In this way, the additional cost of producing two draft translations would be offset by the considered contributions the translators can bring to review assessments. Since they are already familiar with the translation challenges in the texts, they make the review more effective. Split translation arrangements can still capitalize on the advantages of having more than one translator in the review discussion but avoid the cost of full or double translations. The advantages and disadvantages of each approach are discussed under guidelines 3 and 4 (see too,  and ).
The specifics of team translation procedures are considered below. For other aspects of translation production, please refer to:
Finding, Selecting, and Briefing Translation Team Members
Translation Management and Budgeting
Figure 2 shows translation within the survey production process lifecycle
(survey lifecycle) as represented
in these guidelines. The lifecycle begins with establishing study structure
and Operational Structure) and ends with data dissemination (Data
Dissemination). In some study designs, the lifecycle may be completely or partially repeated. There might also be iteration within a production process.
The order in which survey production processes are shown in the lifecycle does not represent a strict order to their actual implementation, and some processes
may be simultaneous and interlocked (e.g., sample design and contractual work). Quality
and ethical considerations are relevant to all processes throughout the
survey production lifecycle. Survey quality can be assessed in terms of fitness for
intended use (also known as fitness for purpose), total survey error, and
the monitoring of survey production process quality, which may be affected by survey infrastructure, costs, respondent and interviewer burden, and study design
specifications (see Survey Quality).
Figure 2. The Survey Lifecycle
Goal: To create and follow optimal procedures to standardize, assess, and document the processes and outcomes of survey questionnaire translation.
- Plan translation as an integral part of the study design. This planning should include all the elements that will be part of the translation procedures (e.g., selection of team members, language harmonization), and should accommodate them in terms not only of procedural steps but with regard to hiring, training, budgeting, time schedules, and the questionnaire and translation production processes.
Survey translation efforts are part of the target language instrument development and should be treated accordingly. In addition, when translations are produced in order to take part in a larger comparative project, forethought and a clear direction to planning and implementing translation will help produce translations across multiple locations which comply with project requirements.
- Define the following:
- The larger vision (e.g., a successfully implemented survey).
- The concrete goal (e.g., a well-developed translation for the various contexts and populations).
- Important quality goals (e.g., a population-appropriate translation,
source questionnaire, efficiency and feasibility of translation procedures, timeliness).
- Relevant factors (e.g., schedules, budget, personnel available, unexpected events).
- Tasks involved (e.g., assembling personnel and the translation documents; preparing tools, such as templates; training personnel; producing and reviewing translations;
- Identify core team members (those people required for the team translation effort). See Appendix B for specific tasks of each core team member and other team players identified below.
- Identify any other team players who may be required, based upon the size of the project, the mode of data collection, etc.
- Substantive experts
- Other experts, such as visual design experts, adaptation experts
- External assessors
- Back-up personnel
- Determine whether regional variance in a language or shared languages need to be accommodated; decide on strategies for this as needed (see Language Harmonization).
- Select, brief, and train personnel (see Finding, Selecting, and Briefing Translation Team Members). Identify the in-house and external staff and consultant needs on the project and follow appropriate selection, briefing, and training procedures for each person or group.
- Identify, acquire, and prepare the materials for translation. In addition to the source questionnaire, these may include advertising material, interviewer manuals, programmer instructions, and any supporting materials such as "showcards," as well as statements of informed consent.
- Clarify payment arrangements for all involved (see Translation Management and Budgeting).
- Create a time schedule and identify project phases and milestones for members of the team (see Translation Management and Budgeting).
- Arrange for back-up team members in the event of unavailability or illness.
- Decide on the mode and schedule of meetings (face-to-face, web casting, or conference calls) and materials to be used at meetings (e.g., shared templates, software tools, documents deposited in e-room facilities, paper-and-pencil note-taking).
- Decide on other communication channels and lines of communication (reporting delays, illness, completion, deadlines).
- Decide whether each translator will prepare a full translation (double translation) or whether the material to be translated will be divided among the translators (split translation).
- In major efforts, the bigger picture must first be considered to confirm which routine or special tasks are vital and which are not. It is easy to focus on procedures which are familiar and thus inadvertently miss other vital elements. For example, if consistency in terminology across versions is not something a project leader has usually considered, procedures to check for this might be overlooked in planning.
- The number of translations required varies among multilingual survey projects. The Afrobarometer Survey , the Asian Barometer Survey , and the European Social Survey Source  specify that every language group that is likely to constitute at least 5% of the sample should have a translated questionnaire.
- Planning and quality assurance and quality control should go hand-in-hand. When planning the project or procedure, it is also time to plan the quality assurance and quality control steps. For example, in planning the translation of answer scales, steps to check that scales are not reversed or a response category omitted can be incorporated into a translation template.
- Have two or more translators produce first draft translations. If possible, have each translator
produce a full translation; if that is not possible, aim to create overlap in the split translation
sections each translator produces.
Having more than one translator work on the draft translation(s) and be part of the review team encourages more discussion of alternatives in the review procedure. It also helps reduce idiosyncratic preferences or unintended regional preferences. In addition, including the translators in the review process who produced the initial drafts not only improves the review but may speed it up as well.
- Determine lines of reporting and document delivery and receipts.
- Translation coordinators typically deliver materials to translators. Coordinators should keep records of the delivery of materials and require receipt of delivery. This can be done in formal or less formal ways, as judged suitable for the project complexity and the nature of working relationships.
- The project size and complexity and the organizational structure (whether centralized, for example) will determine whether translation coordinators or someone else actually delivers materials and how they are delivered.
- Determine the protocol and format for translators to use for note-taking and providing comments on source questions, on adaptations needed, and translation decisions. See Appendix A for documentation templates.
- Establish deadlines for deliveries, including partial translations (see below), and all materials for the review session.
- Require each translator to deliver the first 10% of his/her work by a deadline to the coordinator
(senior reviewer or other supervisor) for checking. Reviewing performance quickly enables the supervisor
to modify instructions to translators in a timely fashion and enables hiring decisions to be revised if necessary.
- Following the established protocol for production procedures and documentation, each translator produces his/her translation and delivers it to the relevant supervisor.
- Either have translators produce a full draft of the questionnaire and other materials to be translated or require each to produce some portion of the material (double or full translation or split translation).
- After receiving the translated materials, have the coordinator/senior reviewer prepare for the review session by identifying major issues or discrepancies in advance.
- Develop procedures for recording and checking consistency across the questionnaire at the finish of each
stage of review or adjudication. (See
Appendix A for documentation examples.)
- The more complex the project (e.g., number of translations), the more careful planning, scheduling, and documentation should be (see Translation Management and Budgeting).
- Since the aim of review is to improve the translation wherever necessary, discussion and evaluation are at the heart of the review process. The senior
reviewer or coordinator of the review meetings must, if necessary, help members focus on the goal of
improvement. In line with this, people who do not respond well to criticism of their work are not likely to make good team players for a review.
- Review of the first 10% of the draft translation may indicate that a given translator is not
suitable for the project because it is unlikely that serious deficiencies in translation quality can be remedied by more training or improved instructions. If this is the case, it is
probably better to start over with a new translator.
- Draft translation is only the first step in a team approach. Experience shows that many translations proposed in drafts will be changed during review.
- If translators are new to team translation or the whole team is new, full translation rather than a split procedure is recommended whenever possible to better foster discussion at the review and avoid fixation on "existing" text rather than "possible" text.
- Translations that are fine from the translation point of view may not be strong enough from a measurement perspective. For instance, a translator might be inclined to reduce the numerous adverbial references in the following: "Generally speaking, how much television do you usually watch all in all on an average weekday?" Translators might feel that any one of "generally speaking," "usually," or "all in all," or possibly the adjective "average" could usefully be omitted to make the sentence clearer. Rightly or wrongly, the question designer presumably felt it important to include each of these phrases to "guide" the respondent in what to consider. As things currently stand, there is little basic research into what and how respondents specifically process questions with such multiple "signposts."
- It is important to inform team members that changes to draft translations are the rule rather than the exception. The aim of a review is to review AND improve translations. Changes to draft
translations should be expected and welcomed.
- Providing templates to facilitate note-taking will encourage team members to do just this. Notes collected in a common template can be displayed more readily for all to see at meetings.
- It may seem cheaper only to work with one translator and to eschew review sessions, since at face value, only one translator is paid for his or her translation and there are no review teams
or team meetings to organize and budget for. In actuality, unless a project takes the considerable risk of just accepting the translation as delivered, one or more people will be engaged in some
form of review.
- A professional review team may involve more people and costs than an ad hoc informal review but it is a central and deliberate part of
quality assurance and monitoring in the team translation procedure.
In addition, even in a team translation procedure, translation costs will make up a very small part of a survey budget and cannot reasonably be looked at as a place to cut costs. Experience
gained in organizing translation projects and selecting strong translators and other experts is
likely to streamline even these costs (see Translation Management and Budgeting).The
improvements that team translations offer justify the additional translator(s) and experts employed.
- The burden of being the only person with language and translation expertise in a group of multiple other experts can be extreme. If more than one translator is involved in review, their
contributions may be more confident and consistent and also be recognized as such.
- When translators simply "hand over" the finished assignment and are excluded from the review discussion, the project loses the chance to have translator input on the review and any discussion
of alternatives. This seems an inappropriate place to exclude translator knowledge.
- Relying on one person to provide a questionnaire translation is particularly problematic if the review is also undertaken by individuals rather than a team.
- Even if only one translator can be hired, one or more persons with strong bilingual skills could be involved in the review process. (The number might be determined by the range of regional
varieties of a language requiring consideration for the translation. Bilinguals might not be able to produce a useable translation but could probably provide input at the review after having gone
through the translation ahead of the meeting.)
- If possible, have new teams work with two or more full translations.
Having new teams work with two or more full translations is the most thorough way to avoid the disadvantages of a
single translation. It also provides a richer input for review sessions than the split translation procedure,
reduces the likelihood of unintentional inconsistency, and constantly prompts new teams to consider
alternatives to what is on paper.
- Have several translators make independent full translations of the same questionnaire, following
the steps previously described in Guideline 2.
- At the review meeting, have translators and a translation reviewer and anyone else needed at that session
go through the entire questionnaire, question by question. In organizing materials for the review, depending on how material is shared for discussion, it may be useful to merge documents and notes
in the template (see Appendix A).
- The translation(s) required will determine whether more than two translators are required. Thus if the goal is to produce a questionnaire that is suitable for Spanish-speaking people from many different countries, it is wise to have translators with an understanding of each major regional variety of Spanish required. If, as a result, 4 or 5 translators are involved, full translation can become very costly and splitting the translation material is probably the more viable option.
- Translators usually enjoy not having to carry sole responsibility for a version once they have experienced team work.
- To save time and funds, have experienced teams produce split translations.
- Divide the translation among translators in the alternating fashion used to deal cards in many card games.
- This ensures that translators get a spread of the topics and possibly different levels of difficulty present in the instrument text.
- This is especially useful for the review session—giving each translator material
from each section avoids possible translator bias and maximizes translator input evenly
across the material. For example, the Survey on Health, Ageing, and Retirement in Europe (SHARE) questionnaire has modules on financial topics, relationships, employment,
health, and other topics .
- By splitting the questionnaire (more or less) page for page, each translator is exposed to trying to translate a variety of topics and better able to contribute directly
during review as a result.
- Whenever possible, divide the questionnaire up in a way that allows for some overlap in the material each translator receives (see the first two "lessons learned" for this guideline).
- Keep an exact record of which translator has received which parts of the source documents.
- Have each translator translate and deliver the parts he/she has been given for the review meeting.
- Use agreed formats or tools for translation delivery for the review session. For example, if a template is agreed upon, then different versions and comments can be entered in the template to make comparison easier during review. (See examples in Appendix A.)
- Develop a procedure to check for consistency across various parts of the translation.
- At the review meeting, have translators and the review team go through the entire questionnaire. When organizing materials for the review, depending on how material is shared for discussion, it may be useful to merge documents and notes (see Appendix A).
- Take steps to ensure that material or terms which recur across the questionnaire are translated consistently. For example, it is conceivable that two translators translate the same expression and come up with suitable but different translations. Source instrument references to a person's (paid) work might be rendered with "employment" by one translator, with "job" by another, and with "profession" by a third. In a given context, more than one of these could be acceptable. If this is the case, choose one term and use it consistently.
- It is often necessary to split the material to address issues of time, budget, or language variety. Even observing the card-dealing division of the material
( ), there is often no direct
overlap in split translations between the material the different translators translate. Translators are
thus less familiar with the challenges of the material that they did not translate than the sections they translated. This can reduce the detail of input at the question-by-question
review meeting. The senior reviewer must therefore take care to stimulate discussion of any section(s)
where only one translation version is available.
- Budget and schedules permitting, it is useful to create some modest overlap in material translated. This allows the review team, including translators, to have an increased sense of
whether there are large differences in translating approaches between translators or in their understanding of source text components at the draft production level.
- Giving people time to prepare the materials for the review meeting and making sure that they prepare is important for the meeting's success. Ad hoc suggestions and responses to
translations are usually insufficient.
- Consistency checks can ensure that one translator's translation can be harmonized with another
translator's possibly equally good but different rendering.
- In checking for consistency, it is important to remember this procedure must not be only mechanical (for example, using a find function in software). The source text may use one and the
same term in different contexts with different meanings, while other language versions may need to choose different terms for different contexts. The opposite may also hold. Automatic
harmonization based on "words" is thus not a viable procedure. For example, the English word "government" may need to be translated with different words in another language depending on
what is meant. In reverse fashion, English may use different words for different notions which are covered by a single word or phrase in other languages. Examples: English "ready" and
"prepared" can in some circumstances be one word in German; "he" and "she" are differentiated in English but not in Turkish or Chinese.
- Checks for general tone consistency are also needed. There is, for instance, a difference in tone in English between talking about a person's "job" and a person's "profession," or
in referring to a young person as a "child" or a "kid."
- Review and refine draft translations in a team meeting. Review meetings may be in person, virtual, or a mix of the two. The time involved depends upon the length and complexity of a
questionnaire, the familiarity of the group with procedures, and disciplined discussion. The work may call for more than one meeting.
The team meeting brings together all those with the necessary expertise to discuss alternatives and collaborate in refining the draft translations—translation
reviewers, survey experts, and any others that a specific project requires.
- Make all the translated draft materials available to team members in advance of the review meeting(s) to allow preparation.
- Provide clear instructions to members on expected preparation for the meeting and their roles and presence at the meeting.
- Arrange for a format for translations and documentation that allows easy comparison of versions.
- Appoint a senior reviewer with specified responsibilities.
- Have the senior reviewer specifically prepare to lead the discussion of the draft translations in advance. Prior to the meeting, this reviewer makes notes on points of difficulty across
translations or in the source questionnaire and reviews translators' comments on their translations and the source
documents with a view to managing.
- Ask other team members to review all the draft materials and take notes in preparation for the meeting. The time spent on preparation will be of benefit at the meeting.
- Have the senior reviewer lead the discussion.
- The lead person establishes the rules of the review process.
- He/she emphasizes, for example, that most likely the team will change existing translations, and that the common aim is to collaborate towards finding the best solutions.
- Have the senior reviewer appoint two revision meeting note-takers (any careful and clear note-taker with the appropriate language skills, and often the senior reviewer).
- Have the team go through each question, answer scale, instruction, and any other components, comparing draft suggestions, and considering other alternatives. Members aim to identify weaknesses
and strengths of proposed translations and any issues that arise such as comparability with the source text,
adaptations needed, difficulties in the source text, etc.
- Ensure that changes made in one section are also made, where necessary, in other places. Some part of this may be more easily made after the review meeting on the basis of notes taken.
- Whenever possible, finalize a version for adjudication.
- If a version for adjudication cannot be produced, the review meeting documentation should note problems preventing resolution.
- After review, before adjudication, copyedit the reviewed version in terms of its own accuracy
(consistency, spelling, grammar, etc.).
- After review, before adjudication, copyedit the reviewed version against the source questionnaire, checking for any omissions, incorrect filtering or instructions, reversed order items in a battery or answer scale labels, etc.
- Guidelines are only as good are their implementation. Quality monitoring plays an essential role. However, evaluation of survey quality begs many issues. Translators asked to assess other translators’ work may, for example, be hesitant to criticize or, if not, may apply standards which work in other fields but are not appropriate for survey translation. In the worst instance, they may follow criteria required by people who do not understand survey translation.
- Much remains to be established with regard to survey translation quality. Group dynamics are important. The lead person/senior reviewer leads the discussion. When two suggested
versions are equally good, it is helpful to take up one person's suggestion one time and another person's the next time. Given the objectives of the review, however, translation quality obviously
takes priority in making decisions about which version to accept.
- Time-keeping is important. The senior reviewer should confirm the duration of the meeting at the start and pace progress throughout. Otherwise much time may be spent on early questions, leaving
too little for later parts of the questionnaire.
- It is better to end a meeting when team members are tired and re-convene than to review later parts of the questionnaire with less concentration.
- Practice taking documentation notes on points not yet resolved or on compromised solutions. (See
Finding, Selecting, and Briefing Translation Team Members).
- Not everyone needs to be present for all of a review meeting. Members should be called upon as needed. Queries for substantive experts, for example, might be collected across the instrument and
discussed with the relevant expert(s) in one concentrated sitting.
- Have the adjudicator sign-off on the final version for pretesting.
Official approval may simply be part of the required procedure, but it also emphasizes the importance of this step and the significance of translation procedures in the project.
- If the adjudicator has all the skills needed (strong language ability in the source language and target language, knowledge of the study and also survey measurement and design issues), have him or her take part in the review session if this is possible. Even in this case, whenever possible it is advisable to delay official signing-off to another day, thus leaving time for final checking of the decisions taken .
- If the adjudicator does not have special relevant expertise, have him or her work with consultants to check that all the procedures have been followed, that appropriate people were involved, that documentation was kept, etc., according to procedural requirements. To assess the quality of review outputs, for example, the adjudicator can ask to have a list of all the perceived challenges and request to have concrete examples of these explained.
- If the expertise of the adjudicator lies somewhere between these extremes, consider having him or her review the translation with the senior
reviewer on the basis of the review meeting documentation.
- Ensure again that changes made in one section are also made, if necessary, in other places.
- Emphasizing the value of finding mistakes at any stage in the production is useful. At the same time, a team effort usually shares responsibility. If things are missed, it is best in any instance if no one is made to feel solely responsible.
- If a translation mistake means a question is excluded from analysis in a national study, the costs and consequences are high; in a comparative survey, the costs and consequences are even higher. Making team members aware of this may help focus attention. The German mistranslation in a 1985 ISSP question regarding participation in demonstrations meant both the German and the Austrian data on this question could not be compared with other countries . (Austria had used the German translation, complete with the mistranslation.)
- Pretest the version resulting from adjudication.
One purpose of pretesting is to test the viability of the translation and to inform its refinement, as necessary, in preparation for final fielding.
All instruments should be pretested before use. The best possible version achievable by the team development process should be targeted before pretesting (see Pretesting).
- Review, revise, and re-adjudicate the translation on the basis of
Pretesting results may show that changes to the translation are needed. Changes can be
implemented as described below.
- Decide on the team required to develop revisions. This will differ depending on the nature and number of problems emerging from the
pretest and on whether or not solutions are presented along with the problems.
- If a one- or two-person team is chosen that does not include one of the translators, share any changes (tracked or highlighted) with a
translator and a "typical target population person" for final commentary, explaining the purpose of the revision.
- Review the documentation from the pretest, considering comments for each question or element concerned.
- Ensure that changes made in one section are also made, where necessary, in other places.
- Copyedit the revised version in terms of its own accuracy (consistency, spelling, grammar, etc.).
Target language competence is required for this.
- Copyedit the revised version in its final form against the source questionnaire, checking for any omissions, incorrect filtering or instructions, reversed order items or answer scale labels, etc. Competence in both target and source language is required for this.
- Check in programmed applications that hidden instructions have also undergone this double copyediting (see Instrument Technical Design).
- Present the revised version for final adjudication. The adjudication procedures for this are as before. Project specifics will determine in part who is involved in the final adjudication.
- It is extremely easy to overlook mistakes in translations and in copyediting. The review and adjudication
steps offer repeated appraisals which help combat this, as do the documentation tools.
- It is often harder to find certain kinds of mistakes if one is familiar with the text. It is better if the
copyeditors are not the people who produced the texts.
- Although copyediting is a learnable skill, good copyeditors must also have a talent for noticing small details. The senior reviewer should ensure people selected for copyediting work have
- If the people available to copyedit have helped produce the translations, allow time to elapse between their producing the translation and carrying out copyediting. Even a few days may
- Problems with incorrect instructions, numbering, filters, and omitted questions are quite common. They are often the result of poor copyediting, cut and paste errors, or inadvertent omissions, rather than "wrong" translation. Thus, for example, reversed presentation of answer scale categories is a matter of order rather than a matter of translation. It can be picked up in checking, even if the reversal may have occurred during translation.
- Use a system of checking-off (ticking) material that has itself been tested for efficiency and usability. In iterative procedures such as review and revision, this checking-off of achieved milestones and versions and the assignment of unambiguous names to versions reduces the likelihood of confusing a preliminary review/adjudication with a final one.
- Automatic copyediting with Word will not discover typographical errors such as for/fro, form/from, and if/of/off. Manual checking is necessary.
- Organize survey translation work within a quality assurance and
quality control framework and document the entire process.
Defining the procedures used and the protocol followed in terms of how these can enhance the translation refinement process and the ultimate translation product is the most certain way to achieve the translation desired. Full documentation is necessary for internal and external quality assessment. At the same time, strong procedures and protocols do not resolve the question of what benchmarks should be applied for quality survey translation.  discusses the need for research in this area.
The steps involved in organizing a team translation are not repeated here. The focus instead is on what can
be targeted in terms of translation quality.
- Define survey translation quality in terms of fitness for use:
- Produce survey translations in a manner that adequately and efficiently documents the translation process and the products for any users of the documentation at any required stage in production (review, version production control, language harmonization, later questionnaire design).
- The effort required to implement a well-structured and well-documented procedure and process will be repaid by the transparency and
quality control options it makes possible. Thus even simple Word or Excel templates make it easier to
track the development of translations, to check that certain elements have not been missed, and to verify if and how certain problems have been resolved. These might begin with
translator notes from the draft productions and evolve into aligned translations in templates for review,
later becoming templates for adjudication with translations proposed and comments on these.
 provides examples of how Excel templates help guide quality control and assurance steps.
- Once procedures become familiar and people gain practice in following protocols, the effort involved to produce documentation is reduced.
Template 1 is typical of templates used in the European Social Survey (ESS) in rounds 1-4 for draft translations. The source questionnaire has been entered in the template in distinct sections. Each
translator enters his or her translation in the template and provides commentary. For later stages in the
translation process, similar templates retained information from each preceding stage and added columns for outcomes and comments on the current step (see Template 2).
Template 1: Extract from a translation template from the ESS Round 4 for one draft translation (core module B)
Template 2 illustrates possible headings for a template bringing together two draft translations for a review meeting based on Template 1.
Template 2: Headings required for a team review meeting
- Prepare individual translations in preparation for the review session.
- Take notes on translation and source texts in preparation for the review session (documentation to inform the review).
- Participate in review sessions with other members of the review team.
- Consult on any translation revisions at later stages.
- May assess source questionnaires for comparative viability.
- May assess other survey translations.
- May assist in copyediting.
- Participate in review sessions at times identified as relevant depending on their role.
- Contribute their individual area of expertise to developing and refining the translated instrument.
- Organize review session meetings (unless a co-coordinator does this).
- Organize materials for the review session(s) (unless a co-coordinator does this).
- Lead review sessions, including attending to group dynamics, appointing note takers, coordinating contributions to the discussion, ensuring the meeting runs according to
schedule, and ensuring each relevant topic is discussed and resolved or noted as unresolved.
- Organize and supervise the documentation of review session outputs. Review session outputs will principally consist of refined translation versions and accompanying
documentation, queries, and comments; they may also include action points arising from the review meeting(s), such as the need to consult with question designers or other subject
- Appraise and officially sign off on translations, usually after the review meeting(s).
- Appraise the review outputs probably in consultation with a senior advisor (the senior reviewer or other consultant) and approve a final version for
pretesting and fielding. If the adjudicator is also the senior
reviewer, review and
adjudication may follow directly upon one another.
- If the senior person on a project who is officially required to sign off on a translation is not appropriate to appraise translation quality and decisions, this nominal adjudicator
may delegate adjudication to another senior person better suited for this task. Alternatively, in the same situation, the adjudicator may use consultants and documentation from the
review session(s), to work through the translation and documented decision points and notes before signing off.
- Check for correctness in the target language, including spelling, omissions, wrong formatting,
consistency of formulation, and repeated phrases (e.g., "please tick one box"), and for completeness
of revision. When multiple versions are in circulation, teams can become unclear, for example, about which version is indeed intended to be the final version. Copyeditors should also check this.
- Check against the source document for such errors as inadvertent omissions or additions or question and answer option reversals,
mistakes resulting from copy-and-paste activities, misread source questions, and filter numbering correctness.
- Large translation efforts, centrally organized studies, or efforts conducted within a large organization may have a coordinator to manage the translation effort in an
organizational management sense (schedule coordination, personnel identification, budgeting, and so forth).
- In other instances the senior reviewer may organize the translation effort.
Substantive and other experts
- Substantive experts may be needed to provide advice on a variety of matters, such as the suitability of indicators or the formulation of questions with regard to measurement goals.
- Question design experts might be consulted about changes in format necessitated by translation.
- Interviewers might be consulted for fielding matters relevant to translation.
- Visual design experts might, for example, be consulted about cross-cultural aspects of visual presentation.
- If the questionnaire is computer-assisted, consultation with programmers, or those familiar with programming requirements, is needed to ensure that the translation document
or file is marked appropriately. Numerous programming details may need to differ from one language to another to accommodate different language structure requirements
(see Questionnaire Design).
- Projects sometimes run beyond agreed times of availability of personnel. Personnel may also become unavailable for a variety of reasons. It is a good idea to have back-up
personnel in place.
- If some parts of the translation process or translation outputs are to be subjected to external assessment, suitable assessment personnel will be required (see
- Changing existing materials (e.g., management plans, contracts, training manuals,
questionnaires, etc.) by deliberately altering some content or design component to make the resulting materials more suitable for another socio-cultural context or a particular population.
- The translation evaluation step at which a translation is signed off and released for whatever follows next such as pretesting or final fielding (see
Translation). When all review and refinement procedures are completed, including any revisions after pretesting and copyediting, a final signing off/adjudication is required. Thus, in any translation effort there will be one or more signing-off steps ("ready to go to client," "ready to go to fielding agency," for example).
- The person who signs-off on a finalized version of a questionnaire (see
- The systematic difference over all conceptual trials between the expected value of the survey estimate of a population parameter and the true value of that parameter in the
- The extent to which differences between survey statistics from different countries, regions, cultures, domains, time periods, etc., can be attributable to differences in population true values.
- Consent (informed consent)
- A process by which a sample member voluntarily confirms his or her willingness to participate in a study, after having been informed of all aspects of the study that are relevant to the decision to participate. Informed consent can be obtained with a written consent form or orally (or implied if the respondent returns a mail survey), depending on the study protocol. In some cases, consent must be given by someone other than the respondent (e.g., an adult when interviewing children).
- Consistency is achieved when the same term or phrase is used throughout a translation to refer to an object or an entity referred to with one term or phrase in the source text. In many cases, consistency is most important with regard to technical terminology or to standard repeated components of a questionnaire. Reference to "showcard" in a source questionnaire should be consistently translated, for example. The translation of instructions which are repeated in the source text should also be repeated (and not varied) in the target text.
- A legally binding exchange of promises or an agreement creating and defining the obligations between two of more parties (for example, a survey organization and the
coordinating center) written and enforceable by law.
- Coordinating center
- A research center that facilitates and organizes cross-cultural or multi-site research activities.
- The person who reviews a text and marks up any changes required to correct style, punctuation, spelling, and grammar errors. In many instances, the copyeditor may
also make the corrections needed.
- Fitness for intended use
- The degree to which products conform to essential requirements and meet the needs of users for which they are intended. In literature on quality, this
is also known as "fitness for use" and "fitness for purpose."
- Full translation (double translation)
- Each translator translates all of the material to be
translated. It stands in contrast to split translations.
- The third step in the concept/construct/indicator/question model. They relate to behaviors, attitudes, reported facts, etc., considered to provide indirect measurement
of constructs. Several indicators might be used for a given construct. For example, price, durability, the attractiveness of packaging, and purchasing convenience (ease), can be indicators to measure a
construct centered on customer satisfaction with a given product.
- Researchers differ greatly in how they use this term. It is usually and most correctly used to refer to the statements in Likert-type batteries. Example: The Government should
provide jobs for everyone who wants to work.
- Language harmonization
- Language harmonization can be understood as the procedures and result of trying to find a common version (vocabulary and/or structure) across questionnaires
for different regional varieties of a "shared" language.
- Mean Square Error (MSE)
- The total error of a survey estimate; specifically, the sum of the variance
and the bias squared.
- Measurement error
- Survey error (variance and
bias) due to the measurement process; that is, error introduced by the survey instrument, the interviewer, or the respondent.
- Method of data collection.
- Overlap in the split translations
- A compromise solution between split and
full translations is to ensure that some overlap exists between materials divided among translators. The
material is split up the way cards are dealt in many games, everyone getting a spread of the material. Each translator
could then receive the last one or two questions of another translators "piece." This allows the review team members to have an increased sense of whether differences in translating approaches between
translators and their understanding of source text components at the draft production level.
- A collection of techniques and activities that allow researchers to evaluate survey questions, questionnaires and/or other survey procedures before data collection
- The degree to which product characteristics conform to requirements as agreed upon by producers and clients.
- Quality assurance
- A planned system of procedures, performance checks, quality audits, and corrective actions to ensure that the
products produced throughout the survey lifecycle are of the highest achievable quality. Quality assurance planning
involves identification of key indicators of quality used in quality assurance.
- Quality audit
- The process of the systematic examination of the quality system of an organization by an internal or external quality auditor or team. It assesses whether the
quality management plan has clearly outlined
quality assurance, quality
control, corrective actions to be taken, etc., and whether they have been effectively carried out.
- Quality control
- A planned system of process monitoring, verification, and analysis of indicators of quality, and updates to quality
assurance procedures, to ensure that quality assurance works.
- Quality management plan
- A document that describes the quality system an organization will use, including quality assurance and
quality control techniques and procedures, and requirements for documenting the results of those procedures,
corrective actions taken, and process improvements made.
- Person who participates in the review of translations in order to produce a final version (see Appendix A of Translation).
- Source document
- The original document from which other (target) documents are translated or adapted as necessary.
- Source language
- The language in which a questionnaire is available from which a translation is made. This is usually but not always the language in which the questionnaire was designed.
- Source questionnaire
- The questionnaire taken as the text for translation.
- Split translation
- Each translator translates only a part of the total material to be translated in
preparation for a review meeting, in contrast to translating the entire text (see full translation).
- Survey lifecycle
- The lifecycle of a survey research study, from design to data dissemination.
- Target language
- The language a questionnaire is translated into.
- Target population
- The finite population for which the survey sponsor wants to make inferences using the sample statistics.
- Team translation
- Team approaches to survey translation and translation assessment bring together a group of people with different talents and functions in the team so as to ensure the mix of skills and discipline expertise needed to produce an optimal translation version in the survey context. Each stage of the team translation process builds on the foregoing steps and uses the documentation required for the previous step to inform the next. In addition, each phase of translation engages the appropriate personnel for that particular activity and provides them with relevant tools for the work at hand.
- Total Survey Error (TSE)
- Total survey error provides a conceptual framework for evaluating survey quality. It defines quality as the estimation and reduction of the
mean square error (MSE) of statistics of interest.
- The person who translates text from one language to another (e.g., French to Russian). In survey research, translators might be asked to fulfill other tasks such as reviewing and copyediting.
- A measure of how much a statistic varies around its mean over all conceptual trials.
 Acquadro, C., Jambon, B., Ellis, D., & Marquis, P. (1996). Language and translation issues. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trial (2nd ed.) (pp. 575—585). Philadelphia, PA: Lippincott-Raven.
 Dept, S., Ferrari, A., & Wäyrynen, L. (2008, June). Comparative overview of test adaptation and translation verification procedures in three international surveys. Paper presented at the international conference on survey methods in multinational, multiregional, and multicultural contexts (3MC), Berlin, Germany. Retrieved April 16, from http://www.csdiworkshop.org/pdf/3mc2008_proceedings/session_09/Dept.pdf.
 Harkness, J. A. (2010, March). Survey translation and the question of quality. Paper presented at the International Workshop on Comparative Survey Design and Implementation, Lausanne, Switzerland.
 Harkness, J. A. (2008). Comparative survey research: Goals and challenges. In E. D. de Leeuw, J. J. Hox & D. A. Dillman (Eds.),
International handbook of survey methodology (pp. 56-77). New York, NY/London: Lawrence Erlbaum Associates..
 Harkness, J. A. (2000/2008). Round 4 ESS translation strategies and procedures. Retrieved April 16, 2010, from http://www.europeansocialsurvey.org/index.php?option=com_docman&task=doc_download&gid=351&itemid=80
 Harkness, J. A. (2007). Improving the comparability of translation. In R. Jowell, C. Roberts, R. Fitzgerald & G. Eva (Eds.), Measuring attitudes cross-nationally: Lessons from the European Social Survey (pp. 79—94). London: Sage.
 Harkness, J. A. (2003). Questionnaire translation. In J. A. Harkness, F. van de Vijver & P. Ph. Mohler (Eds.), Cross-cultural survey methods (pp. 35—56). Hoboken, NJ: John Wiley & Sons.
 Harkness, J. A., Edwards, B., Hansen, S. E., Miller, D. R., & Villar, A. (2010). Designing questionnaires for multipopulation research. In J. A. Harkness., M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. Ph. Mohler, B-E. Pennell & T. W. Smith (Eds.), Survey methods in multinational, multicultural and multiregional contexts (pp. 33-58). Hoboken, NJ: John Wiley & Sons.
 Harkness, J. A, Pennell, B-E., & Schoua-Glusberg, A. (2004). Questionnaire translation and assessment. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin & J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 453-473). Hoboken, NJ: John Wiley & Sons.
 Harkness, J. A., Schoebi, N., Joye, D., Mohler, P., Faass, T., & Behr, D. (2008). Oral translation in telephone surveys. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. de Leeuw, L. Japec, P. J. Lavrakas, (Eds.), Advances in telephone survey methodology (pp. 231—249). Hoboken, NJ: John Wiley&Sons.
 Harkness, J. A., & Schoua-Glusberg, A. S. (1998). Questionnaires in translation. In J. A. Harkness (Ed.), Cross-cultural survey equivalence (pp.87—126). ZUMA-Nachrichten Spezial, 3. Mannheim, Germany: ZUMA.
 Harkness, J. A., Villar, A., & Edwards, B. (2010). Translation, adaptation and design. In J. A. Harkness., M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. Ph. Mohler, B.-E. Pennell & T. W. Smith (Eds.), Survey methods in multinational, multicultural and multiregional contexts (pp.117—139). Hoboken, NJ: John Wiley & Sons.
 Harkness, J., Villar, A., Kephart, K., Behr, D., & Schoua-Glusberg, A. (2009, March). Research on translation assessment procedures: Back translation and expert review. Paper presented at the International Workshop on Comparative Survey Design and Implementation, Ann Arbor, MI.
 Harkness, J. A., Villar, A., Kephart, K., Schoua-Glusberg, A, & Behr. D. (2009, May). Survey translation evaluation: Back translation versus expert review. Paper presented at the annual conference of the American Association for Public Opinion Research, Hollywood, FL.
 Harkness, J. A., Villar, A., Kruse, Y., Branden, L., Edwards, B., Steele, C., & Wang, Y. (2009, March). Interpreted telephone survey interviews. Paper presented at the International Workshop on Comparative Survey Design and Implementation, Ann Arbor, MI.
 Harkness, J. A, Villar, A., Kruse, Y., Steele, C., Wang, Y., Branden, L., Edwards, B., & Wilson, C. (2009, May). Using interpreters in telephone surveys. Paper presented at the annual conference of the American Association for Public Opinion Research, Hollywood, FL.
 McKay, R. B., Breslow, M. J., Sangster, R. J., Gabbard, S. M., Reynolds, R. W., Nakamoto, J. M. et al. (1996). Translating survey questionnaires: Lessons learned. New Directions for Evaluation, 70(Summer), 93-105.
 Pan, Y., & de la Puente, M. (2005). Census Bureau guidelines for the translation of data collection instruments and supporting materials: Documentation on how the guideline was developed (Research Report Series #2005-06). Statistical Research Division, Washington, DC: U.S. Bureau of the Census.
 Schoua-Glusberg, A. (1992). Report on the translation of the questionnaire for the national treatment improvement evaluation study. Chicago, IL: National Opinion Research Center.
 Willis, G. B., Kudela, M. S., Levin, K. L., Norberg, A., Stark, D. S., & Forsyth, B. H. et al. (2010). Evaluation of a multistep survey translation process. In J. A. Harkness., M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. Ph. Mohler, B-E. Pennell & T. W. Smith (Eds.), Survey methods in multinational, multicultural and multiregional contexts (pp. 137—152). Hoboken, NJ: John Wiley & Sons.
 Afrobarometer Survey. Retrieved July 10, 2010, from http://www.afrobarometer.org/questionnaires/R4surveymanual_4feb08_FINAL.pdf
 Asian Barometer. Retrieved July 10, 2010, from http://www.asianbarometer.org/newenglish/introduction/default.htm
 European Social Survey (ESS). Retrieved July 10, 2010, from http://www.europeansocialsurvey.org
 Survey of Health, Ageing and Retirement in Europe (SHARE). (2010). Retrieved July 10, 2010, from http://www.share-project.org/
Return to top
Previous chapter |
Next chapter |
© 2008 The authors of the Guidelines hold the copyright. Please contact us if you wish to publish any of this material in any form.