Developing and testing the main survey questionnaire

The survey questionnaire should aim to gather information that will fulfil the survey objectives as well as provide data for monitoring current strategies and planning for the future. It needs to be clear how the results of the questionnaire will be used, how they will relate to other results, and whether the inclusion of a particular indicator can potentially affect overall programme management. In circumstances where the results of a specific part of the survey will be compared with the results from a previous survey, the wording of the questions and the population groups addressed should be as similar as possible across the two surveys.

It can be helpful at this stage to develop empty report tables (called “shells”) to show how final results could be presented. Each table can be accompanied by notes describing how the data will be collected and how they relate to the survey objectives.

The questionnaire is usually designed to collect certain types of information. This may include:

  • Information about the main micronutrient-related indicators stated in the survey objectives, for example:
    • confirmation that the biological specimens or food samples were collected from respective individuals or households
    • results of any field-based testing (for example, the haemoglobin concentration or the result of a rapid diagnostic test for malaria).
  • Information that can facilitate cross tabulations, such as:
    • demographic information
    • socioeconomic information
    • information reflecting respondent knowledge, attitudes, and practices (KAP), usually in relation to micronutrient interventions
    • participation in nutrition interventions.
  • Information that helps interpret measurements of micronutrient biomarkers Module 3: Biomarker selection and specimen handling tool for further information), such as
    • pregnancy status
    • lactation status
    • use of supplements
    • cooking practices that may affect the micronutrient content of food
    • time of day that specimens were collected (if biomarkers are subject to diurnal variation)
    • approximate time since last meal
    • altitude of the cluster location
    • whether the respondent is a smoker and, if so, the average number of cigarettes smoked per day.
  • Information to assess the impact of certain nationally-relevant practices on micronutrient status, for example:
    • the frequency of using salt substitutes (such as bouillon), to estimate the potential contribution of this dietary habit to iodized salt intake
    • behavior related to a culturally specific infant feeding practice, to assess the impact of a targeted behaviour change communication campaign
    • the use of cooking ash, which affects food’s acidity and in turn affects the bioavailability of micronutrients in multiple micronutrient powders.

When questionnaire modules are similar to those included in the UNICEF Multiple Indicator Cluster Survey (MICS) and the Demographic and Health Survey (DHS), they should use similar design and wording. Using the same or similar wording of questions and similar tables for presenting results makes it easier to compare with other surveys conducted in the country as well as with surveys conducted in other countries. The UNICEF MICS questionnaires are available in English, Spanish, French, and Russian (http://mics.unicef.org/). The DHS model questionnaires are available in English and French (https://dhsprogram.com/What-We-Do/Survey-Types/DHS-Questionnaires.cfm). Standard methods for analysing the data to fill suggested table shells are provided in the MICS and DHS documentation, available on the respective websites. Suggested wording for questions used in micronutrient surveys among different population groups can be found online in the example questionnaires here.

New questions that have not been part of previous surveys need to be developed systematically, preferably following the cognitive interviewing process described in this module.

The overall format of the questionnaires will vary by survey, depending on the objectives, the number and types of micronutrients being assessed, population groups and interventions of interest. All formats must include standard identifying information for each household and participant so that information for each individual can be related to the respective household data and to samples and specimens collected. Sometimes it may also be useful to link the data for specific household members. Unique barcode labels, which are discussed in more detail in Module 9: Survey equipment and supplies, are the recommended method when collecting this information.

Designing an electronic data collection program, with all the required logic and error checks, can take many months. Developing the electronic data entry systems would ideally take place after the paper-based versions of these tools have been pretested with appropriate target populations (see the sequence of development in Fig. 11.1).

Interviewing should be done in the language most familiar to the participant. This may require translating the main survey questionnaire into several languages. Later sections in this module provide guidance on translating and pretesting questionnaires.

Fig. 11.1. Sample sequence for developing the main survey tool

file

Cognitive interviewing

Cognitive interviewing is used to identify and resolve problems with the wording of survey questions and the flow of the questionnaire. This process should be conducted as part of questionnaire development, in parallel to or immediately after pretesting, as it will lead to greater reliability and better interpretation of the final survey questionnaire and outcomes. The process should be applied with each main language into which the tool has been translated.

Cognitive interviewing also helps to identify problems that are not always obvious when designing survey questions, such as:

  • differences between cultures or individuals in interpreting questions and the intention of answers provided;
  • participants not understanding, or being confused by, certain words;
  • participants not understanding, or not knowing, how to answer;
  • participants not being willing to respond truthfully to a question;
  • ambiguity or poor fit between a question and the options for response;
  • redundancy of questions that add no new useful information; and
  • problems with measuring the concept the question is trying to address, for example, questions that expect a knowledge of typical serving sizes or defining the intended period of time for a recall question.

Box 11.1 provides examples of how cognitive interviewing can improve micronutrient survey questions.

There are several approaches to conducting cognitive interviews. The approach described here is “concurrent cognitive interviewing”, which are one-on-one interviews where an interviewer asks the survey questions and the participant responds. After each question is asked and answered, the interviewer probes about that question and the participant’s response, seeking information to better understand the participant’s interpretation of the question and the basis for the response. Sample probing questions include “What did you think that question was trying to ask?”, “Can you repeat that question using your own words?” and “How did you come to that answer?” General probes often help identify problems that were not anticipated, and tailored probes can focus on areas that are anticipated to be problematic. Interviewers should write notes during the interview to document responses, problems, and potential solutions.

Initially, five to ten interviews are conducted with a range of people from the intended survey participant groups, then responses are analysed, and the questions are revised if needed. Additional interviews are carried out with new participants to test the revised questions. The process is repeated until the final questions are agreed.

Box 11.1 Examples of how cognitive interviewing can improve questions in the main survey tool

  1. Time period recall errors: The common phrase “during the last week” may be interpreted in different ways, for example “from today through last week,” “the last 7 days,” “last Monday to Friday,” or “last Sunday to Saturday”. These interpretations can lead to recalls that cover a time period ranging from five to ten days. Pilot testing might not capture this error because respondents can provide an answer and usually appear to understand the question. Through cognitive interviewing, interviewers ask the participant specifically what they thought “the last week” meant and how many days they considered when responding. This will improve the validity and reliability of the question wording, and can improve interpretation of the data collected.

  2. Development of appropriate language to reflect the intended meaning in the original language: In one example, short recall questions focused on the consumption of salt and sugar by women and, by parental recall, for young children. The questions were harder to phrase in the translated survey language compared to the original language. The intended meaning was extracted using the following clarifications: salt and sugar are ingredients added to foods and are not normally consumed alone, salt and sugar are added both during the cooking process and at the table, and women often prepare foods for others in the household that they do not necessarily consume themselves. Cognitive interviewing helped identify appropriate wording that was understood and that adequately and validly captured consumption by the populations of interest.

  3. Development of the most relevant list of closed-end response options that might be used among different population groups: Conducting the cognitive interviewing process with a range of population groups leads to the inclusion of a comprehensive list of options for answers.

It is important to identify the most essential questions that require testing. These are most likely to be questions that have not been included in previous surveys or not in one or more of the languages of the planned survey, or questions that are challenging to phrase, or those intended to measure prioritized indicators. Box 11.2 provides more detail on the concurrent cognitive interview technique.

Box 11.2. Tips for the Concurrent Cognitive Test Technique

Concurrent cognitive interviewing uses a structured interview guide with general and specific probes for each question. The interview guide helps ensure that all probes are asked.

Interviewers should be encouraged to help identify potential solutions to the problems that may arise by, for example, suggesting and testing revised wording during the test process.

At the beginning of the cognitive interview, it is important that participants understand that:

  • they are being interviewed to develop the survey questionnaires so that the interviewer can determine if there are any weaknesses or mistakes with the questions;
  • they are being asked to describe anything that is difficult to understand, or identify a question that is hard to answer;
  • the interviewer is interested in the ways that they decided on their answers, how sure they are about them, and any problems that they encountered;
  • the interviewer did not write the questions and they should not worry about criticizing or hurting anyone’s feelings by pointing out problems; and
  • there are no “wrong” statements and they should respond truthfully.

The sequence of processes required to finalize the data collection tool is shown in Fig. 11.1.

Translation and back-translation

The World Health Organization (WHO) has guidelines for the translation of survey tools 1 that can be used for micronutrient status survey tools. The key steps described are:

  • Finalize the survey tools in English or another applicable language (such as Spanish, French or Portuguese).
  • Develop translated versions in each of the languages needed for the micronutrient survey. The translation should be conceptually equivalent (have the same meaning), culturally appropriate and easily understood.
  • Back-translate the questionnaires to the original language to ensure that the content and meaning of the questions have been maintained.

Translation and back-translation should be done by two different individuals who are fluent in the language of the original questionnaire, whose mother tongue is that of the translated questionnaire, and who have not been involved in the questionnaire development. Box 11.3 provides more detailed information on the process put forward in the WHO guidance.

Box 11.3 Main Steps in Translation and Back-Translation of Survey Questionnaires

  1. Forward translation: Translators who are fluent in the language of the original questionnaire and whose mother tongue is the language that the questionnaire is being translated into translate the original version of each tool into their mother tongue. These translators should have a health/nutrition background, be familiar with nutrition terms, and have experience in interviewing. Instructions should emphasize the need for conceptual translations that provide the same meaning as the original and can be easily understood.

  2. Group discussion to finalize translations: All those involved with the translations, the Survey Coordinator, and other health or nutrition experts with knowledge of at least one of the translation languages and expertise in developing and translating survey questionnaires should meet together to review the original and translated versions of the questionnaire to identify and resolve any discrepancies in the first translations.

  3. Back-translation: Each questionnaire should then be back-translated into the original language using the same approach as the first step, but with a different independent translator. This version can then be compared with the original and, where needed, changes should be made to the translated questionnaires. It is useful to outline the main questions, terms used, and intended concept where discrepancies in meaning or translation were identified.

  4. Pretesting and cognitive interviewing: The revised version of each translated questionnaire should then be pretested with a sample of individuals who are native speakers of that language and have similar demographic characteristics to the intended survey population group. This will assess the understanding and acceptability of the translated questionnaires. If necessary, cognitive interviewing should also be conducted.

  5. Final translation: Based on the above steps, a final translated version should be produced for each of the languages that will be used for the micronutrient status survey. These versions should be used for the training and pilot testing.

Pretesting the survey questionnaire

The survey questionnaire must be pretested with a range of people from the intended different survey populations, including people of different educational and socioeconomic backgrounds. Testing each type of questionnaire tool with five to ten people should help identify any major concerns (such as erroneous skip patterns).

Pretests should be conducted by one or more trained interviewers. Ideally, these interviewers would be those who are likely to serve as supervisors or Team leaders for survey implementation. Interviewers should note where the participant hesitates or gives incorrect answers, to indicate where the tool could be made clearer. The pretest experience should be discussed with the Survey coordinator and relevant Technical committee members to suggest improvements when revising each tool. Whenever a significant change is made, such as the structure of a form, a second round of pretesting should be conducted.

Other data collection tools

A number of additional tools are needed to support survey implementation and to monitor the progress of the fieldwork. These are described in the following sections. Not all surveys will use all tools.

Anthropometry data collection tool

Anthropometry is usually conducted after having completed the interviews, therefore the anthropometry form often comes at the end of the questionnaire. It is essential to confirm that anthropometry measurements are taken from and recorded for the correct individual. This means that the anthropometry form may include questions about date of birth and about sex even when these were collected at the beginning of the interview. Repeating the questions helps the field team verify that the correct individual is being measured. Any errors would also be picked up during programmed data cross-checks, which may happen in the field with electronic data collection, or at a later stage of data management with paper-based forms. The main data collected on the form should include:

  • date of birth and date of assessment to determine age
  • sex
  • height/length (with indication of which was collected) and unit of measurement
  • weight and unit of measurement
  • response: successful, refused, did not come for measurement
  • codes to identify stadiometer, scale and measurer.

Additional anthropometric data may include:

  • a note of bilateral pitting oedema
  • mid-upper arm circumference (MUAC)

The format of the tool needs to be pilot tested. The form used for training should be the same as that used in the field.

Specimen collection and tracking tools

Biological specimen collection is also usually conducted after the main interview, so the tool to document this process may be located at the end of paper questionnaires or in another format that can be linked to the main paper or electronic questionnaire. Similar to anthropometry, it is essential to check that the specimen ID matches the respective individual ID number.

The specimen collection tool notes:

  • the code of the person collecting the specimens
  • the codes for the equipment used (to check for systematic errors)
  • the time of data collection, and fasting status if applicable
  • whether each of the intended specimens was collected for each individual
  • the value of results (if applicable)
  • referrals to health facilities (if applicable)

A yes/no question for all intended specimen collection should be included at the end of each questionnaire and should also be included in the summary cluster control form (described later in this module).

Specimen tracking forms should be developed to allow the field phlebotomists and laboratory technicians to match the specimens collected at the household and analysed in the field with the specimens transported to the laboratory. The forms can be modified according to the survey. If necessary, an additional form should be developed, depending on what laboratory processing is done in the field. The specimen tracking forms should accompany biological specimens or food samples collected (and processed, where applicable) from the field to the laboratory where they will be further processed, analysed or stored. Tracking forms include a summary of the total number of each type of specimen processed and transported, the time and temperature at each transfer from field collection to arrival at the final analysis laboratory, and the signature of the person responsible at each stage.

In addition, a “laboratory specimen log” should be created to record each specimen that arrives, is stored, and is analysed at that laboratory. To reduce errors in data entry, this would ideally be an electronic spreadsheet with barcode scanning of sample IDs.

Sample ”Specimen Tracking Form for Phlebotomist” and ”Specimen Tracking Form for Laboratory Technician” are found in the online tools.

Temperature monitoring

A section to record temperature is included in the “Specimen tracking form for phlebotomist” online tool. Additional forms need to be developed for each level of the cold chain. Thermometers or temperature monitors must be used to check the temperature of each cool box. It is important to set up a system that allows the temperature to be determined in a way that does not cause repetitive opening of the cool box. The ”Specimen tracking form for phlebotomist” is best placed inside a plastic covering to prevent moisture damage to the paper forms and secured with tape on the outside of each specimen storage container (cool box, refrigerator or freezer).

Referral forms

Some indicators, such as the prevalence of anaemia, can be tested in the field. The Steering committee should agree with the ministry of health on the appropriate action to take if results indicate that a participant has a health condition or nutritional deficiency that needs attention. These conditions are most frequently severe anaemia, malaria or oedema.

In some cases, survey participants may be referred to local health facilities according to criteria determined by the ministry of health. These criteria usually include the availability of clinical guidelines and treatment, the capacity of staff to handle an increased volume of patients, and costs. If survey staff are instructed to refer participants, the ministry of health will need to advise staff in nearby public health facilities about the possibility of participants arriving with survey referrals for specific conditions or illnesses. See the ”Referral Form” online tool for an example referral form.

In all cases, the referral policy should be clear and set by the ministry of health. This policy needs to include a management plan for test results and referral logs that are in accordance with the agreed confidentiality of personal data. Participants need to be clearly informed about how data will be used and by whom. The referral protocols should be tested and practiced in the field. A log of all referrals to health facilities should be maintained by each team. Examples of these can be found in the ”Referral Log for Anemia”) and a ”Referral Log for Malaria” online tools. Additional logs can be developed depending on the specific referral criteria in country.

Cluster control form

The cluster control form provides a list of all households that are to be interviewed in the cluster, and is useful to the interviewers, laboratory personnel and supervisors. The main information usually includes the items in the list below. In cases with prior selection of households based on an earlier census, other details might be included.

  • household and cluster number
  • name of the head of household
  • status of interview/sample collection (not present, refused, complete, incomplete)
  • number of attempts to contact household members
  • call-back information if an interview was not completed
  • referrals made
  • reasons for any non-response.

The information on the cluster control form must be concise and at the same time must include all important information needed to summarize the main data collected. See the generic cluster control form online tool and an example from Malawi “Malawi cluster control form”.

At the end of each day, the supervisor(s) should check all cluster control forms filled out by the survey team members and produce a “Cluster summary sheet” for each cluster. This form can be either paper-based or electronic. If in paper format, it can be photographed when complete and uploaded along with electronic data collection forms, or it can be transported or emailed to the central data location. The information contained in the form should be entered into a database of monitoring information so it can be cross checked with the laboratories that are monitoring data collection within clusters to ensure consistency.

  1. Process of translation and adaptation of instruments. Geneva: World Health Organization; 2010 http://www.who.int/substance_abuse/research_tools/translation/en, accessed 18 July 2019. 

tools
examples