Before the questionnaire is ready for the field, it needs to be pre-tested under field conditions. No researcher can prepare a questionnaire so good that improvements cannot be discovered in field tests. Researchers have reported pre-testing, changing, and pre-testing again for as many as 25 times before they were satisfied with some questionnaires. One pretest is as much, however, as most questionnaires get.
Pre-tests are best done by personal interview, even if the survey is to be handled by mail or telephone. Interviews can note respondent reactions and attitudes which cannot otherwise be obtained. After any pertinent changes in the questionnaire have been made, another pretest can be run by mail or telephone if those methods are to be used in the survey. This latter pretest should uncover any weakness peculiar to the method of communication.
The people interviewed in a pretest should be roughly similar to those who will be covered in the final study. Ordinarily, the number of interviews in a pretest is small perhaps 20 but, if major changes are made as a result, the new questionnaire should be pre-tested again. Only the best interviewers should be used in pre-test work, for they must be able to perceive uneasiness, confusion and resistance among respondents. Poor interviewers may not be aware of these difficulties.
Interviewers in pretests should watch particularly to see if the issue in each question is clear to respondents. Any requests for explanation, comments, or other reactions by the respondents should be noted. After an interview is over, interviewers may discuss it with respondents to get explanations of what meaning they got from individual questions and why they answered don’t know where they did.
Wording of some questions should be improved as a result of the pretest. Interviewers should note words which are not understood by all respondents. If there is any doubt as to the wording of a question, interviewers can try alternate wording and compare reactions of respondents to the different phrasings.
As a result of the pretest, some questions may be eliminated from the questionnaire and others may be added. Interviewers will observe questions that cause embarrassment or resistance, the point at which respondents begin to get bored and impatient, and the places where relaxed cooperation seems to break down.
Pretest should also test question sequence. Do the first questions catch the respondent’s interest? Are some answers influenced by the questions that preceded them? Interviewers should report mechanical difficulties encountered, such as confusion in following question sequence and difficulty in squeezing answers into the spaces allotted.
Revision and Final Draft:
After each significant revision of the questionnaire, another pretest should be run. When the last pretest suggests no new revisions, the researcher is ready to print the actual questionnaires to be used in the survey.
Forms for observational studies:
Forms for the recording of observational data are much simpler to construct than are questionnaires. The psychological impact of the form on the source of the information does not need to be considered. All that is necessary is to develop form that makes it simple for the field workers to record the desired information accurately, that identifies the information properly, and that makes tabulation of the results easy.
A good observation form permits researchers to record individual observations, but it does not force them to make a summary of their observations, which would be subject to error.
Physical layout and reproduction of observation forms should follow the same rules discussed for questionnaires.
Recording of disguised and unstructured observations is more difficult.
Usually in such situations it is not feasible for observers to record observations at the moment they are made for fear of influencing the event of interest. In a shopping observation for example, where observers are posing as prospective customers in an appliance store they can hardly stop after each exchange with the sales person to note what took place. The sales person would begin to wonder what was going on. Under such circumstances, the observation must be written up as soon after the event as possible – in the example, as soon as the customer observer leaves the store. To help in remembering what has taken place, it is useful to have an outline of the desired observations in mind before undertaking the actual observations. The observer might have in mind to note the following events: (1) the salesperson’s opening; (2) the number of sales points made; (3) if certain major sales appeals were brought up; (4) the method of handling selected objections; (5) the salesperson’s understanding of the product mechanics; (6) how the question of price was handled and (7) how the salesperson attempted to close the sale. With these as a framework for observation, observer would be able to recall what occurred if they recorded it immediately. In some cases, it may be possible to record some of the observations by checking items on a list – for example, did salesperson demonstrate the automatic ice cube maker; did salesperson volunteer price?
Mechanical recording of observations is accomplished in some cases by such devices as the Nielsen audiometer or the eye-camera. Moving pictures are used to record consumer action in such shopping activities as picking products form supermarket shelves. These mechanical methods give accurate records of the factors recorded, but the factors may be limited to only part of the total activity.