Doing research is an exercise in trade offs. Richness of information typically comes with reduced generalization. The more a researcher seeks to control for confounding variables the less realistic his or her results are likely to be. High precision, generalization and control almost always translate into higher costs. When researchers make choices about whom they will study where their research will be done, the method they will use to collect data, so on, they must make some concessions. Good research designs are not perfect but they do carefully reflect the questions being addressed. These facts are to be kept in mind as we review the strengths and weakness of five popular research designs: case studies, field surveys, laboratory experiments, field experiments and aggregate quantitative reviews.
You pick up a copy of Soichiro Honda’s autobiography. In it he describes his impoverished childhood; his decisions to open a small garage, assemble motorcycles, and eventually build automobiles; and how this led to the creation of one of the largest and most successful corporations in the world. Or you are in a business class and the instructor distributes a 50 page handout covering two companies: Wal–Mart and K-mart. The handout details the two firms’ histories; describes their corporate strategies, management philosophies, and merchandising plans and includes copies of their recent balance sheets and income statements. The instructor asks the class members to read the handout analyze the data, and determine why Wal-Mart has done so much more successful than Kmart in recent years.
Soichiro Honda’s autobiography and the Wal-Mart and Kmart handouts are case studies. Drawn from real life situations, case studies present an in-depth analysis of one setting. They are thorough descriptions, rich in details about an individual, a group or an organization. The primary sources of information in case studies is obtained occasionally backed up by interviews and a review of records and documents.
Case studies have their drawbacks. They are open to the perceptual bias and subjective interpretations of the observer. The reader of case is captive to what the observer/case writer chooses to include and exclude. Cases also trade off depth of information and richness of detail. Because it is always dangerous to generalize from a sample of one, case studies make it difficult trove or reject a hypothesis. On the other hand, you can’t ignore the in-depth analysis that cases often provide. They are an excellent device for initial exploratory research and for evaluating real life problems in organizations.
A length questionnaire was created to asses the use of ethics policies, formal ethics structures, formalized activities such as ethics training, and executive involvement in ethics programs among billion-dollar corporations. The public affairs or corporate communications office of all Fortune 500 industrial forms and 500 service corporations were contacted to get the name and address of the officer most responsible for dealing with ethics and conduct issues in each firm. The questionnaire, with a cover letter explaining the nature of the study was mailed to these 1,000 officers. Of the total, 254 returned a completed questionnaire, for a response rate just above 25 percent. The results of the survey found, among other things that 77 percent has formal codes of ethics and 54 percent had a single officer specifically assigned it deal with ethics and conduct.
The preceding study illustrates a typical filed survey. A sample of respondents (in this case, 1,000 corporate officers in the largest US publicly held corporations) was selected to represent a larger group that was under examination (billion dollar US business firms). The respondents were then surveyed using a questionnaire or interviewed to collect data on particular characteristics (the content and structure of ethics programs and practices) of interest to the researchers. The standardization of response items allows for data to be easily quantified, analyzed, and summarized and for the researchers it make inferences from the representative sample about the larger population.
The field survey provides economies for doing research. It’s less costly to sample a population than to obtain data from every member of that population. There are, for instance, more than 5,000 US business firms with sales in excess of a billion dollars; and since some of these are privately held and don’t release financial data to the public, they are excluded from the Fortune list). Moreover as the ethics study illustrates field surveys provide an efficient way to find out how people feel about issues or how they say they behave. These data can then be easily quantified.
But the field survey has a number of potential weaknesses. First, mailed questionnaire rarely obtain 100 percent returns. Low pressure rates call into question whether conclusions based on respondents’ answers are in general to non-respondents. Second, the format is better at tapping respondents’ attitudes and perceptions than behaviors. Third, responses can suffer from social desirability; that is, people saying what they think the researchers want to hear. Fourth, since field surveys are designed to focus on specific issues they are a relatively poor means of acquiring depth of information. Finally, the quality of the generalization is largely a factor of the population chosen. Responses from executives at Fortune 500 firms, for instance, tell us nothing about small or medium sized firms or not-for-profit organizations. In summary even a well designed field survey trades off depth of information for breadth generalization and economic efficiencies.