Experimenter Bias

Description:

The argument draws a conclusion from data that has been influenced by the expectations and hopes of the person collecting the data.

Comments:

In most cases this influence is unconscious and unintentional. Bad data may be due to such "honest" mistakes as rounding up or rounding down to favor a certain result, or treating ambiguous results as favoring the preferred result. However, there have also been documented cases of outright fraud. I'm not sure how to classify scientific fraud, except as a really extreme case of Experimenter Bias.

Examples:

"Samuel Morton collected data on cranial capacity, hoping to prove that white races had a larger brain size than dark races. He measured cranial capacity by filling the cranium with mustard seed and measuring how much seed each skull could hold. He performed the measurements himself, knowing which skulls belonged to whites and which belonged to other races. His results confirmed an average cranial capacity of 87 cubic inches for whites, but only 83 cubic inches for Africans." [This difference disappeared when the experiment was re-done using lead shot, and using lab assistants unfamiliar with Morton's collection to do the measurements.]

"In 1955 Cyril Burt published a study of 21 identical twins who were separated at birth and raised in different families. His study confirmed his theory that intelligence and behavior is controlled more by genetics than by environment." [It was later discovered that Burt's data was fabricated, and indeed that many of the pairs of twins included in his study did not even exist!]

Discussion:

Since such tainting may be unconscious, steps must be taken to prevent it. The fallacy of Experimenter Bias may be avoided by using "double blind" techniques, so that experimenters do not know (as they are recording data) which results the data favors. Any experiment that does not employ such techniques may be suspected of committing the error of Experimenter Bias, so experimenters are generally very careful to build such techniqes into their experiments.

Double blind means that both the experimenter and the experimental subjects are "blind" to the meaning of the data being collected. Experiments in which the experimenter is not "blind" may commit the fallacy of Experimenter Bias; experiments in which the experimental subjects are not "blind" commit the fallacy of Tainted Data.

Source: Certainly not the original source, but the most entertaining and comprehensive discussion of this fallacy is in Stephen Jay Gould, The Mismeasure of Man, New York: Norton, 1981. He discusses both Morton's cranial capacity measurements and Cyril Burt's fabricated twin studies.

WELCOME                     EXPLANATION OF PRINCIPLES                                     TABLE OF FALLACIES                        EXERCISES                     INDEX