Home»Experiments»Cities » Aumann Committee Experiment Bookmark and Share

Aumann Committee Experiment

Under the auspices of the Center For the Study of Rationality, Hebrew University, the Aumann committee began its work in 1996. By 1998 it had a list of instructions to be provided to experts who would determine the list of birth and death cities and their Hebrew spellings to be associated with each of the rabbis of the Gans experiment. Two sets of instructions were generated. One, called the "Fresh Test," leaves more discretion to the experts. The other, called the "Replicative Test", follows more closely to the Gans experiment. The committee finished its work on July 16, 2002. Both experiments produced non-significant results. The committee report (with its appendices) is the first document provided.

Aumann Committee Report
Appendix 1: Gans Original Report
Appendix 2: WRR Statistical Science Paper:
Appendix 3: Instructions to the Experts (Hebrew)
Appendix 4: (Hebrew) The City Names Provided By the Experts
Appendix 5: The Bar Natan and Rips Minority Reports

Unfortunately, one expert was not even given the written instructions, but briefed orally. His work, in fact, deviated from the instructions. Furthermore, there were glaring errors in the so-called expert data, errors that are not just one opinion rather than another, but documentable factual errors that the expert himself acknowledged after being shown them. These are discussed in Witztum's report (in Hebrew). In a 159 page report, he documents the blatant errors in the preparation of the Aumann committee data. Because of these errors, Professor Lapides, a committee member, refused to sign the final committee report.

Witztum's report Part 1:
Preface Pages 1-4 (In Hebrew)
Pages 5-11 (In Hebrew)
Pages 12-17 (In Hebrew)
Pages 17-24 (In Hebrew)
Appendix 1 (In Hebrew)
Appendix 2 (In Hebrew)
Appendix 3 (In Hebrew)
Appendix 4 (In Hebrew)
Appendix 5 (In Hebrew)
Appendix 6 (In Hebrew)
Appendix 7 (In Hebrew)

Witztum's report Part 2:
Preface (In Hebrew)
Pages 90-96 (In Hebrew)
Pages 97-105 (In Hebrew)
Appendix 1 (In Hebrew)
Appendix 2 (In Hebrew)
Appendix 3 (In Hebrew)
Appendix 4 (In Hebrew)
Appendix 5 (In Hebrew)
Appendix 6 (In Hebrew)

When the errors became publicly known, including the use of data for a mis-identified rabbi, the committe had to decide what to do. Instead of playing the role of the scientist, who when shown errors in his experiment would immediately correct the errors and redo the experiment, the Aumann committe issued a statement. In this statement Aumann acknowledges problems with the execution of the committee's instructions and acknowledges errors in the data. But he sums it all up by saying that mistakes are taken into account in the experimental design!

Doron Witztum writes that when he first checked a subset of the expert data he found:

"17 instances in which the expert's data did not exist in the references given to him to support his decisions. Moreover, in 6 cases, the information given in the references contradicted the data he claimed to derive from them."

Professor Lapides writes in his statement:

"A thorough check has shown that the work of the experts and of the representatives of the committee was plagued with extreme carelessness, resulting in dozens of errors, and in many essential violations of the instructions of the Committee. The experiments based on those data therefore lacked any scientific meaning."

It is worthwhile to read the full statements of Professor Aumann, Professor Furstenberg, Professor Lapides and Doron Witztum. Their statements can be found below. All are taken unedited from the Aumann Committee post-experiment report.

Lapides Statement
Witztum Statement
Furstenberg Statement
Aumann Statement

Our summary of the Aumann committee experiment :
To err is human. A true blundering failure is when we do not rise up to correct our errors.

Professor Haralick performed his own independent experiment with the committee's data (corrected) and obtained interesting results.