Logo Image
Baseline Replication in Experimental Economics

 

We are two historians of economics working on the history of laboratory experiments in economics. One of our studies is about a specific research practice sometimes called “baseline replication” (explained below). We would like to collect the opinions, impressions and interpretations of experimental economists about that practice. For this purpose, we would be very happy if you could read the following summary of some very general features of our study and then respond to a very short survey. If you want more information about our methodology, our results and our interpretations, they are available in the following working paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3116245

Consent to participate to this research

Your participation to this survey implies your understanding of the following information.

Researchers conducting this survey:
This research is conducted by Dorian Jullien (Associate Researcher at GREDEG, Université Côte d’Azur, France) and Nicolas Vallois (Associate Professor at Université Picardie Jules Verne and member of CRIISEA, France), who can be respectively contacted at: jullien.dorian@gmail.com and nicolas.vallois@u-picardie.fr.

Introductory Statement:
We are inviting experimental economists to express their opinions, impressions and interpretations about one of their daily research practices: “baseline replication”, which is described in the survey.

Purpose of this study:
Our goal is to collect the aforementioned opinions, impressions and interpretations in order to enrich the discussion of the results we obtained in a historical study of replication in experimental economics.

Tasks and time:
You will first have to read a short text (which takes about 3 minutes) and then rate your agreement and disagreement with a number of statements (which takes about 3 minutes) and have the possibility to comment on them (which can take as long as you wish).

Confidentiality:
Participation to this study is anonymous by default. Those who wish to sign their comments with their names are welcome to do so. The data we collect will be used in our research article(s). The data we collect will not be sold and will not be purposefully shared with any entity that has commercial interests in the data.

Non-compensation and voluntary participation:
You will not be compensated for your participation in this study. You are free to refuse to participate to this study or to discontinue your participation at any point during the study.



Remarque sur la protection de la vie privée
Ce questionnaire est anonyme.
L’enregistrement de vos réponses à ce questionnaire ne contient aucune information permettant de vous identifier, à moins que l’une des questions ne vous le demande explicitement. Si vous avez utilisé un code pour accéder à ce questionnaire, soyez assuré qu'aucune information concernant ce code ne peut être enregistrée avec vos réponses. Il est géré sur une base séparée où il sera uniquement indiqué que vous avez (ou non) finalisé ce questionnaire. Il n’existe pas de moyen pour faire correspondre votre code à vos réponses sur ce questionnaire.
Short Text


Historically, replicability and replication have often been presented as "the heart" of the experimental method and as a marker of its scientificity, in economics as well as in other disciplines. Recently, experimental economists have conducted replication studies to see whether or not their discipline is subject to a “replication crisis” (of the sort that is happening in other disciplines). These studies and related writings usually remark that the actual conduct of replication studies has been fairly rare in experimental economics until recently. This claim is about explicit types of replication studies, where at least two conditions – a “baseline” (or “control”) and a “treatment” – of a pre-existing experiment are reproduced (with varying degrees of similarity depending on the specific type of replication intended, e.g., pure replication or robustness check). While this practice is indeed historically rare, we are interested in another form of replication that is more implicit in the daily practices of experimental economists. Most new experiments on a given topic contain a baseline condition that is similar to other baseline conditions in pre-existing experiments, and experimenters sometimes speak of “baseline replication” when their observations in their baseline condition are similar to the observations obtained in the baseline conditions of the pre-existing experiments. It seems that, at least from a historical point of view, the practice of baseline replication is much more at “the heart” of the experimental method than other types of replication.

To investigate this practice from a historical perspective, we conducted a case study on public good game experiments. Drawing on sampling methods previously used in meta-analyses of public good game experiments, we selected 66 papers, the publication dates of which are uniformly distributed on the period 1980-2015. We then collected the main dependent variable in the baseline conditions (i.e., not in the treatment conditions) of these experiments: the average group contribution to the public good in percentage of initial endowment. The distribution of this variable over time is depicted on the following figure:

Linear econometric regressions indicate two significant trends: (a) convergence over time, i.e., average group contributions are more and more similar, and (b) decline over time, i.e., average group contributions are getting smaller.

Among the other data that we collected as controls, only the fact that subjects were economics student is positively correlated (p<0,05) with the convergence trend and the sample size is positively correlated (p<0,1) with the decline trend (the other controls were the number of Google Scholar citations, whether the subjects were students or non-students, and whether the lab was in the U.S.A. or not).


Survey


Please rate your agreement or disagreement with the following propositions (or select "Don't know" if you don't know or if you don't want to answer). We encourage you to comment as much as you want. Your anonymity is guaranteed by default, as we do not ask your name or position or gender etc. at any point. However, if you wish to state your name in some comments so that we may eventually cite you non-anonymously in our article, please do so – we will of course get back to those who may be cited for approval of their citations in context.


On the notion of baseline replication


*

The notion of baseline replication is worth discussing because it is methodologically interesting.

 



Comments:

*

The notion of baseline replication is dangerous because it gives the illusion of good replication practices.


Comments:

*

The notion of baseline replication is empty, it has nothing to do with any meaningful notion of replication.



Comments:


On the results of our study


*

It would have been expected a priori to get a purely random distribution of the average rate of contribution in baseline conditions of public good game experiments. In other words, each baseline condition is supposed to be independent from each other, and there is no reason to expect that observations in these baseline conditions depend on the publication year of the experiment.


Comments:

*

There is nothing to interpret, this is pure randomness. It makes no sense to compare observations across baseline conditions because each is meant to be compared to different treatment conditions, implying a great variation across a multitude of choices regarding the experimental design.


Comments:

*

The results can be interpreted as a sign of scientific progress because various biases such as publication biases (e.g., incentive to publish new and surprising results), experimenter biases (e.g., designing baseline condition to make the expected treatment effect as big as possible) or statistical biases (e.g., lack of power) have been self-corrected over time.


Comments:

*

The results can be interpreted as a sign of scientific progress because early experiments probably overestimated contribution rates to the public good, because of a specific theoretical agenda, against “economic theory”: e.g., authors such as Marwell and Ames (1979, 1981) intended to show that most people (except economists) do actually contribute to public good, contrary to what is predicted by Nash equilibrium.


Comments:

*

The results can be interpreted as a sign of scientific progress because experimental methods have been improved, observations have become more accurate.


Comments:

*

The results can be interpreted as a sign of the standardization of experimental designs over time (e.g., we observed that the number of subjects per group or the value of the marginal per capita return in baseline conditions are increasingly similar over time).



Comments:

*

The standardization and the scientific progress interpretations are mutually exclusive.


Comments:

*

The results can be interpreted as a sign that individuals in real life have become relatively less cooperative or more selfish than they were in the 1980s.



Comments:

If you think of another interpretation, could you please indicate briefly what you have in mind:


*

Have you read the paper?


If you want to express some general comments and criticisms about the paper, we invite you to do so here:

*

Are you an experimental economist with some basic understanding of what a public good game experiment is?


Please click on "Envoyer" when you are done. (If you click on "Sortir et effacer vos réponses", you will cancel your responses and exit the survey).