Are Survey Experiments as Valid as Field Experiments in Management Research? An Empirical Comparison Using the Case of Ethnic Employment Discrimination

• Date: 01 March 2020
• Published date: 01 March 2020
• DOI: http://doi.org/10.1111/emre.12342

Are Survey Experiments as Valid as Field

Experiments in Management Research? An

Empirical Comparison Using the Case of

Ethnic Employment Discrimination

JESPER N. WULFF

1

and ANDERS R. VILLADSEN

2

1

Department of Economics and Business Economics, Aarhus University Fuglesangs Allé 4,DK-8210, Aarhus V

2

Department of Management, Aarhus University, Fuglesangs Allé 4, DK-8210, Aarhus V

Field experiments have long been the gold standard in studies of organizational topics such as ethnic

discrimination in recruitment. The recent use of survey experiments, also known as experimental vignettes, suggests

that some researchers believe that survey experiments could be used as an alternative to field experiments. In this

study we put this notion to the test. We perform a field experiment followed by two survey experiments on ethnic

discrimination in recruitment. While the results of our field experiment are consistent with previous evidence on

discrimination, one survey experiment concludes no difference between native and immigrant employees while

another concludes positive discrimination. These results should invoke cautionin researchers wanting to investigate

organizational topics using survey experiments.

Keywords: Research methods; field experiment; survey experiment; discrimination

Introduction

Experiments are the gold standard for investigating

causal relationships. In organizational research, they

remain the most powerful technique we have for

determining the existence, direction and magnitude of

a causal relation between two variables. The random

assignment of research subjects to either a treatment or

control group eliminates possible alternative effects of

unobserved factors, whose existence is almost always a

concern when dealing with observational data (Hamilton

and Nickerson, 2003).

Experiments come in many shapes and sizes. Most

experiments in management have been done in the

laboratory (Scandura and Williams, 2000). While lab

experiments provide the desirable control over inessential

influences,they may be criticised for being too artificial to

actually generalise to conditions in organizations outside

the laboratory (Fromkin and Streufert, 1976). Field

experiments offer a potential solution to the problem of

artificiality in lab experiments by trading control for

naturalism (Whitley, 1996). The organizational researcher

manipulates the variable of interest and observes its

potential effect in natural settings. This increases the

realism of the results.Field experiments have been widely

used in studies of employment discrimination where

researchers have applied for real job openings with

artificial applications that vary on a dimension of interest

such as name, gender, or picture (for one of the first

applications see Bertrand and Mullainathan, 2004).

One particular boundary might prevent a wider

adoption of the field experimental approach: the question

of feasibility. Some research questions simply require

impractical field experimental research designs. For

instance, in the study of discrimination in the hiring

process, the researcher may want to experimentally

manipulate not only the characteristics of the applicant,

but also the characteristics of the job (Greenberg and

Tomlinson, 2004). Other field experiments may be

excessively expensive or unethical to employ. It would,

for instance, be difficult to imagine researchers randomly

assigning some firms to unethical behaviour (say use of

child labour) to observe the consequences for recruitment

when compared to a control group.

Correspondence: JesperN. Wulff, Department of Economicsand Business

Economics, Aarhus University, Fuglesangs Allé 4, DK-8210 Aarhus V,

+4587165287.E-mail jwulff@econ.au.dk

DOI: 10.1111/emre.12342

©2019 European Academy of Management

European Management Review, Vol. 17, –, (2020)

347 3 6

5