Lessons learned from a partial replication of an experiment in the context of a software engineering course

Abstract

Replications are an integral component of experimentation through which the validity and reliability of the observed outcome in a previous experiment can be probed. In a strict replication, the experiment is executed in the same conditions as the original by following the same protocol and thus the evidence is strengthened statistically by means of increased sample size. Another objective for running replications is generalizing the experimental results beyond the limitations of one study and its context. For this purpose, certain elements of the original experiment, such as experimenters, experimental objects, and construct operationalization are altered and their impact is investigated. This paper presents lessons learned from a replication that was conducted as a part of an undergraduate university course in Serbia. The focus of the experiment was investigating the effectiveness of writing tests during the development process. The original experiment investigated the effectiveness of test-first programming and was conducted in Italy (Politecnico di Torino) with thirdyear computer science students during an intensive Java course. Lessons learned from this partial replication are that the given task descriptions and structure has an impact on the experiment outcome and that variations in metrics collection can occur when multiple researchers analyse the data, which requires metrics consolidation.