Popper and CI systems

By following a convention for structuring the files of a project, experiment execution and validation can be automated without the need for manual intervention. In addition to this, the status of an experiment (integrity over time) can be tracked by a CI service. In this section we describe the workflow that one follows in order to make an experiment suitable for automation on CI systems.

Experiment Folder Structure

A minimal experiment folder structure for an experiment is shown below:

$> tree -a paper-repo/experiments/myexp
paper-repo/experiments/myexp/
|-- README.md
|-- .popper.yml
|-- run.sh
|-- setup.sh
|-- validate.sh
|-- teardown.sh

Every experiment has setup.sh, run.sh and validate.sh scripts that serve as the interface to the experiment. All these return non-zero exit codes if there’s a failure. In the case of validate.sh, this script should print to standard output one line per validation, denoting whether a validation passed or not. In general, the form for validation results is [true|false] <statement> (see examples below).

[true]  algorithm A outperforms B
[false] network throughput is 2x the IO bandwidth

The PopperCLI tool includes an experiment init subcommand that can be executed to scaffold an experiment with the above structure. The syntax of this command is the following:

popper experiment init <name>

Where <name> is the name of the experiment to initialize.