The popper.yml configuration file

The popper command reads the .popper.yml file in the root of a project to figure out how to execute pipelines. While this file can be manually created and modified, the popper command makes changes to this file depending on which commands are executed.

The project folder we will use as example looks like the following:

$> tree -a -L 2 my-paper
my-paper/
├── .git
├── .popper.yml
├── paper
└── pipelines
    ├── arbitrary_stages
    ├── myanalysis
    └── myexp

The .popper.yml for this project looks like:

myexp:
  env: ubuntu-16.04
  stages: [setup.sh, run.sh, post-run.sh, validate.sh, teardown.sh]
myanalysis:
  env: host
  stages: [setup.sh, run.sh, validate.sh]
arbitrary_stages:
  env: host
  stages: [preparation, execution, validation]
  finally: [cleanup]
paper:
  env: host
  stages: [build.sh]

At the top-level of the YAML file there are entries named similarly to the subfolder names contained in the pipelines folder. A special paper pipeline is located on the root of the folder that corresponds to the one generating a (optional) manuscript.

Environments

The env entry in .popper.yml specifies the environment that a pipeline is executed on when invoking the popper check command. The available environments are:

  • host. The experiment is executed directly on the host.
  • alpine-3.4, ubuntu-16.04 and centos-7.2. For each of these, popper check is executed within a docker container whose base image is the given Linux distribution name. The container has docker available inside it so other containers can be executed from within the popper check container.

By default, the host is registered when when executing popper init. The --env flag of popper init can be used to specify another environment. For example:

popper init mypipe --env=alpine-3.4

The above specifies that the pipeline named mypipe will be executed inside a docker container using the alpine-3.4 popper check image.

Stages

The stages YAML entry specifies the sequence of stages that are executed by the popper check command. By default, the popper init command generates scaffold scripts for setup.sh, run.sh, post-run.sh, validate.sh, teardown.sh. If any of those are not present when the pipeline is executed, they are just skipped (without throwing an error). At least one stage needs to be executed, otherwise popper check throws an error.

If arbitrary names are desired for a pipeline, the --stages flag of the popper init command can be used. For example:

popper init arbitrary_stages \
  --stages 'preparation,execution,validation' \
  --finally 'cleanup'

The above line generates the configuration for the arbitrary_stages pipeline showed in the example. If a sequence of stages should execute always regardless of any failures, the finally YAML entry can specify the name of such stages. In the example, the arbitrary_stages pipeline specifies that a cleanup phase is always executed.

Lastly, the special paper pipeline is generated by executing popper init paper and has by default a single stage named build.sh.