The popper.yml configuration file

The popper command reads the .popper.yml file in the root of a project to figure out how to execute pipelines. While this file can be manually created and modified, the popper command makes changes to this file depending on which commands are executed.

The project folder we will use as example looks like the following:

$> tree -a -L 2 my-paper
my-paper/
├── .git
├── .popper.yml
├── paper
└── pipelines
    ├── arbitrary_stages
    ├── myanalysis
    └── myexp

That is, it contains three pipelines named arbitrary_stages, myanalysis and myexp. The .popper.yml for this project looks like:

envs:
  myexp: ubuntu-16.04
  myanalysis: host
  arbitrary_stages: centos-7.2
  paper: host

stages:
  myexp: [setup.sh, run.sh, post-run.sh, validate.sh, teardown.sh]
  myanalysis: [setup.sh, run.sh, validate.sh]
  arbitrary_stages: [preparation, execution, validation]
  paper: [build.sh]

At the top-level of the YAML file there are entries named envs and stages. For each of these, there are sub-entries for each pipeline (subfolders contained in the pipelines folder). A special paper pipeline corresponds to the one generating an (optional) manuscript.

Environments

The envs entry in .popper.yml specifies the environment that a pipeline is used when the pipeline is executed as part of the popper check command. The available environments are:

  • host. The experiment is executed directly on the host.
  • alpine-3.4, ubuntu-16.04 and centos-7.2. For each of these, popper check is executed within a docker container whose base image is the given Linux distribution name. The container has docker available inside it so other containers can be executed from within the popper check container.

The popper init command can be used to initialize a pipeline. By default, the host is registered when using popper init. The --env flag of popper init can be used to specify another environment. For example:

popper init mypipe --env=alpine-3.4

The above specifies that the pipeline named mypipe will be executed inside a docker container using the alpine-3.4 popper check image.

Stages

The stages YAML entry specifies the sequence of stages that are executed by the popper check command. By default, the popper init command generates scaffold scripts for setup.sh, run.sh, post-run.sh, validate.sh, teardown.sh. If any of those are not present when the pipeline is executed using popper check, they are just skipped (without throwing an error). At least one stage needs to be executed, otherwise popper check throws an error.

If arbitrary names are desired for a pipeline, the --stages flag of the popper init command can be used. For example:

popper init arbitrary_stages \
  --stages 'preparation,execution,validation' \

The above line generates the configuration for the arbitrary_stages pipeline showed in the example.

Lastly, the special paper pipeline is generated by executing popper init paper and has by default a single stage named build.sh.