The popper.yml configuration file

The popper command reads the .popper.yml file in the root of a project to figure out how to execute pipelines. While this file can be manually created and modified, the popper command makes changes to this file depending on which commands are executed.

The project folder we will use as example looks like the following:

$> tree -a -L 2 my-paper
my-paper/
├── .git
├── .popper.yml
├── paper
└── pipelines
    ├── analysis
    └── data-generation

That is, it contains three pipelines named data-generation and analysis. The .popper.yml for this project looks like:

pipelines:
  paper:
    envs:
    - host
    path: paper
    stages:
    - build
  data-generation:
    envs:
    - host
    path: pipelines/data-generation
    stages:
    - first
    - second
    - post-run
    - validate
    - teardown
  analysis:
    envs:
    - host
    path: pipelines/analysis
    stages:
    - run
    - post-run
    - validate
    - teardown

metadata:
  author: My Name
  name: The name of my study

popperized:
  - github/popperized
  - github/ivotron/quiho-popper

At the top-level of the YAML file there are entries named pipelines, metadata and popperized.

pipelines

The pipelines YAML entry specifies the details for all the available pipelines. For each pipeline, there is information about:

  • the environment(s) in which the pipeline is be executed.
  • the path to that pipeline.
  • the various stages that are present in it.

The special paper pipeline is generated by executing popper init paper and has by default a single stage named build.sh.

envs

The envs entry in .popper.yml specifies the environment that a pipeline is used when the pipeline is executed as part of the popper run command. The available environments are:

  • host. The experiment is executed directly on the host.
  • alpine-3.4, ubuntu-16.04 and centos-7.2. For each of these, popper check is executed within a docker container whose base image is the given Linux distribution name. The container has docker available inside it so other containers can be executed from within the popper check container.

The popper init command can be used to initialize a pipeline. By default, the host is registered when using popper init. The --env flag of popper init can be used to specify another environment. For example:

popper init mypipe --env=alpine-3.4

The above specifies that the pipeline named mypipe will be executed inside a docker container using the alpine-3.4 popper check image.

To add more environment(s):

popper env myexp --add ubuntu-xenial,centos-7.2

To remove enviroment(s):

popper env myexp --rm centos-7.2

stages

The stages YAML entry specifies the sequence of stages that are executed by the popper run command. By default, the popper init command generates scaffold scripts for setup.sh, run.sh, post-run.sh, validate.sh, teardown.sh. If any of those are not present when the pipeline is executed using popper run, they are just skipped (without throwing an error). At least one stage needs to be executed, otherwise popper run throws an error.

If arbitrary names are desired for a pipeline, the --stages flag of the popper init command can be used. For example:

popper init arbitrary_stages \
  --stages 'preparation,execution,validation' \

The above line generates the configuration for the arbitrary_stages pipeline showed in the example.

metadata

The metadata YAML entry specifies the set of data that gives information about the user’s project. It can be added using the popper metadata --add command For example :

popper metadata --add authors='Dennis Ritchie'

This adds a metadata entry ‘authors’ to the the project metadata.

To view the metadata of a repository type:

popper metadata

To remove the entry ‘authors’ from the metadata:

popper metadata --rm authors

popperized

The popperized YAML entry specifies the list of Github organizations and repositories that contain popperized pipelines. By default, it points to the github/popperized organization. This list is used to look for pipelines as part of the popper search command.