popper.yml configuration file¶
popper command reads the
.popper.yml file in the root of a
project to figure out how to execute pipelines. While this file can be
manually created and modified, the
popper command makes changes to
this file depending on which commands are executed.
The project folder we will use as example looks like the following:
$> tree -a -L 2 my-paper my-paper/ ├── .git ├── .popper.yml ├── paper └── pipelines ├── analysis └── data-generation
That is, it contains three pipelines named
.popper.yml for this project looks
pipelines: paper: envs: - host path: paper stages: - build data-generation: envs: - host path: pipelines/data-generation stages: - first - second - post-run - validate - teardown analysis: envs: - host path: pipelines/analysis stages: - run - post-run - validate - teardown metadata: author: My Name name: The name of my study popperized: - github/popperized - github/ivotron/quiho-popper
At the top-level of the YAML file there are entries named
pipelines YAML entry specifies the details for all the available
pipelines. For each pipeline, there is information about:
- the environment(s) in which the pipeline is be executed.
- the path to that pipeline.
- the various stages that are present in it.
paper pipeline is generated by executing
popper init paper and has by default a single stage named
envs entry in
.popper.yml specifies the environment that a
pipeline is used when the pipeline is executed as part of the
popper run command. The available environments are:
host. The experiment is executed directly on the host.
centos-7.2. For each of these,
popper checkis executed within a docker container whose base image is the given Linux distribution name. The container has
dockeravailable inside it so other containers can be executed from within the
popper init command can be used to initialize a pipeline. By
host is registered when using
popper init. The
--env flag of
popper init can be used to specify another
environment. For example:
popper init mypipe --env=alpine-3.4
The above specifies that the pipeline named
mypipe will be executed
inside a docker container using the
alpine-3.4 popper check image.
To add more environment(s):
popper env myexp --add ubuntu-xenial,centos-7.2
To remove enviroment(s):
popper env myexp --rm centos-7.2
stages YAML entry specifies the sequence of stages that are
executed by the
popper run command. By default, the
command generates scaffold scripts for
teardown.sh. If any of those are not
present when the pipeline is executed using
popper run, they are
just skipped (without throwing an error). At least one stage needs to
be executed, otherwise
popper run throws an error.
If arbitrary names are desired for a pipeline, the
--stages flag of
popper init command can be used. For example:
popper init arbitrary_stages \ --stages 'preparation,execution,validation' \
The above line generates the configuration for the
pipeline showed in the example.
metadata YAML entry specifies the set of data that gives information
about the user’s project. It can be added using the
popper metadata --add command
For example :
popper metadata --add authors='Dennis Ritchie'
This adds a metadata entry ‘authors’ to the the project metadata.
To view the
metadata of a repository type:
To remove the entry ‘authors’ from the
popper metadata --rm authors
popperized YAML entry specifies the list of Github organizations and repositories that contain popperized pipelines. By default, it points to the
github/popperized organization. This list is used to look for pipelines as part of the
popper search command.