Workflow Syntax and Execution Runtime

This section introduces the YAML syntax used by Popper, describes the workflow execution runtime and shows how to execute workflows in alternative container engines.

NOTE: Popper also supports the now-deprecated HCL syntax that was introduced in the alpha version of Github Action Workflows. We strongly recommend the use of Popper’s own YAML syntax.

Syntax

A Popper workflow file looks like the following:

version: '1'
steps:
- uses: docker://alpine:3.9
  args: ["ls", "-la"]

A workflow specification contains one or more steps in the form of a YAML list named steps. Each item in the list is a dictionary containing at least a uses attribute, which determines the docker image being used for that step.

Workflow steps

The following table describes the attributes that can be used for a step. All attributes are optional with the exception of the uses attribute.

Attribute Description
uses The Docker image that will be executed for that step. For example,
uses: docker://node:10. See "Referencing images in a step" section below for more examples.
runs Specifies the command to run in the docker image. If runs is omitted, the
command specified in the Dockerfile's ENTRYPOINT instruction will execute.
Use the runs attribute when the Dockerfile does not specify an ENTRYPOINT
or you want to override the ENTRYPOINT command. The runs attribute does not
invoke a shell by default. Using runs: "echo $VAR" will not print the value
stored in $VAR, but will instead print \"\$VAR.\". To use environment
variables with the runs instruction, you must include a shell to expand
the variables, for example: runs: ["sh", "-c", "echo $VAR"]. If the value of runs
refers to a local script, the path is relative to the workspace folder (see The workspace section below)
args The arguments to pass to the command. This can be a string or array. If you
provide args in a string, the string is split around whitespace. For example,
args: "--flag --arg value" or args: ["--flag", "--arg", "value"]. If the value of args
refers to a local script, the path is relative to the workspace folder (see The workspace section below).
env The environment variables to set inside the container's runtime environment. If
you need to pass environment variables into a step, make sure it runs a command
shell to perform variable substitution. For example, if your runs attribute is
set to ["sh", "-c"], the value of args will be passed to sh -c and
executed in a command shell. Alternatively, if your Dockerfile uses an
ENTRYPOINT to run the same command ("sh -c"), args will execute in a
command shell as well. See ENTRYPOINT for more details.
secrets Specifies the names of the secret variables to set in the runtime environment
which the container can access as an environment variable. For example,
secrets: ["SECRET1", "SECRET2"].
id Assigns an identifier to the step. By default, steps are asigned a numerid id
corresponding to the order of the step in the list, with 1 identifying
the first step.
needs Identifies steps that must complete successfully before this step will be
invoked. It can be a string or an array of strings.

Referencing images in a step

A step in a workflow can reference a container image defined in a Dockerfile that is part of the same repository where the workflow file resides. In addition, it can also reference a Dockerfile contained in public Git repository. A third option is to directly reference an image published a in a container registry such as DockerHub. Here are some examples of how you can refer to an image on a public Git repository or Docker container registry:

Template Description
./path/to/dir The path to the directory that contains the Dockerfile. This is a relative
path with respect to the workspace directory (see The workspace section below).
Example: ./path/to/myimg/.
{url}/{user}/{repo}@{ref} A specific branch, ref, or SHA in a public Git repository. If url
is ommited, github.com is used by default.
Example: https://bitbucket.com/popperized/ansible@master.
{url}/{user}/{repo}/{path}@{ref} A subdirectory in a public Git repository at a specific branch, ref,
or SHA.
Example: git@gitlab.com:popperized/geni/build-context@v2.0.
docker://{image}:{tag} A Docker image published on Docker Hub.
Example: docker://alpine:3.8.
docker://{host}/{image}:{tag} A Docker image in a public registry other than DockerHub. Note
that the container engine needs to have properly configured to
access the referenced registry in order to download from it.
Example: docker://gcr.io/cloud-builders/gradle.

It’s strongly recommended to include the version of the image you are using by specifying a SHA or Docker tag. If you don’t specify a version and the image owner publishes an update, it may break your workflows or have unexpected behavior.

In general, any Docker image can be used in a Popper workflow, but keep in mind the following:

  • When the runs attribute for a step is used, the ENTRYPOINT of the image is overridden.
  • The WORKDIR is overridden and /workspace is used instead (see The workspace section below).
  • The ARG instruction is not supported, thus building an image from a Dockerfile (public or local) only uses its default value.
  • While it is possible to run containers that specify USER other than root, doing so might cause unexpected behavior.

Referencing private Github repositories

You can reference Dockerfiles located in private Github repositories by defining a GITHUB_API_TOKEN environment variable that the popper run command reads and uses to clone private repositories. The repository referenced in the uses attribute is assumed to be private and, to access it, an API token from Github is needed (see instructions here). The token needs to have permissions to read the private repository in question. To run a workflow that references private repositories:

export GITHUB_API_TOKEN=access_token_here
popper run -f wf.yml

If the access token doesn’t have permissions to access private repositories, the popper run command will fail.

Execution Runtime

This section describes the runtime environment where a workflow executes.

The workspace

When a step is executed, a folder in your machine is bind-mounted (shared) to the /workspace folder inside the associated container. By default, the folder being bind-mounted is $PWD, that is, the working directory from where popper run is being invoked from. If the -w (or --workspace) flag is given, then the value for this flag is used instead.

For example, let’s look at a workflow that writes to a myfile in the workspace:

version: '1'
steps:
- uses: docker://alpine:3.9
  args: [touch, ./myfile]

Assuming the above is stored in a wf.yml file, the following writes to the current working directory:

cd /tmp
popper run -f /path/to/wf.yml

In the above, /tmp/myfile is created. If we provide a value for -w, the workspace path changes and thus the file is written to that location:

cd /tmp
popper run -f /path/to/wf.yml -w /path/to

The above writes the /path/to/myfile. And, for completeness, the above is equivalent to:

cd /path/to
popper run -f wf.yml

Filesystem namespaces and persistence

As mentioned previously, for every step Popper bind-mounts (shares) a folder from the host (the workspace) into the /workspace folder in the container. Anything written to this folder persists. Conversely, anything that is NOT written in this folder will not persist after the workflow finishes, and the associated containers get destroyed.

Environment variables

A step can define, read, and modify environment variables. A step defines environment variables using the env attribute. For example, you could set the variables FIRST, MIDDLE, and LAST using this:

version: '1'
steps:
- uses: "docker://alpine:3.9"
  args: ["sh", "-c", "echo my name is: $FIRST $MIDDLE $LAST"]
  env:
    FIRST: "Jane"
    MIDDLE: "Charlotte"
    LAST: "Doe"

When the above step executes, Popper makes these variables available to the container and thus the above prints to the terminal:

my name is: Jane Charlotte Doe

Note that these variables are only visible to the step defining them and any modifications made by the code executed within the step are not persisted between steps (i.e. other steps do not see these modifications).

Exit codes and statuses

Exit codes are used to communicate about a step’s status. Popper uses the exit code to set the workflow execution status, which can be success, neutral, or failure:

Exit code Status Description
0 success The step completed successfully and other tasks that depends on it can begin.
78 neutral The configuration error exit status (EX_CONFIG) indicates that the step
terminated but did not fail. For example, a filter step can use a neutral status
to stop a workflow if certain conditions aren't met. When a step
returns this exit status, Popper terminates all concurrently running steps and
prevents any future steps from starting. The associated check run shows a
neutral status, and the overall check suite will have a status of success
as long as there were no failed or cancelled steps.
All other failure Any other exit code indicates the step failed. When a step fails, all concurrent
steps are cancelled and future steps are skipped. The check run and
check suite both get a failure status.

Container Engines

By default, steps in Popper workflows run in Docker. In addition, Popper can execute workflows in other runtimes by interacting with their corresponding container engines. An --engine <engine> flag for the popper run is used to invoke alternative engines, where <engine> is one of the supported engines. When no value for this flag is given, Popper executes workflows in Docker. Below we briefly describe each container engine supported (besides Docker), and lastly describe how to customize their configuration.

Supported engines

Singularity

Popper can execute a workflow in systems where Singularity 3.2+ is available. To execute a workflow in Singularity containers:

popper run --engine singularity
Limitations
  • The use of ARG in Dockerfiles is not supported by Singularity.
  • The --reuse flag of the popper run command is not supported.

Vagrant

While technically not a container engine, executing workflows inside a VM allows users to run workflows in machines where a container engine is not available. In this scenario, Popper uses Vagrant to spawn a VM provisioned with Docker. It then executes workflows by communicating with the Docker daemon that runs inside the VM. To execute a workflow in Vagrant:

popper run --engine vagrant
Limitations

Only one workflow can be executed at the time in Vagrant runtime, since popper assumes that there is only one VM running at any given point in time.

Host

There are situations where a container runtime is not available and cannot be installed. In these cases, a step can be executed directly on the host, that is, on the same environment where the popper command is running. This is done by making use of the special sh value for the uses attribute. This value instructs Popper to execute the command or script given in the runs attribute. For example:

version: '1'
steps:
- uses: "sh"
  runs: ["ls", "-la"]

- uses: "sh"
  runs: "./path/to/my/script.sh"
  args: ["some", "args", "to", "the", "script"]

In the first step above, the ls -la command is executed on the workspace folder (see “The workspace” section). The second one shows how to execute a script. Note that the command or script specified in the runs attribute are NOT executed in a shell. If you need a shell, you have to explicitly invoke one, for example:

version: '1'
steps:
- uses: sh
  runs: [bash, -c, 'sleep 10 && true && exit 0']

The obvious downside of running a step on the host is that, depending on the command being executed, the workflow might not be portable.

Custom engine configuration

Other than bind-mounting the /workspace folder, Popper runs containers with any default configuration provided by the underlying engine. However, a --conf flag is provided by the popper run command to specify custom options for the underlying engine in question (see here for more).