| title | authors | creation-date | last-updated | status | |
|---|---|---|---|---|---|
TEP-0010-Optional-Workspaces |
|
2020-08-03 |
2020-10-15 |
implemented |
- Summary
- Motivation
- Requirements
- Proposal
- Design Details
- Test Plan
- Drawbacks
- Alternatives
- References (optional)
Pipelines' current Workspaces implementation dictates that any Workspace declared
by a Task must be provided by a TaskRun. This TEP proposes adding a new field
to Workspace declarations, optional, a boolean that allows a Task to declare
that a Workspace is not required for its successful execution. Some behaviour
of the Task may be gated by the Workspace being provided. For example, performing
an authenticated git clone of a private repo may require that a .ssh directory
Workspace be provided by a TaskRun but cloning public repos could occur without it.
Several broad areas motivate this proposal:
- Credentials
- Supplying optional data to inform a Task's behaviour
- Receiving optional data generated by a Task
Allow Tasks to declare dependence / support for specific optional credentials mounted as files. Example: a .ssh directory with private key and known_hosts.
Optionality is desirable because:
- Multiple credential formats are often supported. E.g. git supporting both .ssh directory or a .gitconfig file. These could be modeled as two separate optional Workspaces.
- Credentials can sometimes be safely omitted. E.g. git does not need credentials to clone a public repo like Tekton Pipelines.
- Tekton currently has multiple competing mechanisms for mounting files, specifically credential files, into a Task: creds-init, Workspaces, volumes/volumeMounts. Writing a Catalog Task to support multiple mechanisms implies each can be optionally omitted.
Allow Tasks to accept an optional set of data that will change the operation of that Task in some way.
Examples of this include:
- accepting an optional linter configuration to change the warnings/errors that are raised from the defaults.
- accepting an optional kubeconfig file that allows a Task to target an external cluster instead of the same cluster it's running in.
- accepting an optional manifest naming specific files to be uploaded instead of simply uploading an entire directory of content.
Allow Tasks to optionally generate some data and supply that to the user or other Tasks in a Pipeline.
Examples of usage include:
- Receiving test results as a file as well as printed to stdout.
- Receiving a manifest of file paths that were copied to external storage.
- Receiving a diff of files that were modified during Task execution.
All of the above are examples of behaviours that could be opt-in: a user/pipeline may not need the results of a test run beyond a pass/fail exit code. A team may not need to care which specific files were copied out to external storage; only that the transfer succeeded. The diff of modifications made during execution may have no bearing on subsequent Tasks in a Pipeline.
A Task can also conceivably skip some portion of work if the user has elected not to receive the generated data. For example, a Task may not need to compute the diff of its modifications or generate a manifest of changed files if the user has not provided a location for that data to be received.
- Allow Tasks to declare optional support for file-based credentials.
- Enable Tasks to offer support of Tekton's multiple competing credential mechanisms.
- Allow Tasks to optionally receive storage with some data to use during execution.
- Allow Tasks to optionally receive storage on to which the Task will place some output data.
- Enable Tasks to detect if any declared optional storage has been provided or not at runtime.
- Validate that "competing" filesystem fields don't overlap in a TaskRun. For example: an optional PipelineResource and optional Workspace should not both be provided by a user if they would both end up sharing the same directory.
- Declaring / binding specific file content on the optional storage.
- Validation of the file contents of the optional storage.
When optional storage is not provided by a TaskRun the Task must be able to detect this within the container / script.
Tekton Variables should fail gracefully in the event that a workspace is
not provided. Using a variable like $(workspaces.foo.path) should not
cause the controller to abandon or fail a TaskRun when the optional workspace
foo is omitted by it. Likewise the literal string "$(workspaces.foo.path)"
should not end up being passed to the Task.
When a Tekton deployment includes configuration for Default TaskRun Workspaces a missing optional Workspace does not result in a Default TaskRun Workspace being injected in its place. The reason for this is that optional behaviour in a Task should not be switched on automatically by the Default TaskRun Workspace mechanism.
Allow Workspaces to be declared "optional" in Tasks. An optional Workspace is not required for successful execution of the Task but can be provided by a TaskRun to configure some optional behaviour, like using a specific kind of credential.
An optional Workspace provides the following variables:
$(workspaces.<workspace-name>.bound): the literal string"true"or"false"reflecting whether the optional workspace has been bound or not.$(workspaces.<workspace-name>.path): the path to the workspace if it has been bound or an empty string if it has not been.
A required Workspace today already provides a path variable. Required Workspaces
will be updated to also expose the following variable:
$(workspaces.<workspace-name>.bound): the literal string"true". A required Workspace must always be bound by runtime.
As a Catalog Task author I want to publish a Task that can support both Workspaces and PipelineResources in the same location on a Step's filesystem so that my Task's users can choose whether to use either field without requiring separate Tasks.
As an application developer writing my team's CI/CD pipelines I want to write TaskRuns to run integration tests that safely ignore any extraneous result files generated by the test Task I use from the Catalog so that I don't have to figure out how to discard that data which my pipeline does not use.
As a Task Author I want to write a single Task that supports both credentials coming from Tekton's creds-init mechanism as well as those passed directly via a Workspace so that my users have flexibility when deciding how best to inject credentials according to their organization's requirements.
As a Task Author I want to declare that my Task supports optional authentication so that users who are using my Task to access public resources do not need to provide a credential but those who are accessing private resources are able to do so.
As a Task Author I want to write a lint Task that accepts an optional configuration file so that users of my Task can opt to switch some errors into warnings in mature projects with legacy codebases.
As a Task Author I want to accept an optional cache of dependencies so that a user can choose to seed this cache and speed up their builds.
- tkn will need to be updated to allow users to optionally provide Workspaces.
- tkn will need to reflect whether a workspace is optional in the
describeoutput.
Workspaces can be declared as optional in a Task:
# task.yaml
spec:
workspaces:
- name: test-results
description: |
A Workspace to receive rich test result data in XML format. If a Workspace
is not supplied the rich test data will not be generated.
optional: trueTaskRuns can omit binding a Workspace if it is optional in the Task spec. Validation does not fail if an optional Workspace is not bound by a TaskRun.
If an optional Workspace is omitted Tekton does not inject a Default Workspace
Binding, even if one is configured in the config-defaults ConfigMap.
Optional Workspaces that receive a binding from a TaskRun are validated in the same ways that required Workspaces are.
A variable is provided that resolves to "true" or "false", indicating whether an optional Workspace was bound or not:
# task.yaml
spec:
workspaces:
- name: test-results
optional: true
script: |
if [ $(workspaces.test-results.bound) == "true" ] ; then
# write verbose test result output to workspace
fiRequired Workspaces also provide the variable indicating bound state but the
value is always "true".
When an optional Workspace is omitted by a TaskRun there should be no
placeholder mounted in its absence. This does not preclude mechanisms like
creds-init from populating that part of the filesystem though. The
rationale for this is that checks in the Task like if [ -d $HOME/.ssh ]
should accurately reflect whether content was placed there or not, be it
from a bound Workspace, creds-init etc. An omitted Workspace must not
result in an empty directory being placed in its absence, resulting in an
incorrectly passing behaviour gate.
The git-clone Catalog Task already supports the creds-init approach to
credential handling. Here is an example updating the Task to include support
for an explicit .ssh directory to be provided as an optional Workspace:
# A number of fields which have no bearing on the behaviour of this example
# have been removed from the original Catalog entry.
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-clone
spec:
workspaces:
- name: output
description: The git repo will be cloned onto the volume backing this workspace
- name: ssh-directory
description: An .ssh directory with private key and known_hosts file.
optional: true
params:
- name: url
description: git url to clone
type: string
- name: revision
description: git revision to checkout (branch, tag, sha, ref…)
type: string
default: master
- name: subdirectory
description: subdirectory inside the "output" workspace to clone the git repo into
type: string
default: ""
results:
- name: commit
description: The precise commit SHA that was fetched by this Task
steps:
- name: clone
image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.14.2
script: |
CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"
# Here we only use the credentials provided via the ssh-directory Workspace
# if creds-init did not initialize any SSH creds already.
if [ ! -d "$HOME/.ssh" ] ; then
if [ "$(workspaces.ssh-directory.bound)" == "true" ]; then
cp -R "$(workspaces.ssh-directory.path)" "$HOME/.ssh"
fi
fi
/ko-app/git-init \
-url "$(params.url)" \
-revision "$(params.revision)" \
-path "$CHECKOUT_DIR"
cd "$CHECKOUT_DIR"
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "$EXIT_CODE" != 0 ]
then
exit $EXIT_CODE
fi
# Make sure we don't add a trailing newline to the result!
echo -n "$RESULT_SHA" > $(results.commit.path)
Unit tests for the behaviour of a Workspace being bound and being omitted when the declaration is optional.
Unit tests for the variable interpolation code when a Workspace is omitted.
Unit tests for the variable interpolation of a Workspace's bound state.
Unit tests for the behaviour of an omitted Workspace when there is a Default Workspace configured.
E2E tests can take the form of YAML examples that demonstrate the syntax for declaring a Workspace optional, providing and omitting a Workspace, as well as usage of optional Workspace variables.
- The
descriptionfield of a Task's workspaces now has to pull double-duty: it has to describe the content that is required or will be generated in the workspace and it has to describe what happens if the workspace is omitted.
There are two existing workarounds to mimic the behaviour of an optional Workspace:
- Supply
emptyDirWorkspaces for any that are not required by the TaskRun.
Doing this results in the emptyDir being mounted by the Task and then thrown away at the end. It results in unneccessary extra volumes and volumeMounts in the Pod, extra documentation for the Task to indicate which Workspaces are optional, and extra configuration required on the part of the TaskRun author.
The Task is unable to effectively detect whether the Workspace it's been provided
was intentionally emptyDir to direct its behaviour or not.
- Set a default Workspace type
By specifying a default Workspace in Tekton's defaults ConfigMap the user is able to effectively sidestep providing any Workspace in a TaskRun. This allows easier configuration at TaskRun time but does not express the same thing that an optional Workspace does - optional Workspaces allow explicit changes in Task function based on whether a Workspace was provided or not. If a default Workspace is injected the Task is unable to check whether one was provided or not and is therefore unable to determine whether it should change its behaviour.