Batch Spec Templating
Understand how to use templating to make your batch changes more powerful.
Certain fields in a batch spec YAML support templating to create even more powerful and performant batch changes. Templating in a batch spec uses the delimiters ${{
and }}
. Inside the delimiters, template variables and template helper functions may be used to produce a text value.
Example batch spec with templating
Here is an excerpt of a batch spec that uses templating:
YAMLon: - repositoriesMatchingQuery: lang:go fmt.Sprintf("%d", :[v]) patterntype:structural -file:vendor steps: - run: comby -in-place 'fmt.Sprintf("%d", :[v])' 'strconv.Itoa(:[v])' ${{ join repository.search_result_paths " " }} # ^ templating starts here container: comby/comby - run: goimports -w ${{ join previous_step.modified_files " " }} # ^ templating starts here container: unibeautify/goimports
Before executing the first run
command, repository.search_result_paths
will be replaced with the relative-to-root-dir file paths of each search result yielded by repositoriesMatchingQuery
. Using the template helper function join
, an argument list of whitespace-separated values is constructed.
The final run
value that will be executed will look similar to this:
YAMLrun: comby -in-place 'fmt.Sprintf("%d", :[v])' 'strconv.Itoa(:[v])' cmd/src/main.go internal/fmt/fmt.go
The result is that comby
only searches and replaces in those files instead of having to search through the complete repository.
Before the second step is executed, previous_step.modified_files
will be replaced with the list of files that the previous comby
step modified. It will look similar to this:
YAMLrun: goimports -w cmd/src/main.go internal/fmt/fmt.go
Fields with template support
Templating is supported in the following fields:
steps.run
steps.env
valuessteps.files
valuessteps.outputs.<name>.value
steps.if
changesetTemplate.title
changesetTemplate.body
changesetTemplate.branch
changesetTemplate.commit.message
changesetTemplate.commit.author.name
changesetTemplate.commit.author.email
Template variables
Template variables are the names that are defined and accessible when using templating syntax in a given context. Different variables are available depending on the context in which templating is used.
For example, in the context of steps
, the template variable previous_step
is available, but not in the context of changesetTemplate
.
steps
context
The following template variables are available in the fields under steps
. They are evaluated before executing each entry in steps
, except for the step.*
variables, which only contain values after the step has executed.
Template Variable | Type | Description |
---|---|---|
batch_change.name | string | The name of the batch change, as set in the batch spec |
batch_change.description | string | The description of the batch change, as set in the batch spec |
repository.search_result_paths | list of strings | Unique list of file paths relative to the repository root directory in which the search results of the on.repositoriesMatchingQuery s have been found. Empty list if a select:repo filter is used in the on.repositoriesMatchingQuery , or if only on.repository entries are specified |
repository.branch | string | The target branch of the repository in which the step is being executed |
repository.name | string | Full name of the repository in which the step is being executed. For example, org_foo/repo_bar |
previous_step.modified_files | list of strings | List of files that the previous steps have modified. Empty list if no files have been modified |
previous_step.added_files | list of strings | List of files that the previous steps have added. Empty list if no files have been added |
previous_step.deleted_files | list of strings | List of files that the previous steps have deleted. Empty list if no files have been deleted |
previous_step.stdout | string | The complete output of the previous step on standard output |
previous_step.stderr | string | The complete output of the previous step on standard error |
step.modified_files | list of strings | Only in steps.outputs : List of files that the just-executed step has modified. Empty list if no files have been modified |
step.added_files | list of strings | Only in steps.outputs : List of files that the just-executed step has added. Empty list if no files have been added |
step.deleted_files | list of strings | Only in steps.outputs : List of files that the just-executed step has deleted. Empty list if no files have been deleted |
step.stdout | string | Only in steps.outputs : The complete output of the just-executed step on standard output |
step.stderr | string | Only in steps.outputs : The complete output of the just-executed step on standard error |
steps.modified_files | list of strings | List of files modified by the steps . Empty list if no files have been modified |
steps.added_files | list of strings | List of files that have been added by the steps . Empty list if no files have been added |
steps.deleted_files | list of strings | List of files deleted by the steps . Empty list if no files have been deleted |
steps.path | string | Path (relative to the root of the directory, no leading / or . ) in which the steps have been executed. Empty if no workspaces have been used and the steps were executed in the root of the repository |
changesetTemplate
context
The following template variables are available in the fields under changesetTemplate
. They are evaluated after the execution of all entries in steps
.
Template Variable | Type | Description |
---|---|---|
batch_change.name | string | The name of the batch change, as set in the batch spec |
batch_change.description | string | The description of the batch change, as set in the batch spec |
repository.search_result_paths | list of strings | Unique list of file paths relative to the repository root directory in which the search results of the on.repositoriesMatchingQuery s have been found. Empty list if a select:repo filter is used in the on.repositoriesMatchingQuery , or if only on.repository entries are specified |
repository.branch | string | The target branch of the repository in which the step is being executed |
repository.name | string | Full name of the repository where the step is being executed. For example, org_foo/repo_bar |
steps.modified_files | list of strings | List of files that have been modified by the steps . Empty list if no files have been modified |
steps.added_files | list of strings | List of files that have been added by the steps . Empty list if no files have been added |
steps.deleted_files | list of strings | List of files deleted by the steps . Empty list if no files have been deleted |
steps.path | string | Path (relative to the root of the directory, no leading / or . ) in which the steps have been executed. Empty if no workspaces have been used and the steps were executed in the root of the repository |
outputs.<name> | depends on outputs.<name>.format , default: string | Value of an output set by steps . If the outputs.<name>.format is yaml or json and the value a data structure (i.e., array, object, ...), then subfields can be accessed too. See Templating Examples below |
batch_change_link | string | Only available in changesetTemplate.body Link back to the batch change that produced the changeset on Sourcegraph. If omitted, the link will be automatically appended to the end of the body. Requires Sourcegraph CLI 3.40.9 or later |
Template helper functions
${{ join repository.search_result_paths "\n" }}
- Joins the list of strings given as the first argument with the separator as the last argument${{ join_if "---" "a" "b" "" "d" }}
- Uses the first argument as a separator to join the remaining arguments, ignoring blank strings${{ replace "a/b/c/d" "/" "-" }}
- Replaces occurrences of the second argument in the first one with the last one${{ split repository.name "/" }}
- Splits the first argument into a list of strings at each occurrence of the last argument${{ matches repository.name "github.com/my-org/terra*" }}
- Matches the first argument against the glob pattern in the second argument, returning true/false${{ "${{ repository.name }}" }}
- Outputs the inner expression as a literal string, for example, to ignore the inner set of${{ }}
The features of Go's text/template
package are also available, including conditionals and loops, since it is the underlying templating engine.
Templating Examples
Pass the exact list of search result file paths to a command:
YAMLsteps: - run: comby -in-place -config /tmp/go-sprintf.toml -f ${{ join repository.search_result_paths "," }} container: comby/comby files: /tmp/go-sprintf.toml: | [sprintf_to_strconv] match='fmt.Sprintf("%d", :[v])' rewrite='strconv.Itoa(:[v])'
Run a command for each search result file path:
YAMLsteps: - run: | for file in "${{ join repository.search_result_paths " " }}"; do sed -i 's/mydockerhub-user/ci-dockerhub-user/g;' ${file} done container: alpine:3
Format and fix files after a previous step modified them:
YAMLsteps: | - run: | | | find . -type f -name '*.go' -not -path "*/vendor/*" | \ | xargs sed -i 's/fmt.Println/log.Println/g' container: alpine:3 - run: goimports -w ${{ join previous_step.modified_files " " }} container: unibeautify/goimports
Use the steps.files
combined with template variables to construct files inside the container:
YAMLsteps: | - run: | | | cat /tmp/search-results | while read file; | do | ruplacer --subvert whitelist allowlist --go ${file} | | echo "nothing to replace"; | | ruplacer --subvert blacklist denylist --go ${file} | | echo "nothing to replace"; | done container: ruplacer files: /tmp/search-results: ${{ join repository.search_result_paths "\n" }}
Put information in environment variables, based on the output of previous step steps.env
also
YAMLsteps: - run: echo $LINTER_ERRROS >> linter_errors.txt container: alpine:3 env: LINTER_ERRORS: ${{ previous_step.stdout }}
If you need to escape the ${{
and }}
delimiters you can simply render them as string literals:
YAMLsteps: - run: cp /tmp/escaped.txt . container: alpine:3 files: /tmp/escaped.txt: ${{ "${{" }} ${{ "}}" }}
Accessing the outputs
set by steps
in subsequent steps
and the changesetTemplate
:
YAMLsteps: - run: echo "Hello there!" container: alpine:3 outputs: myFriendlyMessage: value: "${{ step.stdout }}" - run: echo "We have access to the output here: ${{ outputs.myFriendlyMessage }}" container: alpine:3 outputs: stepTwoOutput: otherMessage: "here too: ${{ outputs.myFriendlyMessage }}" changesetTemplate: # [...] body: | The first step left us the following message: ${{ outputs.myFriendlyMessage }} The second step left this one: ${{ outputs.otherMessage }}
Using the steps.outputs.<name>.format
field, it's possible to parse the value of output as JSON or YAML and access it as a data structure instead of just text:
YAMLsteps: - run: cat .goreleaser.yml container: alpine:3 outputs: goreleaserConfig: value: "${{ step.stdout }}" # The step's output is parsed as YAML, making it accessible as a YAML # object in the other templating fields. format: yaml goreleaserConfigExists: # We can use the power of Go's text/template engine to dynamically produce complex values value: "exists: ${{ gt (len step.stderr) 0 }}" format: yaml changesetTemplate: # [...] # Since templating fields use Go's `text/template` and `goreleaserConfig` was # parsed as YAML we can iterate over every field: body: | This repository has a `gorelaserConfig`: ${{ outputs.goreleaserConfigExists.exists }}. The `goreleaser.yml` defines the following `before.hooks`: ${{ range $index, $hook := outputs.goreleaserConfig.before.hooks }} - `${{ $hook }}` ${{ end }}
Using the steps.if
field to conditionally execute different steps in different repositories:
YAMLsteps: # `if:` is true, step always executes. - if: true run: echo "name of repository is ${{ repository.name }}" >> message.txt container: alpine:3 # `if:` checks for repository name. Only runs in github.com/sourcegraph/automation-testing - if: ${{ eq repository.name "github.com/sourcegraph/automation-testing" }} run: echo "hello from automation-testing" >> message.txt container: alpine:3 # `if:` uses glob pattern to match repository name. - if: ${{ matches repository.name "*sourcegraph-testing*" }} run: echo "name contains sourcegraph-testing" >> message.txt container: alpine:3 # Checks for go.mod existence and saves to outputs - run: if [[ -f "go.mod" ]]; then echo "true"; else echo "false"; fi container: alpine:3 outputs: goModExists: value: ${{ step.stdout }} # `if:` uses the just-set `outputs.goModExists` value as condition - if: ${{ outputs.goModExists }} run: go fmt ./... container: golang # `if:` checks for path, in case steps are executed in workspace. - if: ${{ eq steps.path "sub/directory/in/repo" }} run: echo "hello workspace" >> workspace.txt container: golang
Combine the template helper functions with the helper functions built into Go's text/template
library:
YAMLchangesetTemplate: # [...] body: | The host of the repository: ${{ index (split repository.name "/") 0 }} The org of the repository: ${{ index (split repository.name "/") 1 }}
Render the batch change link at the beginning of the changeset body:
YAMLchangesetTemplate: # [...] body: | ${{ batch_change_link }} This is the rest of my changeset description.