Add option to 'flatten' included taskfiles

[tommyalatalo]

It would be great to be able to set an option to avoid having to call the tasks using the task includename:task syntax when including a taskfile.

So this example assumes you have files like:
./Taskfile.yml
./common/Taskfile.yml

The use case for this is when creating a main Taskfile for a repository and importing some common tasks, lets say "list", "clean" and "version", right now I need to then call task common:clean etc to run them.

Instead I would like to be able to do something like this in my main taskfile:

includes:
  common: ./common
    flatten:true
    override: false

Where the 'flatten' property would define that the tasks from 'common' should be available as if they were present in the main taskfile, so "task clean" would actually call "task common:clean". This would probably have to throw a conflict error if the "clean" task exists in both the main taskfile and the 'common' one, meaning that anything from an included taskfile that would be flattened out needs to be unique. Possibly an "override" property could be added instead, to let the included task take precedence over any task with the same name in the main taskfile.

This kind of functionality would allow you to add taskfiles as a submodule to a repository, create a main taskfile for that repo where you include the submodule taskfiles, and still be able to run common tasks with a short syntax without having to point out the sub-taskfile. Or in other words include other taskfiles as transparent "libraries".

[andreynering]

Hi @tommyalatalo, thanks for opening this issue!

I'm still not convinced to add this feature. I think this brings too little benefit to be worth it: mostly allowing shorter task names (i.e. build instead of common:build).

I like how namespaced tasks allow you to immediately know where a task if defined (you just have to follow the import).

Am I missing something on other ways this would be useful?

[tommyalatalo]

Hi @tommyalatalo, thanks for opening this issue!

I'm still not convinced to add this feature. I think this brings too little benefit to be worth it: mostly allowing shorter task names (i.e. build instead of common:build).

I like how namespaced tasks allow you to immediately know where a task if defined (you just have to follow the import).

Am I missing something on other ways this would be useful?

I think there is a valid use case here.
If I want to do multiple includes from a main taskfile, and those includes can vary, lets say like this:

Taskfile.yml
- build
- test

Taskfile.go.yml
- build
- test

Taskfile.another.yml
- build
- test

Then the main taskfile could benefit from the "flatten" and "override" function in that it would allow you to set default build and tests tasks in the main file, but you can include another taskfile which would override them as desired, depending on the content of the repository you're working in. That is to say if I work in a go repo I just import the "go" taskfile and set it to flatten and override, then it would automatically take over the targets in the main taskfile.

Another use for this is to clean up the --list output.
When including multiple taskfiles from the main you could end up with something like this:

Taskfile.yml
- build
  -> go:build
- test
  -> go:test
- clean
  -> go:clean

Taskfile.go.yml
- build
- test
- clean

This would result in duplicate output of commands in --list like:

build - build binary
go:build - build binary
test - run tests
go:test - run tests
clean - remove build artifacts
go:clean - remove build artifacts

So you see in the above case I would be able to override all the non-"go" targets to keep the list output clean, since this list can get very long if including multiple other taskfiles. This duplicate list output can be confusing to someone who doesn't know as much about the importing of other taskfiles, and is simply cluttering. The main point of this would be to keep the standard targets "build", "test" and "clean" in this case always the same, so that you don't have to manually edit them to point to the "go" targets or whatever else you're importing.

I think this relates somewhat to #268 since if you do an override of a task, it would still be beneficial to see whether that task points to another task or if its still the default, so adding an "Alias" column to the --list output would provide that information in a good way.

[ghostsquad]

I kind of disagree that flattening should exist, as there's a couple things that I think might be confusing, how does a user reasonably figure out what tasks refer to what, especially since you can reference super.<task> using :taskname in an included file:

Taskfile.included.yaml

version: '3'

tasks:
  build:
    cmds:
      - task: welcome
      - task: :build # this calls the task named `build` of the `Taskfile` that _imports_ this Taskfile. (i.e. super.build)
      - task: fairwell

  welcome:
    cmds:
      - echo "Hello There!"

  fairwell:
    cmds:
      - echo "Until Next Time!"

Taskfile.yml

version: "3"

includes:
  i:
    taskfile: Taskfile.included.yml

tasks:
  build:
    cmds:
      - echo "building..."
      - echo "build complete!"

I don't think it would be very clear to an end user, if flatting was a thing, to figure out what's actually going on. The more I get into Golang development, the more I'm realizing that readability counts quite a bit. This is a case that would likely significantly hurt readability.

[Gowiem]

@pd93 thanks for sharing the dup and closing out my ticket #1271. I was actually just reading through this issue right after filing mine. Should've searched better before writing the whole thing up.

Anyway, I wanted to share that I think there is value in this and why. Particularly in the case that a lot of orgs are using Task (#770), which is where there is a centralized repo of tasks that get pulled into other repos as shared tasks and there aren't really any tasks in the consuming repos. In that use-case, we're creating a namespace hierarchy that is unnecessary and creates tedium. I get the argument against that everything should be explicit, but there is value in terseness as well.

[marverix]

Hi, I just was looking for Gradle alternative in my organization, which currently is using Gradle as a task manager. Currently, the lack of the "flattening" possibility in Task is a blocker for me to use it as a Gradle alternative. The use case is very similar to my previous speakers. As a CI/CD tech lead, I want to provide one file, which would be a Convention that we use in the company. For example, I can declare a Taskfile:

PipelineConvention.yaml

version: '3'

tasks:
  stage-init:
    desc: Stage run to initialize the pipeline
    cmd: exit 0
    silent: true
    
  stage-build:
    desc: Stage run to build your project
    cmd: docker build -f Dockerfile .
    silent: true
    
  stage-test:
    desc: Stage run to test your project
    cmd: dgoss run something something
    silent: true

And with that, I could do a git submodule in any repo of the organization and create Taskfile:

version: '3'

includes:
  PipelineConvention:
    taskfile: ./common-taskfiles/PipelineConvention.yaml
    flatten: true

tasks:
  stage-init:
    cmd: echo "I'm overwriting the task stage-init in this repo!"

This would give me the possibility to create a Jenkins Pipeline that will always just run for example task stage-init. In my opinion, this is a must-have feature to allow bigger organizations standardize tasks across all repos.

As I said, in Gradle it's easily achievable (and I'm thinking about Gradle mainly as a task manager, not the builder - because I don't care about Java, and it wasn't mine decision to use Gradle), because you can overwrite any task - even subtask.

So at least if you don't want to implement flatten, then maybe at least allow to overwrite included (namespaced) task from the taskfile that includes. Eg.

version: '3'

includes:
  PipelineConvention:
    taskfile: ./common-taskfiles/PipelineConvention.yaml
    flatten: true

tasks:
  "PipelineConvention:stage-init":
    cmd: echo "I'm overwriting the task stage-init in this repo!"

Br,
Marek

[vmaerten]

We have the same use case as @marverix in my company

[Chi-teck]

My use case:
There is a number of tasks included to the Taskfile that is stored in Git repository.

* build/alpha:  Builds alpha
* build/beta:  Builds beta

I need to be able to include additional tasks from local git ignored Task file.
So the resulting list of tasks would look like follows

* build/alpha:  Builds alpha
* build/beta:  Builds beta
* build/gamma:  Builds gamma # Comes from local Taskfile

For consistency it's highly desired that all tasks share same namespace. No matter where they come from.

[Chi-teck]

I agree that that 'flatten' behavior should be optional and disabled by default.

[elocke]

This feature would be massive benefit to our team!

[cenk1cenk2]

With the inclusion of the remote task files, I would argue that this would be a great addition instead of specifying an URL for each task command and just including some templates at a root level for similar repositories, even though it increases the abstraction a bit, and the user has to pay attention to their task names.

[tomharrisonjr]

My use-case for this is simple: I have common commands across projects that insulate devs from dealing with docker commands, so task up runs docker compose up -d then docker compose logs <container>, or clean that runs docker system prune -af and so on.

They're all the same across projects, so a natural case for includes, so I avoid repetition and centralize logic.

Maybe this is a different use-case than namespaces or aliases as they exist today.

[marverix]

Dear authors and/or maintainers of the repo, can we know what is the cause of the reluctance towards this feature? I have created a PR as a working PoC, but no one even looked at it. Now there is a ton of conflicts.
#1281

[andreynering]

Hi @marverix,

I'm sorry that your PR got unreviewed. The fact is that more issues and PRs are opened than we're able to keep up. We're volunteers and have limited time to dedicate to the project.

I'm not against the idea of flattened includes. The codebase got quite a bit of refactor in the past months, which explains the conflicts in your PR. It may be necessary to start from the ground up.

[pd93]

Just wanted to add that besides the time constraints. One of the reasons I've personally been holding off on this is because of the large refactors being made. It would have added complexity to the changes that were being made if we'd started introducing large changes like this at the same time.

It's worth noting that there are still some big changes coming to AST merging in the future (which is where this feature would currently go). As always, we can't make any promises about when we will get time to look at this, but it has not gone unnoticed that this is one of the most upvoted feature requests.

[marverix]

Thanks for the reply and you hard work!

[trulede]

In some cases, this kind of solution may be helpful. Its based on one of the techniques we use ... you have to figure out the sh bit for yourself, but you get the idea - it is actually possible.

version: '3'

includes:
  tests:
    taskfile: ./blue/Taskfile.yml
    optional: true

tasks
  red:
    vars:
      USEBLUE:
        sh: 'if test -f ./blue/Taskfile.yml; then echo "BLUE" fi'
    cmds:
      - task: "{{if .USEBLUE}}blue:red{{else}}red{{end}}"

The other interesting thing might be the alias option applied to an imported task. How exactly that is handled. Could be a neater solution, upon import (i.e. the includes statement) to alias the imported commands into the importer namespace.

But its very difficult when you have more mature/complex taskfiles. Which is why I suggest to try the more explicit approach above.

Also, one can consider a bootstrapping approach, where a first Taskfile "compiles" a second Taskfile and then dispatches task commands to that second Taskfile (via the task command). What "compiles" means is up to you, in our case, we clone a git repo that has a common Taskfile collection for our tools, and then programmatically build another Taskfile with our higher level task/commands.

So the developer only ever has to type task X and X runs .... task takes care of the work in setting that up.

And before you say, too hard ... no its not! Very easy to write some go code to emit a Taskfile using either the AST or even simple templating.

[fetinin]

I’ve made the following work-around to send all commands to namespace of included taskfile, until this issue is fixed. Hope it will help someone.

version: "3"

includes:
  default: path/to/other/Taskfile.yml

# all commands are caught and sent to "default" namespace.
  '*':
    vars:
      COMMAND: '{{index .MATCH 0}}'
    cmds:
      - task: default:{{.COMMAND}}
        vars:
          FROM_STAR_CMD: true
    preconditions:
        # If command not found, it will be caught by '*' again and it will lead to infinite recursion.
        # This check will catch this.
      - sh: test -z "{{.FROM_STAR_CMD}}"
        msg: "Command {{.COMMAND}} not found. Use `task -a` to list all available commands"

[vmaerten]

It has been merged, it'll be available in the next release 🚀