Unifying Task-Runner Discussion - Recommend cargo-make

[AshtonStephens]

Workspace DevOps Script Choices - Final Choice Make

1. Introduction

1.1 Glossary

Term	Definition
task runner	A task runner is a software tool that automates the repetitive tasks involved in software development. This includes tasks like compiling code, minifying files, linting code, and running tests.
CI pipeline	A CI pipeline is a set of automated processes that help us build, test, and deploy our software more reliably. In our case this is usually a github workflow.

1.2 Purpose / Scope

This document compares three different task runners for rust projects and proposes that we utilize cargo-make consistently for each of our repositories. The core repositories to consider:

• https://github.com/stacks-network/sbtc
• https://github.com/stacks-network/stacks-blockchain
• https://github.com/stacks-network/sbtc-docs
• https://github.com/Trust-Machines/stacks-sbtc
• etc.

1.3 Why Add a Task Runner?

Consolidating our commands and scripts with the same task runner will have a few benefits:

• Improved Collaboration: A developer moving from one repository to another will be familiar with the development commands of each repo.
• Reproducability: A task-runner can serve as single point of truth for local development and continuous integration commands so that there are no surprises on pushed code.
• Scalability: A long string of commands required to set up and run a test within a large repo can be broken down into constituent logical steps organized in the language of the task runner.

By choosing a task runner and using it across all our repositories we will be able to unify the developers and integration pipeline designs across projects.

2. Recommendation - cargo-make

I recommend cargo-make because it's a simple framework designed for rust but can support arbitrary commands if needed. It is available anywhere where rust code can be compiled and is specified with a toml file, making it a seemless addition to the project workspace.

It’s slightly less extensible than xtask but easier to maintain and faster to iterate on. It has domain specific features that reduce complexity over make.

3. Options

3.1 Cargo-make - github

Cargo-make is a task runner configured in a toml file with extra rust specific features.

3.1.1 Example

# Taken from https://github.com/sagiegurari/cargo-make
[tasks.format]
install_crate = "rustfmt"
command = "cargo"
args = ["fmt", "--", "--emit=files"]

[tasks.clean]
command = "cargo"
args = ["clean"]

[tasks.build]
command = "cargo"
args = ["build"]
dependencies = ["clean"]

[tasks.test]
command = "cargo"
args = ["test"]
dependencies = ["clean"]

[tasks.my-flow]
dependencies = [
    "format",
    "build",
    "test"
]

3.1.2 Pros and Cons

3.1.2.1 Pros

• Simple
• Well integrated with Cargo
• Efficient
• More popular than Xtask
• Good for large projects

3.1.2.2 Cons

• Requires an extra install
• Can be difficult to debug
• Less popular than Make

3.2 XTask - github

Xtask is an idiomatic task runner design pattern to add free-form automation to a Rust project like make, npm run, or bespoke bash scripts. This is the most extensible option but has more overhead and can become complicated. It's really a program that we write ourselves, but with a recommended format.

3.2.1 Example

Repo with a pre-built Xtask package: https://github.com/jondot/xtaskops

#[derive(Parser, Debug)]
#[command()]
struct Cli {
	build: bool,
}

fn main() -> anyhow::Result<()> {
	let cli = Cli::parse();
	if cli.build {
        // Other commands will need to be.
	    cmd!("cargo", "build");
    }
}

3.2.2 Pros and Cons

3.2.2.1 Pros

• Flexibility
• Scalability
• Integration with CI is straightforward
• Written in rust / integrated with cargo

3.2.2.2 Cons

• Relatively new idea / less used
• Can be slower that other task-runners because it needs to compile to run
• Greater overhead to getting started

3.3 Make - wikipedia

Make is the traditional task runner that comes with unix and unix-like operating systems. It is less integrated into the rust ecosystem than the previous two options but has history and flexibility.

3.3.1 Example

# Taken from an example in https://makefiletutorial.com
files := file1 file2
some_file: $(files)
	echo "Look at this variable: " $(files)
	touch some_file

file1:
	touch file1
file2:
	touch file2

clean:
	rm -f file1 file2 some_file

3.3.2 Pros and Cons

3.3.2.1 Pros

• Flexible - Calling shell scripts means we can do anything that we can also do in shell
• Can be simple for small projects
• Well-documented

3.3.2.2 Cons

• Not well integrated with Cargo
• Not suitable for large projects
• Brittle and difficult to debug
• Large makefiles are difficult to read and maintain
• Complicated tasks will likely be broken into .sh files which can be platform dependent

4. Conclusion

I recommend that we use cargo-make as our task runner for all of our Rust projects. It is a simple, efficient, and well-integrated option that is suitable for large projects and can scale as our solutions scale, but there are other good options too. cargo-make won't restrict what we can do while still keeping our scripting simple

I'm happy to discuss different options but am hoping to come to a near consensus on approach by mid to late next week.

[AshtonStephens]

Cargo Make in practice

I've included a cargo-make task runner and a CI using the task-runner within stacks-network/sbtc#52.
Overall, I regret recommending cargo-make and would suggest using the xtask idiom instead of
cargo-make.

1. Predefined tasks in cargo-make force unexpected makefile boilerplate

cargo-make includes some commands such as coverage,build, format, test, and clean by default.
Overwriting the commands with your own version isn't enough to get your own behavior because it will
just extend the commands on top of the boilerplate. To disable them you need to add the following:

[config]
skip_core_tasks = true

This domain specific knowledge will probably confuse anyone trying to set up a new project with simple but custom cargo-make tasks, and can easily get lost when starting a new project because the default commands (like coverage or format) will appear to work but will actually use some defaults of the core tasks instead of just the user defined traits.

2. Workspace Makefiles are not default

cargo-make by default will run makefiles within the cargo module roots. If you want a workspace level Makefile.toml to be included at all you need to add this env variable to the top of the workspace Makefile:

[env]
CARGO_MAKE_EXTEND_WORKSPACE_MAKEFILE = true

Moreover, if you'd like the task you specified to run only once per workspace instead of once per module, you'll need to additionally include the following:

[config]
default_to_workspace = true

3. Poor support for watching for changes

Cargo make includes a watching option for tasks natively, but when it comes to workspaces this is misleading because the makefiles for each module are run in series. This means that only a single module will be watched at a time. The official solution with [plugins.impl.parallel-members] is insufficient and broken because killing the parent process will leave the others going.

4. Instructions

Cargo make requires the users to either read the cargo-make or read a the readme of the repository to know the useful commands. Using xtask and writing helpful CLI notes (which should happen organically in rust anyway) that will display on the help command is a powerful way to ensure the documentation of the task runner and cli functionality are properly paired and well documented.

Also - the xtask --help workflow makes it obvious where instructions can be located, which is incredibly powerful when onboarding new developers.

---

I would recommend going back to using xtask instead for future task running.

[wileyj]

The biggest concern i have here is adding a new dependency (xtask), and the associated learning curve to utilizing it.

I can see the benefits of it, but i'm unsure if there are benefits to using it over generic a make - can you explain (outside of xtask being rust native) that it would be better to use than a historic Makefile?

For the build/testing steps in the stacks-blockchain repo, I imagine a standard Makefile will be more than sufficient (i'm less concerned about watching a filesystem for changes here as well).

Looking at the "cons" for using make, i disagree slightly with your points - but it's all in context. for most operations that i can envision using a makefile in the stacks-blockchain repo, i don't see the same cons.

for xtask, again - i see the appeal, but is that the best option for what we need? on the sbtc repo side, it may have a very different answer than the stacks-blockchain side and that may be biasing my opinion.

might be time to fork some repos and try these out in practice.

I'm also a bit more interested in cargo-make after reading about Cargo Make in practice - i understand the caveats you've identified, but i don't think any of them are blockers necessarily (depending on how strongly you feel about watching the filesystem for changes).

[AshtonStephens]

After this message @wileyj and I discussed briefly in Discord. Some additional considerations that we discussed:

• We want a unified task runner - which will include non-rust repositories (namely typescript)
• We do care about the watch feature

Ultimately we are interested in the opinions of the devs.

[AshtonStephens]

When I get a chance I'll give creating a Makefile for these repos a shot, but I personally may not get to that for some time. If someone would like they could take a crack at it before I do.

My final recommendation is xtask but there are some really important drawbacks to doing that. Some additional notes:

Pros for xtask:

1. The --help command is crucial for any non standard degree of complexity. Last week while working on [Design]: Signer Communication / Coordinator Selection LLD sbtc#53 we starting using the clarinet integrate command from the clarinet repo, which is a rust written cli like the xtask option is. When it came time to use it for integration testing the on-boarding time to run the different options was faster any local build/integration task runner I've ever seen. The clarinet integrate command is not a task runner but the principle is the same. This was entirely because of the --help command that is easy to create and maintain within a rust cli, and the clarinet integrate --help command does not have an unusually high standard in its help documentation.
2. Makefile is still a domain specific language, which means a smaller set of people can easily maintain it than a Makefile.

Cons for xtask:

1. We'd to have alignment on the structure of the xtask cli, which could deviate from repo to repo. (Though the --help command makes this issue less important)
2. xtask is only for cargo workspaces. If we wanted to use a similar method for other repositories we'd need to break idiom and use cargo for typescript packages. That said, I would recommend using the native task runner for non-rust projects.

[jferrant]

My 8 satoshis:
I like the idea of this. I do not have strong opinions about HOW this gets done ultimately.
I have seen good makefiles and terrible makefiles. If we implement the makefile correctly, I am sure its fine. However, the larger the project, the higher liklihood that the makefile becomes unreadable so for that reason I am leaning a bit towards xtask just because I don't have any prior biases/haven't used it yet.

[cylewitruk]

I've personally used cargo-make and enjoy it, and also chose it over XTask. It is more mature and has many useful features for cross-platform.

The fact that cargo-make supports workspaces and sub-makefiles helps with larger projects.

[AshtonStephens]

What do you make of this cargo-make Makefile.toml? https://github.com/stacks-network/sbtc/blob/master/Makefile.toml

I think this is pretty rough, and wouldn't want to extend it because it will become confusing very quickly. Can you show me a cargo-make Makefile.toml file for a full workspace project that you think does cargo-make justice?

[AshtonStephens]

Additionally - what makes cargo-make superior when in a large complicated workspace? From my observations it becomes very complicated at the workspace level when trying to do more than the bare minimum. I'm not saying you're wrong, I just haven't seen any evidence to support it being better for larger projects but would love to know what it is.

[cylewitruk]

I don't think it looks too bad, easy to understand imo. But you could have created e.g. a build-install.sh and build-install.bat and done something like:

[task.install]
script = "./build-install.sh"
[task.install.windows]
script = "./build-install.bat"

and skipped the "fancy" syntax.

The workspace support is intended to have separate Makefile.toml's for each "project"/crate, where e.g. one crate maybe has special versioning or publishing steps. If you were to call cargo make build for example on the root, it would then iterate through the subdirectories with Makefile.toml's and execute cargo make build as if you had run it manually from the subdirectory. The CARGO_MAKE_EXTEND_WORKSPACE_MAKEFILE environment variable you specified makes the top-level tasks available to the sub-makefiles (e.g. each project makefile effectively "extends" the workspace makefile).

In the end it's a lot of personal preference - I like it, others will prefer something else, etc. But nothing in cargo-make forces you to use their helper-syntax, you could just as well write each task in shellscript/powershell and just use cargo-make for managing ordering/dependency management.

[kantai]

I think this is a great idea. The stacks-blockchain repo could definitely benefit from this, as so many tasks are really defined in github actions or dockerfiles, but would benefit from being defined in a task runner that could be directly used in other contexts.

I'd be much more in favor of something more along the lines of make or just than cargo-make or xtask.

Both cargo-make and xtask are fairly rust-centric, and trying to get them to play nicely with typescript or whatever else seems pretty painful. They also both use TOML, which seems like a bad language for describing tasks. make and just provide a domain-specific language that is in practice much closer to simply writing shell commands. just seems slightly more ergonomic than make, but make is more ubiquitous and has a much longer history of support.

[wileyj]

I don't feel strongly about any of the tools proposed, but i do see some additional benefits of using make personally.
I know @AshtonStephens has said it may result in a complicated and hard to read makefile - but the question i keep coming back to is would xtask/cargo-make have easier instructions/directives, and is that enough of an argument to justify them over make?

without a clear benefit to using xtask/cargo-make over make, outside of it being rust-native, i think there is more to gain by using make vs the other options.

I also think that if we get started using make as the default - and find that there's something we don't like, it wouldn't be too challenging to backtrack and use a different tool.

[AshtonStephens]

For me, the huge benefit of xtask is the --help command. I'm starting to agree with the make argument because it's built for this (I haven't seen anything about just but now I'll give it a look) but I really really wish there was an effective way to make a makefile self documenting.

[wileyj]

maybe something like this?
https://www.freecodecamp.org/news/self-documenting-makefile/

[AshtonStephens]

The workflow of just cloning a repo and then running xtask --help to see every build command in the repo is very appealing to me. The --help command can literally never be out of sync with the cli options unless you manually make the commands invisible. Even if it's poorly documented the variable names, if even vaguely reasonable, give most of the context for the task runner.

That said, I'm not sure if the added complexity of doing everything in rust is that worth it. I'm thinking more from the dev side than the makefile maintenance side.

I'm leaning towards just at this point.

[wileyj]

just learning about just myself. very interesting.
the only drawback i see is that it's not widely used, but the syntax inspired by make is compelling - i may take a stab at working up a few simple examples of just/make and seeing how they compare in practice

[jcnelson]

Seconding @kantai. I'd prefer a Makefile. Just don't go overboard -- a task-runner's job is only to run tasks in the right order. It's not meant to be a programming language for said tasks :P

[bestmike007]

Have you considered about bazel (or similar build system)?
It can detect changes, load intermediate artifacts from cache to skip build/test tasks when possible.

[netrome]

I love Bazel, but it feels quite overkill for our use cases. My impression is that Bazel excels when working with a large mono repo in multiple languages.

[AshtonStephens]

Final Decision: "Make"

After much discussion, we've settled on using Make for our stacks-network task runner.

1. It's tried and true
2. It doesn't overfit for any specific programming language
3. It doesn't require specialty knowledge that many programmers don't already have