Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Taskfile: Download and set up Meteor.js as part of building the package; Refactor tarball download and extraction into reusable task. #363

Merged
merged 25 commits into from
May 2, 2024
Merged

Taskfile: Download and set up Meteor.js as part of building the package; Refactor tarball download and extraction into reusable task. #363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1a09b0c
Taskfile: Add Meteor setup task in Taskfile.yml.
junhaoliao Apr 24, 2024
6dd6bd2
Remove Meteor setup steps from GitHub workflow.
junhaoliao Apr 24, 2024
062fc5c
Revert task "webui-node-modules" back to internal in Taskfile.yml.
junhaoliao Apr 24, 2024
531c5c7
Refactor Taskfile and lint-tasks.yml to extract common download and e…
junhaoliao Apr 24, 2024
930ebbe
Merge branch 'refs/heads/main' into task-meteor-setup
junhaoliao Apr 24, 2024
d3fe1c5
Added webui-node-modules.md5 to sources of task `webui` in Taskfile.yml.
junhaoliao Apr 24, 2024
bc7e85f
Remove temp files resulted from `meteor build` before checksum.
junhaoliao Apr 24, 2024
85409c9
Merge branch 'refs/heads/main' into task-meteor-setup
junhaoliao Apr 26, 2024
60be034
Apply suggestions from code review
junhaoliao Apr 30, 2024
1a13036
Mark task `webui-nodejs` as internal.
junhaoliao Apr 30, 2024
c13f267
Reduce sources array into one-line syntax for `download-tar-and-extra…
junhaoliao Apr 30, 2024
5c62bd5
Add "meteor" to task `webui`'s deps.
junhaoliao Apr 30, 2024
822250d
Rename `download-tar-and-extract` -> ``download-and-extract-tar`.
junhaoliao Apr 30, 2024
705b74a
Rename `FILE_TAR_NAME` -> `TAR_NAME`.
junhaoliao Apr 30, 2024
6242a8f
Refactor `FILE_BASE_DIR` -> `EXTRACTED_DIR` = `OUTPUT_DIR` + ``EXTRAC…
junhaoliao Apr 30, 2024
023f7ad
Change mv command syntax from multiline to single-line string.
junhaoliao Apr 30, 2024
fa2c517
Move "meteor" above "validate-checksum" to stay alphabetical.
junhaoliao Apr 30, 2024
6e0e263
Extract first temporary path then move to preseve hidden files.
junhaoliao Apr 30, 2024
ee73432
Added "run: once" attribute to the task `meteor` to prevent it for ru…
junhaoliao Apr 30, 2024
aedc2fe
Alphabetize vars{}.
junhaoliao Apr 30, 2024
c65e238
Apply suggestions from code review
junhaoliao May 1, 2024
f20b158
Added preconditions to task meteor to prevent running it on Linux ARM.
junhaoliao May 1, 2024
52883fe
Apply suggestions from code review
junhaoliao May 1, 2024
dd2f5b9
Remove Node.js and Meteor.js from the building prerequisites in the d…
junhaoliao May 1, 2024
8a4b223
Merge branch 'refs/heads/main' into task-meteor-setup
junhaoliao May 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions .github/workflows/clp-lint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,6 @@ jobs:
with:
python-version: "3.11"

- id: "parse_meteor_version"
run: |-
release_file="./components/webui/.meteor/release"
meteor_version=$(grep --only-matching --perl-regexp "[\d.]+" $release_file)
echo "meteor_version=${meteor_version}" | tee --append $GITHUB_OUTPUT

- uses: "meteorengineer/setup-meteor@v1"
with:
meteor-release: "${{steps.parse_meteor_version.outputs.meteor_version}}"

- name: "Install task"
shell: "bash"
run: "npm install -g @go-task/cli"
Expand Down
102 changes: 73 additions & 29 deletions Taskfile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@ vars:
# Paths
G_BUILD_DIR: "{{.ROOT_DIR}}/build"
G_CORE_COMPONENT_BUILD_DIR: "{{.G_BUILD_DIR}}/core"
G_METEOR_BUILD_DIR: "{{.G_BUILD_DIR}}/meteor"
G_PACKAGE_BUILD_DIR: "{{.G_BUILD_DIR}}/clp-package"
G_PACKAGE_VENV_DIR: "{{.G_BUILD_DIR}}/package-venv"
G_WEBUI_BUILD_DIR: "{{.G_BUILD_DIR}}/webui"
G_WEBUI_NODEJS_BUILD_DIR: "{{.G_BUILD_DIR}}/webui-nodejs"
G_WEBUI_NODEJS_BIN_DIR: "{{.G_WEBUI_NODEJS_BUILD_DIR}}/node/bin"
G_WEBUI_NODEJS_BIN_DIR: "{{.G_WEBUI_NODEJS_BUILD_DIR}}/bin"

# Versions
G_PACKAGE_VERSION: "0.0.3-dev"
Expand Down Expand Up @@ -170,6 +171,7 @@ tasks:
webui:
deps:
- "init"
- "meteor"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
Expand All @@ -183,20 +185,24 @@ tasks:
cmds:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haiqi96 I noticed we have this platform attribute for task webui. Does it mean the task only runs on x86 platforms? Now that we also set up the arch-specific meteor and nodejs, do we still need the restriction here? (My insights are limited as I'm not too sure how the webui is built currently for your workflow

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does meteor build --architecture now support ARM platform IDs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the oversight. It doesn't seem to be the case.
The aarch64 related handling was added into the install script together with the 2.14 -> 2.15 upgrade so I mistakenly thought it was available for download. https://static.meteor.com/packages-bootstrap/2.15/meteor-bootstrap-os.linux.aarch64.tar.gz just doesn't open.

ARM support is expected to be available in meteor 3.0 though.

- "rm -rf '{{.OUTPUT_DIR}}'"
- "mkdir -p '{{.OUTPUT_DIR}}'"
- "meteor build --directory '{{.OUTPUT_DIR}}'"
- "PATH='{{.G_METEOR_BUILD_DIR}}':$PATH meteor build --directory '{{.OUTPUT_DIR}}'"
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
- >-
rsync -a
"{{.OUTPUT_DIR}}/bundle/"
launcher.js
settings.json
"{{.OUTPUT_DIR}}/"
- "rm -rf '{{.OUTPUT_DIR}}/bundle/'"
# Remove temp files generated by `meteor build` before checksum
- "find node_modules -type f -name '.meteor-portable-2.json' -exec rm {} +"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.G_BUILD_DIR}}/meteor.md5"
- "{{.G_BUILD_DIR}}/webui-node-modules.md5"
- "{{.TASKFILE}}"
- "*"
- ".meteor/*"
Expand All @@ -207,59 +213,100 @@ tasks:
generates: ["{{.CHECKSUM_FILE}}"]

webui-nodejs:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal to leave it since we'll soon clean up the ordering of tasks, but we could move webui-nodejs after webui-node-modules to maintain an alphabetical ordering of these artifact-building internal tasks.

Copy link
Member Author

@junhaoliao junhaoliao May 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave it to the future PR then. Earlier I also wanted to move task meteor to some upper position but was confused by the current ordering of the non-internal tasks.

internal: true
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "{{.G_WEBUI_NODEJS_BUILD_DIR}}"
cmds:
- task: "nodejs"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
NODEJS_VERSION: "v14.21.3"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"

core-submodules:
internal: true
dir: "components/core"
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "submodules"
deps:
- "init"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- task: "nodejs"
vars:
NODEJS_VERSION: "v14.21.3"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"
- "tools/scripts/deps-download/download-all.sh"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.TASKFILE}}"
- ".gitmodules"
- "tools/scripts/deps-download/**/*"
generates: ["{{.CHECKSUM_FILE}}"]

core-submodules:
download-and-extract-tar:
internal: true
dir: "components/core"
requires:
vars: ["CHECKSUM_FILE", "EXTRACTED_DIR_NAME", "TAR_NAME", "OUTPUT_DIR", "URL_PREFIX"]
label: "{{.TASK}}-{{.TAR_NAME}}"
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "submodules"
OUTPUT_TMP_DIR: "{{.OUTPUT_DIR}}-tmp"
EXTRACTED_DIR: "{{.OUTPUT_TMP_DIR}}/{{.EXTRACTED_DIR_NAME}}"
TAR_PATH: "{{.OUTPUT_TMP_DIR}}/{{.TAR_NAME}}"
deps:
- "init"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- "tools/scripts/deps-download/download-all.sh"
- "rm -rf '{{.OUTPUT_DIR}}' '{{.OUTPUT_TMP_DIR}}'"
- "mkdir -p '{{.OUTPUT_TMP_DIR}}'"
- >-
curl --fail --location --show-error
"{{.URL_PREFIX}}/{{.TAR_NAME}}"
--output "{{.TAR_PATH}}"
- "tar xf '{{.TAR_PATH}}' --directory '{{.OUTPUT_TMP_DIR}}'"
- "mv '{{.EXTRACTED_DIR}}' '{{.OUTPUT_DIR}}'"
- "rm -rf '{{.OUTPUT_TMP_DIR}}'"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.TASKFILE}}"
- ".gitmodules"
- "tools/scripts/deps-download/**/*"
sources: ["{{.TASKFILE}}"]
generates: ["{{.CHECKSUM_FILE}}"]

meteor:
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
run: "once"
preconditions:
- sh: >-
(test "$(uname -m)" != "aarch64") || (test "$(uname -s)" != "Linux")
msg: "Meteor 2.x does not support aarch64 on Linux"
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
METEOR_ARCH: "{{ if eq ARCH \"arm64\" }}arm64{{ else }}x86_64{{ end }}"
METEOR_PLATFORM: "{{ if eq OS \"darwin\" }}osx{{ else }}linux{{ end }}"
METEOR_RELEASE: "2.15"
cmds:
- task: "download-and-extract-tar"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
EXTRACTED_DIR_NAME: ".meteor"
OUTPUT_DIR: "{{.G_METEOR_BUILD_DIR}}"
TAR_NAME: "meteor-bootstrap-os.{{.METEOR_PLATFORM}}.{{.METEOR_ARCH}}.tar.gz"
URL_PREFIX: "https://static.meteor.com/packages-bootstrap/{{.METEOR_RELEASE}}"

nodejs:
internal: true
deps: ["init"]
requires:
vars: ["NODEJS_VERSION", "OUTPUT_DIR"]
vars: ["CHECKSUM_FILE", "NODEJS_VERSION", "OUTPUT_DIR"]
vars:
NODEJS_ARCH: "{{ if eq ARCH \"arm64\" }}arm64{{ else }}x64{{ end }}"
NODEJS_VERSION_BASE_URL: "https://nodejs.org/dist/{{.NODEJS_VERSION}}/"
Expand All @@ -272,19 +319,14 @@ tasks:
--max-count 1
"node-v\\d+\\.\\d+\\.\\d+-linux-{{.NODEJS_ARCH}}"
| head --lines 1
NODEJS_FILE_TAR_NAME: "{{.NODEJS_FILE_BASE_NAME}}.tar.xz"
cmds:
- "rm -rf '{{.OUTPUT_DIR}}/node'"
- "mkdir -p '{{.OUTPUT_DIR}}'"
- >-
curl -fsSL
"{{.NODEJS_VERSION_BASE_URL}}{{.NODEJS_FILE_TAR_NAME}}"
-o "{{.OUTPUT_DIR}}/{{.NODEJS_FILE_TAR_NAME}}"
- "tar xf '{{.OUTPUT_DIR}}/{{.NODEJS_FILE_TAR_NAME}}' --directory '{{.OUTPUT_DIR}}'"
- >-
mv "{{.OUTPUT_DIR}}/{{.NODEJS_FILE_BASE_NAME}}"
"{{.OUTPUT_DIR}}/node"
- "rm -f '{{.OUTPUT_DIR}}/{{.NODEJS_FILE_TAR_NAME}}'"
- task: "download-and-extract-tar"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
TAR_NAME: "{{.NODEJS_FILE_BASE_NAME}}.tar.xz"
EXTRACTED_DIR_NAME: "{{.NODEJS_FILE_BASE_NAME}}"
URL_PREFIX: "{{.NODEJS_VERSION_BASE_URL}}"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"

package-venv:
internal: true
Expand Down Expand Up @@ -355,19 +397,21 @@ tasks:
dir: "{{.WEBUI_SRC_DIR}}"
deps:
- "init"
- "meteor"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- "rm -rf '{{.OUTPUT_DIR}}'"
- "meteor npm install --production"
- "PATH='{{.G_METEOR_BUILD_DIR}}':$PATH meteor npm install --production"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.G_BUILD_DIR}}/meteor.md5"
- "{{.TASKFILE}}"
- ".meteor/packages"
- "package.json"
Expand Down
3 changes: 0 additions & 3 deletions docs/src/dev-guide/building-package.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ prebuilt version instead, check out the [releases](https://github.com/y-scope/cl
extra configuration.
* Python 3.8 or newer
* python3-venv
* [Node.js 14](https://nodejs.org/download/release/v14.21.3/) (Meteor.js only
[supports](https://docs.meteor.com/install#prereqs-node) Node.js versions >= 10 and <= 14)
* [Meteor.js](https://docs.meteor.com/install.html#installation)
* [Task](https://taskfile.dev/)

## Setup
Expand Down
18 changes: 2 additions & 16 deletions lint-tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: "3"

vars:
G_LINTER_NODEJS_BUILD_DIR: "{{.G_BUILD_DIR}}/linter-nodejs"
G_LINTER_NODEJS_BIN_DIR: "{{.G_LINTER_NODEJS_BUILD_DIR}}/node/bin"
G_LINTER_NODEJS_BIN_DIR: "{{.G_LINTER_NODEJS_BUILD_DIR}}/bin"
G_LINT_VENV_DIR: "{{.G_BUILD_DIR}}/lint-venv"

tasks:
Expand Down Expand Up @@ -146,26 +146,12 @@ tasks:
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK | replace \":\" \"#\"}}.md5"
OUTPUT_DIR: "{{.G_LINTER_NODEJS_BUILD_DIR}}"
deps:
- ":init"
- task: ":validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- task: ":nodejs"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
NODEJS_VERSION: "latest"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"
# This command must be last
- task: ":compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.ROOT_DIR}}/Taskfile.yml"
- "{{.TASKFILE}}"
generates: ["{{.CHECKSUM_FILE}}"]

linter-node-modules:
internal: true
Expand Down
Loading