Add tasks to build separate releases for each storage engine. #376
Conversation
…ks depend on sources instead of utility package task.
|
@haiqi96 note the task name changes. |
|
LGTM. Let me merge #363 first and re-validate this PR. |
|
Thanks for the note, need to think about how this fits into my flow. |
|
Based on our offline discussion, I've reverted to building a single package and only building clp-json/text releasable tar balls. |
kirkrodrigues
requested review from
junhaoliao
and removed request for
junhaoliao
Taskfile.yml
Outdated
| deps: ["package"] | ||
| cmds: | ||
| - "rm -rf '{{.OUTPUT_DIR}}' '{{.OUTPUT_FILE}}'" | ||
| - "cp -r '{{.G_PACKAGE_BUILD_DIR}}' '{{.OUTPUT_DIR}}'" |
| @@ -328,6 +319,43 @@ tasks: | |||
| URL_PREFIX: "{{.NODEJS_VERSION_BASE_URL}}" | |||
| OUTPUT_DIR: "{{.OUTPUT_DIR}}" | |||
|
|
|||
| package-tar: | |||
There was a problem hiding this comment.
It might be trivial: can we remove the .gitignore in the package before packaging it? It might be confusing to see the file present in the released tar.
(On the other hand, is there a reason we put an empty .gitignore in components/package-template?)
There was a problem hiding this comment.
Ah, it's obsolete so I removed it.
Co-authored-by: Junhao Liao <[email protected]>
| @@ -76,7 +67,7 @@ tasks: | |||
| - "webui" | |||
| - "webui-nodejs" | |||
| cmds: | |||
| - task: "clean-package" | |||
| - "rm -rf '{{.OUTPUT_DIR}}'" | |||
There was a problem hiding this comment.
Just thinking out loud, we have talked about having a package where task overwrites and rsync it to another package where we actually run start_clp to avoid removing the database and compressed archive. Any chance we can add this flow to the task file?
Anyway, it doesn't need to be in this PR
There was a problem hiding this comment.
Yeah, I thought about it a bit, but just haven't done any work on it. I'll add a task for it.
| @@ -35,28 +35,19 @@ tasks: | |||
| vars: | |||
| COMPONENT: "job-orchestration" | |||
|
|
|||
| clean-package: | |||
There was a problem hiding this comment.
Are we changing the behavior such that we can't run task:clean-package is not valid target anymore?
It makes sense for me since anyway running task package will just remove the old package anyway. but what if we want to add more fancy checks, such as make sure no existing containers are running?
There was a problem hiding this comment.
Yeah, we're removing it for now. If we have more fancy things, I'm happy to add it back.
There was a problem hiding this comment.
LGTM. How about:
Add tasks to build separate releases for each storage engine. (#376)
kirkrodrigues
changed the title
Description
CLP currently has two storage engines:
clpfor managing text logs.clp-sfor managing json logs.This PR adds new Taskfile tasks for building releasable tar balls where the default storage engine is one or the other (the only difference between the package binaries is a config value change). Specifically, this PR adds:
clp-json-pkg-tarfor assembling a versioned tar for theclp-jsonpackageclp-text-pkg-tarfor assembling a versioned tar for theclp-textpackageAs a result, the
package-tartasks has been parameterized and made internal.This PR also removes the
clean-packagetask since it was only used in one place and just deleted the package output directory anyway.This PR also adds a utility task for replacing text and ensures that the
validate-checksumtask handles the case where the data to be checksummed doesn't exist.Validation performed
Validated the following sequence of commands were successful:
task cleantaskbuilds the packagetaskdetectedwebui-node-moduleswas changed (due to a known issue fixed in Taskfile: Download and set up Meteor.js as part of building the package; Refactor tarball download and extraction into reusable task. #363) and rebuilt the relevant artifactstaskdetected no changestask cleantask clp-json-pkg-tarbuilt the clp-json packagetask clp-text-pkg-tarbuilt the clp-text packageValidated that
clp-json-packageused theclp-sstorage engine andclp-text-packageused theclpstorage engine.Validated that compression and search were functional for both
clp-json-packageandclp-text-package.