Various fixes to workflows, especially kfp (#664)

* disable test workflow when none code files change Signed-off-by: David Wood <[email protected]> * one more path-ignore in test.yml Signed-off-by: David Wood <[email protected]> * one more fix for path-ignore in test.yml Signed-off-by: David Wood <[email protected]> * test universal transform separately Signed-off-by: David Wood <[email protected]> * rename test universal workflow Signed-off-by: David Wood <[email protected]> * add comments to noop src to trigger new universal test workflow Signed-off-by: David Wood <[email protected]> * fix paths in test universal workflow Signed-off-by: David Wood <[email protected]> * addj back ignore paths in test universal workflow Signed-off-by: David Wood <[email protected]> * another noop comment Signed-off-by: David Wood <[email protected]> * move ignored paths to paths in univesal test workflow Signed-off-by: David Wood <[email protected]> * test-universal workflow name changes Signed-off-by: David Wood <[email protected]> * noop comments Signed-off-by: David Wood <[email protected]> * noop readme change' Signed-off-by: David Wood <[email protected]> * change test universal not paths Signed-off-by: David Wood <[email protected]> * disable all but new noop and doc_id test workflows Signed-off-by: David Wood <[email protected]> * code change in noop Signed-off-by: David Wood <[email protected]> * remake test transforms Signed-off-by: David Wood <[email protected]> * add individual test transform workflows Signed-off-by: David Wood <[email protected]> * noop README change Signed-off-by: David Wood <[email protected]> * better ignore of .md on test transform workflows Signed-off-by: David Wood <[email protected]> * noop readme change Signed-off-by: David Wood <[email protected]> * noop test transform worklow 1 ignore Signed-off-by: David Wood <[email protected]> * noop readme Signed-off-by: David Wood <[email protected]> * split out the tests into test-kfp/lib/misc and remove test.yml, add readme Signed-off-by: David Wood <[email protected]> * test-kfp only on kfp/** Signed-off-by: David Wood <[email protected]> * noop code change to trigger build Signed-off-by: David Wood <[email protected]> * comments in workflows Signed-off-by: David Wood <[email protected]> * updated workflow readme Signed-off-by: David Wood <[email protected]> * only run build-library workflow on data-processing-lib changes Signed-off-by: David Wood <[email protected]> * try and ignore docs in build-library, test-kfp/lib Signed-off-by: David Wood <[email protected]> * workflow title changes for consistency Signed-off-by: David Wood <[email protected]> * test change on filter source Signed-off-by: David Wood <[email protected]> * change to lib readme Signed-off-by: David Wood <[email protected]> * change to lib source Signed-off-by: David Wood <[email protected]> * minor job name changes in transform workflows Signed-off-by: David Wood <[email protected]> * noop readme Signed-off-by: David Wood <[email protected]> * test-lib workflow ignores Signed-off-by: David Wood <[email protected]> * top level readme Signed-off-by: David Wood <[email protected]> * noop test source Signed-off-by: David Wood <[email protected]> * filter source change' Signed-off-by: David Wood <[email protected]> * updated all transform tets workflows Signed-off-by: David Wood <[email protected]> * fix typo in test template on check_images Signed-off-by: David Wood <[email protected]> * noop src change Signed-off-by: David Wood <[email protected]> * check for makefile in test transform workflow Signed-off-by: David Wood <[email protected]> * automatically determine transforms in transforms directory for which to generate test workflows Signed-off-by: David Wood <[email protected]> * worklow readme, transform existence verification, disable build-library, and tools tests Signed-off-by: David Wood <[email protected]> * workflow readme details on kfp and misc tests Signed-off-by: David Wood <[email protected]> * backing out change to dpk lib code Signed-off-by: David Wood <[email protected]> * restore filter code Signed-off-by: David Wood <[email protected]> * restore noop code Signed-off-by: David Wood <[email protected]> * workflow readme Signed-off-by: David Wood <[email protected]> * really restore noop code Signed-off-by: David Wood <[email protected]> * check for makefile in transform test-src testing Signed-off-by: David Wood <[email protected]> * don't include lib test dependencies in transform test workflows Signed-off-by: David Wood <[email protected]> * noop code change Signed-off-by: David Wood <[email protected]> * disable noop, don't include lib test-data in transform dependencies Signed-off-by: David Wood <[email protected]> * use job.id.if on Makefile to enable transform test job Signed-off-by: David Wood <[email protected]> * use job.id.if on Makefile to enable transform test job Signed-off-by: David Wood <[email protected]> * restore noop Makefile Signed-off-by: David Wood <[email protected]> * exclude kfp_ray from transfor test workflow and change noop code Signed-off-by: David Wood <[email protected]> * remove if: from test workflows Signed-off-by: David Wood <[email protected]> * backout noop code change Signed-off-by: David Wood <[email protected]> * backout noop code change Signed-off-by: David Wood <[email protected]> * only build spark image for transform image tests that need it Signed-off-by: David Wood <[email protected]> * header_cleanser code change Signed-off-by: David Wood <[email protected]> * only build spark image for transform image tests that need it Signed-off-by: David Wood <[email protected]> * update workflows to trigger on .make.* changes Signed-off-by: David Wood <[email protected]> * blank line added to .make.versions to test new workflows Signed-off-by: David Wood <[email protected]> * remove change to .make.versions Signed-off-by: David Wood <[email protected]> * remove test-universal-html2parquet.yml since it is now moved to language Signed-off-by: David Wood <[email protected]> * mv kfp blacklist definition to check-workflows.sh script and have it check for kfp workflows Signed-off-by: David Wood <[email protected]> * license_select_wf.py comment to trigger kfp tests here Signed-off-by: David Wood <[email protected]> * have transform top level makefile check for kfp_ray directory before recursing into Signed-off-by: David Wood <[email protected]> * fix doc_quality Makefile kfp rules Signed-off-by: David Wood <[email protected]> * fix tabbing in recent Makefile updates Signed-off-by: David Wood <[email protected]> * fix kfp workflows to only build the target transform Signed-off-by: David Wood <[email protected]> * switch workflows from ubuntu-22.04 to ubuntu-latest to try and avoid excessive job queuing Signed-off-by: David Wood <[email protected]> * remove non-change from license_select_wf.py to trigger new ci/cd Signed-off-by: David Wood <[email protected]> * one more fix to kfp transform workflows to not workflow-build in all transforms Signed-off-by: David Wood <[email protected]> * fix repo_level_ordering/ray/Makefile to support kind/kfp testing targets Signed-off-by: David Wood <[email protected]> * change kfp test workflows to not test if transform's Makefile or kfp_ray dir is not present Signed-off-by: David Wood <[email protected]> * set cancel-in-progress=true in workflows Signed-off-by: David Wood <[email protected]> --------- Signed-off-by: David Wood <[email protected]>
IBM · Oct 3, 2024 · a5f86da · a5f86da
1 parent 0a9fe37
commit a5f86da
Show file tree

Hide file tree

Showing 78 changed files with 1,924 additions and 1,019 deletions.
diff --git a/.github/workflows/Makefile b/.github/workflows/Makefile
@@ -8,7 +8,6 @@ LANG_TRANSFORMS=doc_chunk doc_quality lang_id pdf2parquet pii_redactor text_enco
 
 
 # A list that holds transforms that should not be tested with KFP
-KFP_BLACK_LIST="doc_chunk,pdf2parquet,pii_redactor"
 
 transform-tests:
 	$(MAKE) TRANSFORM_SUBDIR=universal .transform-tests 
@@ -29,13 +28,19 @@ transform-tests:
 	done
 
 .transform-kfp-tests:
-	@for i in $$(find ../../transforms/$(TRANSFORM_SUBDIR) -mindepth 1 -maxdepth 1 -type d); do	\
+	@KFP_BLACK_LIST=$$(cd ../..; bash scripts/check-workflows.sh -show-kfp-black-list);		\
+	for i in $$(find ../../transforms/$(TRANSFORM_SUBDIR) -mindepth 1 -maxdepth 1 -type d); do	\
 		dir=$$(basename $$i);				\
-		z=$$(echo ${KFP_BLACK_LIST} | grep -v $$dir);     \
-		if [ ! -d ../../transforms/$(TRANSFORM_SUBDIR)/$$dir/kfp_ray ] || [ -z "$$z" ]; then		\
+		yml=test-$(TRANSFORM_SUBDIR)-$$dir-kfp.yml;                                     \
+		if [ ! -d ../../transforms/$(TRANSFORM_SUBDIR)/$$dir/kfp_ray ]; then		\
+			echo No kfp_ray directory for $$dir. Skipping generation of $$yml;                                                      \
+			continue;			\
+		fi;					\
+		z=$$(echo $${KFP_BLACK_LIST} | grep  $$dir);     \
+		if  [ ! -z "$$z" ]; then		\
+			echo $$dir is black listed. Skipping generation of $$yml;                                                      \
 			continue;			\
 		fi;					\
-		yml=test-$(TRANSFORM_SUBDIR)-$$dir-kfp.yml;                                     \
 		echo Generating $$yml;                                                      \
 		cat test-kfp-transform.template | sed -e "s?@TARGET_TRANSFORM_DIR@?transforms/$${TRANSFORM_SUBDIR}/$$dir?g" > $$yml;    \
 	done

diff --git a/.github/workflows/deploy-docs.yml b/.github/workflows/deploy-docs.yml
@@ -8,7 +8,7 @@ on:
             - "releases/**"
 jobs:
     deploy:
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         env:
             REPO_URL: "https://github.com/${{ github.repository }}"
             REPO_BRANCH: "dev"

diff --git a/.github/workflows/deploy-library.yml b/.github/workflows/deploy-library.yml
@@ -14,7 +14,7 @@ permissions:
 jobs:
     build-package:
         name: Build Ray data processing libraries
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -30,7 +30,7 @@ jobs:
         name: Publish packages to test.pypi.org
         # disabled
         if: false
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         needs: build-package
 
         steps:
@@ -47,7 +47,7 @@ jobs:
 
     publish-pypi:
         name: Publish release to pypi.org
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         needs: build-package
         # disabled as of now
         if: false

diff --git a/.github/workflows/deploy-transforms.yml b/.github/workflows/deploy-transforms.yml
@@ -9,7 +9,7 @@ on:
 jobs:
     build-images:
         name: Build and check images
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -23,7 +23,7 @@ jobs:
         name: Publish packages to quay.io
         # disabled
         if: false
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         needs: build-images
 
         steps:

diff --git a/.github/workflows/test-code-code2parquet-kfp.yml b/.github/workflows/test-code-code2parquet-kfp.yml
@@ -12,6 +12,8 @@ on:
         tags:
             - "*"
         paths:
+            - ".make.*"
+            - "transforms/.make.workflow"
             - "transforms/code/code2parquet/**"
             - "!kfp/**" # This is tested in separate workflow
             - "!data-processing-lib/**" # This is tested in separate workflow
@@ -24,6 +26,8 @@ on:
             - "dev"
             - "releases/**"
         paths:
+            - ".make.*"
+            - "transforms/.make.workflow"
             - "transforms/code/code2parquet/**"
             - "!data-processing-lib/**" # This is tested in separate workflow
             - "!kfp/**" # This is tested in separate workflow
@@ -32,10 +36,14 @@ on:
             - "!**/images/**"
             - "!**.gitignore"
 
+# taken from https://stackoverflow.com/questions/66335225/how-to-cancel-previous-runs-in-the-pr-when-you-push-new-commitsupdate-the-curre
+concurrency:
+    group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+    cancel-in-progress: true
 
 jobs:
     test-kfp-v1:
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -51,30 +59,34 @@ jobs:
             - name: Test KFP libs (shared and v1) and run a workflow
               timeout-minutes: 120
               run: |
-                  export REPOROOT=$PWD
-                  export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
-                  source $K8S_SETUP_SCRIPTS/requirements.env
-                  export PATH=$PATH:/tmp/
-                  curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
-                  chmod 777 /tmp/kind
-                  curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
-                  chmod 700 /tmp/get_helm.sh
-                  HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
-                  chmod 777 /tmp/helm
-                  curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
-                  chmod 777 /tmp/kubectl
-                  curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
-                  chmod +x /tmp/mc
-                  export DEPLOY_KUBEFLOW=1
-                  make -C $K8S_SETUP_SCRIPTS setup
-                  make -C kfp/kfp_support_lib test
-                  make -C transforms workflow-build
-                  source $K8S_SETUP_SCRIPTS/common.sh
-                  make -C transforms/code/code2parquet workflow-test
-                  echo "Run transforms/code/code2parquet completed"
+                  if [ -e "@TARGET_TRANSFORM_DIR/Makefile" -a -d "@TARGET_TRANSFORM_DIR/kfp_ray" ]; then
+                      export REPOROOT=$PWD
+                      export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
+                      source $K8S_SETUP_SCRIPTS/requirements.env
+                      export PATH=$PATH:/tmp/
+                      curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
+                      chmod 777 /tmp/kind
+                      curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+                      chmod 700 /tmp/get_helm.sh
+                      HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
+                      chmod 777 /tmp/helm
+                      curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
+                      chmod 777 /tmp/kubectl
+                      curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
+                      chmod +x /tmp/mc
+                      export DEPLOY_KUBEFLOW=1
+                      make -C $K8S_SETUP_SCRIPTS setup
+                      make -C kfp/kfp_support_lib test
+                      make -C transforms/code/code2parquet workflow-build
+                      source $K8S_SETUP_SCRIPTS/common.sh
+                      make -C transforms/code/code2parquet workflow-test
+                      echo "Run transforms/code/code2parquet completed"
+                   else
+                      echo "Skipping transforms/code/code2parquet kfp test for lack of Makefile and/or kfp_ray"
+                   fi
 
     test-kfp-v2:
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -90,25 +102,29 @@ jobs:
             - name: Test KFP libs (shared and v2) and run a workflow
               timeout-minutes: 120
               run: |
-                  export REPOROOT=$PWD
-                  export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
-                  source $K8S_SETUP_SCRIPTS/requirements.env
-                  export PATH=$PATH:/tmp/
-                  curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
-                  chmod 777 /tmp/kind
-                  curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
-                  chmod 700 /tmp/get_helm.sh
-                  HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
-                  chmod 777 /tmp/helm
-                  curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
-                  chmod 777 /tmp/kubectl
-                  curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
-                  chmod +x /tmp/mc
-                  export DEPLOY_KUBEFLOW=1
-                  export KFPv2=1
-                  make -C $K8S_SETUP_SCRIPTS setup
-                  make -C kfp/kfp_support_lib test
-                  make -C transforms workflow-build
-                  source $K8S_SETUP_SCRIPTS/common.sh
-                  make -C transforms/code/code2parquet workflow-test
-                  header_text "Run transforms/code/code2parquet completed"
+                  if [ -e "@TARGET_TRANSFORM_DIR/Makefile" -a -d "@TARGET_TRANSFORM_DIR/kfp_ray" ]; then
+                      export REPOROOT=$PWD
+                      export K8S_SETUP_SCRIPTS=$PWD/scripts/k8s-setup
+                      source $K8S_SETUP_SCRIPTS/requirements.env
+                      export PATH=$PATH:/tmp/
+                      curl -Lo /tmp/kind https://kind.sigs.k8s.io/dl/v${KIND_VERSION}/kind-linux-amd64
+                      chmod 777 /tmp/kind
+                      curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+                      chmod 700 /tmp/get_helm.sh
+                      HELM_INSTALL_DIR=/tmp/ /tmp/get_helm.sh -v v${HELM_VERSION} --no-sudo
+                      chmod 777 /tmp/helm
+                      curl -L https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /tmp/kubectl
+                      chmod 777 /tmp/kubectl
+                      curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /tmp/mc
+                      chmod +x /tmp/mc
+                      export DEPLOY_KUBEFLOW=1
+                      export KFPv2=1
+                      make -C $K8S_SETUP_SCRIPTS setup
+                      make -C kfp/kfp_support_lib test
+                      make -C transforms/code/code2parquet workflow-build
+                      source $K8S_SETUP_SCRIPTS/common.sh
+                      make -C transforms/code/code2parquet workflow-test
+                      echo "Run transforms/code/code2parquet completed"
+                  else
+                      echo "Skipping transforms/code/code2parquet kfp test for lack of Makefile and/or kfp_ray"
+                  fi
diff --git a/.github/workflows/test-code-code2parquet.yml b/.github/workflows/test-code-code2parquet.yml
@@ -12,6 +12,8 @@ on:
         tags:
             - "*"
         paths:
+            - ".make.*"
+            - "transforms/.make.transforms"
             - "transforms/code/code2parquet/**"
             - "data-processing-lib/**"
             - "!transforms/code/code2parquet/**/kfp_ray/**" # This is/will be tested in separate workflow
@@ -26,6 +28,8 @@ on:
             - "dev"
             - "releases/**"
         paths:
+            - ".make.*"
+            - "transforms/.make.transforms"
             - "transforms/code/code2parquet/**"
             - "data-processing-lib/**"
             - "!transforms/code/code2parquet/**/kfp_ray/**" # This is/will be tested in separate workflow
@@ -36,13 +40,18 @@ on:
             - "!**/images/**"
             - "!**.gitignore"
 
+# Taken from https://stackoverflow.com/questions/66335225/how-to-cancel-previous-runs-in-the-pr-when-you-push-new-commitsupdate-the-curre
+concurrency:
+    group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+    cancel-in-progress: true
+
 jobs:
     check_if_push_image:
         # check whether the Docker images should be pushed to the remote repository
         # The images are pushed if it is a merge to dev branch or a new tag is created.
         # The latter being part of the release process.
         # The images tag is derived from the value of the DOCKER_IMAGE_VERSION variable set in the .make.versions file.
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         outputs:
             publish_images: ${{ steps.version.outputs.publish_images }}
         steps:
@@ -59,7 +68,7 @@ jobs:
                   fi
                   echo "publish_images=$publish_images" >> "$GITHUB_OUTPUT"
     test-src:
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         steps:
             - name: Checkout
               uses: actions/checkout@v4
@@ -81,7 +90,7 @@ jobs:
                   fi
     test-image:
         needs: [check_if_push_image]
-        runs-on: ubuntu-22.04
+        runs-on: ubuntu-latest
         timeout-minutes: 120
         env:
             DOCKER_REGISTRY_USER: ${{ secrets.DOCKER_REGISTRY_USER }}