generate DRA job configs from a Jinja template #34010

bart0sh · 2024-12-19T13:34:54Z

Implemented job configs generation
added make rules to generate and verify generated jobs
generated DRA canary jobs

/cc @pohly @kannon92 @SergeyKanzhelev @haircommander

k8s-ci-robot · 2024-12-19T13:34:56Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

config/jobs/kubernetes/sig-node/dynamic-resource-allocation-canary.ini

pohly

Looks very promising.

How to solve indention was my biggest concern when thinking about how to use Jinja. I am not sure whether this is addressed here (need to check test results).

config/jobs/kubernetes/sig-node/Makefile

config/jobs/kubernetes/sig-node/dynamic-resource-allocation.conf

config/jobs/kubernetes/sig-node/dynamic-resource-allocation.jinja

bart0sh · 2024-12-20T21:27:36Z

@pohly @kannon92 @SergeyKanzhelev @haircommander

Looks very promising.

Thank you. After fixing review comments, I'm going to remove -pull and -ci yamls from this PR, so we can only test -canary.
It would be great if SIG-Node folks would look at this and confirm that this approach is at least acceptable.

I personally like it. Using it would allow us to

have presubmit job for every periodic
keep them synchronized
easily generate canary jobs for testing purposes (e.g. kubetest2)
make less mistakes as job configs are automatically generated
do less typing and copypasting :)
etc.

WDYT guys?

config/jobs/kubernetes/sig-node/dra-presubmit.yaml

hack/generate-jobs.py

bart0sh · 2025-01-08T20:56:09Z

@pohly good news - all new -canary jobs succeeded for my test PR.

I triggered -pull jobs to check if both run the same set of test cases.

bart0sh · 2025-01-08T21:35:55Z

@pohly I've also fixed inconsistent naming of the crio jobs: *-kubernetes-node-e2e-cgrpv1-crio-dra -> *-kubernetes-node-e2e-crio-cgrpv1-dra. Now names of the new jobs are the same as of the current jobs.

Co-authored-by: Patrick Ohly <[email protected]>

pohly

/lgtm
/assign @dims

Can you perhaps have a look and approve? You have some context on the usage of Jinja for kops.

bart0sh · 2025-01-09T17:32:31Z

@dims I've presented this solution on SIG-Node CI meeting yesterday and received a generally positive feedback. There are certain concerns that presented solution may work for smaller scope of different jobs and can become harder to maintain when amount of jobs and their variety increases. However, we can't agree or disagree with this without trying, which is what I proposed - to try it out with DRA jobs and then gradually expand to the rest of SIG-Node jobs and people agreed.

/cc @kannon92 @SergeyKanzhelev @haircommander

dims · 2025-01-09T18:01:49Z

@bart0sh thanks for doing this! it will definitely help us going forward.

/approve
/lgtm

k8s-ci-robot · 2025-01-09T18:02:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bart0sh, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pohly · 2025-01-09T19:18:42Z

/hold cancel

k8s-ci-robot · 2025-01-09T19:20:39Z

@bart0sh: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key dra-canary.yaml using file config/jobs/kubernetes/sig-node/dra-canary.yaml
key dynamic-resource-allocation.yaml using file ``
key dra-ci.yaml using file config/jobs/kubernetes/sig-node/dra-ci.yaml
key dra-presubmit.yaml using file config/jobs/kubernetes/sig-node/dra-presubmit.yaml
key sig-node-presubmit.yaml using file config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml

In response to this:

Implemented job configs generation

added make rules to generate and verify generated jobs

generated DRA canary jobs

/cc @pohly @kannon92 @SergeyKanzhelev @haircommander

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pohly · 2025-01-09T20:10:50Z

config/jobs/kubernetes/sig-node/dra-ci.yaml

@@ -145,7 +137,7 @@ periodics:
        - '--node-test-args=--feature-gates=DynamicResourceAllocation=true --service-feature-gates=DynamicResourceAllocation=true --runtime-config=api/beta=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
        - --node-tests=true
        - --provider=gce
-        - '--test_args=--timeout=1h --label-filter="Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf { Beta, DynamicResourceAllocation } && !Flaky"'
+        - '--test_args=--timeout=1h --label-filter="Feature: containsAny DynamicResourceAllocation && Feature: isSubsetOf { Beta, DynamicResourceAllocation } && !Flaky && !Slow"'


This seemed okay, but actually was an intentional difference between periodic and presubmit: in a periodic job it's okay to run longer, but presubmits should be fast.

This is relevant for example for kubernetes/kubernetes#129543 because raising the limit potentially affects a slow test.

=> #34113

rifelpet · 2025-01-10T04:08:53Z

👋🏻 from the Kops side, I've been following along to see how you end up integrating the job generation scripts in to CI. I'd like to do the same with kops' build_jobs.py.

Do you all have a preference on if/how I integrate our scripts?

I was thinking of adding new run-in-python-container.sh executions to hack/make-rules/verify/generated-jobs.sh and hack/make-rules/update/generated-jobs.sh.

I can also hold off until after you're confident you dont need to revert.

bart0sh · 2025-01-10T09:41:12Z

@rifelpet The way you've proposed looks good to me. You can modify the scripts. All DRA generated jobs are "green" now.

k8s-ci-robot requested review from haircommander, kannon92, pohly and SergeyKanzhelev December 19, 2024 13:34

bart0sh force-pushed the PR060-generate-job-configs branch from 7c3a83c to 2f75bbd Compare December 19, 2024 13:37

bart0sh mentioned this pull request Dec 19, 2024

DRA: add canary jobs and sync mechanism #33993

Closed

bart0sh force-pushed the PR060-generate-job-configs branch 3 times, most recently from d030275 to c5999e8 Compare December 19, 2024 14:13

pohly reviewed Dec 19, 2024

View reviewed changes

config/jobs/kubernetes/sig-node/dynamic-resource-allocation-canary.ini Outdated Show resolved Hide resolved

config/jobs/kubernetes/sig-node/dynamic-resource-allocation-canary.ini Outdated Show resolved Hide resolved

bart0sh force-pushed the PR060-generate-job-configs branch 3 times, most recently from 3259e4d to 499379c Compare December 20, 2024 12:38

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 20, 2024

bart0sh force-pushed the PR060-generate-job-configs branch from 499379c to 2e1e253 Compare December 20, 2024 15:08

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 20, 2024

pohly reviewed Dec 20, 2024

View reviewed changes

bart0sh force-pushed the PR060-generate-job-configs branch 2 times, most recently from 4234630 to 28eda1b Compare December 20, 2024 21:33

pohly reviewed Jan 8, 2025

View reviewed changes

config/jobs/kubernetes/sig-node/dra-presubmit.yaml Outdated Show resolved Hide resolved

bart0sh force-pushed the PR060-generate-job-configs branch from b1ba970 to 17e5c8a Compare January 8, 2025 17:13

pohly reviewed Jan 8, 2025

View reviewed changes

hack/generate-jobs.py Outdated Show resolved Hide resolved

bart0sh force-pushed the PR060-generate-job-configs branch from 17e5c8a to 985c885 Compare January 8, 2025 21:26

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 8, 2025

bart0sh force-pushed the PR060-generate-job-configs branch from 985c885 to bdac7d1 Compare January 8, 2025 21:32

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 8, 2025

bart0sh and others added 4 commits January 9, 2025 16:06

hack: implement job config generator

a9ebddc

Co-authored-by: Patrick Ohly <[email protected]>

sig-node: generate DRA jobs

4f0fe81

Co-authored-by: Patrick Ohly <[email protected]>

Makefile: generate DRA jobs

b7ee818

Co-authored-by: Patrick Ohly <[email protected]>

Makefile: verify generated jobs

15ae942

bart0sh force-pushed the PR060-generate-job-configs branch from bdac7d1 to 15ae942 Compare January 9, 2025 14:23

pohly reviewed Jan 9, 2025

View reviewed changes

k8s-ci-robot assigned dims and pohly Jan 9, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 9, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 9, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 9, 2025

k8s-ci-robot merged commit c94cb9d into kubernetes:master Jan 9, 2025
8 checks passed

pohly reviewed Jan 9, 2025

View reviewed changes

rifelpet mentioned this pull request Jan 11, 2025

Verify kOps generated jobs in CI #34121

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generate DRA job configs from a Jinja template #34010

generate DRA job configs from a Jinja template #34010

bart0sh commented Dec 19, 2024 •

edited

Loading

k8s-ci-robot commented Dec 19, 2024

pohly left a comment

bart0sh commented Dec 20, 2024 •

edited

Loading

bart0sh commented Jan 8, 2025

bart0sh commented Jan 8, 2025 •

edited

Loading

pohly left a comment

bart0sh commented Jan 9, 2025 •

edited

Loading

dims commented Jan 9, 2025

k8s-ci-robot commented Jan 9, 2025

pohly commented Jan 9, 2025

k8s-ci-robot commented Jan 9, 2025

pohly Jan 9, 2025

rifelpet commented Jan 10, 2025

bart0sh commented Jan 10, 2025

generate DRA job configs from a Jinja template #34010

generate DRA job configs from a Jinja template #34010

Conversation

bart0sh commented Dec 19, 2024 • edited Loading

k8s-ci-robot commented Dec 19, 2024

pohly left a comment

Choose a reason for hiding this comment

bart0sh commented Dec 20, 2024 • edited Loading

bart0sh commented Jan 8, 2025

bart0sh commented Jan 8, 2025 • edited Loading

pohly left a comment

Choose a reason for hiding this comment

bart0sh commented Jan 9, 2025 • edited Loading

dims commented Jan 9, 2025

k8s-ci-robot commented Jan 9, 2025

pohly commented Jan 9, 2025

k8s-ci-robot commented Jan 9, 2025

pohly Jan 9, 2025

Choose a reason for hiding this comment

rifelpet commented Jan 10, 2025

bart0sh commented Jan 10, 2025

bart0sh commented Dec 19, 2024 •

edited

Loading

bart0sh commented Dec 20, 2024 •

edited

Loading

bart0sh commented Jan 8, 2025 •

edited

Loading

bart0sh commented Jan 9, 2025 •

edited

Loading