DAOS-12950 test: auto-determine test tags #13315

daltonbohning · 2023-11-08T21:07:24Z

Skip-test: true
Skip-build: true

Add a tool to:
- automatically determine test tags based on files modified.
- run a lint check on ftest tags
Add a GHA job for runnings ftest tag lint
Update Jenkinsfile Get Commit Message stage to inject Test-tag from the new utility

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2023-11-08T21:07:49Z

Bug-tracker data:
Ticket title is 'Automatically determine which test tags to use'
Status is 'In Progress'
Labels: 'test_infra,triaged'
https://daosio.atlassian.net/browse/DAOS-12950

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-09T00:11:38Z

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/7/execution/node/148/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-09T01:09:35Z

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/8/execution/node/149/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-09T20:20:24Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/9/execution/node/749/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-13T23:51:08Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/10/execution/node/749/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-17T05:07:21Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/11/execution/node/764/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

Skip-func-hw-test: true Skip-unit-tests: true Skip-fault-injection-test: true - Add a tool to: - automatically determine test tags based on files modified. - run a lint check on ftest tags - Add a GHA job for runnings ftest tag lint - Update Jenkinsfile Get Commit Message stage to inject Test-tag from the new utility Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daltonbohning · 2023-11-17T22:16:32Z

Jenkinsfile

@@ -326,7 +326,7 @@ pipeline {
        stage('Get Commit Message') {
            steps {
                script {
-                    env.COMMIT_MESSAGE = sh(script: 'git show -s --format=%B',
+                    env.COMMIT_MESSAGE = sh(script: 'ci/get_commit_message.py --target origin/' + target_branch,


The reason for doing it this way is to prevent needing to modify pipelinelib.

ashleypittman · 2023-11-20T13:32:24Z

.github/workflows/linting.yml

+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - name: Check DAOS ftest tags.
+        run: ./src/tests/ftest/tags.py lint


This needs to be protected against PRs where this check has landed but PRs aren't including it yet. I can send you code for this.

I don't follow. Do GHA for PRs use the master workflows?

Added a check

ashleypittman · 2023-11-20T13:35:44Z

ci/get_commit_message.py

+    return os.linesep.join(commit_message_split)
+
+
+if __name__ == '__main__':


Code after this should be in a main() or similar function. That avoids namespace issues with the "locals" in this code block and any surprises with global variables.

ashleypittman · 2023-11-20T13:36:21Z

debian/changelog

I don't know why this file is needed in the RPM but I'd back out this change for now to avoid conflicts.

Generally, it would useful to have this utility available to probe for things like "what tags do I need to run this test"

ashleypittman · 2023-11-20T13:38:52Z

src/tests/ftest/tags.py

Would this file be better in ci or similar? It's purpose is to choose what to run in CI from a PR and inherently by the time you have a RPM it's too late to call it, plus it's not really part of ftest, it's something that's used to drive ftest and frankly this directory is already crowded and a bit confusing.

One advantage of that would be you could call it from the jira_comment.py script and comment to the PR what test-tags it thinks are appropiate.

Maybe it doesn't belong in ftest, but I wouldn't say it belongs in ci. It's written to be useful locally as well, and we would like to expand functionality for things other than CI. I don't think the core functionality should be tightly coupled with our CI infrastructure.

It's purpose is to choose what to run in CI from a PR

That's not exactly true. That's the purpose of ci/get_commit_message.py. The purpose of ftest/tags.py is to provide an interface to the functional test (ftest) tags. All of these tags are avocado tags, and we only use avocado in ftest. In fact, we can't even properly figure out the tags without parsing through test_* methods. So the tags are rightly coupled with the ftest structure.

One advantage of that would be you could call it from the jira_comment.py script and comment to the PR what test-tags it thinks are appropiate

Do you mean jira_query.py? 1) That only comments once AFAIK, so it wouldn't be useful for subsequent commits. 2) It would be noisy to comment for every commit.
But yes, part of the remaining work here is to somehow propagate the tags to the PR/owner

All this said, I'm open to moving this file to somewhere that makes more sense. I'm just not sure where that is

ashleypittman · 2023-11-20T13:40:57Z

ci/get_commit_message.py

I like the idea of an API to query the test tags from something, be that a commit message, a string or a target sha etc. I think that's what this file is but if that's the case then it needs to be better documented and probably a clearer interface. I'm also sceptical if re is required or just split(:) would suffice.

I'm also sceptical if re is required or just split(:) would suffice.

To rely on just split(':') we would need logic to loop over each line and match lines starting with Test-tag: and Features: (and case insensitive). Which basically means writing a simplified version of our own re. I don't think that's the way to go.

There's a function to do it in https://github.com/daos-stack/daos/pull/8483/files but it's really very simple.

For reference, this is with re

import re features = re.findall(r'Features: (.*)', message, flags=re.MULTILINE | re.IGNORECASE) test_tags = re.findall(r'Test-tag: (.*)', message, flags=re.MULTILINE | re.IGNORECASE)

And this is manually:

features = [] test_tags = [] for line in message.lower().splitlines(): if line.startswith('features:'): features.append(line.split(':', maxsplit=1)[1]) if line.startswith('test-tag:'): test_tags.append(line.split(':', maxsplit=1)[1])

Personally, I think re is more readable and more extensible

daltonbohning · 2023-11-20T17:01:45Z

Looks interesting. Will be important to get contributions on the "good set" of tests per directory.

Definitely. This whole thing warrants a discussion in the PDT, and the idea is that developers could and should contribute to that list

daltonbohning · 2023-11-20T17:20:38Z

Some TODO items here

Get developer contributions for the tags.yaml mapping
Move tags.py?
If a commit message already specified Test-tag, do not overwrite them. The idea is that this functionality chooses the default Test-tag, but someone could overwrite for whatever reason

Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

- Merge Features and Test-tag since both are not supported - If commit has Test-tag, just leave a comment with the recommended Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1 · 2023-11-20T17:44:38Z

Test stage NLT on EL 8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-13315/16/display/redirect

daosbuild1

LGTM. No errors found by checkpatch.

daltonbohning · 2023-11-20T17:47:18Z

ci/get_commit_message.py

+    if test_tags:
+        # Keep the commit tags but leave a comment with the recommended
+        commit_message_split.insert(2, f'Test-tag: {" ".join(sorted(commit_tags))}')
+        commit_message_split.insert(3, '# Auto-recommended Test-tag')
+        commit_message_split.insert(4, f'# Test-tag: {" ".join(sorted(rec_tags))}')
+        commit_message_split.insert(5, '')
+    else:
+        # If there were "Feature" tags, merge into the recommended.
+        # Otherwise, just use the recommended.
+        rec_tags.update(commit_tags)
+        commit_message_split.insert(2, '# Auto-recommended Test-tag')
+        commit_message_split.insert(3, f'Test-tag: {" ".join(sorted(rec_tags))}')
+        commit_message_split.insert(4, '')


Still need to shake out the details but the idea is

Features and Test-tag are not both supported. If the commit has both, merge as a convenience. I've seen people try to specify both.

If the commit has Test-tag, just leave a comment with the recommended. This allows people to override for whatever reason. E.g. when trying iterating on a broken test and not wanting to run everything for every commit.

The end result is that the modified commit never has Features. Only Test-tag

When you say "supported" this is all our code I assume, there's some logic that we own in launch.py that extracts this and passes it to the avocado command line?

I'm not saying this is wrong or requesting changes but just trying to understand the requirements and limitations.

No. pipeline-lib extracts commit pragmas and converts the Features and Test-tag to a single line that gets passed to launch.py

Well, pipeline-lib is also within our control but harder to modify than here certainly.

Why is it harder to modify? It even tests the modifications being made with all currently supported branches to try to prevent unexpected regressions.

It's a separate repo, so I'd have to first update pipeline lib and verify it's working

Yes. Always have to verify changes being made work as intended. But this isn't really any different than if it were on the same project.

and backwards compatible with other branches.

pipeline-lib's CI does that for you. It tests the pipeline-lib against daos' master, release/2.4, weekly-testing and weekly-2.4-testing branches. Might want to think about adding the new daily* branches in there but just those 4 branches is pretty good coverage. You can even define a Test-tag: you want to be used during all of that testing.

And I would still need a daos PR to verify expected behavior of passing both pragmas.

Again, that sounds like it's just a normal process of testing/validating one's changes, regardless of which repo they are in.

Anyway, just hoping to clear up any misconceptions on the perceived difficulty of updating pipeline-lib.

Yes. Always have to verify changes being made work as intended. But this isn't really any different than if it were on the same project.

By using just the daos repo, I can run and debug some code locally to verify it's working. Testing pipeline-lib code locally is virtually impossible.

I would need TWO PRS - one in pipeline-lib, and then one in daos that uses the pipeline-lib one. And every time I update the pipeline-lib PR, I would have to retrigger my daos PR.

And I would still need a daos PR to verify expected behavior of passing both pragmas.

Again, that sounds like it's just a normal process of testing/validating one's changes, regardless of which repo they are in.

I disagree. If I have a daos PR, I only need ONE PR, and I only need to verify my changes on that PR for that ONE branch. With pipeline-lib, I would have to verify all supported branches. Sure, pipeline-lib has automated steps for some of this, but it's still more complex that just ONE daos PR. And I would still have to manipulate the Test-tag and Features commit pragmas for those auto-spawned jobs to verify it works. AND I would still need a separate daos PR to verify it works in "PR mode", since pipeline-lib has different behavior depending on whether it's a timed build, PR, upstream, etc. AND it takes much longer to wait for testing from 4 branches than for one branch.

Another benefit to having just a daos PR is that the change is atomic. One PR, one commit - instead of co-dependent PRs in both daos and pipeline-lib.

In all, I find pipeline-lib incredibly cumbersome to work with.

In all, I find pipeline-lib incredibly cumbersome to work with.

Not disagreeing that it's certainly more cumbersome than having everything in a single repo on a single branch. But then when you do branch you have to start duplicating changes across branches. The exact reason libraries were invented.

That said, library is not really even our primary reason for pipeline-lib (as much as we do benefit from it being a common library to many projects and branches and greatly reducing duplication). It's primary reason for existence is Jenkins itself and it's limitations. We've had to move soooo much code out of a daos branch's Jenkinsfile into pipeline-lib just because there is an upper limit to the size of Jenkinsfile.

But then when you do branch you have to start duplicating changes across branches. The exact reason libraries were invented.

But proper libraries are versioned and you can specify which version you want to use. With pipeline-lib, all daos branches are forced to use the same version. Which goes back to the previous problem of having to make sure all active branches still work.
I don't think putting logic in the daos repo is duplicating code - it's just versioned for different branches. We don't want to do the exact same thing for every branch.

there is an upper limit to the size of Jenkinsfile

FWIW I don't think it's the Jenkinsfile that has a limitation on size. It's the pipeline object that is limited in size. But either way, this is an external python script and doesn't affect that.

I.e. when this PR merges to master but branches don't have the changes yet Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-21T06:50:27Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/18/execution/node/1157/log

ashleypittman · 2023-11-28T10:47:43Z

.github/workflows/linting.yml

+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}


Normal behaviour is to checkout a merge commit with the target branch, this code will checkout the branch itself.

The difference comes in annotating against line numbers, with a merge commit they might be wrong although of course with a branch checkout you're not testing what would be landed. Unless you're annotating the PR against specific lines in files you probably don't want this, if you're just failing the build with logging or annotating against files (which I think is done against "line 0" then you don't want to specify a ref here.

Thanks. The ftest lint just fails with logging + exit(1). No specific lines or files are annotated. So I think I want the default behavior

ashleypittman · 2023-11-28T10:49:32Z

ci/get_commit_message.py

There's a function to do it in https://github.com/daos-stack/daos/pull/8483/files but it's really very simple.

ashleypittman · 2023-11-28T10:53:16Z

ci/get_commit_message.py

+        commit_message = re.sub(
+            r'^ *Test-tag:.*$', '', commit_message, flags=re.MULTILINE | re.IGNORECASE)
+
+    # Put "Test-tag" after the title


I've always put them at the end. Put the human-readable bit at the beginning where it can be seen and the machine readable bit at the end out of the way.

Like Dalton says however this doesn't appear to matter but it does highlight that the way this code is communicating with ftest is something we might be able to improve.

ashleypittman · 2023-11-28T10:55:18Z

ci/get_commit_message.py

+    if test_tags:
+        # Keep the commit tags but leave a comment with the recommended
+        commit_message_split.insert(2, f'Test-tag: {" ".join(sorted(commit_tags))}')
+        commit_message_split.insert(3, '# Auto-recommended Test-tag')
+        commit_message_split.insert(4, f'# Test-tag: {" ".join(sorted(rec_tags))}')
+        commit_message_split.insert(5, '')
+    else:
+        # If there were "Feature" tags, merge into the recommended.
+        # Otherwise, just use the recommended.
+        rec_tags.update(commit_tags)
+        commit_message_split.insert(2, '# Auto-recommended Test-tag')
+        commit_message_split.insert(3, f'Test-tag: {" ".join(sorted(rec_tags))}')
+        commit_message_split.insert(4, '')


When you say "supported" this is all our code I assume, there's some logic that we own in launch.py that extracts this and passes it to the avocado command line?

I'm not saying this is wrong or requesting changes but just trying to understand the requirements and limitations.

daltonbohning · 2023-11-28T21:02:11Z

ci/get_commit_message.py

@@ -0,0 +1,127 @@
+#!/usr/bin/env python3


TODO: add copyright

Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

Skip-func-hw-test: true Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-29T02:04:41Z

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/19/execution/node/764/log

brianjmurrell

Now that we have the GHA alternative to running functional tests, which exists primarily (at this point at least) for external contributors, who would benefit the most from this concept, we should include this functionality in those test runs.

Perhaps it's as simple as changing

daos/.github/workflows/rpm-build-and-test.yml

Line 61 in 188afe6

git show -s --format=%B | escape_single_quotes >> $GITHUB_OUTPUT;

in a similar way that

daos/Jenkinsfile

Line 322 in 188afe6

    
           env.COMMIT_MESSAGE = sh(script: 'ci/get_commit_message.py --target origin/' + target_branch,

was changed.

daltonbohning · 2023-11-29T22:09:17Z

Now that we have the GHA alternative to running functional tests, which exists primarily (at this point at least) for external contributors, who would benefit the most from this concept, we should include this functionality in those test runs.

Ah, yes! I will do that once we've settled on the interface between the Jenkinsfile and utility

daltonbohning · 2023-12-06T17:49:21Z

ci/get_commit_message.py

+    git_root = git_root_dir()
+    result = subprocess.run(
+        ['git', 'diff', target, '--name-only', '--relative'],
+        stdout=subprocess.PIPE, cwd=git_root, check=True)
+    return [os.path.join(git_root, path) for path in result.stdout.decode().split('\n') if path]


This isn't correct in the case where the branch isn't up-to-date with the target. It needs logic similar to find_base.sh

Skip-func-hw-test: true Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daltonbohning · 2024-01-19T23:14:39Z

I've started a simpler version of this work in #13642. It adds most of the core utilities and a linter, but doesn't change the commit message yet. I'd like to get that through first, then we can focus on the details of the interface here.

daosbuild1 · 2024-01-20T02:48:26Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13315/20/execution/node/763/log

daltonbohning · 2024-04-17T18:58:21Z

closed in favor of #13964

daltonbohning self-assigned this Nov 8, 2023

daltonbohning force-pushed the dbohning/daos-12950 branch 6 times, most recently from acf6905 to 6c5a703 Compare November 9, 2023 00:05

daosbuild1 reviewed Nov 9, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from 6c5a703 to 5859040 Compare November 9, 2023 01:01

daosbuild1 reviewed Nov 9, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from 5859040 to 57b4e09 Compare November 9, 2023 16:37

daosbuild1 reviewed Nov 9, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from 57b4e09 to d08f925 Compare November 13, 2023 17:31

daosbuild1 reviewed Nov 13, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from d08f925 to bd1d1df Compare November 17, 2023 01:21

daosbuild1 reviewed Nov 17, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from bd1d1df to ebfae55 Compare November 17, 2023 16:58

daosbuild1 reviewed Nov 17, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from ebfae55 to cd07133 Compare November 17, 2023 18:03

daosbuild1 reviewed Nov 17, 2023

View reviewed changes

daltonbohning force-pushed the dbohning/daos-12950 branch from cd07133 to c6f6acb Compare November 17, 2023 18:07

daosbuild1 reviewed Nov 17, 2023

View reviewed changes

daltonbohning commented Nov 17, 2023

View reviewed changes

ashleypittman reviewed Nov 20, 2023

View reviewed changes

daltonbohning added 2 commits November 20, 2023 17:21

address review comments

da74716

Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

typo

e8c4f3b

Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1 reviewed Nov 20, 2023

View reviewed changes

don't overwrite user Test-tag

14a77be

- Merge Features and Test-tag since both are not supported - If commit has Test-tag, just leave a comment with the recommended Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1 reviewed Nov 20, 2023

View reviewed changes

daltonbohning commented Nov 20, 2023

View reviewed changes

protect against script not existing

661dbd4

I.e. when this PR merges to master but branches don't have the changes yet Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1 reviewed Nov 20, 2023

View reviewed changes

ashleypittman reviewed Nov 28, 2023

View reviewed changes

daltonbohning commented Nov 28, 2023

View reviewed changes

ci/get_commit_message.py

@@ -0,0 +1,127 @@

#!/usr/bin/env python3

Copy link

Contributor Author

daltonbohning Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: add copyright

daltonbohning added 2 commits November 28, 2023 22:13

Merge remote-tracking branch 'origin/master' into dbohning/daos-12950

2b3af31

Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

update lint check workflow

188afe6

Skip-func-hw-test: true Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daosbuild1 reviewed Nov 28, 2023

View reviewed changes

brianjmurrell reviewed Nov 29, 2023

View reviewed changes

daltonbohning commented Dec 6, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into dbohning/daos-12950

102c4f1

Skip-func-hw-test: true Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

daltonbohning mentioned this pull request Jan 22, 2024

DAOS-12950 test: add ftest tag linter #13642

Merged

18 tasks

daltonbohning closed this Apr 17, 2024

daltonbohning deleted the dbohning/daos-12950 branch April 17, 2024 18:58

		return os.linesep.join(commit_message_split)


		if __name__ == '__main__':

DAOS-12950 test: auto-determine test tags #13315

DAOS-12950 test: auto-determine test tags #13315

Conversation

daltonbohning commented Nov 8, 2023

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Nov 8, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 9, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 9, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 9, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 13, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 17, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daltonbohning Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daltonbohning commented Nov 20, 2023

daltonbohning commented Nov 20, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 20, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 29, 2023

brianjmurrell left a comment

Choose a reason for hiding this comment

daltonbohning commented Nov 29, 2023

Choose a reason for hiding this comment

daltonbohning commented Jan 19, 2024

daosbuild1 commented Jan 20, 2024

github-actions bot commented Nov 8, 2023 •

edited

Loading

daltonbohning Nov 20, 2023 •

edited

Loading