[WIP] update licenses #675

MartinsNadia · 2024-08-19T14:29:52Z

Open the new one to fix the origin branch

eessi-bot · 2024-08-19T14:29:57Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-bot · 2024-08-19T14:29:57Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-software

MartinsNadia · 2024-08-19T14:34:12Z

WIP

adding a new licenses update script PR 457;
adding draft yml file

ocaisa

Just had an initial look, perhaps you can add the current output of the script in a gist and link to this issue?

ocaisa · 2024-10-15T09:07:22Z

licenses/check_licenses.yml

+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4


Should use specific version here, see other workflows

ocaisa · 2024-10-15T09:07:49Z

licenses/check_licenses.yml

+          fi 
+
+      - name: Create a PR (if changes detected)
+        uses: peter-evans/create-pull-request@v5


Also needs specific commit

ocaisa · 2024-10-15T09:08:21Z

licenses/check_licenses.yml

+        commit-message: "Auto PR: Update licenses"
+        title: "Auto PR: Update licenses"
+         body: ${{ steps.check_licenses.outputs.patch }}
+         branch: main #fork branch


This is not the correct branch for EESSI

ocaisa · 2025-02-03T11:22:25Z

.github/workflows/check_licenses.yml

+      - name: Create a PR (if changes detected)
+        id: create_pull_request
+        uses: peter-evans/create-pull-request@5e914681df9dc83aa4e4905692ca88beb2f9e91f # v7.0.5
+        if: steps.check_licenses.outputs.patch != ''
+        with: 
+          commit-message: "Auto PR: Update licenses"
+          title: "Auto PR: Update licenses"
+          body: ${{ steps.check_licenses.outputs.patch }}
+          branch: update-licenses-${{ github.run_number }}
+          base: [ "*-software.eessi.io" ]


Doesn't this mean that if a PR adds software, then it will trigger another PR to add the licence of the software to the yaml file? Instead, wouldn't you want to make a suggestion that they add the licence as part of the original PR?

ocaisa · 2025-02-03T11:24:39Z

.github/workflows/check_licenses.yml

+  pull_request:
+    branches:  [ "*-software.eessi.io" ]
+permissions:
+  contents: write # set permissions for writing 


This action is being given a lot of power at a global level, do we really need it to have write permissions? Isn't it enough that we get a notification if it fails?

ocaisa · 2025-02-03T11:26:57Z

.github/workflows/check_licenses.yml

+      - name: Apply patch (if no PR created)
+        if: steps.create_pull_request.outputs.number == '' && steps.check_licenses.outputs.patch != ''        
+        run: |
+          if [ -f license_update.patch ] && [ -s license_update.patch ]; then
+            git apply license_update.patch
+          else
+            echo "No changes to apply"
+          fi
+          git add licenses.json
+          git diff --cached --exit-code || git commit -m "Update licenses.json"
+          git push
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}


Rather than auto-apply a patch, can a code suggestion be made (or a PR to the source repo using a token without write permission to EESSI)?

ocaisa · 2025-02-03T11:27:55Z

licenses/check_licenses.yml

@@ -0,0 +1,59 @@
+name: Check and update licenses


Duplicate file?

ocaisa · 2025-02-03T11:29:35Z

.github/workflows/check_licenses.yml

+          python update_licenses.py --source=pypi TensorFlow
+          python update_licenses.py --source=github:easybuilders/easybuild EasyBuild


You are going to need to extract the software name from the easystack file, that is complicated. Perhaps instead you should add easystack (and easyconfig) support to update_licenses.py?

This could also help you decide where to look for licenses (since you have access to source_urls)

The other option is to gather software (and extension) names via Lmod and then check them against the yaml file.

ocaisa · 2025-02-03T11:34:38Z

.github/workflows/check_licenses.yml

+          branch: update-licenses-${{ github.run_number }}
+          base: [ "*-software.eessi.io" ]
+
+      - name: Apply patch (if no PR created)


Why wouldn't the PR be created? Wouldn't that mean the previous step will have failed?

hvelab · 2025-02-27T18:02:34Z

spare thoughts right now:

i kept the retrieval date because the original script did so, but i dont see much sense in keeping it, I would keep the license url instead
there are a lot of not found but i expect them to be less of them as right now some source_urls are still formatted with EB syntax but it my priority to fix it tomorrow
need to fix the "needs manual action" part as its case sensitive right now with the "Other" licenses

ocaisa · 2025-02-28T15:09:17Z

licenses/licenses.json

+    },
+    "ATK/2.38.0-GCCcore-13.2.0": {
+        "License": [
+            "Other"


How did this end up with Other? I see it is GPL v2 at https://gitlab.gnome.org/Archive/atk

ocaisa · 2025-02-28T15:15:38Z

licenses/licenses.json

+    },
+    "Archive-Zip/1.68-GCCcore-12.2.0": {
+        "License": [
+            "Other"


Hmm, some of these are going to need some discussion. The license of this is at https://github.com/redhotpenguin/perl-Archive-Zip?tab=License-1-ov-file#readme and it is GPL1+ or an artistic licence. If we are going to put Other we'll need a link to the licence if possible and possibly another field permission_to_redistribute

Agree, this is the plain response from the ecosyste.ms API: https://repos.ecosyste.ms/api/v1/repositories/lookup?url=https%3A%2F%2Fgithub.com%2Fredhotpenguin%2Fperl-Archive-Zip

Will work today on a "smarter" way to do this and to scrape other sources because there are much more not found licenses than found or other

ocaisa · 2025-02-28T15:18:53Z

licenses/licenses.json

+    },
+    "Arrow/16.1.0-gfbf-2023b": {
+        "License": [
+            "Other"


License for this is https://github.com/apache/arrow/blob/main/LICENSE.txt ("Apache-2.0")

ocaisa · 2025-02-28T15:31:44Z

licenses/licenses.json

+        "Retrieved From": "not found"
+    },
+    "Bazel/6.3.1-GCCcore-12.3.0": {
+        "License": "not found",


Apache 2.0 from https://github.com/bazelbuild/bazel/blob/master/LICENSE

ocaisa · 2025-02-28T15:33:33Z

licenses/licenses.json

+    },
+    "BeautifulSoup/4.12.2-GCCcore-12.3.0": {
+        "License": [
+            "Other"


MIT from https://pypi.org/project/beautifulsoup4/

ocaisa · 2025-02-28T15:35:23Z

licenses/licenses.json

+        "Retrieved From": "https://www.fftw.org"
+    },
+    "FFmpeg/6.0-GCCcore-13.2.0": {
+        "License": "not found",


https://www.ffmpeg.org/legal.html

hvelab · 2025-03-03T16:15:23Z

Updates:

Fixed bug and now finds more licenses
Now shows the real api call from where it got the license
For the "Other" and "not found", does an scraping to go find the LICENSE file -> we need to find a way to retrieve the spdx from there
This needs to be improved for the packages with "Others"
For the totally "not found", shows source_urls and homepages, seems that most of them need to be sanitized or do the scraping from there, add more keywords

update licenses

062b624

ocaisa reviewed Oct 15, 2024

View reviewed changes

MartinsNadia added 2 commits December 9, 2024 20:31

fix commits, branches, identation. Add content and ids

e517a4e

adding licenses yml

1480ae1

ocaisa requested changes Feb 3, 2025

View reviewed changes

needs more refinement but to show the progress for now

bbb8ee9

updated a bit

3355eb2

ocaisa reviewed Feb 28, 2025

View reviewed changes

progress in the license retrieval

2d7826e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] update licenses #675

[WIP] update licenses #675

MartinsNadia commented Aug 19, 2024

eessi-bot bot commented Aug 19, 2024

eessi-bot bot commented Aug 19, 2024

MartinsNadia commented Aug 19, 2024 •

edited

Loading

ocaisa left a comment

ocaisa Oct 15, 2024

ocaisa Oct 15, 2024

ocaisa Oct 15, 2024

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

ocaisa Feb 3, 2025

hvelab commented Feb 27, 2025

ocaisa Feb 28, 2025

ocaisa Feb 28, 2025

hvelab Mar 3, 2025

ocaisa Feb 28, 2025

ocaisa Feb 28, 2025

ocaisa Feb 28, 2025

ocaisa Feb 28, 2025

hvelab commented Mar 3, 2025 •

edited

Loading

		python update_licenses.py --source=pypi TensorFlow
		python update_licenses.py --source=github:easybuilders/easybuild EasyBuild

[WIP] update licenses #675

Are you sure you want to change the base?

[WIP] update licenses #675

Conversation

MartinsNadia commented Aug 19, 2024

eessi-bot bot commented Aug 19, 2024

eessi-bot bot commented Aug 19, 2024

MartinsNadia commented Aug 19, 2024 • edited Loading

ocaisa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvelab commented Feb 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvelab commented Mar 3, 2025 • edited Loading

MartinsNadia commented Aug 19, 2024 •

edited

Loading

hvelab commented Mar 3, 2025 •

edited

Loading