diff --git a/CHANGELOG.md b/CHANGELOG.md index b1c8a36..c25f5e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,15 +8,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - [``GDI-SOP_github-introduction-for-maintainers.md``](docs/GDI-SOP_github-introduction-for-maintainers.md) - Introductory guide for GDI SOP Repository maintainers. -- [``review_reminder.yml``](.github/workflows/review_reminder.yml) - GH Workflow to preiodically (monthly) and automatically (through [check_sop_reviews.py](scripts/check_sop_reviews.py)) create SOP review reminders (GH issues) + +### Modified +### Fixed + +## [1.0.0](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/releases/tag/v1.0.0) - 2024-10-28 + +### Added +- [``GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md``](sops/european-level/GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md) - European-level SOP describing how the 1+MG DAC is to review data access requests and recommend their rejection or approval. +- [``GDI-SOP0002_NCPs-veto-EDIC-decision.md``](sops/node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md) - Node-specific SOP Template describing how NCPs may veto EDIC decisions on data requests. +- [``GDI-SOP0007_SOP-template-creation.md``](sops/european-level/GDI-SOP0007_SOP-template-creation.md) - European-level SOP describing how to develop new SOP templates. +- [``review_reminder.yml``](.github/workflows/review_reminder.yml) - GH Workflow to periodically (monthly) and automatically (through [check_sop_reviews.py](scripts/check_sop_reviews.py)) create SOP review reminders (GH issues) - [``check_sop_reviews.py``](scripts/check_sop_reviews.py) - Script to automatically check if SOPs are due for review and create GitHub issues if so - [``GDI-SOP_organisational-roles-and-responsibilities.md``](docs/GDI-SOP_organisational-roles-and-responsibilities.md) - Documentation containing Organisational Roles and Responsibilities for GDI SOPs. +- [``GDI-SOP_information-service-management.md``](GDI-SOP_information-service-management.md) - Framework designed to systematically manage SOPs information flows across the GDI infrastructure - [``GDI-SOP_review-checklist.md``](docs/GDI-SOP_review-checklist.md) - Documentation guidelines in the form of checklists, for reviewers, approvers and authorizers of SOPs. -- [``GDI-SOP_charter.md``](docs/GDI-SOP_charter.md) - Documentation Charter of the task 4.3 - [``compare_index.py``](scripts/compare_index.py) - Script to automatically check if the SOP index table is up to date - [``sop_index.py``](scripts/sop_index.py) - Script to automatically create the SOP index table - [``utils.py``](scripts/utils.py) - General functions used by other scripts - [``sops/README.md``](sops/README.md) - Markdown containing the SOP index table +- [``GDI-SOP_charter.md``](docs/GDI-SOP_charter.md) - Documentation Charter of the task 4.3 - [``tests/``](tests/) - Directory containing tests to run GH repo's code - [``requirements.txt``](requirements.txt) - Needed modules to run GH repo's code - [``lint_sops.yml``](.github/workflows/lint_sops.yml) - GH Workflow to trigger linter on PRs @@ -31,5 +42,3 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - [``GDI-SOP_style-guide.md``](docs/GDI-SOP_style-guide.md) - Draft of styling guide - [``README.md``](README.md) - Main repository's readme - [``LICENSE``](LICENSE) - Repository license -- [``CONTRIBUTING.md``](CONTRIBUTING.md) - Guidance for repository contributions -- [``GDI-SOP_information-service-management.md``](GDI-SOP_information-service-management.md) - Framework designed to systematically manage SOPs information flows across the GDI infrastructure diff --git a/docs/images/GDI-SOP0002_1-summary-diagram.jpg b/docs/images/GDI-SOP0002_1-summary-diagram.jpg new file mode 100644 index 0000000..c6ef97b Binary files /dev/null and b/docs/images/GDI-SOP0002_1-summary-diagram.jpg differ diff --git a/docs/images/GDI-SOP0007_1-trigger-gh-issue.png b/docs/images/GDI-SOP0007_1-trigger-gh-issue.png new file mode 100644 index 0000000..b3b5b41 Binary files /dev/null and b/docs/images/GDI-SOP0007_1-trigger-gh-issue.png differ diff --git a/docs/images/GDI-SOP0007_2-label-assignment.png b/docs/images/GDI-SOP0007_2-label-assignment.png new file mode 100644 index 0000000..22dca7d Binary files /dev/null and b/docs/images/GDI-SOP0007_2-label-assignment.png differ diff --git a/docs/images/GDI-SOP0007_3-zenhub-ticket-management.png b/docs/images/GDI-SOP0007_3-zenhub-ticket-management.png new file mode 100644 index 0000000..0a1cbac Binary files /dev/null and b/docs/images/GDI-SOP0007_3-zenhub-ticket-management.png differ diff --git a/docs/images/GDI-SOP0007_4-Draft-SOP-document.gif b/docs/images/GDI-SOP0007_4-Draft-SOP-document.gif new file mode 100644 index 0000000..3a0f045 Binary files /dev/null and b/docs/images/GDI-SOP0007_4-Draft-SOP-document.gif differ diff --git a/docs/images/GDI-SOP0007_5-doc-to-md.png b/docs/images/GDI-SOP0007_5-doc-to-md.png new file mode 100644 index 0000000..d4531c4 Binary files /dev/null and b/docs/images/GDI-SOP0007_5-doc-to-md.png differ diff --git a/docs/images/GDI-SOP0007_6-PR-image.png b/docs/images/GDI-SOP0007_6-PR-image.png new file mode 100644 index 0000000..d495fd7 Binary files /dev/null and b/docs/images/GDI-SOP0007_6-PR-image.png differ diff --git a/docs/images/GDI-SOP0007_7-GH-workflows-checks.png b/docs/images/GDI-SOP0007_7-GH-workflows-checks.png new file mode 100644 index 0000000..a704a45 Binary files /dev/null and b/docs/images/GDI-SOP0007_7-GH-workflows-checks.png differ diff --git a/scripts/sop_linter.py b/scripts/sop_linter.py index 17ac8df..d76aa2e 100644 --- a/scripts/sop_linter.py +++ b/scripts/sop_linter.py @@ -110,13 +110,20 @@ def lr_check_metadata_table(self, soup: BeautifulSoup, file_path: str): for key, value in table_dict.items(): # Linting rules for node-specific SOPs should not apply for european-level ones - if key in ["gdi node", "instance version"]: + node_specific_keys = ["gdi node", "instance version"] + if key in node_specific_keys: try: if table_dict["template sop type"].lower() == "European-level SOP".lower(): continue except: pass + # We only want to evaluate each key Node-specific key format if any is present. + # Otherwise, it could be a Node-specific SOP template (correct without these keys) + if all(table_dict.get(key) in [None, "", []] for key in node_specific_keys): + self.report_issue(f"At the metadata table, value column for '{key}' was empty. If the SOP is a Node-specific SOP Instance (not a template), it should have a value.", file_path, warning=True) + continue + if key in expected_metadata: # Depending on the type of format rules for each row, we apply them differently if isinstance(expected_metadata[key], str) and not re.match(expected_metadata[key], value): diff --git a/sops/README.md b/sops/README.md index 68a3b99..0353d33 100644 --- a/sops/README.md +++ b/sops/README.md @@ -1,5 +1,7 @@ # European GDI SOP Index Below you can find the full list of SOPs in this repository. -| Name | Identifier | Template version | Topic | Type | GDI Node | Instance version | Nº steps | Last modified | -|:----------------------------------------------------------------------------------------------------|:-------------|:-------------------|:---------------------------|:-------------------|:-----------|:-------------------|-----------:|:----------------| -| | | | | | | | | | \ No newline at end of file +| Name | Identifier | Template version | Topic | Type | GDI Node | Instance version | Nº steps | Last modified | +|:----------------------------------------------------------------------------------------------------------------------------|:-------------|:-------------------|:---------------------------|:-------------------|:-----------|:-------------------|-----------:|:----------------| +| [GDI-SOP0002_NCPs-veto-EDIC-decision.md](./node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md) | GDI-SOP0002 | v1 | Helpdesk & operations | Node-specific SOP | | | 7 | 2024.09.27 | +| [GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md](./european-level/GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md) | GDI-SOP0003 | v1 | Data & metadata management | European-Level SOP | | | 10 | 2024.10.24 | +| [GDI-SOP0007_SOP-template-creation.md](./european-level/GDI-SOP0007_SOP-template-creation.md) | GDI-SOP0007 | v1 | Helpdesk & operations | European-Level SOP | | | 7 | 2024.07.08 | \ No newline at end of file diff --git a/sops/european-level/.gitignore b/sops/european-level/.gitignore deleted file mode 100644 index 8b13789..0000000 --- a/sops/european-level/.gitignore +++ /dev/null @@ -1 +0,0 @@ - diff --git a/sops/european-level/GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md b/sops/european-level/GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md new file mode 100644 index 0000000..842ce0e --- /dev/null +++ b/sops/european-level/GDI-SOP0003_SOP-1+MG-DAC-Recommendation-Approval.md @@ -0,0 +1,227 @@ +# European GDI - 1+MG DAC Recommendation Approval Process + +| Metadata | Value | +|------------------------|--------------------------------------------------| +| Template SOP number | ``GDI-SOP0003`` | +| Template SOP version | ``v1`` | +| Topic | Data & metadata management | +| Template SOP Type | European-Level SOP | +| GDI Node | | +| Instance version | | + +## Index + +1. [Document History](#1-document-history) +2. [Glossary](#2-glossary) +3. [Roles and Responsibilities](#3-roles-and-responsibilities) +4. [Purpose](#4-purpose) +5. [Scope](#5-scope) +6. [Introduction and Background Information](#6-introduction-and-background-information) +7. [Summary or Context Diagram](#7-summary-or-context-diagram) +8. [Procedure](#8-procedure) +9. [References](#9-references) + +### 1. Document History + +| Template Version | Instance version | Author(s) | Description of changes | Date | +|------------------|------------------|-------------------|----------------------------------|------------| +| ``v1.0.1`` | | Marcos Casado Barbero | Transform document to markdown and complete version | ``2024.10.24`` | +| ``v1`` | | Marcos Casado Barbero | Transform document to markdown and complete version | ``2024.10.04`` | +| ``v0`` | | Dylan Spalding | First version of SOP drafted | ``2024.04.11`` | + +### 2. Glossary +Find GDI SOPs common Glossary at the [**charter document**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md). + +For this SOP in particular, the source of truth for most definitions and terms is the [1+MG Data Access Governance for Research](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU). +| Abbreviation | Description | +|--------------|-----------------------------------------------------------------------------------------------------------| +| DAC | Data Access Committee, responsible for reviewing access requests and approving or denying the requests | +| 1+MG DAC | 1+MG Data Access Committee - Data Access Committee, which may include several domain-specific sub-committees, provided by 1+MG that receives and reviews access requests and moderates consensus discussions on access requests where necessary | +| EDIC | European Digital Infrastructure Consortium, will serve as the DAC for managing access requests to data in GDI | +| NCP | National Coordination Point | + +| Term | Definition | +|---------------|-----------------------------------------------------------------------------------------------------------| +| | | + +### 3. Roles and Responsibilities +See the qualifications and responsibilities of the roles at the [**Organisational Roles and Responsibilities**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_organisational-roles-and-responsibilities.md) document. + +| Role | Full name | GDI/node role | Organisation | +|------------|-----------------------|-------------------------|--------------| +| Author | Dylan Spalding | Finland / Pillar II co-lead | CSC | +| Author | Marcos Casado Barbero | Task 4.3 Lead | EMBL-EBI | +| Reviewer | Regina Becker | LU / Pillar I co-lead | LNDS | +| Reviewer | Gergő Csarnai | LU / Senior ELSI Specialist | LNDS | +| Approver | Gabriele Rinck | Task 4.3 member | EMBL-EBI | + +### 4. Purpose +The purpose of this SOP is to define the process for **requesting and granting access to controlled data within the European Genomic Data Infrastructure (GDI)**. This SOP ensures that: + +- **Access to data is granted only in justified cases**, following thorough review by the 1+MG Data Access Committee (DAC) and involving all necessary stakeholders to protect the rights and freedoms of the data subjects. +- The 1+MG DAC **reviews applications** to ensure **compliance with the 1+MG Data Governance requirements**, and **makes recommendations** to National Coordination Points (NCPs), who may agree or exercise their veto rights. +- The **process is transparent**, and **disputes are resolved in a timely manner**, adhering to the allocated timeframes for each request. + +### 5. Scope +This SOP covers the process **from when a data access application is received by the 1+MG DAC to when the application is approved or rejected** by the 1+MG DAC. The **input** is a completed application form, and the **output** is a notification of approval or rejection of the request for authorisation to access the controlled access data described in the application. + +### 6. Introduction and Background Information +To access controlled access data within the European Genomic Data Infrastructure a user must apply for authorisation to access the data from the 1+ Million Genomes Data Access Committee (1+MG DAC). The 1+MG DAC will **review applications** to ensure they conform to the 1+MG Data Governance requirements, and **make recommendations** to the relevant National Coordination Points (NCPs) who may agree or disagree with the recommendation. + +Additionally, the 1+MG DAC will follow a review process when the NCP disagrees with its recommendation (see [GDI-SOP0002_NCPs-veto-EDIC-decision.md](../node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md)). Once an agreement is reached, or the review process completes, the data user will be informed of the decision. + +### 7. Summary or Context Diagram +The following diagram summarises the step-by-step process for the recommendation by the 1+MG DAC of approval or rejection of data access applications. +````mermaid +graph TD + A[**Start**: Data Access
Application Received] + B1{**Step 1**:
Application Checked
by 1+MG DAC} + B2[**Step 2**: Application
Recommended for Approval] + B2.2{Response
from NCPs?} + B3[**Step 3**: No NCP Response
- Approval Confirmed] + B4[**Step 4**: NCP Agrees
- Approval Confirmed] + B5[**Step 5**: NCP Disagrees
- NCP Review
Process Initiated] + B6[**Step 6**: NCP Review
Upholds Recommendation
- Approval Confirmed] + B7[**Step 7**: NCP Review
Overrules Recommendation
- Application Rejected] + B8[**Step 8**: Review Delayed
- Inform Data User] + B9[**Step 9**: Approval confirmed
- Inform Data User] + B10[**Step 10**: Rejection confirmed
- Inform Data User] + + A --> B1 + B1 -->|Application
is approved| B2 + B2 --> B2.2 + B2.2 -->|No NCP Response| B3 + B2.2 -->|NCP Agrees| B4 + B2.2 -->|NCP Disagrees| B5 + B3 --> B9 + B4 --> B9 + B6 --> B9 + B5 -->|Recommendation Upheld| B6 + B5 -->|Recommendation Overruled| B7 + B5 -->|Review Delayed| B8 + B1 -->|Application
is rejected| B10 + B7 --> B10 + B8 -->|Follow up
review process| B5 +```` + +### 8. Procedure +#### 8.1. Initiate application check +| Step identifier | When | Who | +|-----------------|--------------------------------------------|--------------| +| 1 | On receipt of notification of a new data access application | 1+MG DAC | + +As the 1+MG DAC, **check the requester's application**, based on the criteria in the [**Data Access Review**](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU/edit?tab=t.0#bookmark=kix.9o5agsar78sq) section of the Data Governance document. The application form, which is used to help the data requester enter valid and complete information for the application, must be checked to ensure that: +- The research has [institutional sign-off](https://docs.google.com/document/d/1LJa_vWhqUtpNnw8hW7cmPoihImqELv9dgK7Bd826EHg/edit#heading=h.94zixl2wbth), if applicable. +- The **data analysis plan** is present and suitable for the proposed research. +- The **data minimisation principle** is met. +- **Algorithms** (such as artificial intelligence) **are suitable** for the proposed purpose and transparent. +- The **ethical approval** has been obtained for the proposed research, where applicable, and that the approval is applicable for the proposed research and conclusions of the ethics review are followed. +- The **funding mechanisms** for the proposed research are in place (if required). +- The **project description** corresponds with the data analysis plan, and the **ethical approval and legal requirements** exist with respect to the requested data. +- Any **peer review** of the scientific validity of the proposed research exists. In case it does not, follow [**section II.3**](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU/edit?tab=t.0#bookmark=id.utnaek5ua0aa) of the Data Governance document. + +You must **finish this initial assessment within 5 working days** (review [**agreed timelines**](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU/edit?tab=t.0#bookmark=kix.9o5agsar78sq) if needed). After this period,you shall **transmit the outcome of the review and a decision recommendation** to the 1+MG NCPs who can channel it to relevant bodies on the national level. + +Be mindful that **this timeline may be affected** where the planned research affects vulnerable subjects or groups, as specified in [**section III.2**](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU/edit?tab=t.0#bookmark=id.xez4tqfoy9hq) of the Data Governance document. + +Depending on the outcome of this initial evaluation: +- If all steps are correct, move to **[step 2](#82-recommend-application-for-approval)**. +- If **any step fails**, the **application must be rejected**: + - Inform the relevant NCPs, who may veto the rejection (see [GDI-SOP0002_NCPs-veto-EDIC-decision.md](../node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md)). In your communication to the NCPs, clearly state the date for which their veto expires. Also inform NCPs about the possibility of extending the veto period, in exceptional and justified cases (as defined in the [1+MG Data Access Governance for Research](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU)). + - After 6 _calendar_ days since NCPs were notified, **send a reminder** to the NCPs. + - After 11 working days (``DATE:A``) since NCPs were notified, if no response was given from NCPs, move to **[step 10](#810-inform-data-requestor-rejection-confirmed)**. + +#### 8.2. Recommend application for approval +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 2 | Once a completed application has been reviewed by the 1+MG DAC and recommended for approval | 1+MG DAC | + +Once initial evaluation of data request is positive, **mark application** as 'Recommended for Approval'. + +**Inform the relevant NCP(s)** of the review outcome, and pass the application to them for further action as per [GDI-SOP0002_NCPs-veto-EDIC-decision.md](../node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md). Record the date of recommendation and set a reminder for 11 working days (``DATE:A``). + +Depending on the response from NCPs: +- If **no response** before the end of DATE:A, move to **[step 3](#83-confirm-approval-no-ncp-response)**. +- If **NCPs agree** with recommendation, move to **[step 4](#84-confirm-approval-ncp-agrees)**. +- If **NCPs disagree** with recommendation, move to **[step 5](#85-initiate-ncp-review-process)**. + +#### 8.3. Confirm approval (no NCP response) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 3 | DATE:A is reached or past, and no response from the NCP has been received | 1+MG DAC | + +After the end of ``DATE:A`` is reached with no response from relevant NCPs, **confirm the approval of the data access request**: move to **[step 9](#89-inform-data-user-approval-confirmed)**. + +#### 8.4. Confirm approval (NCP agrees) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 4 | DATE:A has not been passed, and the NCP agrees with the 1+MG DAC recommendation | 1+MG DAC | + +If the **NCPs agree** with the recommendation, then **confirm the approval of the data access request**: move to **[step 9](#89-inform-data-user-approval-confirmed)**. + +#### 8.5. Initiate NCP review process +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 5 | DATE:A has not been passed, and the NCP disagrees with the 1+MG DAC recommendation | 1+MG DAC | + +If the NCPs disagree with the recommendation (see [GDI-SOP0002_NCPs-veto-EDIC-decision.md](../node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md)), **initiate the review process for handling NCP disagreements** as per SOP [NCP review process of disagreement with 1+MG DAC recommendation](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/33). This process must be completed by the time limit set in that SOP (``DATE:B``). + +Depending on the outcome of the NCP review process: +- If **no response** before the end of ``DATE:B``, move to **[step 8](#88-inform-data-user-process-delayed)**. +- If **NCPs upholds recommendation**, move to **[step 6](#86-confirm-approval-recommendation-upheld)**. +- If **NCPs overrules recommendation** (see [GDI-SOP0002_NCPs-veto-EDIC-decision.md](../node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md)), move to **[step 7](#87-reject-application-recommendation-overruled)**. + +#### 8.6. Confirm approval (recommendation upheld) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 6 | DATE:B has not been passed, and the NCP review process upholds the 1+MG DAC recommendation | 1+MG DAC | + +If the outcome of the NCP review process is positive, then **confirm the approval of the data access request**: move to **[step 9](#89-inform-data-user-approval-confirmed)**. + +#### 8.7. Reject application (recommendation overruled) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 7 | DATE:B has not been passed, and the NCP review process overrules the 1+MG DAC recommendation | 1+MG DAC | + +If the NCPs overrule the initial recommendation, **reject the application**: move to **[step 10](#810-inform-data-requestor-rejection-confirmed)**. + +#### 8.8. Inform data user (process delayed) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 8 | DATE:B has passed, and the review process has not returned a decision within 2 months | 1+MG DAC | + +After the end of ``DATE:B``, if no decision is obtained **after 2 months**, **inform the data user that the application is still under review**. + +Move back to **[step 5](#85-initiate-ncp-review-process)**. + +#### 8.9. Inform data requestor (Approval confirmed) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 9 | After confirmation of approval is reached | 1+MG DAC | + +Regardless of the path of confirmation, once the approval has been confirmed: +- **Record the decision and reasons**. +- **Inform the data requestor**. + +This concludes the process resulting in confirmation of the data request. + +#### 8.10. Inform data requestor (Rejection confirmed) +| Step identifier | When | Who | +|-----------------|------------------------------------------------|----------| +| 10 | After rejection of request is reached | 1+MG DAC | + +Regardless of the path of rejection: +- **Record the decision and reasons** (e.g., missing documents, invalid data, non-compliance with ethical standards...). Include options to rectify these issues for the data requestor to amend and re-submit the application. +- **Inform the data requestor**. + +This concludes the process resulting in rejection of the data request. + +### 9. References +| Reference | Description | +|-----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------| +| [1](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md) | European GDI - SOP Charter | +| [2](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_organisational-roles-and-responsibilities.md) | European GDI - Organisational Roles and Responsibilities (ORR) | +| [3](../node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md) | European GDI SOP - NCPs veto EDIC Decision | +| [4](https://docs.google.com/document/d/19HVf7SP2R_fCI-KugVStZR4yEAkV6vhvSbVd6a2ZviU) | 1+MG Data Access Governance for Research | +| [5](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/33) | European GDI - NCP review process of disagreement with 1+MG DAC recommendation | +| [6](https://docs.google.com/document/d/1LJa_vWhqUtpNnw8hW7cmPoihImqELv9dgK7Bd826EHg/edit#heading=h.94zixl2wbth) | European GDI - Obtain Institutional Sign-off | \ No newline at end of file diff --git a/sops/european-level/GDI-SOP0007_SOP-template-creation.md b/sops/european-level/GDI-SOP0007_SOP-template-creation.md new file mode 100644 index 0000000..ac7f7e5 --- /dev/null +++ b/sops/european-level/GDI-SOP0007_SOP-template-creation.md @@ -0,0 +1,480 @@ +# European GDI - SOP Template creation + +| Metadata | Value | +|----------------------|--------------------| +| Template SOP number | ``GDI-SOP0007`` | +| Template SOP version | ``v1`` | +| Topic | Helpdesk & operations | +| Template SOP Type | European-Level SOP | +| GDI Node | | +| Instance Version | | + +## Index + +1. [Document History](#1-document-history) +2. [Glossary](#2-glossary) +3. [Roles and Responsibilities](#3-roles-and-responsibilities) +4. [Purpose](#4-purpose) +5. [Scope](#5-scope) +6. [Introduction and Background Information](#6-introduction-and-background-information) +7. [Summary or Context Diagram](#7-summary-or-context-diagram) +8. [Procedure](#8-procedure) +9. [References](#9-references) + +### 1. Document History +| Template Version | Instance version | Author(s) | Description of changes | Date | +|---------|-----------|-----------|------------------------------|------------| +| ``v1`` | | Marcos Casado Barbero | Created first version of the SOP | ``2024.07.08`` | + +### 2. Glossary +Find GDI SOPs common Glossary at the [**charter document**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md). + +| Abbreviation | Description | +|---------------|-----------------------------------------| +| EBI | European Bioinformatics Institute | +| EMBL | European Molecular Biology Laboratory | +| GH | GitHub | +| HRI | Health Research Infrastructure | +| IST | Instituto Superior Técnico | +| NBIS | National Bioinformatics Infrastructure Sweden | +| PR | Pull Request | +| RFC | Request For Comments | +| SLA | Service Level Agreements | +| UU | University of Uppsala | + +| Term | Definition | +|---------------|-----------------| +| | | + +### 3. Roles and Responsibilities +See the qualifications and responsibilities of the roles at the [**Organisational Roles and Responsibilities**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_organisational-roles-and-responsibilities.md) document. + +| Role | Full name | GDI/node role | Organisation | +|------------|-----------------|-----------------|--------------| +| Author | Marcos Casado Barbero | Task 4.3 member | EMBL-EBI | +| Reviewer | Pedro Ferreira | Task 4.3 member | IST | +| Reviewer | Bianca Hendriksze | Task 4.3 member | HRI | +| Reviewer | Elisavet Torstensson | Task 4.3 member | UU / NBIS | +| Reviewer | Mattias Strömberg | Task 4.3 member | UU / NBIS | +| Approver | Markus Englund | Task 4.3 member | UU / NBIS | +| Approver | Dylan Spalding | WP5 Leader | CSC | +| Authorizer | _TBD: to be brought up in a monthly MB meeting, to appoint a MB authorizer_ | Management Board | _TBD_ | + +### 4. Purpose +This SOP will help maintainers of the [``GenomicDataInfrastructure/standard-operating-procedures``](https://github.com/GenomicDataInfrastructure/standard-operating-procedures) repository (i.e., GDI's Operations Committee and Security and Data Protection Committee) **transform SOP Requests into SOP templates**. In doing so, the process will be reproducible and straightforward, ensuring consistency and quality in SOP development across the GDI project + +### 5. Scope +This SOP **starts with the need to create a new GDI SOP Template**, triggered by the creation of a new GH issue through the [``New SOP Request``](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/new?assignees=&labels=new-sop-request%2Cenhancement&projects=&template=new_sop_request.yaml&title=%5BSOP+Request%5D+%3CShort+title%3E) issue template. + +The SOP encompasses the steps after the need for an SOP has been manifested, until the addition of the markdown SOP within the GitHub repository itself. The **output of this SOP will be the finalized document added to the repository**. + +### 6. Introduction and Background Information +Given the size of GDI as a project, in order to minimize the heterogeneity among GDI nodes, this repository contains SOPs, templates, and instances, standardizing procedures for GDI members to follow. How these SOPs are created is a process in itself, which task 4.3 of the project aims to define. This document aids the creators and maintainers of these SOP Templates along that process. + +### 7. Summary or Context Diagram +```mermaid +graph TB + start( ) + input1[SOP Request] + Zenhub1[ZenHub ticket] + subgraph 'Operations Committee' and 'Security and Data Protection Committee' + step1{Is SOP
request valid?} + step_subsChange{Does this SOP represent
a 'substantial' change?} + step3{More information
needed?} + stepj(Request more
information from user) + step4(Justify rejection
and close GH issue) + step2(Update ZenHub ticket
& GH issue) + step5{"`Will the template be created + in **Google Docs** first?`"} + step6(Create Google
Document draft) + step7(Create Markdown
document draft) + step8(Assign SOP Template authors) + step9(Contact the authors) + step13_prev(Request nomination of approvers) + step13{Document
approved?} + r3(Contact authors) + step13-1(Contact
Management Board) + + step-rfc1(Create RFC discussion) + step-rfc2(Update RFC discussion) + step-rfc3(Update RFC discussion) + + step14(Prepare final
SOP Markdown document) + step15(Create PR against
'dev' branch) + step16(Review and merge PR) + end + step-rfc1 --> step2 + step8 -..->|Set as Active|step-rfc2 + step16 -..->|Set as Landed|step-rfc3 + subgraph Management Board + step-mb1{Document
authorization
vetoed?} + r1(Contact authors) + end + subgraph Authors + step10(Fill in SOP Draft) + end + step10 -->|Request review| step12 + subgraph Reviewers + step12{Document
reviewed?} + r2(Contact authors) + end + step12 -->|Yes|step13_prev + step13_prev -->|2 OC and 2 SDPC members
are nominated| step13 + step13 -->|Yes|step13-1 + step13-1 --> step-mb1 + ending(" ") + + start -..->|GDI member creates
SOP request| input1 + input1 -..->|ZenHub automatically
creates ticket in Board|Zenhub1 + Zenhub1 -..->|OC/SDPC
Notices the request| step1 + step1 -->|Yes| step_subsChange + step_subsChange -->|No| step2 + step_subsChange -->|Yes| step-rfc1 + step1 -->|No| step3 + step3 -->|Yes| stepj + stepj -->|Wait for user's answer| step1 + step3 -->|No| step4 + step2 --> step5 + step5 -->|Yes| step6 + step5 -->|No| step7 + step6 --> step8 + step7 --> step8 + step8 -->|Authors are given the SOP draft| step9 + step9 -->|Authors are requested to fill in the draft| step10 + + step-mb1 -->|Requested
changes| r1 + r1 -->|Amendments| step-mb1 + step12 -->|Requested
changes| r2 + r2 -->|Amendments| step12 + step13 -->|Requested
changes| r3 + r3 -->|Amendments|step13 + + step-mb1 -->|No changes
requested|step14 + step14 -->|SOP is accessioned| step15 + step15 -->|Request review from
other OC/SDPC members | step16 + step-rfc3 -..- ending +``` + +### 8. Procedure +#### 1. Evaluate SOP request +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``1`` | When GH issue (_New SOP Request_) is created | OC/SDPC | + +A GH user creates a GH issue like the following, requesting a new SOP: + +![GH Issue](../../docs/images/GDI-SOP0007_1-trigger-gh-issue.png) + +The first step is for the Operations Committee to **evaluate the new SOP request**, following these criteria: +- **Existing content**: Is the requested SOP not already in the GH repository? +- **Request**: Is the request correctly made? Is there missing information? Is the given information comprehensive enough? +- **Motivation**: Is the creation of the SOP justified and valid? Would GDI benefit from the creation of this SOP? Is the SOP covering a repetitive process of the GDI workflow? + +🔀 Depending on the answer to all previous questions: +- If the answers are **affirmative**: + - Add the tag ``accepted`` to the GitHub issue + ![Adding "accepted" label to GH issue](../../docs/images/GDI-SOP0007_2-label-assignment.png) + - In the ZenHub board, move the ticket (i.e., the request) to the ``Product backlog`` column. + - Proceed to [step 2](#2-create-rfc-discussion). +- If the answers are **negative**, either: + - Request more information from the user, and repeat the review if necessary. + - Justify the rejection of the request in the GH issue, and close it. + +#### 2. Create RFC discussion +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``2`` | After new SOP request has been deemed valid by the Operations Committee | OC/SDPC | + +Once the SOP request was accepted, it is time for the idea to be built from within GDI. **Depending on the relative importance of the SOP**, you may need to bring it forth in the shape of an **RFC**, to host a GDI-led discussion. Follow the checklist below (_#! TODO: this list will be at the [RFC's README](https://github.com/GenomicDataInfrastructure/rfcs?tab=readme-ov-file#when-to-follow-this-process) at some point_) to decide the course of action: + +- **Impact on Multiple Nodes or Stakeholders:** + - Does the SOP affect multiple GDI nodes or have the potential to influence operations across the network? + - Will the SOP change how multiple stakeholders (e.g., operations, data management, security) interact with or use GDI infrastructure? + +- **Introduction of New Features or Processes:** + - Does the SOP introduce a new feature, process, or operational procedure that was not previously part (or similar) of GDI operations? + - Is this SOP setting a precedent for future SOPs that would fundamentally alter existing workflows? + +- **Modification of Existing GDI Procedures:** + - Does the SOP significantly change the semantics or behavior of an existing procedure that is widely used or critical to the GDI infrastructure? + - Does the SOP propose removing or deprecating a feature or procedure that is currently in use by GDI nodes? + +- **Impact on GDI Service Levels (SLAs):** + - Could the SOP influence key Service Level Agreements (SLAs) within GDI, such as system uptime, helpdesk response times, or software update protocols? + - Does the SOP align with or potentially conflict with existing SLAs, requiring a review to ensure consistency and clarity for users? + +- **Cross-Functional Dependencies:** + - Does the SOP require coordination between different functions or teams within GDI, such as security, operations, and data management? + - Will the SOP impact how different GDI committees (e.g., OC, SDPC) operate or collaborate? + +- **Potential for User Confusion or Misalignment:** + - Could the SOP lead to confusion among users or stakeholders regarding their roles, responsibilities, or the functionality of GDI services? + - Is there a risk that the SOP might lead to a misalignment between user expectations and the services provided by GDI? + +- **Risk and Compliance Considerations:** + - Does the SOP introduce new risks or compliance requirements that need to be carefully reviewed and managed? + - Is there a significant risk that the SOP might not meet the necessary compliance standards without thorough review and consensus? + +- **Strategic Importance:** + - Is the SOP strategically important for the long-term goals or sustainability of the GDI? + - Does the SOP align with or diverge from the broader strategic vision of GDI, requiring input and consensus from a wider audience? + +- **Community Feedback and Consensus Building:** + - Would the SOP benefit from broader community feedback to ensure it is well-received and effectively implemented? + - Has there been significant interest or concern from the community that would warrant a more formal review and discussion process? + +- **Complexity and Scope of Change:** + - Is the SOP complex enough that its implementation could have unforeseen consequences or require extensive coordination? + - Does the SOP cover a wide scope, making it difficult to assess its impact without a more structured and collaborative review process? + +🔀 Depending on the answer to previous questions: +- If some answers are **affirmative**: + - Continue with this step, creating the RFC discussion. +- If all are **negative** or you consider the SOP not to be substantial: + - Skip the rest of step 2, and jump straight to [step 3](#3-draft-sop-document). + +Go to [**GDI's RFC GH repository**](https://github.com/GenomicDataInfrastructure/rfcs?tab=readme-ov-file#what-the-process-is) and familiarize yourself with the process of **RFC creation**. As the first step, **create a [new discussion](https://github.com/GenomicDataInfrastructure/rfcs/discussions/new/choose)** choosing the ``RFC Discussions`` category. The goal of this discussion is for the SOP, as a substantial change to GDI, to be introduced and discussed by the GDI community. + +_#! TO-DO: the process of adding an RFC to the repository is still not fully defined, but the idea is for a discussion to be opened in GH, for consensus to be reached and a final markdown RFC document to be merged with the repo._ + +#### 3. Draft SOP document +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``3`` | After RFC discussion has reached consensus, if an RFC was created | OC/SDPC | + +Depending on the course of action of [step 2](#2-create-rfc-discussion), this step may start without the need of an RFC, or when the RFC discussion has reached consensus. Regardless of the origin, the OC/SDPC member shall **prepare the SOP draft**. This document will be a modified copy of the [general SOP template](../../docs/GDI-SOP_sop-template.md). + +Depending on the product backlog, a request may need to wait until it is picked for drafting. Once this step is started, **add yourself as an assignee** to the ZenHub ticket (i.e., the request) and move it to the ``In Progress`` column. + +![ZenHub ticket management](../../docs/images/GDI-SOP0007_3-zenhub-ticket-management.png) + +The **format** of the drafted document can vary, based on the convenience of all the roles intervening in the writing and reviewing processes: +- **Markdown document**. The draft can be started plainly in markdown, by making a copy of the [general SOP template](../../docs/GDI-SOP_sop-template.md) (already in markdown). This format is recommended only if everyone involved in the process is familiar with Git and Markdown's syntax. While less agile than a live collaborative platform, the benefit is that there are no format conversions, and the draft evolves directly into the final SOP document. +- **Google Document**. The template may be transformed from its native markdown to a Google Document, where it will be edited live as a draft, and then reformatted back into a markdown document later on. While reformatting back to Markdown will be needed at [step 6](#6-prepare-final-sop-markdown-document), this is likely the most common path, given its simplicity by making use of Google Drive features. To create the draft, follow these steps: + - Either (_option 1_) directly copy-paste the markdown from the [general SOP template](../../docs/GDI-SOP_sop-template.md) into a new document at **GDI's [SOP Drafts](https://drive.google.com/drive/u/0/folders/131kJLHDk8L2oGgnRzRBT5AR0Ofpbn2qS) directory**; or (_option 2_) duplicate the existing SOP Template Google Document in the same directory. See video snippet below. + - Name the new file following the format of ``< YYYYMMDD > - GDI-SOP_draft-< SOP title >`` (e.g., ``20240702 - GDI-SOP_draft-SOP Template creation``). + - In Google Documents, tables can have a **header**, but you must specify it as such for each table (or copy-paste the one you configure): right-click on a table, click on ``Table properties``, click on ``Row``, and then tick the box ``Pin header row(s)`` (number of header rows commonly is 1). This little trick will save you a lot of headaches when the time comes to transform the document to markdown. + +![Drafting SOP template in Google Drive](../../docs/images/GDI-SOP0007_4-Draft-SOP-document.gif) + +Regardless of the format, **fill out the draft with as much information** (e.g., background, purpose, summary...) **as possible**, to the best of your knowledge. This content will be finished by the authors later on. + +#### 4. Contact Authors +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``4`` | After SOP document has been drafted | OC/SDPC | + +Once the SOP document has been drafted, experts are required to fill in the gaps and finalize it. These **authors are to be appointed and contacted by the OC/SDPC**. Who the authors are will depend on the background and requirements of each SOP, and thus it is the responsibility of the OC/SDPC to **find the best-suited people for the task**. The only requirements are for authors to be part of the GDI project and to know about, or be part of, the subject the SOP revolves around. An approach to identify authors is to go through the contributors of the RFC discussion at [step 2](#2-create-rfc-discussion). + +The communication may vary depending on the selected authors. For example, if the experts are part of the OC/SDPC themselves, then it may be best to let the group know through GDI's Slack workspace or mailing lists (``gdi-oc [at] elixir-europe.org`` and ``gdi-sdpc [at] elixir-europe.org``). On the other hand, if authors are external to these two committees, the following email template could be used to contact them. + +Remember to CC the OC and SDPC mailing lists: ``gdi-oc [at] elixir-europe.org`` and ``gdi-sdpc [at] elixir-europe.org``. +```` +Subject: [GDI T4.3] Requesting expert input for drafted SOP +```` +```` +Dear < Recipient's Name(s) >, + +We hope this message finds you well. + +The GDI's Operations Committee (OC) and Security and Data Protection Committee (SDPC) have drafted a new Standard Operating Procedure (SOP): "< SOP Title >". We now seek your expertise to fill in the gaps and help us finalize it. Given your background and involvement in the GDI project, we believe you are well-suited as an author of the SOP. + +We kindly request you to review the SOP draft and provide the necessary input. Please find the draft SOP document here: < URL of drafted document > + +We will aid you during the subsequent rounds of review, approval and authorization of the SOP. You can find more information about GDI SOPs here: https://github.com/GenomicDataInfrastructure/standard-operating-procedures + +Should you know any other GDI members who could assist as authors as well, please let us know to get in contact with them. + +Thank you for your attention and collaboration. We look forward to your valuable contributions. + +Best regards, + +< Your Name > +< Your GDI Node > +Operations Committee (OC) / Security and Data Protection Committee (SDPC) +```` + +Depending on the format of the draft, the above-mentioned ``< URL of drafted document >`` will change: +- **Google Document**. You can simply paste the URL of the Google Document, making sure to add the recipients as editors of that particular document in Google Drive. +- **Markdown document**. The easiest way to share the document would be by creating a drafted PR in GH, from either a personal fork or branch to the ``dev`` branch. Make sure to create it as a _Draft pull request_. If done this way, at [step 7](#7-create-pr-review-and-merge) you will simply have to convert the draft into a _Ready for review_ PR. Check the [accessioning guide](../../docs/GDI-SOP_sop-accessioning.md#file-naming-conventions) to know how to name and where to place this new file. + +Remember to **leave a comment in the GitHub issue**, briefly mentioning that authors have been contacted. Be mindful of the information you share (e.g., no email addresses), since _anything_ that is typed through ZenHub in the ticket, will be **publicly displayed** in the GitHub issue! + +This step of the process ends when enough GDI members accepted authoring the SOP. For this to happen, it may be needed for the OC/SDPC member to engage in conversations to find the best-suited authors. These conversations will be useful when reviewers are needed. + +#### 5. Monitor SOP development +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``5`` | After SOP authors have been appointed | OC/SDPC | + +It is your duty, as a member of the OC/SDPC, to monitor the entire SOP development process, ensuring that: +- **Authors** are engaged with the development of the SOP. Checking the content of the SOP or contacting the authors recurrently may be required. +- **Reviewers and approvers are appointed**, diverse across GDI nodes, and engaged in the process. Beyond the role definitions in the [ORR](../../docs/GDI-SOP_organisational-roles-and-responsibilities.md), the rule of thumb is for these roles to span multiple GDI Pillars and nodes. Reviewers may be selected directly by the OC/SDPC or by the authors themselves. Approvers will be part of the OC/SDPC itself. +- **Communication is efficient** throughout the process. For example, ensuring that GDI MB receives the request to authorize the final SOP, or that authors are aware of any requested changes. +- All **people involved are included in the _Roles and Responsibilities_ section** of the drafted SOP. This includes all authors, reviewers, approvers, authorizers and any other role pertinent to the SOP. +- Every **gate-keeping event** (review, approval, authorization) **is documented in the GH issue and RFC discussion**. In other words, when the SOP draft passes through any of these steps, the OC/SDPC should document it in the GH issue and RFC discussion. For example: ``SOP draft has been reviewed by reviewers X, X and X...``. + +In each of the subsequent sub-steps, back and forth communication between all actors may be needed to address requests. + +##### 5.1 Appoint reviewers + +Similar to how authors were nominated and contacted, you shall **appoint reviewers and contact them** requesting their review. Depending on the SOP, who are the reviewers and how to contact them will vary. If by email, you can use the email template below. + +Remember to CC the OC and SDPC mailing lists: ``gdi-oc [at] elixir-europe.org`` and ``gdi-sdpc [at] elixir-europe.org``. +```` +Subject: [GDI T4.3] Requesting SOP review +```` +```` +Dear < Recipient's Name(s) >, + +We hope this message finds you well. + +A new GDI Standard Operating Procedure (SOP) is in development. We now seek your expertise to review the drafted SOP to ensure that its content meets GDI's quality standards. Given your background and involvement in the GDI project, we believe you are well-suited as a reviewer of this SOP, titled "< SOP Title >". + +We kindly request you to review the SOP draft and provide your feedback. Please find the draft SOP document here: < URL of drafted document > + +You can find more information about GDI SOPs here: https://github.com/GenomicDataInfrastructure/standard-operating-procedures + +Should you know any other GDI members who could assist as reviewers as well, please let us know to get in contact with them. + +Thank you for your attention and collaboration. We look forward to your valuable contributions. + +Best regards, + +< Your Name > +< Your GDI Node > +Operations Committee (OC) / Security and Data Protection Committee (SDPC) +```` + +You shall **follow through the communication between authors and reviewers**, to ensure that once authors have drafted the SOP, reviewers are notified and proceed with their reviews. + +##### 5.2 Request OC/SDPC approval +Once the drafted SOP has been filled in by the authors, and has passed the inspection of reviewers, the SOP document shall go through approval of the OC/SDPC. **These committees need to nominate 2 members each** (4 in total), that will be responsible for approval of the SOP. + +Use the following template to **send an email to the OC (``gdi-oc [at] elixir-europe.org``) and SDPC (``gdi-sdpc [at] elixir-europe.org``) requesting approval**: +```` +Subject: [GDI T4.3] Requesting SOP approval +```` +```` +Dear OC/SDPC, + +I hope this message finds you well. + +A new GDI Standard Operating Procedure (SOP) is in development. It was drafted by the OC/SDPC, authored, and reviewed by GDI members, and is pending approval from across the Operations Committee (OC) and Security and Data Protection Committee (SDPC). + +Please appoint 2 members of each committee (4 in total) to take the role of approvers of the SOP. These shall provide a formal approval or justified rejection at the earliest convenience. Find the current SOP document here: < URL of SOP document >. + +You can find more details about this SOP development at the following sources: +- [Charter](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md) +- [Information Service Management](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_information-service-management.md). +- SOP Request: < URL of the GH issue, the SOP request > +- RFC Discussion: < URL of the RFC discussion created for this SOP, if applicable > + +Thank you for your attention and collaboration. + +Best regards, + +< Your Name > +< Your GDI Node > +Operations Committee (OC) / Security and Data Protection Committee (SDPC) +```` + +**Follow through the approval process**, ensuring that all three GDI Pillars are aware of the new SOP, and have given their formal approval. It may be needed for you to bring it up at committee meetings, or to chase members to approve the SOP. + +##### 5.3 Contact Management Board +Once approved by the OC/SDPC, **communicate the formal authorization request to the GDI MB**. Make use of the template below to craft the email sent to ``gdi-mb [at] elixir-europe.org``. + +Remember to CC the OC and SDPC mailing lists: ``gdi-oc [at] elixir-europe.org`` and ``gdi-sdpc [at] elixir-europe.org``. +```` +Subject: [GDI T4.3] Requesting SOP Authorization +```` +```` +Dear GDI Management Board, + +We hope this message finds you well. + +The Operations Committee (OC) and Security and Data Protection Committee (SDPC) are pleased to inform you that a new GDI Standard Operating Procedure (SOP) has been developed. It was drafted by the OC/SDPC, authored, and reviewed by GDI members, and finally approved by the OC/SDPC. We now present this SOP to the Management Board for formal authorization. + +As part of the authorization process, if no issues are raised within four weeks of this notice, the OC/SDPC will consider the SOP to be authorized and we will proceed to add it to the GDI SOP GitHub repository. + +Please, also let us know if: +- An extension to this 4-week period is required to evaluate whether to veto or not the SOP. +- You are certain the veto of the SOP will not be exercised before these 4 weeks. This will help us resume the process as early as possible. + +Please, find the SOP document here: < URL of SOP document >. + +More information about the SOP development process can be found within the [Charter](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md) and [Information Service Management](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_information-service-management.md) at our repository. +Thank you for your attention and collaboration. + +Best regards, + +< Your Name > +< Your GDI Node > +Operations Committee (OC) / Security and Data Protection Committee (SDPC) +```` + +This **step shall finish** either: +- When the **period to raise issues (4 full weeks) concluded without any complaints from the MB**. +- The **MB explicitly authorized the SOP** before the 4-week period finished. + +If comments are given from this body, the step shall finish when they are addressed, starting a new period of review altogether. Similar consideration is to be taken if the MB requests a period extension. It will be your responsibility, as a member of the OC/SDPC, to keep track of the status of development and to make sure requested changes are addressed (e.g., contacting authors). + +#### 6. Prepare final SOP Markdown document +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``6`` | After receiving authorization from GDI MB | OC/SDPC | + +Now that we have the content reviewed and ready, we need to format it to fit into the GH repository. The difficulty of this step will vary depending on the chosen format for the draft at [step 3](#3-draft-sop-document). In both cases, it is assumed that you are familiar with GH and already have a local copy of the GDI SOP repository (i.e., you have cloned it). + +If the **document was drafted using Google Drive**, its format must be modified before it is added to the repository. On the other hand, if the **document was drafted in markdown** format natively, there are no format changes required and thus you can skip the first two of the following: +1. [_Only if the document was drafted using Google Drive_] **Copy the whole Google Document and paste it** into the left box at [**gdoc2md**](https://gdoc2md.com/). There are many tools to format a Google Document into markdown but, in our experience, this one keeps the markdown format the closest to the native template, which is especially relevant regarding tables. Bear in mind that anything copied here would be processed by the tool deployed at someone else's server. If the document contains information that should not be public (yet), consider other choices, like locally installing [gdoc2md](https://github.com/mr0grog/google-docs-to-markdown) or [pandoc](https://pandoc.org/installing.html). + +2. [_Only if the document was drafted using Google Drive_] **Copy the raw markdown into a new file**. Check the [accessioning guide](../../docs/GDI-SOP_sop-accessioning.md#file-naming-conventions) to know how to name and where to place this new file. + +![Converting Google Document to Markdown](../../docs/images/GDI-SOP0007_5-doc-to-md.png) + +3. Manually **inspect that the markdown file complies with the [Style and Format guide for GDI SOPs](../../docs/GDI-SOP_style-guide.md)** before proceeding to [step 7](#7-create-pr-review-and-merge). Some **format changes** may be due depending on the document, especially if it was drafted originally in Google Drive. These changes should not affect the content that was reviewed and authorized previously, just the format. + +At this point, **assign an accession to the new SOP**. This includes modifying both the SOP's filename and metadata table inside it. See the [accessioning guide](../../docs/GDI-SOP_sop-accessioning.md#file-naming-conventions) on how to. This accession should be unique, and thus you shall check what the existing accessions are at the [SOP Index table](../README.md), and other incoming accessions in open PRs (from other SOPs also in development). + +Finally, update secondary documentation of the repository: +- **Update the [SOP Index table](../README.md)** with the new SOP. You can either do it manually, or copy the output of the following snippet (assuming you have the required libraries installed): +```` +python3.8 scripts/sop_index.py sops/ -v 1 +```` +- Add **any new changes to the [CHANGELOG.md](../../CHANGELOG.md)** document. + +#### 7. Create PR, review and merge +| Step identifier | When| Who | +|:----------------|:----|:----| +| ``7`` | After final document was created | OC/SDPC | + +Once the markdown file has the required content in the proper format, it is time to add it to the other SOPs in the repository. To do so, **create a PR against the ``dev`` branch**, containing the new SOP document. + +![PR creation](../../docs/images/GDI-SOP0007_6-PR-image.png) + +Remember to **comment on the GH issue** (i.e., its URL) of the SOP request in this PR, so that it is automatically tracked by GH. + +Members of the OC/SDPC are to be listed as **reviewers** in the PR. Given that the content is not supposed to be modified, this review is merely for format changes that occurred (or should have occurred) between the formal authorization and the final document ([step 6](#6-prepare-final-sop-markdown-document)). + +Furthermore, any **GH workflows** (e.g., SOP linter) **should be addressed** (i.e., assert that they pass) before merging. See in the image below where to look at: + +![Checking GH workflows](../../docs/images/GDI-SOP0007_7-GH-workflows-checks.png) + +Finally, **once the PR was reviewed** and there are no conflicting checks, **merge it** against the ``dev`` branch. At this point, **this SOP is finished**. The new content will stay there until it is to be released and merged with ``main``. + +In case there are merge conflicts, resolve them either through GH's user interface or command line before merging (see [documentation](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-on-github)). + +At this point, **resolve any possible loose ends** (e.g., GH issues, RFC discussions, email threads...). + +### 9. References +| Reference | Description | +|-----------|------------------------------------------------------| +| [1](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md) | European GDI - SOP Charter | +| [2](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_information-service-management.md) | European GDI - Procedures for Information Service Management (ISM) for SOPs | +| [3](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_organisational-roles-and-responsibilities.md) | European GDI - Organisational Roles and Responsibilities (ORR) | +| [4](https://drive.google.com/drive/u/0/folders/131kJLHDk8L2oGgnRzRBT5AR0Ofpbn2qS) | European GDI - GDI's SOP Drafts directory | +| [5](https://gdoc2md.com/) | gdoc2md - Document formatter from Google Document to Markdown| +| [6](https://github.com/GenomicDataInfrastructure/standard-operating-procedures) | European GDI - standard-operating-procedures GH repository| +| [7](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/new?assignees=&labels=new-sop-request%2Cenhancement&projects=&template=new_sop_request.yaml&title=%5BSOP+Request%5D+%3CShort+title%3E) | European GDI - New SOP request GH issue template | +| [8](https://github.com/GenomicDataInfrastructure/rfcs?tab=readme-ov-file) | European GDI - Request for comments (RFC) | \ No newline at end of file diff --git a/sops/node-specific/.gitignore b/sops/node-specific/.gitignore deleted file mode 100644 index 8b13789..0000000 --- a/sops/node-specific/.gitignore +++ /dev/null @@ -1 +0,0 @@ - diff --git a/sops/node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md b/sops/node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md new file mode 100644 index 0000000..ce4f877 --- /dev/null +++ b/sops/node-specific/GDI-SOP0002_NCPs-veto-EDIC-decision.md @@ -0,0 +1,148 @@ +# European GDI - NCPs veto EDIC decision + +| Metadata | Value | +| -- | -- | +| Template SOP number | ``GDI-SOP0002`` | +| Template SOP version | ``v1`` | +| Topic | Helpdesk & operations | +| Template SOP Type | Node-specific SOP | +| GDI Node | | +| Instance version | | + +## Index + +1. [Document History](#1-document-history) +2. [Glossary](#2-glossary) +3. [Roles and Responsibilities](#3-roles-and-responsibilities) +4. [Purpose](#4-purpose) +5. [Scope](#5-scope) +6. [Introduction and Background Information](#6-introduction-and-background-information) +7. [Summary or Context Diagram](#7-summary-or-context-diagram) +8. [Procedure](#8-procedure) +9. [References](#9-references) + +### 1. Document History + +| Template Version | Instance version | Author(s) | Description of changes | Date | +|------------------|------------------|-------------------|--------------------------------------|------------| +| ``v1`` | | Marcos Casado Barbero | Transform document to markdown | ``2024.09.27`` | +| ``v0.1`` | | Dylan Spalding | Comments addressed and first draft of SOP | ``2024.08.14`` | + +### 2. Glossary +Find GDI SOPs common Glossary at the [**charter document**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_charter.md). + +| Abbreviation | Description | +|--------------|----------------------------------------------------| +| 1+MG | 1+ Million Genomes | +| DAC | Data Access Committee | +| EDIC | European Digital Infrastructure Consortium | +| NCP | National Coordination Point | +| SOP | Standard Operating Procedure | + +| Term | Definition | +|---------------|----------------------------------------------------| +| Permit Authority | National body responsible for data access decisions | + +### 3. Roles and Responsibilities +See the qualifications and responsibilities of the roles at the [**Organisational Roles and Responsibilities**](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/docs/GDI-SOP_organisational-roles-and-responsibilities.md) document. + +| Role | Full name | GDI/node role | Organisation | +|------------|-----------------------|-------------------------|--------------| +| Author | Dylan Spalding | Finland / Pillar II co-lead | CSC | +| Author | Marcos Casado Barbero | Task 4.3 Lead | EMBL-EBI | +| Reviewer | Regina Becker | LU / Pillar I co-lead | LNDS | +| **Approver** | Gabriele Rinck | Task 4.3 member | EMBL-EBI | +| **Approver** | Markus Englund | Task 4.3 member | UU / NBIS | + +### 4. Purpose +The purpose of this SOP is to delineate a standardised process for NCPs to review and potentially veto decisions made by the 1+MG EDIC regarding data access requests. This will ensure that data access is consistent with national regulations and ethical standards. + +### 5. Scope +This SOP applies to all NCPs and Permit Authorities involved in the GDI project across participating countries, and relates to the 1+MG Data Governance (See [1+MG Data Governance for Research](https://docs.google.com/document/d/1P_nzGxMXG4CWzqkVbceY2fA3MQ7PEAwW)). It covers the procedures for reviewing and vetoing data access decisions made by the 1+MG DAC. + +### 6. Introduction and Background Information +The GDI project aims to facilitate the integration and accessibility of genomic data across Europe, underpinned by the 1+MG initiative. The 1+MG EDIC is a central legal entity established through a Commission Implementing Decision. It orchestrates the infrastructure implementation of the multi-country project based on the 1+MG Declaration, central to overseeing data access requests. Given the strategic importance and sensitivity of genomic data, the role of NCPs and Permit Authorities are critical in ensuring that access decisions by the 1+MG DAC adhere to both national and European standards. + +### 7. Summary or Context Diagram + +![1+MG DAC, NCP, and Permit Authority processes for approval or veto of a 1+MG recommendation](../../docs/images/GDI-SOP0002_1-summary-diagram.jpg) +**Figure 1**: 1+MG DAC, NCP, and Permit Authority processes for approval or veto of a 1+MG recommendation  + +### 8. Procedure +#### 1. NCP Distribute the application recommendation +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 1 | Receipt of 1+MG DAC decision | NCP | + +Once the 1+MG DAC issues a decision, as the NCP, you should **distribute the application recommendation** to the appropriate 1+MG Data Holders and Permit Authority(ies), who will approve or deny the recommendation. Remind them to respond within 10 working days. + +Depending on the response that you receive from these authorities and data holders: +- If a decision is **not** communicated within 10 working days, move to step **step 4**. +- If a decision is communicated within 10 working days: + - If veto is not exercised in the response you receive, let the 1+MG DAC know and move to **step 4** + - If **veto is exercised** in the response you receive, move to **step 2**. + +#### 2. Communicate justification of veto +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 2 | After step 1 and within 1 working day | NCP | + +If the veto of the 1+MG DAC's recommendation is exercised, as the NCP you shall **communicate a detailed justification to the 1+MG DAC within one day**. In this communication, you shall cite discrepancies between the ethical assessment of the 1+MG DAC and the national decision maker(s). + +Move to **step 3**. + +#### 3. Review of veto by 1+MG DAC +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 3 | After step 2 | 1+MG DAC | + +If veto is exercised on some or all of the relevant data, as the 1+MG DAC, **review the justification provided** by the NCP. + +Depending on the outcome of this evaluation: +- If **veto is accepted**, move to **step 7**. +- If the **justification is not accepted** in full, move to **step 6**. +- If **no veto was exercised**, or the **time lapse** from step 1 is greater than **10 working days**, move to **step 4**. + +#### 4. Inform NCP and User about the decision +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 4 | After step 3 or 6 | 1+MG DAC | + +As the 1+MG DAC, **inform the NCP and User** (whose request was initially recommended) of the decision: approved access request. + +Move to **step 5**. + +#### 5. Inform data subjects +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 5 | After step 4 | NCP | + +As the NCP, **inform the relevant data subjects** of the proposed research **within 1 working day**. + +This concludes the process resulting in final approval of the data access request. + +#### 6. Consensus building process +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 6 | After step 3 | 1+MG DAC | + +If the justification is not accepted in full, as the 1+MG DAC, **initiate the consensus-building process** between the 1+MG DAC and the national data holders and/or permit authorities. + +Depending on the outcome of the process: +- If **consensus is reached**, move to **step 4**. +- If consensus is **not** reached, move to **step 7**. + +#### 7. Inform User and other NCPs of the final decision +| Step identifier | When | Who | +|:----------------|:-------------------------------|:----------| +| 7 | After step 3 or 6 | 1+MG DAC | + +As the 1+MG DAC, **inform** the User and the User’s institution of the **final decision**, as well as other NCPs for whom the decision to veto may be relevant. + +This concludes the process resulting in the rejection of the data access request. + +### 9. References +| Reference | Description | +| -- | -- | +| [1](https://docs.google.com/document/d/1b887HMySeKnJt1pN1pZspGRbEnrJ6Sfk-aAwiskYJK0/edit) | European GDI - SOP Charter (including Glossary) | +| [2](https://docs.google.com/document/d/1P_nzGxMXG4CWzqkVbceY2fA3MQ7PEAwW/edit) | 1+MG Data Governance for Research |