Stage1 data cleaning

[ZoeMZou]

PR: Refinements for Script Structure & Data Processing

(Based on discussions on 28th January 2025)

🔹 1. Structural Changes
📌 1.1 Naming Conventions

• Removed _stage1 from scripts to improve clarity and consistency.

• The old stage1_data_cleaning.R script was handling:

  1. Setting reference levels for categorical variables
  2. Applying quality assurance
  3. Applying inclusion/exclusion criteria

• The new structure separates these tasks into three dedicated functions:

  • fn-ref.R → Sets reference levels
  • fn-qa.R → Applies quality assurance
  • fn-inex.R→ Applies inclusion/exclusion criteria

• The main script data_cleaning.R calls these functions, ensuring: Project-specific edits are made in function scripts and data_cleaning.R remains clear and workflow-focused.

📌 1.2 Output Dataset Naming

• The cleaned dataset output is now: input_{cohort}.rds

• The preprocessed dataset output is now:
  input_{cohort}.rds → input_{cohort}_0.rds (to avoid YAML conflicts)

🔹 2. fn-ref.R — Reference Level Settings

📌 2.1 Data Formatting Fixes

• Some categorical variables were not factors, and some numeric variables were not numeric. I added code back to preprocess_data.R to ensure correct formatting:
  🔗 See fix in preprocess_data.R

📌 2.2 Code Deletions from Old Repo
Index Date: Already correct—no correction needed.
🔗 Old repo reference
Deprivation (IMD): No need for re-categorization; new repo includes IMD (1–5) directly.
🔗 Old repo reference

📌 2.3 Sex Category Adjustments
Old repo: Only male (M) and female (F).
New repo: Four levels → female, male, intersex, unknown.
Fix:
Set non-male/non-female values to missing. Retain three levels: female, male, unknown.

🔹 3. fn-qa.R — Quality Assurance Fixes

📌 3.1 Fixing "Date of Death" Message
The original message incorrectly stated that NA values were being removed.
Fix: Message now correctly states:
"Quality assurance: Date of death is invalid (on or before 1/1/1900 or after current date)"

🔗 Old repo reference

📌 3.2 Fixing Missing Data Handling
Issue: Patients with missing sex were marked as missing for the entire record, which was unintended.
Fix: The new script preserves records and only marks sex as missing.
🔗 Updated QA code

📌 3.3 Update to qa_bin_prostate_cancer
Issue:
The current method introduced missing values due to logic operations with NA values. NA values were assigned to alive patients who had no recorded prostate cancer diagnosis but were affected by the logic.
The check for cause of death due to prostate cancer was unnecessary. Patients who died from prostate cancer before the index date are already excluded later in the study. Removing this check ensures all alive participants receive a valid classification (TRUE/FALSE) rather than NA.
🔗 Updated method

🔹 4. fn-inex.R — Inclusion/Exclusion Criteria Fixes

📌 4.1 Code Corrections
The new repo includes inex_bin_alive, so we can use it directly instead of recalculating.
🔗 Old repo reference
🔗 New implementation

📌 4.2 Variable Renaming
death_date → cens_date_death
has_follow_up_previous_6months → inex_bin_6m_reg
deregistration_date → cens_date_dereg

📌 4.3 Removal of Unnecessary Code
Issue: The old repo had an unnecessary step for active registration at index.
Fix:
Removed redundant filtering for active registration, as inex_bin_6m_reg already ensures this. I rewrote the print message to reflect this.
🔗 Old repo reference
🔗 Updated function

📌 4.4 Fix for Exclusion Criteria in the Vax Cohort
Issue: Errors in the vax cohort exclusion criteria due to incorrect variable types.
Fix:
Redefined data types to ensure they are numeric before calculating vax_mixed.
🔗 Updated fix

🔹 5. Other Updates

📌 5.1 Cause of Death Extraction Simplified
A more efficient method for extracting cause of death has become available in OpenSAFELY.
Fix: Updated our code to align with the latest OpenSAFELY documentation.
🔗 Updated function

[ZoeMZou]

Hi @venexia,

This PR is now ready for your review. It runs successfully locally. Since the revisions in this script are quite detailed, I’ve listed the key changes at the beginning of the PR to make it easier to navigate and review.

Thank you very much.

Best wishes,
Zoe

[venexia]

Hi @ZoeMZou. Great work - well done! Just a few things to address:

• Some of the QA criteria are out of date - these should be updated to match the current protocol
• Some of the code for QA and inex could be simplified by using dplyr::case_when and/or removing double negatives
• The naming structure for the files through the pipeline is not very intuitive - we should agree an approach for this when we next meet the post-covid-events team
• The currently generated dummy data does not have the variation to test this code properly - we should resolve this so we can double check we are removing the correct people (I did this for the first QA criterion but haven't gone through each criteria in this review)
• The formatting of the scripts is inconsistent and should be reviewed - we can automate this using linting but I need to look up how to set it up

[venexia]

I would make this something like:

print('Quality assurance: Date of death is invalid (on or before earliest expected date or after current date)')

It is only on or before 1/1/1900 when study_dates$earliest_expec=1/1/1900.

[venexia]

Though on further reading of the protocol, I notice this criteria is listed as 'Remove individuals whose date of death is after today' and does not mention invalid death dates.

[ZoeMZou]

Decision from today's meeting (February 11, 2025): The repository will be revised to ensure consistency with the protocol regarding inclusion criteria and quality assurance.

[venexia]

I am in two minds about including | is.na(input$cov_cat_sex) - the people with missing sex might include, for example, pregnancy/birth codes for men. This applies to all sex specific QA criteria.

[ZoeMZou]

Yes, individuals with missing sex should be included here. Only pregnancy/birth codes for men (HRT or COCP meds for men; prostate cancer codes for women) are excluded. We will handle the exclusion of individuals with missing sex in fn-inex.R to ensure accurate counts. Excluding them here would lead to inconsistencies in the reported number of missing sex cases.

[venexia]

I think we could make this more readable using dplyr::case_when().

[ZoeMZou]

We will use this PR to merge preprocess and data_cleaning for better efficiency. Key steps to follow:

• Apply Inclusion Criteria First: Ensure inclusion criteria are applied before quality assurance checks.

• Improve Readability: Use case_when for filtering datasets to enhance clarity.

• Step-by-Step Validation: Check the dataset at each stage to confirm that the code is functioning as expected.

[venexia]

We should discuss whether the dummy data is appropriate to test these scripts when we next meet. I have everyone born in the year 1975 in my dataset so nobody matches the first QA criterion and it is hard to tell if it is working without modifying the dummy data (which we could do).

[venexia]

Is this intended as a permanent change or was it for testing purposes? The life course of the file is now input_cohort.csv.gz > input_cohort_0.rds > input_cohort.rds, which is a little confusing as you expect the first and third file to be the same?

[ZoeMZou]

Decision from today's meeting (11th Feb 2025):

• We will not create an intermediate dataset from preprocess.

• After merging preprocess and data_cleaning, the only output will be input_cohort_clean.rds.