Task: cleanup new pipeline

[nlebovits]

Pipeline Outstanding Items Checklist

• Include z-scored density values for KDE outputs (not just percentiles).
  • This ensures continuous, interpretable values for ML and community partners.
• Set the number of workers dynamically based on machine cores for parallel processing.
  • Avoid hardcoding values based on a single machine's configuration.
• Add data documentation:
  • Create a DAG.
  • Create a data dictionary.
• Add a "create date" field for the final dataset.
• Run ruff formatting and linting, and address all errors.
• Ensure all code in the pipeline has accurate:
  • Docstrings.
  • Typing.
• Develop a better method for implementing data QC.

[nlebovits]

In data dictionary, add a known_issues column with notes on known data quality/source issues (e.g., missing polygons in pwd_parcels.

[nlebovits]

Also update the development rank calculation to use the standard KDE function rather than the census bg bins (plus z-score and percentile cuts).

[nlebovits]

• Consider updating guncrime priority to things 2 std above the mean--or will that be uninterpretable?

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity.

[CodeWritingCow]

@nlebovits can we close this ticket?