Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider breaking harvest DAG into separate DAGs #116

Open
peetucket opened this issue Jan 22, 2025 · 0 comments
Open

Consider breaking harvest DAG into separate DAGs #116

peetucket opened this issue Jan 22, 2025 · 0 comments

Comments

@peetucket
Copy link
Member

  • One desirable reason for breaking up the DAG is to allow harvesting to take place independently of how the harvested data is joined together, and subsetted to create different reports.
  • It might be useful to express the user data collection (currently a manual CSV drop) as a separate workflow?
  • It also might be feasible to treat the unique-list-of-DOI dataset generation as a separate workflow?
  • These independent workflows would still have dependencies between them, and it might still be useful to be able to kick off all of them at once still?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@peetucket