Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial pipeline implementation #14

Merged
merged 4 commits into from
Feb 6, 2025
Merged

Initial pipeline implementation #14

merged 4 commits into from
Feb 6, 2025

Conversation

tillywoodfield
Copy link
Collaborator

@tillywoodfield tillywoodfield commented Feb 4, 2025

For each registered dataset

  1. Download dataset
  2. Validate dataset
  3. Save valid dataset to JSON file

Closes #2

@tillywoodfield tillywoodfield requested a review from Bjwebb February 5, 2025 10:13
Base automatically changed from setup to live February 5, 2025 12:47
Copy link
Member

@Bjwebb Bjwebb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we'd raise a custom exception class, rather than Exception directly, but I don't think that needs to block merging this.

@tillywoodfield
Copy link
Collaborator Author

I think ideally we'd raise a custom exception class, rather than Exception directly, but I don't think that needs to block merging this.

@Bjwebb good point, I've been using Exception for the next few PRs as well, but I'll make a note to rework these into some custom exceptions when I work on the following issues which are related to error handling: #6, #9.

@tillywoodfield tillywoodfield merged commit 38fc1e6 into live Feb 6, 2025
1 check passed
@tillywoodfield tillywoodfield deleted the 2-file-ingestion branch February 6, 2025 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement basic file ingestion
2 participants