New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor tweetset_loader.py for greater flexibility #134

Open

dolsysmith opened this issue Aug 3, 2021 · 1 comment

Assignees

Labels

enhancement medium effort level

Milestone

Contributor

dolsysmith commented Aug 3, 2021

Move from argparse to click
Move less-commonly used options to config
Add logic for creating extracts without loading

dolsysmith added this to the 2.2 milestone

dolsysmith self-assigned this

dolsysmith changed the title ~~Refactor tweetset_loader.py~~ Refactor tweetset_loader.py for greater flexibility

dolsysmith added enhancement medium effort level labels

lwrubel modified the milestones: 2.2, 2.3

Collaborator

lwrubel commented Sep 13, 2021 •

edited

Loading

Add option to specify concatenating JSON files up to a size.

Also move parse_size() from spark_utils.py to utils.py since it will need to be called to supply to_size value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment