Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor tweetset_loader.py to use Spark DataFrame API #128

Closed
dolsysmith opened this issue Jul 28, 2021 · 1 comment · Fixed by #138
Closed

Refactor tweetset_loader.py to use Spark DataFrame API #128

dolsysmith opened this issue Jul 28, 2021 · 1 comment · Fixed by #138
Assignees
@dolsysmith
Labels
enhancement major effort level pull request
Milestone
2.2

Comments

@dolsysmith
Copy link
Contributor

References #122, #84, #117

  • Upgrading to Spark 2.4
  • Using Spark SQL to transform Tweet JSON for Elasticsearch indexing
  • Using Spark SQL to create full extracts at time of load
@dolsysmith dolsysmith added this to the 2.2 milestone Jul 28, 2021
@dolsysmith
Copy link
Contributor Author

Upgrading to Spark 3 in order to accommodate Python 3.8.

@dolsysmith dolsysmith self-assigned this Aug 6, 2021
This was referenced Aug 6, 2021
Closed
Closed
@dolsysmith dolsysmith linked a pull request Aug 13, 2021 that will close this issue
T128 refactor loader #128, #122 #138
Merged
@lwrubel lwrubel mentioned this issue Aug 26, 2021
Closed
dolsysmith added a commit that referenced this issue Aug 27, 2021
@dolsysmith
Merge pull request #138 from gwu-libraries/t128-refactor-loader
5229297 
T128 refactor loader #128, #122
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dolsysmith dolsysmith

Labels
enhancement major effort level pull request
Projects
None yet
Milestone
2.2
Development

Successfully merging a pull request may close this issue.

T128 refactor loader #128, #122 gwu-libraries/TweetSets
1 participant
@dolsysmith