Code

T128 refactor loader #128, #122#138

Code

Merged

dolsysmith merged 28 commits intomastergwu-libraries/TweetSets:masterfrom t128-refactor-loadergwu-libraries/TweetSets:t128-refactor-loaderCopy head branch name to clipboardAug 27, 2021

+11,678-16

Commits

Commits on Aug 19, 2021

• Added code for using Spark DataFrame API

  dolsysmith

  committed
  9a05868

  Copy full SHA for 9a05868

• Fixed pyspark version

  dolsysmith

  committed
  5ef31cd

  Copy full SHA for 5ef31cd

• Adding schema and SQL files

  dolsysmith

  committed
  ceffc9f

  Copy full SHA for ceffc9f

• Upgraded to twarc 1.12 for column name fix

  dolsysmith

  committed
  0723af2

  Copy full SHA for 0723af2

• Working with Spark 3/Java 11/Python 3.8

  dolsysmith

  committed
  57d1ad7

  Copy full SHA for 57d1ad7

• Updated README

  dolsysmith

  committed
  0fab820

  Copy full SHA for 0fab820

• Adding headers to files

  dolsysmith

  committed
  f58028a

  Copy full SHA for f58028a

• Using elasticsearch instead of elasticsearch_dsl to trigger dependency install for Spark submit

  dolsysmith

  committed
  1d53efc

  Copy full SHA for 1d53efc

• Fixed issues in creation of CSV

  dolsysmith

  committed
  e32c6b1

  Copy full SHA for e32c6b1

• Updating Spark SQL to reflect json2csv 1.12

  dolsysmith

  committed
  1311d8a

  Copy full SHA for 1311d8a

• Mapping to json2csv 1.12

  dolsysmith

  committed
  7e012a3

  Copy full SHA for 7e012a3

• Updating SQL for json2csv 1.12

  dolsysmith

  committed
  09a36d7

  Copy full SHA for 09a36d7

• Fixed syntax errors

  dolsysmith

  committed
  412ddbb

  Copy full SHA for 412ddbb

• Fixed logic to capture all URL's

  dolsysmith

  committed
  c5e76ba

  Copy full SHA for c5e76ba

• Fixed logic to capture all URL's

  dolsysmith

  committed
  0ff89e9

  Copy full SHA for 0ff89e9

• Removing extra comma

  dolsysmith

  committed
  751ea73

  Copy full SHA for 751ea73

• Hashtags from retweeted_status

  dolsysmith

  committed
  66b3889

  Copy full SHA for 66b3889

• Order of hashtag fields

  dolsysmith

  committed
  f21d5ed

  Copy full SHA for f21d5ed

• Added readable version of schema

  dolsysmith

  committed
  0739b6f

  Copy full SHA for 0739b6f

Commits on Aug 20, 2021

• Changing JSON handling for tweet data

  dolsysmith

  committed
  f7bab20

  Copy full SHA for f7bab20

• Fixing indentation

  dolsysmith

  committed
  3ee1211

  Copy full SHA for 3ee1211

• textFile method requires string, not list

  dolsysmith

  committed
  52cd0e6

  Copy full SHA for 52cd0e6

• Added support for full_text element to schema

  dolsysmith

  committed
  a42cea2

  Copy full SHA for a42cea2

Commits on Aug 24, 2021

• JSON of tweet now correctly indexed; removed extraneous columns from ES mapping

  dolsysmith

  committed
  d81c832

  Copy full SHA for d81c832

Commits on Aug 25, 2021

• Refactored to avoid join

  dolsysmith

  committed
  9148842

  Copy full SHA for 9148842

• Fixed typo

  dolsysmith

  committed
  fe401d0

  Copy full SHA for fe401d0

Commits on Aug 27, 2021

• Using RDD for ES load; DataFrame for extracts; no JSON extract

  dolsysmith

  committed
  fc33807

  Copy full SHA for fc33807

• Removed extraneous import

  dolsysmith

  committed
  e845cf5

  Copy full SHA for e845cf5