Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Update mantis 3rd party dependency specifications #349

Draft
wants to merge 26 commits into
base: master
Choose a base branch
from

Conversation

mabelbot
Copy link

@mabelbot mabelbot commented Feb 28, 2023

Context

Changes made by module

  1. mantis-common: jctoolsVersion : "1.+" moved to root build.gradle ext.versions (only occurrence), nettyVersion = '4.1.34.Final' moved to root build.gradle ext.versions (bumped, requires updates to other submodules later), snappyVersion : "1.1.+" moved to root build.gradle ext.versions (only occurrence). All other dependencies made consistent.
  2. mantis-connectors:mantis-connector-iceberg: bumped hadoop '2.7.3' to "2.7.7", icebergVersion : "0.14.+" moved to root build.gradle ext.versions (only occurrence), avroVersion : "1.10.2" moved to root build.gradle ext.versions (only occurrence), parquetVersion : "1.12.0" moved to root build.gradle ext.versions (only occurrence).
  3. mantis-connectors:mantis-connector-job: gsonVersion : "2.8.+" moved to root build.gradle ext.versions (only occurrence)
  4. mantis-connectors:mantis-connector-kafka: Was using spectatorVersion = '0.82.+', so new variable has been given for major version (0) in root build.gradle '0.134.0' to match other modules using v0. Was using archaiusVersion = '2.3.+' whereas all other submodules are using latest.release so it has been bumped to use the latest.version specified in root build.gradle. kafkaVersion = '2.2.+' moved to root build.gradle ext.versions (only occurrence).

mantis-control-plane:mantis-control-plane-core:

  1. Moved to root build.gradle ext.versions: configMagicVersion = "0.11" is shared by the only 2 submodules that have it specified, so this has been moved to root build.gradle ext.versions, same with cliParserVersion = '1.1.1' and mesosVersion = '1.7.2'. hdrHistogramVersion = '2.+', jodaTimeVersion = '2.+', jsonVersion = '20180813'
  2. Major version new variable created: None
  3. Minor/patch bump: curatorVersion = '2.11.0' doesn't seem to be used in this build.gradle but it has been bumped to match the highest version = '2.12.+' (minor version)

mantis-control-plane:mantis-control-plane-server:

  1. Moved to root build.gradle ext.versions: akkaVersion = '2.6.15', akkaHttpVersion = '10.2.7', fenzoVersion = '0.13.8', rxJavaReactiveStreamsVersion = '1.+', scalaBinaryVersion = '2.12'
  2. Major version new variable created: None
  3. Minor/patch bump: curatorVersion = '2.11.0' to 2.12.+, testngVersion = '6.+' bumped to testngVersion = '6.14.+'

mantis-examples

  1. mantis-examples-groupby-sample: bump spectator-api:1.3.+ to use latest.release. fix guava under configurations.all.
  2. mantis-examples-jobconnector-sample: move mantisConnectorsVersion = '1.2.5' to root build.gradle and introduce new variable mantisConnectorJob.
  3. mantis-examples-mantis-publish-sample: bump spectator 0 spectatorVersion = '0.96.0' to 0.134.0 (tbd making 0.134.0 a variable), create variable spectatorPlugin0, move make mockneatVersion = '0.4.8' a variable in root build.gradle.
  4. mantis-examples-mantis-publish-web-sample: create variable javaxServletApi, bump spectator 0 spectatorVersion = '0.96.0' to 0.134.0 (tbd making 0.134.0 a variable), create variable guiceServlet
  5. mantis-examples-sine-function: spectator-api 1.3.+ bumped to latest.release
  6. mantis-examples-synthetic-sourcejob: mqlVersion = '3.2.2' bumped to 3.4.+, mockNeatVersion = '0.3.7' bumped to 0.4.8, mockito-all:1.9.5 made into own variable since major version
  7. mantis-examples-twitter-sample: move httpComponentsVersion = '4.5.9' to root build.gradle, spectator-api 1.3.+ bumped to latest.release, create new variable hbcCore
  8. mantis-examples-wordcount: spectator-api 1.3.+ bumped to latest.release

mantis-network

  1. Moved to root build.gradle ext.versions: mqlVersion = '3.4.+' (requires bumping of minor version for other modules that have this dependency).
  2. Major version new variable created: None
  3. Minor/patch bump: nettyVersion = '4.1.17.Final' to "4.1.34.Final"
  4. Library vars added to root build.gradle: mqlVersion, mqlJvm, nettyHandler
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

:mantis-publish:mantis-publish-core

  1. Moved to root build.gradle ext.versions/already there from above refactors: archaiusVersion = 'latest.release' (was already present from previous refactor), versionMockito = '2.18.+' - existing root build.gradle variable bumped from 2.0.+ to 2.18.+, root build.gradle variable for v0 bumped to spectatorVersion = '0.134.0' (note if mantis-publish-core can be bumped to v1.3+ we can delete some variables, tbd)
  2. Major version new variable created: None
  3. Minor/patch bump: mqlVersion = '3.2.2' bumped to '3.4.+'
  4. Library vars added to root build.gradle:
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

:mantis-publish:mantis-publish-netty

  1. Moved to root build.gradle ext.versions/already there from above refactors: spectatorVersion = 'latest.release' was found in the ext, thus I changed the root build.gradle to bump from its current 1.3.+ to latest.release.
  2. Major version new variable created: None
  3. Minor/patch bump: spectatorExtIpc0
  4. Library vars added to root build.gradle: nettyAll, spectatorExtIpc
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

:mantis-publish:mantis-publish-netty-guice

  1. Moved to root build.gradle ext.versions/already there from above refactors: archaiusVersion = 'latest.release' (already there), guiceVersion = '4.2.2'
  2. Major version new variable created: None
  3. Minor/patch bump: spectator version 0.96.0 bumped to use version already in root build.gradle 0.134.0
  4. Library vars added to root build.gradle: archaius2Guice, guice, spectatorPlugin
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

mantis-remote-observable

  1. Moved to root build.gradle ext.versions/already there from above refactors: rxJavaMathVersion = '0.20.6'
  2. Major version new variable created: None
  3. Minor/patch bump: nettyVersion = '4.1.17.Final' bumped to use version already in root build.gradle "4.1.34.Final"
  4. Library vars added to root build.gradle: rxJavaMath
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

mantis-server:mantis-server-worker - tbd if need to fix exclude group lines if there is no version

  1. Moved to root build.gradle ext.versions/already there from above refactors: mantisRxControlVersion = '1.3.+', mesosVersion = '1.7.2' (already there), httpComponentsVersion = '4.5.6' (bumped to 4.5.9)
  2. Major version new variable created: None
  3. Minor/patch bump: curatorVersion = '2.11.0' bumped to use version already in root build.gradle 2.12.+'
  4. Library vars added to root build.gradle: httpClient, sketchesCore, moa, hamcrestAll
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

mantis-server:mantis-server-worker-client
All fixes made with existing root build.gradle vars and/or without deletions to ext

mantis-shaded
I grouped some but not all jackson dependencies (due to the comments), but tbd if they need to split back out due to Issue.

  1. Moved to root build.gradle ext.versions/already there from above refactors: jacksonVersion = '2.12.+', guavaFailureAccessVersion = '1.0.1', guavaVersion = '31.1-jre', curatorVersion (already there), zookeeperVersion = '3.4.+', jlineVersion = '0.9.94'
  2. Major version new variable created: this is the only module that is using nettyVersion 3 so I created a new variable for v3 (although tbd, if the version numbering should be named more consistently)
  3. Minor/patch bump:
  4. Library vars added to root build.gradle: jackson (see note above), guava, failureaccess, curator (grouped), zookeeper, jline, netty
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

mantis-source-jobs:mantis-source-job-kafka

  1. Moved to root build.gradle ext.versions/already there from above refactors:
  2. Major version new variable created: None
  3. Minor/patch bump: bump mqlVersion = '3.2.2' bumped to use version already in root build.gradle '3.4.+'
  4. Library vars added to root build.gradle:
  5. All other fixes made with existing root build.gradle vars and/or without deletions to ext

No fixes to the following submodules that I observed seem to be already specified properly (list will update if version changes due to ext variables are necessary):

  • mantis-client
  • mantis-common-serde
  • mantis-connector-publish
  • mantis-control-plane:mantis-control-plane-client
  • mantis-discovery-proto
  • mantis-examples:mantis-examples-core
  • mantis-runtime
  • mantis-runtime-loader
  • mantis-server:mantis-server-agent
  • mantis-source-jobs:mantis-source-job-publish

Note: ./gradlew build does not compile correctly (prior to any changes) when compiled locally, however I have checked with ../../gradlew execute on mantis-examples which is able to build (@sundargates is aware of issue)

Checklist

  • ./gradlew build compiles code correctly
  • Added new tests where applicable
  • ./gradlew test passes all tests
  • Extended README or added javadocs where applicable
  • tbd - update hardcoded versions in root build.gradle

Refactor Checklist

  • 'mantis-client'
  • 'mantis-common'
  • 'mantis-common-serde'
  • 'mantis-connectors:mantis-connector-iceberg'
  • 'mantis-connectors:mantis-connector-job'
  • 'mantis-connectors:mantis-connector-kafka'
  • 'mantis-connectors:mantis-connector-publish'
  • 'mantis-control-plane:mantis-control-plane-client'
  • 'mantis-control-plane:mantis-control-plane-core'
  • 'mantis-control-plane:mantis-control-plane-server'
  • 'mantis-discovery-proto'
  • 'mantis-examples:mantis-examples-core'
  • 'mantis-examples:mantis-examples-groupby-sample'
  • 'mantis-examples:mantis-examples-jobconnector-sample'
  • 'mantis-examples:mantis-examples-mantis-publish-sample'
  • 'mantis-examples:mantis-examples-mantis-publish-web-sample'
  • 'mantis-examples:mantis-examples-sine-function'
  • 'mantis-examples:mantis-examples-synthetic-sourcejob'
  • 'mantis-examples:mantis-examples-twitter-sample'
  • 'mantis-examples:mantis-examples-wordcount'
  • 'mantis-network'
  • ':mantis-publish:mantis-publish-core'
  • ':mantis-publish:mantis-publish-netty'
  • ':mantis-publish:mantis-publish-netty-guice'
  • 'mantis-remote-observable'
  • 'mantis-runtime'
  • 'mantis-runtime-loader'
  • 'mantis-server:mantis-server-agent'
  • 'mantis-server:mantis-server-worker'
  • 'mantis-server:mantis-server-worker-client'
  • 'mantis-shaded'
  • 'mantis-source-jobs:mantis-source-job-kafka'
  • 'mantis-source-jobs:mantis-source-job-publish'

chore: update 3rd party dependency specifications for mantis-common and corresponding changes to root build.gradle (issue Netflix#345)
api "io.netty:netty-codec-http:$nettyVersion"
api "io.netty:netty-buffer:$nettyVersion"
api libraries.nettyCodec
api libraries.nettyBuffer
api group: 'io.netty', name: 'netty-transport-native-epoll', classifier: 'linux-x86_64', version: nettyVersion
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would recommend moving this to libraries.nettyEpoll as well even if it's a little tricky.
It would look like this - io.netty:netty-transport-native-epoll:4.1.17.Final:linux-x86_64

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, will do ASAP! Thank you.

@@ -24,11 +24,11 @@ ext {

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ext variable can be safely removed with this change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a concern I had earlier, there were different versions of the same dependencies across different modules (so I kept this part that specifies the specific versions). Would it be better to standardize the versions and move a single version to the root build.gradle's ext.versions?

Copy link
Collaborator

@hmitnflx hmitnflx Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, that'd be desired to realize the full benefits of this work.
The different versions would make it tricky and here's a good starting tip:

  1. For major version difference, Create a new library variable.
  2. For minor and patch, try to bump to the latest version for all subprojects (one variable per library and see if builds correctly).

You could call out the places where you are bumping the minor and patch version in the PR for an easier review.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you! As some other GSoC applicants are having trouble with builds failing (including the Mantis tutorial) before we made any changes, I brought this up to Sundaram yesterday and iirc he said that right now there wasn't a good way for us to validate whether these "good first issue" PRs aren't going to break something - but that it will be fixed by end of March. If needed, I can provide the exact error on Discord so maybe we can get ./gradlew build to work?

The only thing that succeeds at building correctly locally is the synthetic-sourcejob example, since I found a workaround that can allow it to build temporarily. I have been using that to test but it's not ideal.

@mabelbot mabelbot changed the title Update mantis-common 3rd party dependency specifications [WIP] Update mantis 3rd party dependency specifications Feb 28, 2023
chore: update netty-transport-native-epoll dependency specification to libraries.nettyEpoll in mantis-common and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request February 28, 2023 22:43 — with GitHub Actions Failure
refactor mantis-common build.gradle by moving ext versions to root build.gradle and removing ext variable (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request February 28, 2023 22:55 — with GitHub Actions Failure
first pass at refactor 3rd party dependency specifications for mantis-connector-iceberg and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 1, 2023 01:38 — with GitHub Actions Failure
first pass at refactor 3rd party dependency specifications for mantis-connector-job and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 1, 2023 02:10 — with GitHub Actions Failure
first pass at refactor 3rd party dependency specifications for mantis-connector-kafka and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 1, 2023 02:26 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 1, 2023 02:44 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 1, 2023 03:12 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 1, 2023 03:23 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 00:14 — with GitHub Actions Failure
mabelbot added 2 commits March 1, 2023 16:25
first pass at refactor 3rd party dependency specifications for mantis-publish-netty-guice and corresponding changes to root build.gradle. (issue Netflix#345)
first pass at refactor 3rd party dependency specifications for mantis-remote-observable and corresponding changes to root build.gradle. (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 01:10 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 01:15 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 01:29 — with GitHub Actions Failure
mabelbot added 2 commits March 1, 2023 17:33
first pass at refactor 3rd party dependency specifications for mantis-server-worker and corresponding changes to root build.gradle. tbd for lines 31,32 - exclude (issue Netflix#345)
first pass at refactor 3rd party dependency specifications for mantis-server-worker-client and corresponding changes to root build.gradle. (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 01:34 — with GitHub Actions Failure
first pass at refactor 3rd party dependency specifications for mantis-shaded and corresponding changes to root build.gradle. tbd - issue Netflix#194 and grouping (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 01:58 — with GitHub Actions Failure
first pass at refactor 3rd party dependency specifications for mantis-source-job-kafka. (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 02:02 — with GitHub Actions Failure
mabelbot added 2 commits March 1, 2023 18:07
…ations

first pass at refactor 3rd party dependency specifications for mantis-examples-groupby-sample. (issue Netflix#345)
…cifications

first pass at refactor 3rd party dependency specifications for mantis-examples-jobconnector-sample and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 02:17 — with GitHub Actions Failure
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 02:25 — with GitHub Actions Failure
…pecifications

first pass at refactor 3rd party dependency specifications for mantis-examples-mantis-publish-sample and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 02:30 — with GitHub Actions Failure
mabelbot added 3 commits March 1, 2023 18:37
…cy specifications

first pass at refactor 3rd party dependency specifications for mantis-examples-mantis-publish-web-sample and corresponding changes to root build.gradle (issue Netflix#345)
…tions

first pass at refactor 3rd party dependency specifications for mantis-examples-sine-function and corresponding changes to root build.gradle (issue Netflix#345)
…cifications

first pass at refactor 3rd party dependency specifications for mantis-examples-synthetic-sourcejob and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot marked this pull request as draft March 2, 2023 02:56
mabelbot added 2 commits March 1, 2023 19:00
…ations

first pass at refactor 3rd party dependency specifications for mantis-examples-twitter-sample and corresponding changes to root build.gradle (issue Netflix#345)
first pass at refactor 3rd party dependency specifications for mantis-examples-wordcount and corresponding changes to root build.gradle (issue Netflix#345)
@mabelbot mabelbot had a problem deploying to Integrate Pull Request March 2, 2023 03:01 — with GitHub Actions Failure
@mabelbot
Copy link
Author

mabelbot commented Mar 2, 2023

Update: Finished the bulk of the refactoring, still going back over to make sure I didn't miss anything, also I had a few questions/tbd changes which I will leave comments about here soon (before requesting review again)

@sundargates
Copy link
Contributor

@mabelbot Is this ready for review?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Specify all 3rdparty dependencies in the root build.gradle file
3 participants