HADOOP-19405. hadoop-aws and hadoop-azure tests have stopped running. #7335
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@ahmarsuhail @steveloughran The impact I can think of is due to the introduction of this dependency. I am currently working on the upgrade from JUnit 4 to JUnit 5, but it will take some time to complete. |
|
@ahmarsuhail @steveloughran If we add the JUnit 4 dependency in these two modules, the unit tests will run successfully. Can we temporarily use this workaround to resolve the issue until the JUnit 5 upgrade is completed?
|
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
@anujmodi2021 After adding the JUnit 4 dependency, the unit tests can be successfully executed. |
slfan1989
changed the title
Thanks @slfan1989 this indeed is working for us. |
|
Others distcp +probably more. Given junit 4 and 5 can apparently coexist, why not just add it everywhere? |
@steveloughran I’m continuing to follow up and improve this PR by adding the necessary configurations for the modules that haven’t been upgraded to JUnit 5 yet. Currently, I’m preparing a PR to upgrade from JUnit 4 to JUnit 5, and I plan to submit the code for the hadoop-common module. After submission, I’d like to discuss a few specific issues with everyone to ensure a smooth upgrade process and address any potential compatibility problems. |
steveloughran
approved these changes
Jan 30, 2025
There was a problem hiding this comment.
+1 .
Let's merge as soon as the build is done, unless it is very unhappy. If there are any other modules being skipped -they can be fixed as they are found
ran an s3a test against s3 london -the tests ran and passed
with both a mvn test and mvn verify invocation.
On abfs same outcome: test and verify both happy.
I've kicked off a full parallel run, all tests are going except for some test failures which I'm assuming are unrelated as it is now running the tests to find these :)
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization.testOpenFileAuthorized Time elapsed: 0.01 s <<< ERROR!
java.lang.NullPointerException
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization.runTest(ITestAzureBlobFileSystemAuthorization.java:280)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAuthorization.testOpenFileAuthorized(ITestAzureBlobFileSystemAuthorization.java:139)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
💔 -1 overall
This message was automatically generated. |
|
There are still some issues with the javadoc and unit tests, and I will continue to follow up and fix them. |
anujmodi2021
approved these changes
Feb 1, 2025
|
After the merge of HADOOP-15984, we noticed that some unit tests stopped running. To resolve this, we found a solution that allows the unit tests to run again. During the process of completing this PR, we validated its effectiveness for modules like hadoop-aws and hadoop-azure. Currently, this PR contains some unit test and Javadoc errors, which are primarily caused by other components such as MapReduce and YARN. To address these issues step by step, I plan to split this PR into multiple independent PRs, each focusing on resolving issues in specific modules. I will split this PR into 3 separate PRs:
|
74 tasks
|
💔 -1 overall
This message was automatically generated. |
4 tasks
|
💔 -1 overall
This message was automatically generated. |
slfan1989
force-pushed
the
HADOOP-19405
branch
from
76b878c to
629debf
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
cnauroth
approved these changes
Feb 2, 2025
|
@steveloughran @anujmodi2021 @cnauroth Thank you very much for helping to review this PR! If there are no further comments, I will merge it tomorrow. |
|
This PR is a collaborative effort, and I've added @steveloughran @anujmodi2021, and @cnauroth as co-authors. Thank you for your review and validation! Merged into trunk. |
|
Awesome, I appreciate your contribution @slfan1989 . |
|
thanks |
|
This is awesome. Thanks for the efforts @slfan1989 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: HADOOP-19405. hadoop-aws tests have stopped running.
How was this patch tested?
Junit Test &
mvn clean testFor code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?