HADOOP-19289. Protobuf-Java 3.25.5

[pjfanning]

Description of PR

HADOOP-19289

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 18s		trunk passed
+1 💚	compile	9m 6s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	8m 19s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	mvnsite	15m 8s		trunk passed
+1 💚	javadoc	5m 42s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	5m 13s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	30m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	17m 50s		the patch passed
+1 💚	compile	8m 45s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	8m 45s		the patch passed
+1 💚	compile	8m 19s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	8m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	9m 6s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	5m 38s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	5m 16s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	31m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	583m 0s	/patch-unit-root.txt	root in the patch failed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		759m 30s

Reason	Tests
Unreaped Processes	root:2
Failed junit tests	hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/1/artifact/out/Dockerfile
GITHUB PR	#7072
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 74a50cbe0d09 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0f0398c
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Unreaped Processes Log	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/1/artifact/out/patch-unit-root-reaper.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/1/testReport/
Max. process+thread count	4551 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

Let's do a new release of the third party lib before this

[pjfanning]

Let's do a new release of the third party lib before this

With hadoop-thirdparty, can we call the next release 1.3.1? The upgrade from 1.3.0 isn't big. The current version is 1.4.0-SNAPSHOT.

[steveloughran]

ok

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	6m 23s		Maven dependency ordering for branch
+1 💚	mvninstall	18m 34s		trunk passed
+1 💚	compile	8m 51s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	8m 2s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	mvnsite	12m 20s		trunk passed
+1 💚	javadoc	5m 24s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	4m 55s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	shadedclient	29m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	17m 24s		the patch passed
+1 💚	compile	8m 45s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	8m 45s		the patch passed
+1 💚	compile	8m 4s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	8m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	8m 27s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	5m 31s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	4m 54s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	shadedclient	30m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	291m 34s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		451m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission
	hadoop.hdfs.TestDecommissionWithBackoffMonitor
	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.server.federation.router.TestRouterFaultTolerant

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/2/artifact/out/Dockerfile
GITHUB PR	#7072
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 6408266f8bc1 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / b911afc
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/2/testReport/
Max. process+thread count	4205 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7072/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

failures look like race conditions, e.g. decommissioning failure

(DataXceiver.java:writeBlock(952)) - opWriteBlock BP-1168966658-172.17.0.2-1737771042928:blk_1073741825_1005 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1168966658-172.17.0.2-1737771042928:blk_1073741825_1005 already exists in state FINALIZED and thus cannot be created.
2025-01-25 02:11:40,248 [DataXceiver for client  at /127.0.0.1:57588 [Receiving block BP-1168966658-172.17.0.2-1737771042928:blk_1073741825_1005]] INFO  datanode.DataNode (DataXceiver.java:run(314)) - 127.0.0.1:37517:DataXceiver error processing WRITE_BLOCK operation  src: /127.0.0.1:57588 dst: /127.0.0.1:37517; org.apache.hado

that is not protobuf related

[steveloughran]

+1

[pjfanning]

I've seen those decommission tests fail in a few builds recently - in different PRs so not seemingly caused by the PR changes.

Protobuf-Java 3.25.6 has just come out but has a major breaking change where it fails at runtime if you have Protobuf Generated Java code that is too old (ie generated with a fairly old version of protoc). Unfortunately, this doesn't seem uncommon. HBase looks like it will be quite affected. I came across the issue in Apache Pekko.

apache/pekko#1748

The claim is that the old generated code is vulnerable to DoS.

https://github.com/protocolbuffers/protobuf/pull/20084/files#diff-94b93e396de2304f28fafc97968031b4e2a3a598c7efdab8b252a156ac7c57dbR315

[steveloughran]

thirdparty update it is then -targeting 3.4.2, presumably.