TCK: WebSocket NegDep Challenge #465

volosied · 2024-12-11T17:14:12Z

Challenged tests
Main Package: https://github.com/jakartaee/platform-tck/tree/main/websocket/spec-tests/src/main/java/com/sun/ts/tests/websocket/negdep

wsc_negdep_invalidpathparamtype_srv_onclose_web
wsc_negdep_invalidpathparamtype_srv_onerror_web
wsc_negdep_invalidpathparamtype_srv_onmessage_web
wsc_negdep_invalidpathparamtype_srv_onopen_web
wsc_negdep_malformedpath_web
wsc_negdep_multiplepaths_web
wsc_negdep_onclose_srv_duplicate_web
wsc_negdep_onclose_srv_toomanyarguments_web
wsc_negdep_onerror_srv_duplicate_web
wsc_negdep_onerror_srv_toomanyarguments_web
wsc_negdep_onmessage_pasrv_nomoreendpoints_web
wsc_negdep_onmessage_ppsrv_nomoreendpoints_web
wsc_negdep_onmessage_srv_binarybytebufferint_web
wsc_negdep_onmessage_srv_binaryinputstreamboolean_web
wsc_negdep_onmessage_srv_binaryinputstreamboolean_web
wsc_negdep_onmessage_srv_binarynodecoder_web
wsc_negdep_onmessage_srv_pongboolean_web
wsc_negdep_onmessage_srv_pongduplicate_web
wsc_negdep_onmessage_srv_textbigdecimal_web
wsc_negdep_onmessage_srv_textduplicate_web
wsc_negdep_onmessage_srv_textnodecoder_web
wsc_negdep_onmessage_srv_textreaderboolean_web
wsc_negdep_onmessage_srv_textstringint_web
wsc_negdep_onopen_srv_duplicate_web
wsc_negdep_onopen_srv_toomanyarguments_web

TCK Version
Main Branch of platform-tck/websocket

Description
This is partly a challenge and partly a request for improvement. These web socket tests listed above have moved to use the Arquillian framework. They invoke the application and then expect an exception to occur, as these are negative tests (some misconfiguration exists in the app). 

While testing on Liberty, we have encountered various errors when trying to get these tests to run. By design, these apps trigger DeploymentException, which causes them to not start in the server. As a result, Arquillian and any related plugins we use cannot handle the app (since it doesn’t exist for us). After making a few fixes on our side, the last error we’ve encountered is related to the URL:   

java.lang.RuntimeException: Could not lookup value for field protected java.net.URL com.sun.ts.tests.websocket.common.client.WebSocketCommonClient.url
        at org.jboss.arquillian.test.impl.enricher.resource.ArquillianResourceTestEnricher.enrich(ArquillianResourceTestEnricher.java:68)
        at org.jboss.arquillian.test.impl.TestInstanceEnricher.enrich(TestInstanceEnricher.java:51)
        at org.jboss.arquillian.container.test.impl.ClientTestInstanceEnricher.enrich(ClientTestInstanceEnricher.java:48)
         … 
Caused by: java.lang.RuntimeException: All Providers for type class java.net.URL returned a null value: [org.jboss.arquillian.container.test.impl.enricher.resource.URLResourceProvider@573d07e6]
        at org.jboss.arquillian.test.impl.enricher.resource.ArquillianResourceTestEnricher.lookup(ArquillianResourceTestEnricher.java:126)
        at org.jboss.arquillian.test.impl.enricher.resource.ArquillianResourceTestEnricher.enrich(ArquillianResourceTestEnricher.java:66)

We’re unsure if any other implementations have been tested yet, but we would expect similar issues to occur. Arquillian is trying to do some Test enrichment, but due to the deployment exception, there is no available URL for these tests.

We propose to remove the invoke all in these tests:    

try {
  invoke(false);
} catch (Exception tfe) {
  // DeploymentException
}

and instead rely on the Arquillian’s ShouldThrowException annotation. This has the added bonus of being able to explicitly check for DeploymentException, instead of just catching Exception as is currently done. This is similar to the CDI TCK’s approach, as linked here and here.

Some extra handling would also be needed to verify that any previously deployed endpoints are removed from service. A draft PR can be provided.

Of if anyone has a workaround for the URL issue above, we’d be happy to accept other suggestions.

Additional context
None

The text was updated successfully, but these errors were encountered:

pnicolucci · 2024-12-11T18:51:24Z

Tagged as a challenge. @markt-asf @joakime can you take a look?

markt-asf · 2024-12-12T10:20:53Z

The tests all pass with Tomcat and Eclipse Tyrus. I'm not adverse to improving the tests but this looks more like an issue with the Liberty / Arquillian integration at this point.

brideck · 2024-12-19T17:59:46Z

this looks more like an issue with the Liberty / Arquillian integration at this point.

Which part do you consider to be the issue here? That Liberty fails to start these applications? Or that the Liberty Arquillian plugin considers this to be an Arquillian deployment exception? Since the application doesn't start, there is no valid URL that could be supplied. The plugin could certainly fake one up, but it already knows at this point that no endpoints will exist so why bother?

I'm surprised that other plugin implementations wouldn't treat this sort of case the same way.

jansupol · 2024-12-19T21:48:20Z

@pnicolucci What JDK do you use to run the TCK? Can you try with 21?

brideck · 2024-12-19T21:51:51Z

What JDK do you use to run the TCK? Can you try with 21?

17 & 21. Things work the same in both cases.

jansupol · 2024-12-19T21:54:33Z

Here is Tyrus script with some old glassfish that works with JDK 21.

brideck · 2024-12-20T00:20:04Z

Right, so the fundamental difference here is that Glassfish seems to go ahead and deploy the application anyway:

[2024-12-19T15:28:16.829040-08:00] [GF 8.0.0-M3] [INFO] [] [jakarta.enterprise.system.core] [tid: _ThreadID=51 _ThreadName=admin-listener(3)] [levelValue: 800] [[
  wsc_negdep_invalidpathparamtype_srv_onclose_web was successfully deployed in 186 milliseconds.]]

The WebSocket spec (https://jakarta.ee/specifications/websocket/2.2/jakarta-websocket-spec-2.2#deployment-errors) states:

Other errors arise as a result of a malformed WebSocket application. Chapter 4 provides several examples of WebSocket endpoints that are malformed. In such cases, the container must provide an informative error message to the deployer during the deployment process [WSC-5.2.1-2].

In both cases, a deployment error raised during the deployment process must halt the deployment of the application, any well formed endpoints deployed prior to the error being raised must be removed from service and no more WebSocket endpoints from that application may be deployed by the container, even if they are valid [WSC-5.2.1-3].

Liberty has chosen to fulfill "must halt the deployment of the application" as meaning that the application does not end up deployed at all.

volosied · 2024-12-23T15:39:41Z

Is there any chance these tests could be excluded from this release? This would allow more time to investigate between the implementations and the Arquillian plugins.

The application starting in Glassfish looks a bit concerning since should not start due to a deployment exception. As mentioned, Liberty fails to start the app, and this leads no URL provided error later on in Arqullian.

Additionally, my original suggestion was to use @ShouldThrowException, but, after looking at Tomcat's plugin, I'm not so sure it would handle ShouldThrowException? Glassfish may not either.

volosied · 2025-01-06T16:09:35Z

Following up on my request to exclude the tests. Thanks!

markt-asf · 2025-01-06T17:46:09Z

Looking at this is on my TODO list. I want to understand why the test passes with Tomcat before taking any action on this.

brideck · 2025-01-06T18:01:54Z

The test currently expects the app to deploy, but for none of the endpoints to exist, which I believe is exactly how Tomcat currently performs (I know @volosied did a little investigation of Tomcat awhile back, so perhaps he can recall). That's definitely the case for Glassfish.

Open Liberty doesn't deploy the app (therefore none of the endpoints exist), but that doesn't play nicely with the test, which does not expect a deployment failure.

We actually worked around the app deployment failure in the old TCK by special casing these specific tests in our TCK porting package to ignore the exception and proceed, which is pretty broken. We don't want to do the same thing here because our Arquillian plugin is open source, available for anyone to use, and not specific to TCK testing. The old porting package is proprietary and specific to TCK testing, so that was a marginally acceptable solution there.

volosied · 2025-01-06T18:22:35Z

I was pushing for excluding these tests more due to Glassfish's behavior (deploying the app even with an exception).

As for Tomcat, here's the output below. It does report startup failed due to previous errors. But I don't know how it interacts with Arquillian to determine that startup failed. I would expect no URLs to be available, and thus a similar error as in Liberty.

09-Dec-2024 15:31:32.910 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [280] milliseconds
09-Dec-2024 15:32:12.944 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/Users/volosied/Downloads/apache-tomcat-11.0.2/webapps/wsc_negdep_invalidpathparamtype_srv_onclose_web.war]
09-Dec-2024 15:32:12.992 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Error during ServletContainerInitializer processing
	jakarta.servlet.ServletException: jakarta.websocket.DeploymentException: The type [com.sun.ts.tests.websocket.negdep.StringHolder] is not permitted as a path parameter. Parameters annotated with @PathParam may only be Strings, Java primitives or a boxed version thereof.
		at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:116)
		at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4393)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
		at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:566)
		at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:653)
		at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:933)
		at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1875)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
		at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
		at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:735)
		at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:387)
		at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1642)
		at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:263)
		at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:109)
		at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:940)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1139)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1143)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1121)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
		at java.base/java.lang.Thread.run(Thread.java:833)
	Caused by: jakarta.websocket.DeploymentException: The type [com.sun.ts.tests.websocket.negdep.StringHolder] is not permitted as a path parameter. Parameters annotated with @PathParam may only be Strings, Java primitives or a boxed version thereof.
		at org.apache.tomcat.websocket.pojo.PojoPathParam.validateType(PojoPathParam.java:65)
		at org.apache.tomcat.websocket.pojo.PojoPathParam.<init>(PojoPathParam.java:41)
		at org.apache.tomcat.websocket.pojo.PojoMethodMapping.getPathParams(PojoMethodMapping.java:298)
		at org.apache.tomcat.websocket.pojo.PojoMethodMapping.<init>(PojoMethodMapping.java:193)
		at org.apache.tomcat.websocket.server.WsServerContainer.addEndpoint(WsServerContainer.java:132)
		at org.apache.tomcat.websocket.server.WsServerContainer.addEndpoint(WsServerContainer.java:242)
		at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:113)
		... 26 more
09-Dec-2024 15:32:12.993 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [/wsc_negdep_invalidpathparamtype_srv_onclose_web] startup failed due to previous errors
09-Dec-2024 15:32:12.997 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/Users/volosied/Downloads/apache-tomcat-11.0.2/webapps/wsc_negdep_invalidpathparamtype_srv_onclose_web.war] has finished in [53] ms

brideck · 2025-01-06T21:29:02Z

I'm pretty sure that Deployment of web application archive [/Users/volosied/Downloads/apache-tomcat-11.0.2/webapps/wsc_negdep_invalidpathparamtype_srv_onclose_web.war] has finished in [53] ms means that from an Arquillian perspective there would be no errors encountered, but that's for a Tomcat expert to know for sure.

volosied · 2025-01-13T19:21:15Z

I ran the tomcat TCK locally and looking at this stack trace (which is similar to the one I posted above)

jakarta.servlet.ServletException: jakarta.websocket.DeploymentException: The type [com.sun.ts.tests.websocket.negdep.StringHolder] is not permitted as a path parameter. Parameters annotated with @PathParam may only be Strings, Java primitives or a boxed version thereof.
        at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:116)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4393)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:164)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:566)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:653)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:933)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1875)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:735)
        at org.jboss.arquillian.container.tomcat.embedded.EmbeddedHostConfig.deployWAR(EmbeddedHostConfig.java:63)
        at org.jboss.arquillian.container.tomcat.embedded.Tomcat10EmbeddedContainer.deploy(Tomcat10EmbeddedContainer.java:150)

If you look at the EmbeddedHostConfig.deployWAR, you'll see it doesn't actually throw any exceptions.

This means the DeploymentException from the app is just ignored by the Arquillian integration?

I also don't see any errors in the log for this message: https://github.com/apache/tomcat/blob/main/java/org/apache/catalina/startup/HostConfig.java#L936C37-L936C62

Looking some more... I guess the source of this might be this catch? Though, I don't get how all the pieces fit, but I think this is on the right track. The deployment error is just handled earlier...?

https://github.com/apache/tomcat/blob/2810ac234084e92b3282b0d2364deed4d03c7c21/java/org/apache/catalina/core/StandardContext.java#L4391-L4399

Edit: I realized I used main branch (which is Tomcat 12, but I think it should still apply to 11).

markt-asf · 2025-01-15T11:47:00Z

There is a lot to unpack here. I am going to try and keep the focus fairly narrow.

These tests expect deployment to fail. After the failed deployment, these tests expect:

any attempt to call the invalid WebSocket endpoint to fail; and
any attempt to call a valid WebSocket endpoint packaged with the invalid endpoint to fail.

I don't think there is any disagreement on the above.

The (huge) grey area is what is meant by "failed deployment". There is no definition I can find in either the WebSocket specification or the Platform specification for "failed deployment". While there is a requirement for a DeploymentException to be thrown, it is a checked exception so something is going to need to catch it at some point in the deployment mechanism. There is no definition of what should catch that exception. There is no definition of what should happen once that exception has been caught. And all of that is before you add Arquillian integration into the mix.

Given the ambugiuty around failed deployment, I don't think we can reliably test for DeploymentException being thrown unless the Platform specification gets a lot more specific about the deployment process, including failure modes, and that seems unlikely to me.

For these tests, that leaves us with testing that a client cannot successfully connect to either the invalid endpoint or a valid endpoint that was packaged along with the invalid one. To do that requires a URL. Therefore I think requiring something (the plug-in, the TCK, the product specific TCK integration) to ensure that a correct URL is provided is the best way forward.

I've been inspecting a few Tomcat TCK runs with a debugger. For a successful deployment the URL returned is <scheme>://<host>:<port>/<context-path>/ but for an unsuccessful deployment it is <scheme>://<host>:<port>. That seems wrong to me. The intention of the tests (and, from what I recall, the behaviour of the pre-Arquillian test suite) is that the URL used by the client should always be <scheme>://<host>:<port>/<context-path>/ regardless of whether deployment is successful or not.

I'm going to spend some time looking at options getting the Tomcat TCK tests to use <scheme>://<host>:<port>/<context-path>/ for the WebSocket tests that depend on successful and failed deployment.

markt-asf · 2025-01-15T14:46:13Z

FYI, this is the change I made to Tomcat's TCK integration code to address this: apache/tomcat-tck@1b6769f

For the record, the WebSocket TCK still passes.

brideck mentioned this issue Dec 11, 2024

Add support for tests with intentional app deployment failures OpenLiberty/liberty-arquillian#137

Open

pnicolucci added the challenge label Dec 11, 2024

olamy mentioned this issue Jan 21, 2025

Fix possible NPE for default servlet arquillian/arquillian-container-jetty#234

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TCK: WebSocket NegDep Challenge #465

TCK: WebSocket NegDep Challenge #465

volosied commented Dec 11, 2024 •

edited

Loading

pnicolucci commented Dec 11, 2024

markt-asf commented Dec 12, 2024

brideck commented Dec 19, 2024 •

edited

Loading

jansupol commented Dec 19, 2024

brideck commented Dec 19, 2024

jansupol commented Dec 19, 2024

brideck commented Dec 20, 2024

volosied commented Dec 23, 2024

volosied commented Jan 6, 2025

markt-asf commented Jan 6, 2025

brideck commented Jan 6, 2025 •

edited

Loading

volosied commented Jan 6, 2025

brideck commented Jan 6, 2025

volosied commented Jan 13, 2025

markt-asf commented Jan 15, 2025

markt-asf commented Jan 15, 2025

TCK: WebSocket NegDep Challenge #465

TCK: WebSocket NegDep Challenge #465

Comments

volosied commented Dec 11, 2024 • edited Loading

pnicolucci commented Dec 11, 2024

markt-asf commented Dec 12, 2024

brideck commented Dec 19, 2024 • edited Loading

jansupol commented Dec 19, 2024

brideck commented Dec 19, 2024

jansupol commented Dec 19, 2024

brideck commented Dec 20, 2024

volosied commented Dec 23, 2024

volosied commented Jan 6, 2025

markt-asf commented Jan 6, 2025

brideck commented Jan 6, 2025 • edited Loading

volosied commented Jan 6, 2025

brideck commented Jan 6, 2025

volosied commented Jan 13, 2025

markt-asf commented Jan 15, 2025

markt-asf commented Jan 15, 2025

volosied commented Dec 11, 2024 •

edited

Loading

brideck commented Dec 19, 2024 •

edited

Loading

brideck commented Jan 6, 2025 •

edited

Loading