[Questions] SocketException happens sometimes and messages not consumed

[Luk7c]

Community Support Policy

Hello everyone,
I encountered two problems with my RabbitMQ cluster.

1st problem:

I have a cluster with three RabbitMQ nodes.
Everything works fine until my connection to the cluster is closed when an error appears:

java.net.SocketException: Connection reset by peer

Followed by :

java.net.SocketException: Broken pipe

My logs in rabbit are :

2024-12-10 11:25:25.928085+00:00 [error] <0.1998.0> closing AMQP connection <0.1998.0> (X.X.X.X:43878 -> X.X.X.X:5672 - connectionFactory#19dc0d32:0):
rabbitmq1-1  | 2024-12-10 11:25:25.928085+00:00 [error] <0.1998.0> {writer,send_failed,timeout}

After that, I have to run docker compose down/up to reconnect my application to the cluster. My service consumes messages, but sometimes the connection is closed again with the same errors.

2nd problem:

Sometimes I have zero messages in my queues (0 ready, 0 unacked, 0 total), but the messages are not immediately consumed by my Spring service.

@Override
public void onMessage(Foo message) {
    log.info("Message {} received", message.getName());
}

It feels like there’s a delay — my messages are only consumed 5 minutes later (my log.info appears 5min later) for example.

Any ideas?

Thank you!

[acogoluegnes]

Please provide more information: client and broker versions and logs. A sample project and steps to reproduce the issue are useful as well.

Make sure connection recovery is activated (Spring AMQP has its own recovery mechanism and deactivates the one from the RabbitMQ Java client library).

[michaelklishin]

2024-12-10 11:25:25.928085+00:00 [error] <0.1998.0> {writer,send_failed,timeout}

is clear evidence of a genuine TCP connection problem. This message is logged when RabbitMQ tries to send a serialized protocol frame on a connection and that operation fails (in this case, with a timeout).

A traffic capture with Wireshark or tcpdump (that can be loaded into Wireshark for inspection on a different machine) would provide a lot of relevant information at the TCP level and below.

This is expected from time to time in practice. This is why all messaging and streaming protocols RabbitMQ supports provide heartbeats (even if the feature is named differently), and why publisher confirms, connection recovery in client libraries exist.

Our community support policy clearly states that we will not troubleshoot networking for users without a commercial support license.