Sarama TLS not working: client has run out of available brokers to talk to: 3 errors occurred:\n\t* unexpected EOF
#3037
Comments
Larleyt
changed the title
client has run out of available brokers to talk to: 3 errors occurred:\n\t* unexpected EOF
|
More logs with Sarama debug enabled: |
|
@Larleyt hmm, so an "unexpected EOF" at connection time when enabling TLS on the client like this usually suggests that the brokers are expecting plain traffic. If I point a tls-enabled sarama client at a non-TLS broker then I similarly get EOF Can you try running this simple standalone "apiversions" sample? https://gist.github.com/dnwe/8cc61dad3e4fdabf857d03f20368dce5 If you go build and then try (within your aws environment so you have access to the hosts) doing: ./apiversions -broker b-1.t1devkafka.u91w03.c17.kafka.us-east-1.amazonaws.com:9094 and ./apiversions -tls -tls-skip-verify -broker b-1.t1devkafka.u91w03.c17.kafka.us-east-1.amazonaws.com:9094 and share the output from each, that would be useful |
|
Hello @dnwe, thanks for your reply! For first command ( 2024/12/23 14:18:40 unexpected EOFFor ,APIs,
0,ProduceRequest,9
1,FetchRequest,13
2,ListOffsetsRequest,7
3,MetadataRequest,12
4,LeaderAndIsrRequest,7
5,StopReplicaRequest,4
6,UpdateMetadataRequest,8
7,ControlledShutdownRequest,3
8,OffsetCommitRequest,8
9,OffsetFetchRequest,8
10,FindCoordinatorRequest,4
11,JoinGroupRequest,9
12,HeartbeatRequest,4
13,LeaveGroupRequest,5
14,SyncGroupRequest,5
15,DescribeGroupsRequest,5
16,ListGroupsRequest,4
17,SaslHandshakeRequest,1
18,ApiVersionsRequest,3
19,CreateTopicsRequest,7
20,DeleteTopicsRequest,6
21,DeleteRecordsRequest,2
22,InitProducerIdRequest,4
23,OffsetForLeaderEpochRequest,4
24,AddPartitionsToTxnRequest,3
25,AddOffsetsToTxnRequest,3
26,EndTxnRequest,3
27,WriteTxnMarkersRequest,1
28,TxnOffsetCommitRequest,3
29,DescribeAclsRequest,3
30,CreateAclsRequest,3
31,DeleteAclsRequest,3
32,DescribeConfigsRequest,4
33,AlterConfigsRequest,2
34,AlterReplicaLogDirsRequest,2
35,DescribeLogDirsRequest,4
36,SaslAuthenticateRequest,2
37,CreatePartitionsRequest,3
38,CreateDelegationTokenRequest,3
39,RenewDelegationTokenRequest,2
40,ExpireDelegationTokenRequest,2
41,DescribeDelegationTokenRequest,3
42,DeleteGroupsRequest,2
43,ElectLeadersRequest,2
44,IncrementalAlterConfigsRequest,1
45,AlterPartitionReassignmentsRequest,0
46,ListPartitionReassignmentsRequest,0
47,OffsetDeleteRequest,0
48,DescribeClientQuotasRequest,1
49,AlterClientQuotasRequest,1
50,DescribeUserScramCredentialsRequest,0
51,AlterUserScramCredentialsRequest,0
56,AlterPartitionRequest,2
57,UpdateFeaturesRequest,1
58,EnvelopeRequest,0
60,DescribeClusterRequest,0
61,DescribeProducersRequest,0
65,DescribeTransactionsRequest,0
66,ListTransactionsRequest,0
67,AllocateProducerIdsRequest,0As I understand, it doesn't work with plaintext (first cmd), but works with TLS enabled and skip-verify. As expected. But I have |
|
@Larleyt hmm thanks, so that test confirms that generally the brokers are working, they are correctly TLS-enabled (and Sarama can successfully TLS handshake with them) and they have matching support api versions to what you'd see with the official Apache Kafka 3.4.0 release. |
|
Here's what I've got now with LogsConfluent consumers don't work anymore too, when Sarama consumer/producer exist. As soon as I remove Sarama client Confluent work fine (with TLS enabled): Logs
This link is in private repo, no access. |
|
If I leave only Sarama client, it doesn't work: Logs |
|
It looks to me (from last logs) like it can only create one client for Sarama and I need 3 (in the original code in first post you can see I use |
|
This is the result of your script when only ➜ ./kafkabrokerfinder -tls -tls-skip-verify -verbose -broker localhost:9094Logs |
|
Do you have access to the broker logs? Unexpected EOF from the client-side typically means that Kafka has forcibly closed the connection and that would usually be accompanied by some warning or error message in the broker logs |
|
Here it is, exactly after running the command Kafka broker logs |
|
yep, but running that with my script worked fine right? There weren't any unexpected EOF, just a single client connected, did a metadata request and a describe configs request and then disconnected. You'd need to run one of your other examples where an unexpected EOF was seen |
|
Here are Kafka Broker logs when I deploy my own code and it tries to connect. I see only authentication error, but don't understand why it occurs with the settings as in first post. Confluent Kafka works with the same. Kafka Broker logs |
|
@Larleyt hmm as per the Kafka docs:
My understanding from our earlier discussion was that you weren't using mTLS right, as you hadn't configured anything in rdkafka for that either? If so, then I can only assume it's an issue with the cipher negotiation between modern Go and the Java runtime under which Kafka is being run |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I have a service that already had completely working Confluent Kafka Consumers and Producers with SSL enabled in the environment I've got the following bug. Then I've added a goka package which only supports Sarama.
When I'm trying to turn on Sarama's TLS (without cert verifying – the same as Confluent consumer), I get the following error:
Error creating the kafka client: kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* unexpected EOF\n\t* unexpected EOF\n\t* unexpected EOF\n"Versions
I also tried different
sarama.V0_11_0_0,V2_8_*, etc. Nothing workedConfiguration
The code that utilizes the Sarama and not starting:
code: CLICK ME
Logs
logs: CLICK ME
Additional Context
As you can see from the logs, Confluent Kafka is starting absolutely fine (first part of the logs), but Sarama does not.
Additionally, I wonder why bootstrap servers' string split by comma (
bootstrap_servers_splitin logs) have this[](Confluent accepts single string with comma, Goka accepts a slice). But that might be just Zerolog representation of slice in logs, cause otherwise this slice would not work locally (without TLS).The text was updated successfully, but these errors were encountered: