Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken pipe and EOF #2938

Open
errrov opened this issue Jul 18, 2024 · 2 comments
Open

Broken pipe and EOF #2938

errrov opened this issue Jul 18, 2024 · 2 comments
Labels
stale Issues and pull requests without any recent activity

Comments

@errrov
Copy link

errrov commented Jul 18, 2024

Description

Started getting errors of broken pipe and EOF when moved to go1.22. Maybe it coincided with changes on kafka server, want to get some information before passing this problem to company's kafka development team
Consuming messages using Consumer Group.
Want to understand when Broken Pipe error appear for Consumer Groups. When it sends offset to server?
Can this problem be caused of consumer group code. Or its more likely kafka server side error?

Versions
Sarama Kafka Go
v1.43.0 v2.7.2.0 1.22
Configuration
config.Consumer.Group.Session.Timeout = 20 * time.Second
config.Consumer.Group.Heartbeat.Interval = 6 * time.Second
config.Consumer.MaxProcessingTime = 500 * time.Millisecond
config.Version = sarama.V2_7_2_0
config.Net.SASL.Mechanism = sarama.SASLTypeSCRAMSHA256
config.Net.SASL.SCRAMClientGeneratorFunc = func() sarama.SCRAMClient {
	return &XDGSCRAMClient{HashGeneratorFcn: scram.SHA256}
}
Logs
logs: CLICK ME

{"level":"error","error":"kafka: error while consuming testtopic/2: EOF","time":"2024-07-18T08:00:25+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/8: EOF","time":"2024-07-18T08:02:05+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/5: EOF","time":"2024-07-18T08:03:10+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/2: EOF","time":"2024-07-18T08:25:30+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/8: EOF","time":"2024-07-18T08:27:27+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/5: EOF","time":"2024-07-18T08:28:19+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/2: write tcp ip1->ip2: write: broken pipe","time":"2024-07-18T08:39:37+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/8: write tcp ip1->ip2: write: broken pipe","time":"2024-07-18T08:41:33+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/5: write tcp ip1->ip2: write: broken pipe","time":"2024-07-18T08:42:46+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/2: EOF","time":"2024-07-18T09:43:39+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/8: EOF","time":"2024-07-18T09:45:23+03:00","message":"Error occurred during consumer life-cycle"}
{"level":"error","error":"kafka: error while consuming testtopic/5: EOF","time":"2024-07-18T09:46:25+03:00","message":"Error occurred during consumer life-cycle"}

Additional Context
@ciphersaw
Copy link

The same errors occur in my kafka clients consuming messages by Consumer Group.
The feature is that the usage rate of CPU of kafka client host is high and even up to 100% when the errors occur, which may be a reason causing Broken Pipe.
In addition, the usage rate of CPU, memory or disk I/O of kafka server host seems well during these errors.

Copy link

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

@github-actions github-actions bot added the stale Issues and pull requests without any recent activity label Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Issues and pull requests without any recent activity
Projects
None yet
Development

No branches or pull requests

2 participants