Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logstash.outputs.webhdfs【 6.2.4】write hdfs failed after the first time successfully written #9712

Closed
hero1122 opened this issue Jun 7, 2018 · 1 comment

Comments

@hero1122
Copy link

hero1122 commented Jun 7, 2018

[2018-06-07T10:26:37,363][ERROR][logstash.outputs.webhdfs ] Max write retries reached. Events will be discarded. Exception: {"RemoteException":{"message":"Failed to APPEND_FILE /logstash/dt=2018-06-07/logstash-02.log for DFSClient_NONMAPREDUCE_-692957599_23 on 192.168.0.3 because this file lease is currently owned by DFSClient_NONMAPREDUCE_382768740_23 on 192.168.0.3","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
[2018-06-07T10:26:40,009][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x415fbb96 sleep>"}

For all general issues, please provide the following details for fast resolution:

  • Version: 6.2.4
  • Operating System: centos 6.5
  • Config File (if you have sensitive info, please remove it):
    port => "14000"
    use_httpfs => "true"
    path => "/logstash/dt=%{+YYYY-MM-dd}/logstash-%{+HH}-%{+mm}-%{+ss}.log"
    user => "hadoop"
  • Sample Data:
  • Steps to Reproduce:
    the first time will be successful,but the second time will be failed where append to hdfs!

And hdfs no support append file, we mush call file close method after write data info hdfs file!

@yaauie
Copy link
Member

yaauie commented Jun 18, 2018

I opened an issue on the WebHDFS Output Plugin: logstash-plugins/logstash-output-webhdfs#35

@yaauie yaauie closed this as completed Jun 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants