Unnecessary failovers for HDFS namenode

[dmichal]

Hello,
I've noticed that in my cluster the webhdfs output plugin from time to time performs failover between my HDFS namenodes, despite the fact that the namenodes themselves do not failover. After a bit of investigation I've found that the actual exception causing the failover in plugin is: "Failed to connect to host hd4.local:50075, Net::ReadTimeout.", where hd4 is one of my datanodes, ie. the plugin performs failover even in case of datanode connection error. It is so because the plugin just searches the error string for pattern "Failed to connect". Maybe some more specific matching should be performed, eg. searching for namenode port as well? Unnecessary failovers cause a lot of problems for me, as they sometimes result in HDFS lease problems.

Observed in logstash 6.6.0, HDFS 2.7.3; logstash and hadoop machines are running on CentOS 7.