Error handling and retry behaviour improved

[dmichal]

Several changes in error handling:

• Support for infinite retries added - better fits logstash philosophy of not dropping events in case of errors in order to provide data integrity,
• Properly use 'retry_times' in case of failovers - previously 'retry_times' parameter was ignored in case of failovers thus leading to infinite retries,
• Reducing unnecessary failovers due to conn errors - solves Unnecessary failovers for HDFS namenode #36
• Optional limit for retry interval added - useful in case of infinite retries as retry interval increases with each attempt potentially resulting in extremely high values,
• Docs improved - better description of retry behaviour and retry_interval increments,
• Properly handle errors during file creation - previously errors encountered during file creation were not handled at all; now they are treated just like any other write error

[dmichal]

One more change I'm wondering about is to add sleep before retry in case of failovers, just as it is in case of other errors. This may prove beneficial when both namenodes are down or inaccessible - in current implementation retries are performed immediately after error thus using resources, producing a lot of unnecessary log messages and quickly using up retry limit. On the other hand, in case of one namenode working properly sleeping for a while after failover won't have significant impact on performance since it is going to be performed only once.