-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDFS-17546. Implementing HostsFileReader timeout #6873
Conversation
ca8e2ec
to
8ed85ca
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Just one minor comment.
@Test | ||
public void testReadFileWithTimeoutTimeoutException() throws Exception { | ||
exception.expect(IOException.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can just use @Test(expected = IOException.class)
here since we don't check any other properties of the exception.
Then you can remove the Rule declaration above and the new imports.
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
|
||
import org.apache.hadoop.hdfs.protocol.DatanodeAdminProperties; | ||
import org.apache.hadoop.test.GenericTestUtils; | ||
import org.junit.Before; | ||
import org.junit.Rule; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary import.
@Rule | ||
public final ExpectedException exception = ExpectedException.none(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This variable is not used anywhere.
import org.junit.After; | ||
import org.junit.Test; | ||
import org.mockito.invocation.InvocationOnMock; | ||
import org.mockito.stubbing.Answer; | ||
import org.junit.rules.ExpectedException; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary import.
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Latest test failure looks unrelated to the changes in this PR. I'll wait for the current build to complete though.
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
(cherry picked from commit 2fbbfe3)
(cherry picked from commit 2fbbfe3)
(cherry picked from commit 2fbbfe3)
HDFS-17546. Implementing HostsFileReader timeout (#6873)
Description of PR
In hadoop environments that rely on NAS to house dfs.hosts file, during FS hangs (for any reason) the refreshNodes call would infinitely hang (causing a blocked thread) until the FS returns.
This is due to how Java's Filestream works here:
In order to fix this we introduce a method (+ new config value) which (if set) encapsulates the original method in a
Future.Task
. If the Task -- reading the file -- does not complete withindfs.hosts.timeout
then we raise IOException which would interrupt (close) the thread and return a failure.How was this patch tested?
Updated Unit tests to account for new method, ran through local test suite, and planning to run through remote test suite
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?