HADOOP-19393. ABFS: Returning FileAlreadyExists Exception for UnauthorizedBlobOverwrite Rename Errors #7312
+28
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manika137
changed the title
snvijaya
suggested changes
Jan 22, 2025
| @@ -613,6 +615,11 @@ public AbfsClientRenameResult renamePath( | |||
| throw e; | |||
| } | |||
|
|
|||
| if(op.getResult().getStorageErrorCode() | |||
| .equals(UNAUTHORIZED_BLOB_OVERWRITE.getErrorCode())){ | |||
| throw new FileAlreadyExistsException("File already exists." ); | |||
There was a problem hiding this comment.
The error string should be "File already exists or request not authorized for blob overrides", as this can happen when destination already exists or if SAS is missing "m".
There was a problem hiding this comment.
Makes sense. Taken.
There was a problem hiding this comment.
As discussed, we're observing other 403 errors with mismatching SAS tokens/invalid permissions. We see UNAUTHORIZED_BLOB_OVERWRITE only with rename overwrites. Making changes accordingly.
|
💔 -1 overall
This message was automatically generated. |
|
Aggregated Test Results ============================================================
|
manika137
changed the title
|
🎊 +1 overall
This message was automatically generated. |
manika137
force-pushed
the
renameErrorMapping
branch
from
e5d0c97 to
401052c
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Jan 23, 2025
| @@ -613,6 +615,11 @@ public AbfsClientRenameResult renamePath( | |||
| throw e; | |||
| } | |||
|
|
|||
| if(op.getResult().getStorageErrorCode() | |||
| .equals(UNAUTHORIZED_BLOB_OVERWRITE.getErrorCode())){ | |||
| throw new FileAlreadyExistsException("File already exists."); | |||
There was a problem hiding this comment.
Can you add a comment explaining why, e.g
Write permission checks can happen before probes for validity
of the operation -if there is a path at the destination
it may be rejected with a authorization failure.
Catch and translate. See HADOOP-19393.
| touch(src); | ||
| touch(dest); | ||
|
|
||
| try { |
There was a problem hiding this comment.
use LambdaTestUtils.intercept, e.g
intercept(FileAlreadyExistsException.class, () -> fs.rename(src, dest));
| @@ -58,7 +58,8 @@ public enum AzureServiceErrorCode { | |||
| ACCOUNT_REQUIRES_HTTPS("AccountRequiresHttps", HttpURLConnection.HTTP_BAD_REQUEST, null), | |||
| MD5_MISMATCH("Md5Mismatch", HttpURLConnection.HTTP_BAD_REQUEST, | |||
| "The MD5 value specified in the request did not match with the MD5 value calculated by the server."), | |||
| UNKNOWN(null, -1, null); | |||
| UNKNOWN(null, -1, null), | |||
| UNAUTHORIZED_BLOB_OVERWRITE("UnauthorizedBlobOverwrite", HttpURLConnection.HTTP_FORBIDDEN, "This request is not authorized to perform blob overwrites."); | |||
There was a problem hiding this comment.
nit: split the line with the text on the line below
| @@ -58,7 +58,8 @@ public enum AzureServiceErrorCode { | |||
| ACCOUNT_REQUIRES_HTTPS("AccountRequiresHttps", HttpURLConnection.HTTP_BAD_REQUEST, null), | |||
| MD5_MISMATCH("Md5Mismatch", HttpURLConnection.HTTP_BAD_REQUEST, | |||
| "The MD5 value specified in the request did not match with the MD5 value calculated by the server."), | |||
| UNKNOWN(null, -1, null); | |||
| UNKNOWN(null, -1, null), | |||
| UNAUTHORIZED_BLOB_OVERWRITE("UnauthorizedBlobOverwrite", HttpURLConnection.HTTP_FORBIDDEN, "This request is not authorized to perform blob overwrites."); | |||
There was a problem hiding this comment.
nit: It would be better to move it above Unknown or add with other 403 errors
| fs.rename(src, dest); | ||
| Assertions.fail("Exception expected on rename overwrites."); | ||
| } catch (FileAlreadyExistsException e) { | ||
| Assertions.assertThat(e) |
There was a problem hiding this comment.
Nit: add a description for assert failures here and anywhere else you are using Assertions. Something like this.
Assertions.assertThat(e).describedAs("FileAlreadyExistsException was expected").isInstaceOf();
manika137
force-pushed
the
renameErrorMapping
branch
from
e339293 to
001e137
Compare
Yes, the problem was reproduced before applying the fix. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
I should add, +1 pending those suggested changes and the (trivial) checkstyle fix. No problems with the actual code itself. |
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-19393
ABFS driver adheres to Hadoop's expectations which does not allow rename blob overwrites. Recently we came across the case where UnauthorizedBlobOverwrite error (HTTP 403- Access Denied Exception) is thrown for rename overwrites (with SAS authentication).
Remapping this error to FileAlreadyExists exception for better understanding.
Adding the test results in comments below.
The one test failure present is intermittent and has been brought up here: https://issues.apache.org/jira/browse/HADOOP-19213