diff --git a/hadoop-tools/hadoop-azure/src/site/markdown/fns_blob.md b/hadoop-tools/hadoop-azure/src/site/markdown/fns_blob.md index bf0835ccbe3ae..1141ac0ec1b9c 100644 --- a/hadoop-tools/hadoop-azure/src/site/markdown/fns_blob.md +++ b/hadoop-tools/hadoop-azure/src/site/markdown/fns_blob.md @@ -14,8 +14,6 @@ # ABFS Driver for Namespace Disabled Accounts (FNS: Flat Namespace) -### Note: FNS-BLOB Support is being built and not yet ready for usage. - ## Background The ABFS driver is recommended to be used only with HNS Enabled ADLS Gen-2 accounts for big data analytics because of being more performant and scalable. @@ -79,4 +77,80 @@ configured service type. fs.azure.fns.account.service.type BLOB - ``` \ No newline at end of file + ``` + +5. Auth type supported over FNS Blob: SharedKey, OAuth and fixed SAS. + ```xml + + fs.azure.account.auth.type + SharedKey + + ``` + - How to configure Shared Key + auth: [Shared Key](./abfs.md#a-nameshared-key-autha-default-shared-key) + - How to configure + OAuth: [OAuth](./abfs.md#a-nameoauth-client-credentialsa-oauth-20-client-credentials) + - How to configure fixed + SAS: [Fixed SAS](./abfs.md#using-accountservice-sas-with-abfs) + + OAuth is recommended auth type as it is more secure and flexible. + +### Rename delete configs + +The following configs are related to rename and delete operations. + +- `fs.azure.blob.copy.progress.wait.millis`: Blob copy API is an async API, this + configuration defines polling duration for checking copy status. The default + value is 1 sec i.e. 1000 ms. + +- `fs.azure.blob.copy.max.wait.millis`: Maximum time to wait for a blob copy + operation to complete. The default value is 5 minutes. + +- `fs.azure.blob.atomic.rename.lease.refresh.duration`: Blob rename lease + refresh + duration in milliseconds. This setting ensures that the lease on the blob is + periodically refreshed during a rename operation to prevent other operations + from interfering. + The default value is 60 seconds. + +- `fs.azure.blob.dir.list.producer.queue.max.size`: Maximum number of blob + entries + enqueued in memory for rename or delete orchestration. The default value is 2 + times the default value of list max results, which is 5000, making the current + value 10000. + +- `fs.azure.blob.dir.list.consumer.max.lag`: It sets a limit on how much blob + information can be waiting to be processed (consumer lag) during a blob + listing + operation. If the amount of unprocessed blob information exceeds this limit, + the + producer will pause until the consumer catches up and the lag becomes + manageable. The default value is equal to the value of default value of list + max + results which is 5000 currently. + +- `fs.azure.blob.dir.rename.max.thread`: Maximum number of threads per blob + rename + orchestration. The default value is 5. + +- `fs.azure.blob.dir.delete.max.thread`: Maximum number of thread per + blob-delete + orchestration. The default value currently is 5. + +## Features currently not supported + +1. **User Delegation SAS** feature is currently not supported but we + plan to bring support for it in the future. + Jira to track this + workitem : https://issues.apache.org/jira/browse/HADOOP-19406. + + +2. **Context Provider Key (CPK)** support is currently not available. It refers to the ability to use a +customer-provided encryption key to encrypt and decrypt data in Azure Blob +Storage. This feature allows users to manage their own encryption keys, +providing an additional layer of security and control over their data. + +## Ask all about ABFS Driver + +For any queries related to onboard to FNS Blob or anything related to ABFS +Driver in general, kindly reach out to us at **askabfs@microsoft.com**. \ No newline at end of file diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java index dc9c0e25cc1d3..e563e082b80cc 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java @@ -28,7 +28,6 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; -import org.junit.Assume; import org.junit.Test; import org.mockito.Mockito; import org.mockito.stubbing.Stubber; @@ -39,7 +38,6 @@ import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.LocatedFileStatus; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType; import org.apache.hadoop.fs.azurebfs.constants.FSOperationType; import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations; import org.apache.hadoop.fs.azurebfs.contracts.services.DfsListResultEntrySchema;