release-2023-10-17

bandwith command now includes block count as a column in the csv output.

Example: 16MB upload against gcs through s3a connector, 8 MB clock size

hadoop jar cloudstore-1.0.jar bandwidth -block 8  -csv s3a.stevel.gcs.csv 16m s3a://stevel-gcs/bandwidth.bin

CSV file

"operation","iteration","bytes","total","duration"
"create-file",1,0,0,772
"upload-block",1,8388608,8388608,42
"upload-block",2,8388608,16777216,35
"close-upload",1,0,16777216,6926
"upload",1,8388608,8388608,7788
"open-for-download",1,0,0,159
"download-block",1,8388608,8388608,2888
"download-block",2,8388608,16777216,2650
"download",1,16777216,16777216,5702

release-2023-10-16

abfs diags includes the fs.azure.enable.readahead.v2 option; checks for prefetch safety in path capabilities scan

release-2023-10-03

system properties printed by abfs and gcs include proxy info

Selected System Properties
==========================

[001]  https.proxyHost = (unset)
[002]  https.proxyPort = (unset)
[003]  https.nonProxyHosts = (unset)
[004]  https.proxyPassword = (unset)
[005]  http.proxyHost = (unset)
[006]  http.proxyPort = (unset)
[007]  http.proxyPassword = (unset)
[008]  http.nonProxyHosts = (unset)
[009]  java.net.preferIPv4Stack = "true"
[010]  java.net.preferIPv6Addresses = (unset)
[011]  networkaddress.cache.ttl = (unset)
[012]  networkaddress.cache.negative.ttl = (unset)
[013]  socksProxyHost = (unset)
[014]  socksProxyPort = (unset)
[015]  sun.net.client.defaultConnectTimeout = (unset)
[016]  sun.net.client.defaultReadTimeout = (unset)
[017]  sun.net.inetaddr.ttl = (unset)
[018]  sun.net.inetaddr.negative.ttl = (unset)
[019]  java.version = "1.8.0_362"

release-2023-09-29

release of 2023-09-29

release-2023-09-26: cloudup

Major Cloudup rework

• incremental -update operation to skip files which exist.
• improved output
• options to flush/hflush
• standalone document with more examples
• tuning for maximum s3a performance

see cloudup

Release 2023-09-14. Bandwidth enhancements.

Release 2022-09-14. Bandwidth enhancements.

The bandwidth command now

• can save details to a CSV file
• adds options to set: read policy, block size, whether to flush/hflush after each write
• reports whether progress callbacks were made during (possibly slow) close() operations.

Read the bandwidth documentation for details and example analysis of CSV files using
different S3A tuning parameters (including prefetching).

release-2023-08-21

• analyzes signing algorithm, including warnings about v2 sdk compatibility
• a bit more v2 sdk awareness
• prints UTC timestamp of when the log was collected

release-2023-08-10: AWS SDK v2 awareness

release of 2023-08-10

s3a diagnostics now works with aws v2 sdk branches.

• looks for the different classes
• all code using aws sdk v1 classes wrapped by exception handling
• endpoint analysis recognises fs.s3a.endpoint set to an ipv4 dotted address and comments on that (https won't work, path style access doomed)

Note: the low-level s3 operations only work on v1 sdk hadoop releases; no immediate plans to switch

release-2023-06-15: storediag critical fix

Critical bug fix: bucket propagation

• S3A per-bucket values weren't being printed in storediag,
  even though they were used to configure the target FS.
• the specific AWS credential key which was used to auth with the store
  is now printed (obfuscated)

release-2023-06-14

bandwidth command: add -keep and -rename options