release-2023-03-17

release of 2023-03-17.j

Improved endpoint/region analysis.

it now tries to identify when an attempt is being made to talk to third party or vpce buckets, and review the settings as appropriate, with suggestions

release-2023-03-03pm

Leasing diagnostics checks improved; verified correct handling of situation where a dir has a lease -second attempt to write a file fails

release-2023-03-03

versioning

"restore" command

adds a new command to restore an s3 object by version.

2023-03-01-listversions

Adds a new command listversions to list all versions of files under an s3 path and print out details in a CSV file for review and recovery.

Consult the documentation for the full details.

bin/hadoop jar cloudstore-1.0.jar listversions -separator ,  s3a://stevel-london/test/target

2023-03-01 18:01:11,231 [main] INFO  extra.ListVersions (StoreDurationInfo.java:<init>(56)) - Starting: listversions

Listing s3a://stevel-london/test/target
=======================================

"index","key","path","restore","latest","size","tombstone","directory","date","timestamp","version","etag"
1,"test/target/","s3a://stevel-london/test/target",0,1,0,1,1,"2023-03-01+000002:01:11",1677679271000,"R.YDYPm780vbXX41W4vGcy2cg8PXxE4F",""
2,"test/target/","s3a://stevel-london/test/target",0,0,0,0,1,"2023-03-01+000002:01:11",1677679271000,"sexI0eOxmrHXJnEcqvYhbNwM3eB4bT3T","df73760194643ac2b2968280c3b7ca4b"
3,"test/target/","s3a://stevel-london/test/target",0,0,0,1,1,"2023-03-01+000002:00:46",1677679246000,"QBw3uSgAHEDCP2tJJF3GxQhvI8B8LhIp",""
4,"test/target/","s3a://stevel-london/test/target",0,0,0,0,1,"2023-03-01+000002:00:45",1677679245000,"6lMGdcckhrqu_eJxftBYHQ4eERr5Gq0y","f384715ff40cca0ca36177b937a21d2f"
5,"test/target/testEncryptionOverRename[delete-markers]-0400","s3a://stevel-london/test/target/testEncryptionOverRename[delete-markers]-0400",0,1,0,1,0,"2023-03-01+000002:01:12",1677679272000,"KoAXT3Et71UHkjVxoXXpCT_v8X2JAhT5",""
6,"test/target/testEncryptionOverRename[delete-markers]-0400","s3a://stevel-london/test/target/testEncryptionOverRename[delete-markers]-0400",1,0,1024,0,0,"2023-03-01+000002:01:11",1677679271000,".2RcOYUhtWM2bWFpyFNmJPb_fbwQNCWv","23d45cf31df15bfa3f176e8621103170"
7,"test/target/testEncryptionOverRename[keep-markers]-0400","s3a://stevel-london/test/target/testEncryptionOverRename[keep-markers]-0400",0,1,0,1,0,"2023-03-01+000002:00:46",1677679246000,"ehNAUNAYm_Ee7xRqgtPgByM6kzBmTQZ3",""
8,"test/target/testEncryptionOverRename[keep-markers]-0400","s3a://stevel-london/test/target/testEncryptionOverRename[keep-markers]-0400",1,0,1024,0,0,"2023-03-01+000002:00:45",1677679245000,"TaTKDWWKImCf77FgdtsUqlzLRgABo4LM","abe6cf8b4d38929146d2d1b2b8060622"

Found 8 objects under s3a://stevel-london/test/target with total size 2,048 bytes
Hidden file count 2 with hidden data size 2,048 bytes
Hidden zero-byte file count 0
Hidden directory markers 2
Tombstone entries 4 comprising 2 files and 2 dir markers
2023-03-01 18:01:12,436 [main] INFO  extra.ListVersions (StoreDurationInfo.java:close(115)) - Duration of listversions: 0:01:206

2023-02-16-release

Abfs and s3a diagnostics to included detailed analysis of the local buffer dir

Reviews state of buffer dir, parent dirs, identifies permission issues,

If invoked with the -w parameter, storediag
then tries to create a temporary file through the LocalDirAllocator, just
as the filesystems will -which should trigger directory creation.

2023-01-30-release

obfuscates fs.gs.auth.service.account.private.key

2023-01-27-release

new tarhardened command to verify the release has the fix for "HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files."

bin/hadoop jar cloudstore-1.0.jar tarhardened "file.tar; true"

Bad

Attempting to untar file with name "file.tar; true"
untar operation reported success

2023-01-27 16:42:35,931 [main] INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 0

Although the file doesn't exist, the bash "true" command was executed after the untar, so
the operation was reported as a success.

Good

2023-01-27 16:48:44,461 [main] INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status -1: ExitCodeException exitCode=1: tar: Error opening archive: Failed to open 'file.tar; true'

The file file.tar; true was attempted to be opened; as it is not present the operation failed.
Expect a stack trace in the report

2022-12-19 safeprefetch

Adds the safeprefetch command to validate prefetch safety of an abfs client, as determined from hadoop release features (pathcapabilities probes) and the site configuration.

2022-11-11 release: mkcsv tuning

even more cross referencing in (now shorter) records, in particularly each data column is always a single char repeated length times.

doc updated to match.