Poor checksum/hash performance

[alex-thiessen-for-siemens]

Describe the bug

When using JFrog-CLI to download artifacts from a generic repository, we experience poor performance when files are already present locally.

We use self-hosted build agents which have local directories where downloaded artifacts are stored semi-permanently (caches), so that they don't need to be downloaded very often. This enables us to access company's central servers directly and helps us avoid complications of using site-local Artifactory satellites.

Downside is that it takes JFrog-CLI about twice as long to hash the local files than to download them. Typical figures I observed in the last days (256 files, total size 3.1 GB, residing on an SSD of a machine with an Intel Celeron N2930 @ 1.83GHz and 8 GB of RAM):

• download (without local files): ~ 2 minutes
• download (with local files): ~ 3-5 minutes
• sha256sum run over the local files: 51 seconds
• openssl sha256 run over the local files: 34 seconds

We also observe such issues with Xeon machines (which also lack hardware support for SHA256, as a side note).

Current behavior

Download process indicates that files are present locally (which they are), but it takes roughly twice as long as a data-transferring download does. CPU is stressed during the process, which indicates that the checksum algorithm implementation lacks performance).

Reproduction steps

• Upload a large file to a generic repository
• Download the file to an empty directory, observe good performance
• Repeat the download with the local file present, observe "file present locally" messages but poor performance and high CPU load

Expected behavior

Having data already present locally accelerates downloads, with the performance being in the ballpark of openssl sha256 times.

JFrog CLI version

2.72.2

Operating system type and version

Debian 12 x64

JFrog Artifactory version

JFrog Platform Enterprise Plus 7.98.8 rev 79808900

JFrog Xray version

N/A

[alex-thiessen-for-siemens]

FYI: I observe a wide spread in performance measurements, ranging between 1m 37s and 3m 15s. The machine which runs JFrog-CLI is company-managed, so it runs puppet agent in the background which seems to require substantial CPU capacity. Still, even the best time is far from the openssl sha256 performance.