Problems running bench with 1M files

[HanSolo]

Downloaded the 1M dataset and adjusted the code in DownloadHelper and Bench according to the new file names but when trying to run the bench example using ./mvnw -Pjdk11 compile exec:exec@bench it stops running after a while with a RuntimeException.
RuntimeException_executing_bench_on_1M_dataset.txt
machine_info.txt

Is there anything specific one need to adjust to make the bench example run with the 1M dataset?

[jbellis]

Exception in thread "main" java.lang.RuntimeException: File sizes greater than 2GB are not supported on Windows--contributions welcome

Longer explanation: the easiest way to make a reasonably performant RandomAccessReader class is to use mmap, but Java's standard mmap only supports segments up to 2GB. Hence the comments here:

/**
 * Simple sample implementation of RandomAccessReader.
 * It provides a bare minimum to run against disk in reasonable time.
 * Does not handle files above 2 GB.
 */
public class SimpleMappedReader implements RandomAccessReader {

We plugged in a native library to work around this in MMapReader, but that library does not support Windows.

So your options include (from easiest to hardest)

1. Use mac or linux to test large graphs
2. Update SimpleMappedReader to segment large files into < 2GB chunks
3. Implement a RandomAccessReader using standard io that matches mmap's performance
4. Implement Windows support for the native library we use https://github.com/indeedeng/util/tree/main/mmap

[HanSolo]

Yep I know that but I’m running on Linux x64 which is the reason why I was surprised about the 2GB limit.

[jbellis]

        try {
            return new MMapReaderSupplier(path);
        } catch (UnsatisfiedLinkError|NoClassDefFoundError e) {
            if (Files.size(path) > Integer.MAX_VALUE) {
                throw new RuntimeException("File sizes greater than 2GB are not supported on Windows--contributions welcome");
            }

            return new SimpleMappedReaderSupplier(path);
        }

looks like for some reason it can't find the native mmap library

[HanSolo]

Will do, thx for the hint 👍🏻

[jkni]

Thanks for the report @HanSolo. This identifies an issue with JVector and not your setup. I'll push a PR shortly that provides better diagnostic output and fixes the issue (which I'll link here).

[jkni]

Issue reproduced and resolved locally by the linked PR. If this PR doesn't resolve your issue, it should at least provide a clearer reason for the fallback.