-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial cut for a cuVS Java API #450
Open
chatman
wants to merge
55
commits into
rapidsai:branch-25.02
Choose a base branch
from
SearchScale:java-api
base: branch-25.02
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
c777924
Initial cut for a cuVS Java API
f9c2df7
API changes, serialize optional tmpFile param + other changes
1597158
make topK configurable
6f405e9
multiple fixes
8531be2
pom file update
c9b8891
sample test
9bb865a
sample test update + other changes
d40b3d0
sample test update + use logging
1e03585
variable name updates + other
3222536
Javadoc draft 1
c5536b6
Fixes for Javadocs, reducing visibility of internals, other refactorings
chatman 2d79580
Loading the .so file from the jar instead of a hardcoded path
84c6121
package renaming and formatting
20562ee
moving CuVSResources + other updates
ad11369
code refactoring
6a851ef
javadoc maven config update and bug fixes
68e8b94
Consolidating all Arenas and Linkers to CuVSResources, fixing copyrig…
3852f6e
Bumping version to 24.12, installing to local maven with build.sh & s…
479e488
major performance improvement - reduced creation time for memory segm…
43f5e15
Adding randomized test, fixes for memory allocation and deallocation,…
narangvivek10 3438dbf
Initial cut for a cuVS Java API
4411629
API changes, serialize optional tmpFile param + other changes
b7da0ef
make topK configurable
c4346d9
multiple fixes
cefadb8
pom file update
8492c72
sample test
8e7c62c
sample test update + other changes
eb94a0f
sample test update + use logging
d32f8f9
variable name updates + other
bbcbba1
Javadoc draft 1
696f015
Fixes for Javadocs, reducing visibility of internals, other refactorings
chatman b95bb7f
Loading the .so file from the jar instead of a hardcoded path
9ab2754
package renaming and formatting
638092c
moving CuVSResources + other updates
51eb397
code refactoring
fb295dd
javadoc maven config update and bug fixes
80174d2
Consolidating all Arenas and Linkers to CuVSResources, fixing copyrig…
61ea9ac
Bumping version to 24.12, installing to local maven with build.sh & s…
1803dee
major performance improvement - reduced creation time for memory segm…
7cda1e5
Adding randomized test, fixes for memory allocation and deallocation,…
narangvivek10 53dfa23
Merge branch 'branch-25.02' into java-api
6f708eb
Upgrading to 25.02
b98d2bb
Bruteforce API implementation (#8)
narangvivek10 81b60f1
Bruteforce serialize and deserialize API implementation (#9)
narangvivek10 dca835d
Merge branch 'rapidsai:branch-25.02' into java-api
narangvivek10 e3757c0
`metric` parameter addition to the CAGRA index parameters (#10)
narangvivek10 0aa2665
update year
narangvivek10 a89fe5d
API update - move prefilter from BruteForceIndex to BruteForceQuery
narangvivek10 b44baa9
HNSW API implementation
narangvivek10 98c3c93
Mapping is a single integer list, added an error logging for absurd v…
feed385
Compilation error fix (enum needs to be mentioned as per C standards)
1d80b41
If dataset is smaller than topK, then adjust topK to be the dataset size
2f0f143
Add randomized tests for HNSW and Bruteforce
narangvivek10 2c0e4cd
Adding module-info.java
72bc6e3
Review updates
narangvivek10 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -83,3 +83,8 @@ ivf_pq_index | |
# cuvs_bench | ||
datasets/ | ||
/*.json | ||
|
||
# java | ||
.classpath | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
Prerequisites | ||
------------- | ||
|
||
* JDK 22 | ||
* Maven 3.9.6 or later | ||
|
||
To build this API, please do `./build.sh java` in the top level directory. Since this API is dependent on `libcuvs` it must be noted that `libcuvs` gets built automatically before building this API. | ||
|
||
Alternatively, please build libcuvs (`./build.sh libcuvs` from top level directory) before building the Java API with `./build.sh` from this directory. | ||
|
||
Building | ||
-------- | ||
|
||
`./build.sh` will generate the libcuvs_java.so file in internal/ directory, and then build the final jar file for the cuVS Java API in cuvs-java/ directory. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
VERSION="25.02" | ||
GROUP_ID="com.nvidia.cuvs" | ||
SO_FILE_PATH="./internal" | ||
|
||
cd internal && cmake . && cmake --build . \ | ||
&& cd .. \ | ||
&& mvn install:install-file -DgroupId=$GROUP_ID -DartifactId=cuvs-java-internal -Dversion=$VERSION -Dpackaging=so -Dfile=$SO_FILE_PATH/libcuvs_java.so \ | ||
&& cd cuvs-java \ | ||
&& mvn package \ | ||
&& mvn install:install-file -Dfile=./target/cuvs-java-$VERSION-jar-with-dependencies.jar -DgroupId=$GROUP_ID -DartifactId=cuvs-java -Dversion=$VERSION -Dpackaging=jar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,164 @@ | ||
<!-- | ||
/* | ||
* Copyright (c) 2025, NVIDIA CORPORATION. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
--> | ||
<project xmlns="http://maven.apache.org/POM/4.0.0" | ||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
<modelVersion>4.0.0</modelVersion> | ||
<groupId>com.nvidia.cuvs</groupId> | ||
<artifactId>cuvs-java</artifactId> | ||
<version>25.02</version> | ||
<name>cuvs-java</name> | ||
<packaging>jar</packaging> | ||
|
||
<properties> | ||
<maven.compiler.target>22</maven.compiler.target> | ||
<maven.compiler.source>22</maven.compiler.source> | ||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | ||
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> | ||
</properties> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>commons-io</groupId> | ||
<artifactId>commons-io</artifactId> | ||
<version>2.15.1</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>com.github.fommil</groupId> | ||
<artifactId>jniloader</artifactId> | ||
<version>1.1</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.slf4j</groupId> | ||
<artifactId>slf4j-api</artifactId> | ||
<version>2.0.13</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.slf4j</groupId> | ||
<artifactId>slf4j-simple</artifactId> | ||
<version>2.0.13</version> | ||
<scope>runtime</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>com.carrotsearch.randomizedtesting</groupId> | ||
<artifactId>randomizedtesting-runner</artifactId> | ||
<version>2.8.2</version> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>junit</groupId> | ||
<artifactId>junit</artifactId> | ||
<version>4.13.1</version> | ||
<scope>test</scope> | ||
</dependency> | ||
|
||
</dependencies> | ||
|
||
<build> | ||
<plugins> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-surefire-plugin</artifactId> | ||
<version>2.7</version> | ||
<configuration> | ||
<systemPropertyVariables> | ||
<java.library.path>${project.build.directory}/classes</java.library.path> | ||
</systemPropertyVariables> | ||
</configuration> | ||
</plugin> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-dependency-plugin</artifactId> | ||
<version>2.10</version> | ||
<executions> | ||
<execution> | ||
<id>copy</id> | ||
<phase>compile</phase> | ||
<goals> | ||
<goal>copy</goal> | ||
</goals> | ||
<configuration> | ||
<artifactItems> | ||
<artifactItem> | ||
<groupId>com.nvidia.cuvs</groupId> | ||
<artifactId>cuvs-java-internal</artifactId> | ||
<version>25.02</version> | ||
<type>so</type> | ||
<overWrite>false</overWrite> | ||
<outputDirectory> | ||
${project.build.directory}/classes</outputDirectory> | ||
<destFileName>libcuvs_java.so</destFileName> | ||
</artifactItem> | ||
</artifactItems> | ||
</configuration> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-assembly-plugin</artifactId> | ||
<version>3.4.2</version> | ||
<configuration> | ||
<descriptorRefs> | ||
<descriptorRef>jar-with-dependencies</descriptorRef> | ||
</descriptorRefs> | ||
<archiverConfig> | ||
<duplicateBehavior>add</duplicateBehavior> | ||
</archiverConfig> | ||
</configuration> | ||
<executions> | ||
<execution> | ||
<id>assemble-all</id> | ||
<phase>package</phase> | ||
<goals> | ||
<goal>single</goal> | ||
</goals> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-jar-plugin</artifactId> | ||
<version>2.2</version> | ||
<configuration> | ||
<archive> | ||
<manifest> | ||
<addClasspath>true</addClasspath> | ||
<mainClass> | ||
com.nvidia.cuvs.examples.CagraExample</mainClass> | ||
</manifest> | ||
</archive> | ||
</configuration> | ||
</plugin> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-javadoc-plugin</artifactId> | ||
<version>3.6.2</version> | ||
<configuration> | ||
<excludePackageNames>com.nvidia.cuvs.examples,com.nvidia.cuvs.panama</excludePackageNames> | ||
<reportOutputDirectory>${project.build.directory}</reportOutputDirectory> | ||
</configuration> | ||
</plugin> | ||
</plugins> | ||
</build> | ||
</project> |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should make this smart enough to automatically build libcuvs when the
java
arg is supplied. Just makes the overall experience better for users when they don't need to know all the ins and outs of the build in order to get a build from source.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated.