Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] cuvs v24.06 #168

Merged
merged 51 commits into from
Jun 5, 2024
Merged

[RELEASE] cuvs v24.06 #168

merged 51 commits into from
Jun 5, 2024

Conversation

raydouglass
Copy link
Member

❄️ Code freeze for branch-24.06 and v24.06 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.06 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-24.06 into main for the release

raydouglass and others added 30 commits March 18, 2024 09:27
@raydouglass
DOC v24.06 Updates [skip ci]
0f11d07
@benfred
Merge branch 'branch-24.04' into branch-24.06-merge-24.04
8f73bdb
@benfred
Update devcontainers to use cuda12.2 (#72)
36926d2 
Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #72
@benfred
Enable forward-merger ops-bot plugin (#70)
1e3206a 
Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #70
@cjnolet
Merge branch 'branch-24.06' into branch-24.06-merge-24.04
fa42fea
@cjnolet
Fixing brute-force python build
e7e3f56
@raydouglass
Merge pull request #71 from benfred/branch-24.06-merge-24.04
72db6b9 
Forward merge branch-24.04 to branch-24.06
@cjnolet
Add lucene-cuvs to integrations section of docs (#73)
4488626 
Authors:
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Ben Frederickson (https://github.com/benfred)

URL: #73
@benfred
Add Cagra-Q compression to the python and rust api's (#68)
6ad58e3 
Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #68
@galipremsagar @cjnolet
Enable all tests for arm jobs (#63)
26377ef 
This PR enables all tests in `arm` jobs.

xref: rapidsai/build-planning#36

---------

Co-authored-by: Corey J. Nolet <[email protected]>
@benfred
Merge branch-24.04 into branch-24.06
cc29792
@AyodeAwe
Merge pull request #80 from benfred/branch-24.06-merge-24.04
bc9d2d5 
Forward merge branch-24.04 to branch-24.06
@KyleFromNVIDIA
Update the developer's guide with new copyright hook (#81)
8df8c49 
Issue: rapidsai/build-planning#30

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - Ben Frederickson (https://github.com/benfred)

URL: #81
@robertmaynard
Use static gtest (#83)
a237049 
Allows us to remove the gtest dependency from our conda env for testing.

Authors:
  - Robert Maynard (https://github.com/robertmaynard)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Jake Awe (https://github.com/AyodeAwe)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #83
@benfred
Fix rust docs build (#84)
da1e63b 
The rust API docs aren't being generated on docs.rapids.ai/cuvs . While the `build.sh docs` script was including the rust api docs, the `ci/build_docs.sh` wasn't. Fix.

Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #84
@trxcllnt
Update pip devcontainers to UCX v1.15.0 (#88)
aea7f54 
Authors:
  - Paul Taylor (https://github.com/trxcllnt)

Approvers:
  - Ben Frederickson (https://github.com/benfred)

URL: #88
@hurutoriya
update: fix RAFT URL in README (#91)
5ff588d 
## What
SSIA

## Why
Maintain Docs

Authors:
  - Shunya Ueta (https://github.com/hurutoriya)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #91
@robertmaynard
Remove gtest from dependencies.yaml (#87)
b38dc11 
Missed these files in #83

Authors:
  - Robert Maynard (https://github.com/robertmaynard)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #87
@trxcllnt
add --rm and --name to devcontainer run args (#89)
32235a4 
* Remove the devcontainer when the VSCode window closes
* Adds a descriptive name to the running container:
  ```shell
  $ docker ps -a
  CONTAINER ID   IMAGE         ...  NAMES
  0dbb364fe544   vsc-cuvs-...  ...  rapids-cuvs-24.06-cuda12.2-conda
  
  $ docker rm -f rapids-cuvs-24.06-cuda12.2-conda
  ```

Authors:
  - Paul Taylor (https://github.com/trxcllnt)

Approvers:
  - Ben Frederickson (https://github.com/benfred)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #89
@cjnolet
Moving and renaming distance namespaces from raft -> cuvs (#85)
8db4638 
This PR moves the `raft::distance` APIs over to cuVS. We're ripping the band-aid off quickly here so moving all over at once. 

All of the non-public APIs are being moved to `src/distance` and only the public APIs being moved to `include/cuvs/distance`.

Authors:
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Ben Frederickson (https://github.com/benfred)
  - Ray Douglass (https://github.com/raydouglass)

URL: #85
@bdice
Remove libnvjitlink dependency. (#97)
6399d05 
PR #85 added a dependency on `libnvjitlink`. This was needed only temporarily, due to an issue with a particular build of `cupy` that caused an undesired conda environment. This should be safe to remove.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: #97
@benfred
Add python and rust bindings for Ivf-Pq (#90)
82de6ef 
Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #90
@benfred
Add python and rust bindings for Ivf-Flat (#82)
00616f9 
Authors:
  - Ben Frederickson (https://github.com/benfred)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #82
@mroeschke
Enable Warnings as errors in Python tests (#102)
d44aa39 
Part of rapidsai/build-planning#26, warnings in Python tests will now be treated as errors

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #102
@hcho3
Migrate to {{ stdlib("c") }} (#93)
07791df 
The `sysroot*` syntax is getting phased out (conda-forge/conda-forge.github.io#2102).
The recommendation is to move to `{{ stdlib("c") }}`.

Ref rapidsai/build-planning#39

Authors:
  - Philip Hyunsu Cho (https://github.com/hcho3)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - https://github.com/jakirkham
  - Ray Douglass (https://github.com/raydouglass)

URL: #93
@lowener
Migrate IVF-PQ from RAFT to cuVS (#86)
22f107c 
List of changes I made during the migration:
- The unit tests are testing IVFPQ codepacker and build functions. So these functions had to be exposed (through `cuvs/neighbors/details`)
- The bitset filter is now located under `cuvs/neighbors/bitset_filter.cuh"
- search_with_filter is added to the public API, with bitset_filter
- Bitset is exposed in public API in a `.hpp` header, for inclusion in CUDA-free code. `bitset::test()` function has been temporarily disabled due to the switch from `.cuh` to `.hpp`.

Authors:
  - Micka (https://github.com/lowener)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #86
@vyasr
Pass through raft static CMake var (#111)
9f02d70 
cuvs wheel builds were deviating from the behavior of the rest of RAPIDS before this PR by not linking statically to libraft.so but instead building the full dynamic and static lib (the default behavior) and then presumably bundling libraft.so. This change should fix that linking.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #111
@benfred
Fix rust api docs (#119)
55e60ff 
The rust api docs on docs.rapids.ai are deployed to https://docs.rapids.ai/api/cuvs/nightly/rust_api/index.html but the relative link in the iframe was assuming that these were at  https://docs.rapids.ai/api/cuvs/nightly/rust_api.html. (There seems to be some difference in the url structure depending on whether you build locally or not)

Fix by forcing the url structure to always be `rust_api/index.html`

Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #119
@divyegala
Migrate IVF-Flat from RAFT (#94)
bd7df64 
Authors:
  - Divye Gala (https://github.com/divyegala)
  - Micka (https://github.com/lowener)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #94
@cjnolet
Migrate raft::cluster to cuvs::cluster (#115)
bb193ab 
Authors:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Micka (https://github.com/lowener)
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - Divye Gala (https://github.com/divyegala)

URL: #115
trxcllnt and others added 14 commits May 23, 2024 11:12
@trxcllnt
hide RAFT #pragma deprecation warnings (#114)
ac85fa6 
Define `-DRAFT_HIDE_DEPRECATION_WARNINGS` added in rapidsai/raft#2271.

Authors:
  - Paul Taylor (https://github.com/trxcllnt)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #114
@cjnolet
Changing RAFT_EXPLICT_* to CUVS_EXPLITI_* (#141)
3515f44 
Changing this for now. We'll work on removing this macro in a future release.

Authors:
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Ben Frederickson (https://github.com/benfred)
  - Divye Gala (https://github.com/divyegala)

URL: #141
@benfred
Add pairwise_distance api's for C, Python and Rust (#142)
4b260aa 
Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #142
@trxcllnt
fix devcontainer name for codespaces (#153)
b1f17ed 
This PR fixes launching the devcontainers in GitHub Codespaces: 
![image](https://github.com/rapidsai/cuvs/assets/178183/71d4bab2-82b1-4671-b382-c5f21503dfce)

Authors:
  - Paul Taylor (https://github.com/trxcllnt)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #153
@lowener
Add refine to public API (#154)
b9e8a4b 
Add `cuvs::neighbors::refine` to public API, with it's test

Authors:
  - Micka (https://github.com/lowener)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Tamas Bela Feher (https://github.com/tfeher)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #154
@tfeher
Accept host_mdspan for IVF-PQ build and extend (#148)
1912355 
This PR enables host input arrays for `ivf_pq::build` and `ivf_pq::extend`.

closes #120 
closes #143

Authors:
  - Tamas Bela Feher (https://github.com/tfeher)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #148
@rhdong
[FEA] support of prefiltered brute force (#146)
c533fe3 
- The PR is one part of prefiltered brute force and should work with the PR of raft: rapidsai/raft#2294

Authors:
  - rhdong (https://github.com/rhdong)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #146
@eltociear
chore: update Doxyfile (#162)
5937d54 
efficively -> effectively

Authors:
  - Ikko Eltociear Ashimine (https://github.com/eltociear)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #162
@benfred
Select k instantiations (#159)
758d147 
Since we're now using raft in header only mode, we don't have the compiled select_k instantiations in raft available to us anymore. Instead instantiate inside cuvs so we don't recompile in multiple spots.

Authors:
  - Ben Frederickson (https://github.com/benfred)

Approvers:
  - Tamas Bela Feher (https://github.com/tfeher)
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #159
@hurutoriya
update: replace to cuvs from RAFT in PULL_REQUEST_TEMPLATE (#92)
e0d169e 
## What

SSIA

Authors:
  - Shunya Ueta (https://github.com/hurutoriya)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #92
@cjnolet
cuVS docs updates for release (#161)
a4c3a6f 
Authors:
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Ben Frederickson (https://github.com/benfred)

URL: #161
@benfred
Expose serialization to the python / c-api (#164)
35f8b84 
Authors:
  - Ben Frederickson (https://github.com/benfred)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #164
@mfoerste4
CAGRA API update and allow async host refinement (#131)
13e0732 
This PR updates the CAGRA public API, changes defaults, and improves refinement during IVF-PQ build step.

Updated defaults:
- By default CAGRA would select NN descent. We fall back to IVF-PQ build algorithm if there is not enough memory for NN descent.
- For the IVF-PQ build algo, the search params were updated to use `n_probe = 0.01*nlist`, and both LUT and internal distance type is set to `fp16` (as opposed to `fp8/fp32` previously)

- By default build would create the index that contains both the `graph` and the `dataset` on GPU. If the `dataset` does not fit gpu, then the returned index will only contain the graph (on device). In such case the user is expected to call `index.update_dataset()`  (for example with dataset in managed memory) before we can `search` the index. 

API changes:
- We can specify IVF-PQ build algo parameters the following way:
```c++
cagra::index_params params;
params.graph_degree = 32;
params.intermediate_graph_degree = 48;
auto pq_params = cagra::graph_build_algo::ivf_pq_params(dataset.extents());
// This sets reasonable defaults, but can be updated by the user, e.g.:
pq_params.pq_dim = 32;
// Select IVF-PQ algorithm by passing ivf_pq_params as graph_build_algo
params.graph_build_algo = pq_params;
```

- We can specify NN descent algo parameters a similar way
```c++
cagra::index_params params;
params.graph_degree = 32;
params.intermediate_graph_degree = 48;
// Select NN descent algo by passing nn_descent_params
params.graph_build_algo = cagra::graph_build_params::nn_descent_params(intermediate_degree) ;
```

Additionally, this PR optimizes the IVF-PQ refinement step within the CAGRA graph creation in case the dataset is in host-memory.

* depending on hardware, a certain amount of refinement comes for free
* the change also has a positive effect without refinement as the graph creation is done on host either way

Authors:
  - Malte Förster (https://github.com/mfoerste4)
  - Tamas Bela Feher (https://github.com/tfeher)
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Tamas Bela Feher (https://github.com/tfeher)

URL: #131
@tfeher
Fix CAGRA OOM handling (#167)
66959b1 
Fixes handling OOM error during CAGRA index creation, that was introduced in #131.

Authors:
  - Tamas Bela Feher (https://github.com/tfeher)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #167
@raydouglass raydouglass requested review from a team as code owners May 31, 2024 13:19
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented May 31, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cjnolet cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels May 31, 2024
@raydouglass raydouglass merged commit 8a1b8b2 into main Jun 5, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@raydouglass raydouglass

Labels
ci CMake cpp improvement Improves an existing functionality non-breaking Introduces a non-breaking change Python
Projects
VS/ML/DM Primitives Release Board
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.