Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hnsw index #21186

Merged
merged 260 commits into from
Feb 25, 2025
Merged
Changes from 1 commit
Commits
Show all changes
260 commits
Select commit Hold shift + click to select a range
e86ed1e
ignore license check
cpegeric Jan 10, 2025
5330453
Merge branch 'hnsw_index' of github.com:cpegeric/matrixone into hnsw_…
cpegeric Jan 10, 2025
53d7ea7
update test
cpegeric Jan 10, 2025
a2d0d7c
license
cpegeric Jan 10, 2025
cc40f44
usearch testing
cpegeric Jan 10, 2025
69983d1
fix ut
cpegeric Jan 10, 2025
1a34e07
ut depends on cgo thirdparties
cpegeric Jan 10, 2025
2f9a252
add static library
cpegeric Jan 10, 2025
d4408e3
musl not compatiable with g++
cpegeric Jan 10, 2025
566b0c5
musl
cpegeric Jan 13, 2025
0e60220
musl bug fix
cpegeric Jan 13, 2025
f200286
comments
cpegeric Jan 13, 2025
f61eac5
Merge branch 'main' into hnsw_index
cpegeric Jan 13, 2025
283a712
musl update
cpegeric Jan 14, 2025
e7009f2
Merge branch 'hnsw_index' of github.com:cpegeric/matrixone into hnsw_…
cpegeric Jan 14, 2025
34d92ae
add DEBUG_OPT
cpegeric Jan 14, 2025
fe3c33e
revert DEBUG_OPT
cpegeric Jan 14, 2025
c4f6a37
parser support hnsw index
cpegeric Jan 14, 2025
d16d7c3
Merge branch 'main' into hnsw_index
cpegeric Jan 14, 2025
c938f9c
update
cpegeric Jan 15, 2025
57fd28c
update x86_64 cmake
cpegeric Jan 15, 2025
c5b4169
x86_64 optimize with simsimd
cpegeric Jan 15, 2025
754fde1
Makefile
cpegeric Jan 15, 2025
dc7eaaa
update
cpegeric Jan 15, 2025
f604dd1
update
cpegeric Jan 15, 2025
0148c42
update
cpegeric Jan 15, 2025
96dcd88
update CC
cpegeric Jan 15, 2025
0a30324
disable simsimd
cpegeric Jan 15, 2025
9c6f5e1
disable simsimd for now
cpegeric Jan 15, 2025
ed7e9df
update
cpegeric Jan 15, 2025
5fa6289
Merge branch 'main' into hnsw_dev
cpegeric Jan 15, 2025
1a7fd43
update
cpegeric Jan 15, 2025
70467df
compile code update
cpegeric Jan 15, 2025
762eb05
bug fix
cpegeric Jan 16, 2025
ff587c4
update
cpegeric Jan 16, 2025
2ff6e02
bug fix table type name crashed with ivfflat table type name
cpegeric Jan 16, 2025
7628429
quantization
cpegeric Jan 16, 2025
f53cb3d
update
cpegeric Jan 16, 2025
86d9151
Merge branch 'main' into hnsw_dev
cpegeric Jan 16, 2025
e481029
table_function call end() when end of input
cpegeric Jan 16, 2025
07d2873
apply end
cpegeric Jan 17, 2025
06e103b
table_function
cpegeric Jan 17, 2025
c5bfb42
update makefile
cpegeric Jan 17, 2025
1fad511
enable openmp on linux
cpegeric Jan 17, 2025
9e131ab
check avx512fp16
cpegeric Jan 17, 2025
8b4419c
check avx512fp16
cpegeric Jan 17, 2025
70c026f
checkt avx512fp16
cpegeric Jan 17, 2025
2e1bf6e
Merge branch 'main' into hnsw_index
cpegeric Jan 17, 2025
5ecdebd
merge fix
cpegeric Jan 17, 2025
098411b
table function
cpegeric Jan 17, 2025
f4d8dc3
add types.go
cpegeric Jan 17, 2025
0893107
update
cpegeric Jan 17, 2025
a8886dd
check type when init
cpegeric Jan 17, 2025
80266cd
create index
cpegeric Jan 20, 2025
f9ed8f7
update
cpegeric Jan 20, 2025
d93552f
usearch index config
cpegeric Jan 20, 2025
4503a83
update
cpegeric Jan 20, 2025
63d87f0
update
cpegeric Jan 20, 2025
117f5d5
checksum
cpegeric Jan 20, 2025
64e7bb5
update
cpegeric Jan 20, 2025
cd9ec9a
update
cpegeric Jan 20, 2025
77001af
add cache
cpegeric Jan 21, 2025
f58eead
update
cpegeric Jan 21, 2025
e1a1c7c
update
cpegeric Jan 21, 2025
1ae70a0
update
cpegeric Jan 21, 2025
d9d1ecd
update
cpegeric Jan 21, 2025
c152363
update
cpegeric Jan 21, 2025
5ada712
update
cpegeric Jan 21, 2025
bd9270c
sync.Once
cpegeric Jan 21, 2025
b7ce43d
update
cpegeric Jan 21, 2025
a13ca24
expired
cpegeric Jan 21, 2025
b0857fe
update
cpegeric Jan 21, 2025
897d543
update
cpegeric Jan 21, 2025
a9a3aca
update
cpegeric Jan 21, 2025
42fa247
update
cpegeric Jan 21, 2025
b239c5b
delete from map when Load from database failed
cpegeric Jan 21, 2025
ac155fc
update
cpegeric Jan 21, 2025
48eef00
remove test
cpegeric Jan 21, 2025
7519be1
update
cpegeric Jan 21, 2025
89f519f
update
cpegeric Jan 21, 2025
3b64525
defer ticker.Stop
cpegeric Jan 21, 2025
025d180
update
cpegeric Jan 22, 2025
146711b
search
cpegeric Jan 22, 2025
52792e6
check house keep
cpegeric Jan 22, 2025
a664c09
time in micro
cpegeric Jan 22, 2025
ac21d2f
checksum
cpegeric Jan 22, 2025
42ed6af
thread safe heap
cpegeric Jan 22, 2025
45537f9
int64 id
cpegeric Jan 22, 2025
c0806e9
search with multiple threads
cpegeric Jan 22, 2025
192a8cf
l2sq always positive
cpegeric Jan 22, 2025
5796cf8
unixmicro
cpegeric Jan 22, 2025
ac72b40
add LastUpdate
cpegeric Jan 22, 2025
3efdaf7
remove cache by key
cpegeric Jan 22, 2025
974a0b2
destroy remove all keys
cpegeric Jan 22, 2025
58c6685
bug fix erros.Join
cpegeric Jan 22, 2025
84caf0b
generalized vector index cache
cpegeric Jan 23, 2025
0590238
update
cpegeric Jan 23, 2025
e055e04
cleanup
cpegeric Jan 23, 2025
de53357
cleanup
cpegeric Jan 23, 2025
d43cbe5
cleanup
cpegeric Jan 23, 2025
a676937
cleanup
cpegeric Jan 23, 2025
07bf903
update
cpegeric Jan 23, 2025
1ba7762
cleanup
cpegeric Jan 23, 2025
5fec6ea
comments
cpegeric Jan 23, 2025
334c406
update comments
cpegeric Jan 23, 2025
0aed0de
comments
cpegeric Jan 23, 2025
d8d8036
limit number of threads for search mini-indexes
cpegeric Jan 23, 2025
914418d
bug fix
cpegeric Jan 23, 2025
a57863e
fix indexes is empty
cpegeric Jan 23, 2025
ccbfb38
rename
cpegeric Jan 23, 2025
f570782
cleanup
cpegeric Jan 23, 2025
4d47db9
mocksearch
cpegeric Jan 23, 2025
a2541d5
cache expired
cpegeric Jan 23, 2025
ea0b729
bug fix
cpegeric Jan 23, 2025
88f4b59
more tests
cpegeric Jan 23, 2025
a06d54a
bug fix remove GetIndex
cpegeric Jan 23, 2025
24b0351
race test
cpegeric Jan 23, 2025
d83c819
cleanup
cpegeric Jan 24, 2025
271235b
test
cpegeric Jan 24, 2025
8284468
check status
cpegeric Jan 24, 2025
dd828f3
remove lock from expired
cpegeric Jan 24, 2025
490add5
update tests
cpegeric Jan 24, 2025
1d519f6
add hnsw test
cpegeric Jan 24, 2025
4dc919d
add index file
cpegeric Jan 24, 2025
05b0109
tests
cpegeric Jan 24, 2025
aa7a7bb
update
cpegeric Jan 24, 2025
08bc50b
test Once
cpegeric Jan 24, 2025
6a9c34e
add build test
cpegeric Jan 24, 2025
2800609
stub newHnswAlgo
cpegeric Jan 24, 2025
1a52395
hnsw_create test
cpegeric Jan 24, 2025
d3eb0e6
TODO load file
cpegeric Jan 25, 2025
306eccf
fallocate
cpegeric Jan 25, 2025
54cf8bd
add filesize and load chunk without order by
cpegeric Jan 25, 2025
75a42bc
fix tests
cpegeric Jan 25, 2025
e28845a
update config
cpegeric Jan 27, 2025
03ad23c
hnsw_search
cpegeric Jan 27, 2025
78d250a
ef_search
cpegeric Jan 27, 2025
2befcc6
ef_search
cpegeric Jan 27, 2025
190f34a
ef_search
cpegeric Jan 27, 2025
fa59add
remove ef_search sys var
cpegeric Jan 27, 2025
5b50e29
apply index
cpegeric Jan 27, 2025
2c9b492
update
cpegeric Jan 28, 2025
2ef55cb
update
cpegeric Jan 28, 2025
237f173
working
cpegeric Jan 28, 2025
785ca2d
check limit
cpegeric Jan 28, 2025
3653963
Merge branch 'hnsw_dev' into hnsw_index
cpegeric Jan 28, 2025
3b00137
sca test
cpegeric Jan 28, 2025
6d4731f
Merge branch 'hnsw_dev' into hnsw_index
cpegeric Jan 28, 2025
5d9c883
sca test
cpegeric Jan 28, 2025
f83a202
Merge branch 'hnsw_dev' into hnsw_index
cpegeric Jan 28, 2025
0e30165
sca
cpegeric Jan 28, 2025
52483c0
Merge branch 'hnsw_dev' into hnsw_index
cpegeric Jan 28, 2025
1bdf77c
Merge branch 'main' into hnsw_index
cpegeric Jan 28, 2025
3bf59a6
Merge branch 'main' into hnsw_dev
cpegeric Jan 28, 2025
5795fff
merge fix
cpegeric Jan 30, 2025
2ecb896
Merge branch 'hnsw_dev' into hnsw_index
cpegeric Jan 30, 2025
9213dcb
update cross apply
cpegeric Jan 31, 2025
f151772
remove cross apply
cpegeric Jan 31, 2025
3a8a028
search test
cpegeric Jan 31, 2025
b87300a
reindex
cpegeric Jan 31, 2025
4b9f9b5
comment unused code
cpegeric Jan 31, 2025
414d892
remove hnsw_Refresh
cpegeric Feb 3, 2025
7d28078
clear the cache when create/reindex index
cpegeric Feb 3, 2025
86c0dca
cleanup cache when sigterm
cpegeric Feb 3, 2025
978b5f8
only support bigint pk and float32 vector
cpegeric Feb 3, 2025
429f619
drop column bug fix
cpegeric Feb 3, 2025
16c0d38
bug fix empty records
cpegeric Feb 3, 2025
842a7ca
add bvt test
cpegeric Feb 5, 2025
a492aab
concurrent test
cpegeric Feb 6, 2025
e4b48c7
add lock
cpegeric Feb 6, 2025
b2a77cb
use Lock instead of RLock(). usearch is not thread safe
cpegeric Feb 6, 2025
5bc4725
update
cpegeric Feb 6, 2025
7413245
update usearch 2.17.0
cpegeric Feb 6, 2025
4e3cf7e
remove old usearch
cpegeric Feb 6, 2025
8cf23e1
use mutex to replace rwmutex
cpegeric Feb 6, 2025
1e360d5
failed test cases
cpegeric Feb 6, 2025
c20472c
more tests
cpegeric Feb 6, 2025
a29ffc7
cmake
cpegeric Feb 6, 2025
0612530
add tests
cpegeric Feb 6, 2025
1a5d1e9
bug fix usearch
cpegeric Feb 7, 2025
084a8c2
rwmutex
cpegeric Feb 7, 2025
5dbba94
note for usearch repo
cpegeric Feb 7, 2025
7141b42
usearch fix
cpegeric Feb 7, 2025
f2b9a66
remove message
cpegeric Feb 7, 2025
d23973b
add mutex
cpegeric Feb 7, 2025
b62bfaf
update
cpegeric Feb 7, 2025
ae1de7e
update
cpegeric Feb 7, 2025
499be52
condition
cpegeric Feb 7, 2025
3137258
fix usearch concurrency
cpegeric Feb 10, 2025
45e42d5
cleanup
cpegeric Feb 10, 2025
0a251f1
cleanup
cpegeric Feb 10, 2025
5d5a770
cleanup
cpegeric Feb 10, 2025
017b138
build with multi-threads
cpegeric Feb 10, 2025
3089eae
update
cpegeric Feb 10, 2025
818dba8
use atomic counter instead of calling usearch.Len()
cpegeric Feb 10, 2025
52494a6
multithread build
cpegeric Feb 10, 2025
6a02ce5
update
cpegeric Feb 10, 2025
8c28f48
merge fix
cpegeric Feb 10, 2025
9c1bba0
cleanup
cpegeric Feb 10, 2025
03fe7ae
free memory after save file
cpegeric Feb 10, 2025
beeed48
err_chan don't wait
cpegeric Feb 10, 2025
117deea
copy slice to chan
cpegeric Feb 10, 2025
be00651
bug fix incr counter
cpegeric Feb 10, 2025
a569276
save file out of mutex
cpegeric Feb 11, 2025
5f9ad30
more tests
cpegeric Feb 11, 2025
f673d23
remove Stderr
cpegeric Feb 11, 2025
fe7a050
cleanup
cpegeric Feb 11, 2025
75ec3dc
update
cpegeric Feb 11, 2025
a9f4d57
remove comment
cpegeric Feb 11, 2025
8744499
add tests
cpegeric Feb 12, 2025
8497675
add variables
cpegeric Feb 12, 2025
d2aade5
update tests
cpegeric Feb 12, 2025
74e36e3
bug fix variables
cpegeric Feb 12, 2025
0b8f16b
merge fix
cpegeric Feb 12, 2025
0e6e753
update tests
cpegeric Feb 12, 2025
d5fc4f3
uintptr
cpegeric Feb 13, 2025
a76ab31
change id to string
cpegeric Feb 13, 2025
c349668
id as string
cpegeric Feb 13, 2025
beae333
index_id as string
cpegeric Feb 13, 2025
ccbe8f4
hnsw_threads_build default to 1
cpegeric Feb 13, 2025
2d0aaff
reduce cache ttl to 5 min
cpegeric Feb 13, 2025
2e6fdf4
fix sca
cpegeric Feb 13, 2025
d801a4c
fix parallel id
cpegeric Feb 13, 2025
91c6322
fix parallel id
cpegeric Feb 13, 2025
1356611
update recall test
cpegeric Feb 14, 2025
324a6ab
reduce test time
cpegeric Feb 14, 2025
d54082c
Merge branch 'main' into hnsw_index
cpegeric Feb 14, 2025
bb7f251
reduce dim
cpegeric Feb 14, 2025
e6c0b4c
update test
cpegeric Feb 14, 2025
55200f8
Merge branch 'main' into hnsw_dev
cpegeric Feb 17, 2025
6cd1162
fix build threads
cpegeric Feb 17, 2025
ec44079
remove comments
cpegeric Feb 17, 2025
b4ca360
remove unused code
cpegeric Feb 17, 2025
2e47417
merge fix
cpegeric Feb 20, 2025
0d099bb
install cmake
cpegeric Feb 24, 2025
40845ef
set LD_LIBRARY_PATH
cpegeric Feb 24, 2025
4ccba12
debug
cpegeric Feb 24, 2025
f4eeefb
ldconfig
cpegeric Feb 24, 2025
ef2f691
update
cpegeric Feb 24, 2025
43d41aa
update
cpegeric Feb 24, 2025
94b2848
update
cpegeric Feb 24, 2025
c5ae3b8
update
cpegeric Feb 24, 2025
8a17adf
install omp
cpegeric Feb 24, 2025
a1df2b6
missing library
cpegeric Feb 24, 2025
f62222c
update
cpegeric Feb 24, 2025
3f8e47b
update
cpegeric Feb 24, 2025
0e55204
update
cpegeric Feb 24, 2025
21ae617
update
cpegeric Feb 24, 2025
a46d982
update
cpegeric Feb 24, 2025
7220245
Merge branch 'main' into hnsw_index
mergify[bot] Feb 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
revert DEBUG_OPT
cpegeric committed Jan 14, 2025
commit fe3c33ef4a9d6cbe528cd182887f979a8763f0cb
5 changes: 1 addition & 4 deletions Makefile
Original file line number Diff line number Diff line change
@@ -107,15 +107,12 @@ pb: vendor-build generate-pb fmt

THIRDPARTIES_INSTALL_DIR=$(ROOT_DIR)/thirdparties/install
RACE_OPT :=
DEBUG_OPT :=
CGO_DEBUG_OPT :=
CGO_OPTS :=CGO_CFLAGS="-I$(THIRDPARTIES_INSTALL_DIR)/include"
GOLDFLAGS=-ldflags="-extldflags '-L$(THIRDPARTIES_INSTALL_DIR)/lib -Wl,-rpath,$(THIRDPARTIES_INSTALL_DIR)/lib' -X '$(GO_MODULE)/pkg/version.GoVersion=$(GO_VERSION)' -X '$(GO_MODULE)/pkg/version.BranchName=$(BRANCH_NAME)' -X '$(GO_MODULE)/pkg/version.CommitID=$(LAST_COMMIT_ID)' -X '$(GO_MODULE)/pkg/version.BuildTime=$(BUILD_TIME)' -X '$(GO_MODULE)/pkg/version.Version=$(MO_VERSION)'"
TAGS :=

ifeq ($(DEBUG_OPT),)
DEBUG_OPT :=
endif

.PHONY: cgo
cgo:
@(cd cgo; ${MAKE} ${CGO_DEBUG_OPT})