[Feat] Add Support for Index merge in CAGRA
#618
Conversation
| auto merged_index = | ||
| cagra::build(handle, params, raft::make_const_mdspan(device_updated_dataset_view)); | ||
|
|
||
| if (static_cast<std::size_t>(stride) == dim) { |
There was a problem hiding this comment.
Hi @cjnolet @achirkin , I know these codes are odd, but without them, datasets will be changed after calling cagra::detail::search_main_core, which will cause the test failure. I do not know how the dataset format, matrix ownership, cagra::search interact behind it. Could you have comments here? Many thanks!
There was a problem hiding this comment.
To me, it looks like no one owns the host_updated_dataset or device_updated_dataset beyond the scope of this function, so the data gets destroyed unless the owning update_dataset is called under the if branch here.
Hence, I think, you should call update_dataset unconditionally here.
There was a problem hiding this comment.
To me, it looks like no one owns the
host_updated_datasetordevice_updated_datasetbeyond the scope of this function, so the data gets destroyed unless the owningupdate_datasetis called under theifbranch here. Hence, I think, you should callupdate_datasetunconditionally here.
Thank you, very helpful!
rhdong
added
feature request
| } | ||
|
|
||
| // Allocate the new dataset on device | ||
| auto device_updated_dataset = |
There was a problem hiding this comment.
I found a great API that can be used to fit the situation that device memory is not enough cuvs::neighbors::nn_descent::has_enough_device_memory. I will make it in the next commit
| * | ||
| * @return A new CAGRA index containing the merged indices, graph, and dataset. | ||
| */ | ||
| auto merge(raft::resources const& res, |
There was a problem hiding this comment.
Hey @chatman, I'm working on cagra::merge. Could you review the API design when you have a moment? Any suggestions would be greatly appreciated. Thanks!
No description provided.