Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.0: Separating vector store from the Index #168

Merged
merged 88 commits into from
Aug 5, 2023
Merged

1.0: Separating vector store from the Index #168

merged 88 commits into from
Aug 5, 2023

Conversation

ashvardanian
Copy link
Contributor

@ashvardanian ashvardanian commented Jul 31, 2023

This PR brings the first significant breaking change to USearch.

New Serialization Format and External Storage

From now onwards, the serialized dense index files are prepended by a .bbin matrix containing type-punned vectors.
It makes it easy to debug and check the contents of each index and potentially reconstruct it.
Moreover, this allows us to integrate the underlying index_gt in places like UStore, where the vectors would be stored externally in a separate collection.
Those collections are often stored on disk - in external memory. Random reads can be expensive, so we now allow batch-evaluated metrics to let you prefetch all the relevant vectors simultaneously, resulting in major performance improvements.

Minor Tweaks

ashvardanian and others added 30 commits July 21, 2023 15:01
@ashvardanian ashvardanian changed the title Separating vector store from the index 1.0: Separating vector store from the Index Aug 1, 2023
@ashvardanian ashvardanian mentioned this pull request Aug 1, 2023
2 tasks
@ashvardanian ashvardanian merged commit eeca16a into main Aug 5, 2023
11 checks passed
@ashvardanian
Copy link
Contributor Author

🎉 This PR is included in version 0.23.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants