Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Array::shrink_to_fit(&mut self) to 53.4.0 (#6790) (#6817) #6962

Open
wants to merge 2 commits into
base: 53.0.0_maintenance
Choose a base branch
from

Conversation

emilk
Copy link
Contributor

@emilk emilk commented Jan 10, 2025

This cherry-picks the following PRs into the 53.4.0 minor release:

This adds a feature that is very important at us at Rerun in order for us to migrate away from arrow2 to arrow-rs without regressing significantly (2x) on memory overhead.

This is pure addition, with no change of existing code, so it is perfectly safe to add to a minor release.

Related

emilk and others added 2 commits January 10, 2025 18:08
* Add `Array::shrink_to_fit`

* Test that shrink_to_fit actually frees memory

* Make sure the buffer isn't shared in the test of shrink_to_fit

* Remove `#[inline]`

* Use `realloc` to reallocate the bytes

* Clean up test

* Improve docstring for `Array::shrink_to_fit`

Co-authored-by: Raphael Taylor-Davies <[email protected]>

* `Buffer::shrink_to_fit`: ignore shared buffers

* Improve comment in `ArrayRef::shrink_to_fit`

* Document why `try_realloc` is safe, and actually make it safe :)

* Improve testing of shrink_to_fit

* Fix a few corner cases, and improve test

* Add license header to new test file

---------

Co-authored-by: Raphael Taylor-Davies <[email protected]>
* Support shrink to empty

* Docs

* Format
@github-actions github-actions bot added the arrow Changes to the arrow crate label Jan 10, 2025
@emilk emilk changed the title Add Array::shrink_to_fit(&mut self) (#6790) (#6817) Add Array::shrink_to_fit(&mut self) to 53.4.0 (#6790) (#6817) Jan 10, 2025
@emilk
Copy link
Contributor Author

emilk commented Jan 10, 2025

CI failures are unrelated to the PR - they seem to be because of the recent release of Rust 1.84.0.

It is likely best fixed by adding a rust-toolchain file to the repository to pin the Rust version to the MSRV for arrow.

@alamb
Copy link
Contributor

alamb commented Jan 10, 2025

We can get this in no problem. I'll try and look at the CI failure later today

@alamb
Copy link
Contributor

alamb commented Jan 10, 2025

This PR should fix the CI on the 53.0.0_maintenance branch:

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @emilk -- this looks good to me.

Once we get #6964 in the CI should be clean and we can re-trigger CI for this PR to get a clean run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants