Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding some roadmap to dedupe #1392

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

adding some roadmap to dedupe #1392

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions DEDUPE-TODO
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,22 @@
The storage subsystem usually identifies the similar buffers using
locality-sensitive hashing or other methods.

- Varying compression ratios on a single job
We could accept a list of 2d tuples in form of
[(probability,compression_ratio), ...] such that the compression ratios
are generated according to their set probability

- Rework verification with dedupe and compression.

- Reduce memory required to manage to dedupe_working_set.
Currently we require to maintain a seed (12-16 bytes) per page in
the working set. With large files we waste a lot of memory.
Either leverage disk space for that, or recalculate the seeds during
buffer generation phase

- Dedupe hot spots.
Maintain different probabilities within the dedupe_working_set such that when
generating dedupe buffers we choose the seeds non uniformly in motivation to
simulate real-world use-cases better.

- Add examples of fio jobs utilizing deduplication and/or compression.