chore: blobStore.setDownloadFilter() & createEntriesReadableStream()
#940
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gmaclennan
changed the title
blobStore.setDownloadFilter() & createEntriesReadableStream()
This was referenced Oct 30, 2024
EvanHahn
requested changes
Oct 30, 2024
| const coreManager = createCoreManager(...args) | ||
| const blobStore = new BlobStore({ coreManager }) | ||
| return { blobStore, coreManager } | ||
| test('blobStore.createEntriesReadStream({ live: false })', async (t) => { |
There was a problem hiding this comment.
As I said earlier, do we ever (plan to) use live: false? If not, could we remove this test?
EvanHahn
force-pushed
the
chore/entries-stream
branch
from
8287f62 to
6496672
Compare
Before this change, we turned filters into a list of path prefixes and
then checked entry paths against those prefixes. Rough pseudocode:
```javascript
function doesEntryMatchFilter({ path }, filter) {
const pathPrefixes = pathPrefixesFromFilter(filter)
return pathPrefixes.some((prefix) => path.startsWith(prefix))
}
```
For performance and simplicity, I think it's cleaner if we "look up"
entry paths in the existing filter object. Rough pseudocode:
```javascript
function doesEntryMatchFilter({ path }, filter) {
const { type, variant } = parsePath(path)
return filter[type]?.includes(variant)
}
```
I think this has two advantages:
- It's less code. We don't need to worry about de-duping paths (e.g.,
`/photos/original` versus `/photos/`). We don't need to worry about
sketchy paths (e.g., `/photos/garbage/../original`).
- It's faster (at least, in theory). Rather than having to iterate over
every path prefix, we only need to iterate over the variants of each
path. (This could be further optimized with a `Set`.)
|
Added end-to-end tests in d374b20. This will currently break if deployed. If you set yourself as a non-archive device, sync will be partially broken. I'm going to merge this into the |
EvanHahn
approved these changes
Nov 5, 2024
EvanHahn
added a commit
that referenced
this pull request
Nov 7, 2024
…)` (#940) - Removes `blobStore.download()` and the `LiveDownload` class. - Adds `blobStore.createEntriesReadableStream()` which is a stream of all entries in the blob store, filtered by blob type and variant(s). - Changes behaviour of `BlobStore` to auto-download blobs according to a `downloadFilter`. - Changes the `BlobStore` filter according to `isArchiveDevice` setting. - Changes the behaviour of `CoreManager` to not auto-download blob cores (this responsibility is now with `BlobStore`). Closes [#681]. [#681]: #681 Co-authored-by: Evan Hahn <[email protected]>
Merged
EvanHahn
added a commit
that referenced
this pull request
Nov 20, 2024
This is a squashed commit of: - #940 - #957 - #956 Co-authored-by: Gregor MacLennan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #681
blobStore.download()and theLiveDownloadclass.blobStore.createEntriesReadableStream()which is a stream of all entries in the blob store, filtered by blob type and variant(s).BlobStoreto auto-download blobs according to adownloadFilter.BlobStorefilter according toisArchiveDevicesetting.CoreManagerto not auto-download blob cores (this responsibility is now withBlobStore).TODO: