Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paginate profile feed #750

Merged
merged 38 commits into from
Dec 22, 2023
Merged

Paginate profile feed #750

merged 38 commits into from
Dec 22, 2023

Conversation

mplorentz
Copy link
Member

@mplorentz mplorentz commented Dec 19, 2023

This is the first part of #223. It adds pagination to our relay communication on the profile feed. Here's are the functional differences for the user:

  • Before this the profile feed would fetch the most recent 50 events for the user from relays. Sometimes we would show more than 50 events if we had already downloaded older ones (and for the logged in user we keep all their events), but we would only download the last 50 so you couldn't scroll back very far. Now you can scroll back to the user's first note.
  • Sometimes when a reposted loads in while you are looking at it the cell plays a "slide in from top" animation instead of expanding in place. I spent a while trying to figure it out but I want to wait to solve this issue for a different PR. It seems minor and I don't think it should block all the other benefits we get from pagination.

I ended up using a UICollectionView to do the pagination because it solves the problem of determining when a cell scrolls into view (which is hacky/inefficient in SwiftUI) and it supports the UICollectionViewDataSourcePrefetching protocol which is great for loading cells before they scroll into view. LazyVStack is supposed to load cells before they scroll into view but it seems to do it just milliseconds before which isn't enough time to load expensive network resources. You will find the collection view code in PagedNoteListView which relies on PagedNoteDataSource.

To do the pagination I had to do some refactoring to the RelayService. Instead of mapping one RelaySubscription to a bunch of different Nostr subscriptions on different websockets I refactored so that one RelaySubscription represents one Nostr subscription for one websocket/relay. I also added a SubscriptionCancellable that bundles these subscriptions into a single object that auto-unsubscribes from them when it is deallocated. This inflated the size of the diff as every place we open a subscription changed slightly. This refactor allows me to keep track of the oldest event received for a given subscription per-relay, which means when it comes time to load the next page of events we actually make a slightly different request to each relay to load events older than the oldest one we have received. As far as I can tell this is the only reasonable way to page in Nostr, and this is how other apps are doing it, but it can sometimes result in fetching more events than we need. The PagedRelaySubscription contains most of this logic.

I also started logging the length of the RelayService.parseQueue (filter logs for "parse queue") which was really eye opening as when the app first opens we add tens of thousands of events to the parse queue and it can take ~30 seconds to clear them all out. Switching tabs can cause tens of thousands more to be added to the queue and extend the time further. This causes things like reposts, mentions, and author names to load slowly. I knew it was bad but not this bad. Pagination will reduce the number of events in the queue by at least a factor of 10.

}
Button(Localized.reportNote.string, role: .destructive) {
showingReportMenu = true
if !note.isStub {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little feature that allows us to copy the ID of notes even if they are only stubs, which is helpful for debugging.

@mplorentz mplorentz marked this pull request as ready for review December 20, 2023 23:25
@rabble
Copy link
Contributor

rabble commented Dec 21, 2023

Just testing it out with xcode on my phone... it crashed once with nothing interesting in the logs and then when i went from notifications to view a profile i got this:

1e7764cb418a8bc3da01f3d124d2bbf6c13ae9252b01e6359b639af176d40ddf loading view data
cache hit: outOfNetwork
cache hit: outOfNetwork
cache hit: inNetwork
*** Assertion failure in -[UICollectionView Bug_Detected_In_Client_Of_UICollectionView_Invalid_Number_Of_Items_In_Section:], UICollectionView.m:11000
fault: Serious application error. An exception was caught from the delegate of NSFetchedResultsController during a call to -controllerDidChangeContent:. Invalid update: invalid number of items in section 0. The number of items contained in an existing section after the update (13) must be equal to the number of items contained in that section before the update (13), plus or minus the number of items inserted or deleted from that section (0 inserted, 1 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out). Collection view: <UICollectionView: 0x1468aae00; frame = (0 0; 390 670); clipsToBounds = YES; autoresize = W+H; gestureRecognizers = <NSArray: 0x2826af000>; backgroundColor = UIExtendedGrayColorSpace 0 0; layer = <CALayer: 0x28777d9a0>; contentOffset: {0, 0}; contentSize: {390, 2556}; adjustedContentInset: {0, 0, 0, 0}; layout: <UICollectionViewCompositionalLayout: 0x129838a30>; dataSource: <TtGC3Nos19PagedNoteDataSourceGV7SwiftUI6IDViewGVS1_15ModifiedContentGS3_VS_13ProfileHeaderVS1_23_CompositingGroupEffect_VS1_13_ShadowEffect_Vs16ObjectIdentifier_GS3_GVS1_6VStackGS3_GS3_VS1_4TextVS1_14_PaddingLayout_VS_18ReadabilityPadding__VS1_16_FlexFrameLayout
: 0x113ef8e10>> with userInfo {
NSAssertFile = "UICollectionView.m";
NSAssertLine = 11000;
}
CoreData: fault: Serious application error. An exception was caught from the delegate of NSFetchedResultsController during a call to -controllerDidChangeContent:. Invalid update: invalid number of items in section 0. The number of items contained in an existing section after the update (13) must be equal to the number of items contained in that section before the update (13), plus or minus the number of items inserted or deleted from that section (0 inserted, 1 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out). Collection view: <UICollectionView: 0x1468aae00; frame = (0 0; 390 670); clipsToBounds = YES; autoresize = W+H; gestureRecognizers = <NSArray: 0x2826af000>; backgroundColor = UIExtendedGrayColorSpace 0 0; layer = <CALayer: 0x28777d9a0>; contentOffset: {0, 0}; contentSize: {390, 2556}; adjustedContentInset: {0, 0, 0, 0}; layout: <UICollectionViewCompositionalLayout: 0x129838a30>; dataSource: <TtGC3Nos19PagedNoteDataSourceGV7SwiftUI6IDViewGVS1_15ModifiedContentGS3_VS_13ProfileHeaderVS1_23_CompositingGroupEffect_VS1_13_ShadowEffect_Vs16ObjectIdentifier_GS3_GVS1_6VStackGS3_GS3_VS1_4TextVS1_14_PaddingLayout_VS_18ReadabilityPadding__VS1_16_FlexFrameLayout_: 0x113ef8e10>> with userInfo {
NSAssertFile = "UICollectionView.m";
NSAssertLine = 11000;
}
CoreData: Serious application error. An exception was caught from the delegate of NSFetchedResultsController during a call to -controllerDidChangeContent:. <decode: bad range for [%@] got [offs:350 len:1087 within:0]> with userInfo <decode: bad range for [%@] got [offs:1437 len:71 within:0]>
error: Serious application error. Exception was caught during Core Data change processing. This is usually a bug within an observer of NSManagedObjectContextObjectsDidChangeNotification. Invalid update: invalid number of items in section 0. The number of items contained in an existing section after the update (13) must be equal to the number of items contained in that section before the update (13), plus or minus the number of items inserted or deleted from that section (0 inserted, 1 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out). Collection view: <UICollectionView: 0x1468aae00; frame = (0 0; 390 670); clipsToBounds = YES; autoresize = W+H; gestureRecognizers = <NSArray: 0x2826af000>; backgroundColor = UIExtendedGrayColorSpace 0 0; layer = <CALayer: 0x28777d9a0>; contentOffset: {0, 0}; contentSize: {390, 2556}; adjustedContentInset: {0, 0, 0, 0}; layout: <UICollectionViewCompositionalLayout: 0x129838a30>; dataSource: <TtGC3Nos19PagedNoteDataSourceGV7SwiftUI6IDViewGVS1_15ModifiedContentGS3_VS_13ProfileHeaderVS1_23_CompositingGroupEffect_VS1_13_ShadowEffect_Vs16ObjectIdentifier_GS3_GVS1_6VStackGS3_GS3_VS1_4TextVS1_14_PaddingLayout_VS_18ReadabilityPadding__VS1_16_FlexFrameLayout_: 0x113ef8e10>> with userInfo {
NSAssertFile = "UICollectionView.m";
NSAssertLine = 11000;
}
CoreData: error: Serious application error. Exception was caught during Core Data change processing. This is usually a bug within an observer of NSManagedObjectContextObjectsDidChangeNotification. Invalid update: invalid number of items in section 0. The number of items contained in an existing section after the update (13) must be equal to the number of items contained in that section before the update (13), plus or minus the number of items inserted or deleted from that section (0 inserted, 1 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out). Collection view: <UICollectionView: 0x1468aae00; frame = (0 0; 390 670); clipsToBounds = YES; autoresize = W+H; gestureRecognizers = <NSArray: 0x2826af000>; backgroundColor = UIExtendedGrayColorSpace 0 0; layer = <CALayer: 0x28777d9a0>; contentOffset: {0, 0}; contentSize: {390, 2556}; adjustedContentInset: {0, 0, 0, 0}; layout: <UICollectionViewCompositionalLayout: 0x129838a30>; dataSource: <TtGC3Nos19PagedNoteDataSourceGV7SwiftUI6IDViewGVS1_15ModifiedContentGS3_VS_13ProfileHeaderVS1_23_CompositingGroupEffect_VS1_13_ShadowEffect_Vs16ObjectIdentifier_GS3_GVS1_6VStackGS3_GS3_VS1_4TextVS1_14_PaddingLayout_VS_18ReadabilityPadding__VS1_16_FlexFrameLayout_: 0x113ef8e10>> with userInfo {
NSAssertFile = "UICollectionView.m";
NSAssertLine = 11000;
}
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid update: invalid number of items in section 0. The number of items contained in an existing section after the update (13) must be equal to the number of items contained in that section before the update (13), plus or minus the number of items inserted or deleted from that section (0 inserted, 1 deleted) and plus or minus the number of items moved into or out of that section (0 moved in, 0 moved out). Collection view: <UICollectionView: 0x1468aae00; frame = (0 0; 390 670); clipsToBounds = YES; autoresize = W+H; gestureRecognizers = <NSArray: 0x2826af000>; backgroundColor = UIExtendedGrayColorSpace 0 0; layer = <CALayer: 0x28777d9a0>; contentOffset: {0, 0}; contentSize: {390, 2556}; adjustedContentInset: {0, 0, 0, 0}; layout: <UICollectionViewCompositionalLayout: 0x129838a30>; dataSource: <TtGC3Nos19PagedNoteDataSourceGV7SwiftUI6IDViewGVS1_15ModifiedContentGS3_VS_13ProfileHeaderVS1_23_CompositingGroupEffect_VS1_13_ShadowEffect_Vs16ObjectIdentifier_GS3_GVS1_6VStackGS3_GS3_VS1_4TextVS1_14_PaddingLayout_VS_18ReadabilityPadding__VS1_16_FlexFrameLayout_: 0x113ef8e10>>'
*** First throw call stack:
(0x193b3a6a0 0x18bdebc80 0x193098dd8 0x1962df700 0x195f6020c 0x195f5dac8 0x1007b19b0 0x1007b1f48 0x19bddad5c 0x19bc84760 0x19bc84634 0x19bdd9770 0x193a7d2c8 0x193a7cc90 0x193a7cbd8 0x193a7c128 0x192a127a4 0x19bcaa2a8 0x19bca9ed4 0x19bc7b3e0 0x19bdb4aec 0x19bdb3e9c 0x19bc78f84 0x1002ce7f8 0x1002ce8d8 0x19bcca92c 0x19bd12d2c 0x19bcca838 0x19bcca770 0x19bc85c30 0x19bc84760 0x1062fab34 0x10630c258 0x1062fab34 0x10630afec 0x10630aba0 0x193a8500c 0x193a81d18 0x193a81468 0x1d79024f8 0x195eae004 0x195ead640 0x1988bf4b8 0x1988bf2fc 0x19852fe90 0x1004c12e0 0x1004c1390 0x1b70d2dcc)
libc++abi: terminating due to uncaught exception of type NSException

@mplorentz
Copy link
Member Author

Thanks for testing @rabble. I haven't been able to reproduce this crash but I noticed while trying that deleting a note on your profile doesn't remove the note right away. I'll get that fixed and continue looking for the crash.

Copy link
Member

@martindsq martindsq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work, it works nicely for me!

@@ -151,15 +151,16 @@ enum AuthorError: Error {
return fetchRequest
}

@nonobjc func allPostsRequest() -> NSFetchRequest<Event> {
@nonobjc func allPostsRequest(since: Date = .now) -> NSFetchRequest<Event> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since made my think that it will fetch all events posted after that date, but it actually does the reverse thing. Can you rename this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yes, good catch.

@@ -805,6 +806,77 @@ public class Event: NosManagedObject {
return nil
}

// MARK: - Preloading and Caching
// Probably should refactor this stuff into a view model
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should refactor this, its mixing the model and the view. Can we do it now that we are working on this? Otherwise the code will quickly grow in complexity.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to refactor this too but there are a few reasons I didn't want to do it in this PR:

  • It will make the diff huge because every view that holds a reference to a Event will need to change
  • I'm not sure how to make this work with @FetchRequest which returns NSManagedObjects and notifies us when they change. If we used a view model we would need to wrap @FetchRequest or something to instantiate view models when new objects are inserted? Same with NSFetchedResultsController. We could make a separate object that manages this cache of data but then we have to figure out cache invalidation.

I opened #764 to come back and figure this out later.


/// A handle that holds references to one or more `RelaySubscription`s and provides the ability to cancel these
/// subscriptions. Will auto-cancel them when it is deallocated. Modeled after Combine's `Cancellable`.
class SubscriptionCancellable {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move this class to its own file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mplorentz mplorentz requested a review from martindsq December 22, 2023 19:12
@mplorentz mplorentz merged commit 8903743 into main Dec 22, 2023
@mplorentz mplorentz deleted the paginate-profile-feed branch December 22, 2023 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants