Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SimpleOffsetPaginator fix and Json Selector #59

Merged
merged 1 commit into from
Sep 30, 2024

Conversation

BubbaTLC
Copy link
Contributor

Hello,

This is my first contribution so please give me feedback.

I was attempting to use the new SimpleOffsetPaginator implemented here #48. I encountered a bug in streams.py as the pagination_page_size was not being properly set, so I was unable to specify a different value other than 25. I believe this issue was introduced when trying to fix a different issue in #52

I have also added the ability to select the records in the request, to enable counting of nested records.

My API was returning records like the example I gave in the readme which gave me a count of 2 no mater how many records are in the contacts list.

  {
    "contacts": [
      {
        "id": 52,
        "emailBlacklisted": false,
        "smsBlacklisted": false,
        "createdAt": "2024-09-24T01:00:00.000-00:00",
        "modifiedAt": "2024-09-25T01:00:00.000-00:00",
      }
    ],
    "count": 256
  }

No I can specify pagination_page_size = 1 and offset_records_jsonpath = "$.contacts" to compare the correct number of records.

I added the optional argument offset_records_jsonpath to both top-level and stream level options.

@BubbaTLC BubbaTLC changed the title feat: added config to select records with jsonpath with simple_offset SimpleOffsetPaginator fix and Json Selector Sep 27, 2024
@jlloyd-widen jlloyd-widen merged commit 10ddd9c into Widen:main Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants