Pagination does not iterate through all items available.

[karasjoh000]

Describe the bug
Pagination does not iterate through all items available. It only prints the first iteratation of items that is of size provided by the MaxItems parameter.

Steps to reproduce

def get_folders_and_objects(bucket, prefix): 
    # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html#paginators
    paginator = client.get_paginator('list_objects')
    list_params = {'Bucket': bucket, 'Delimiter': '/'}
    if prefix is not None: 
        list_params['Prefix'] = prefix
    page_iterator = paginator.paginate(PaginationConfig={'MaxItems': 5}, **list_params)

    contents = [] 
    for page in page_iterator:
        if 'Contents' in page.keys():
            for object in page['Contents']: 
                contents.append({'name': object['Key'].split('/')[-1], 'date': object['LastModified'].strftime("%d-%b-%Y %H:%M"), 'size': object['Size'], 'dir': False})
        if 'CommonPrefixes' in page.keys():
            for folder in page['CommonPrefixes']: 
                contents.append({'name': folder['Prefix'].split("/")[-2], 'date': "", 'size': None, 'dir': True })
    return contents

Expected behavior
e.g. If there are 10 objects, the iterator should iterate 2 times for {'MaxItems': 5}

[kdaily]

Hi @karasjoh000,

You would need to use the PageSize parameter, not MaxItems. However, I see you're using Delimiter as a parameter, and there's another open issue around the behavior ofMaxItems that you should note:

#2376

Also, are you specifically using list_object? There's an updated API for this available at list_objects_v2, here's the definition for the paginator.

[karasjoh000]

got it to work with PageSize parameter.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.