InvalidPayloadLength using Kinesis Enhanced Fan-Out consumer

[mjp9zn]

Describe the bug

InvalidPayloadLength('Payload length of 19271862 exceeded the maximum of 16777216')

Code location

We're processing a Kinesis stream using an Enhanced Fan-Out consumer, and see the above error raised by Botocore. This error is being raised because the payload sent by Kinesis is greater than a hard-coded _MAX_PAYLOAD_LENGTH at 16Mb. There is no recovery from this, as the stream will continue to attempt to send the same event that exceeds the payload max length on retry.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would expect that the _MAX_PAYLOAD_LENGTH value be configured to match the limits of Kinesis when processing as an Enhanced Fan-Out consumer. If Kinesis doesn't have an inherent limit, then this value should be easily configurable or recoverable by clients.

Current Behavior

InvalidPayloadLength('Payload length of 19271862 exceeded the maximum of 16777216')

Reproduction Steps

This isn't very useful as reproduction depends on external state in Kinesis

import boto3

client = boto3.client("kinesis", region_name="us-east-1")
subscription_params = dict(
    ConsumerARN=...,
    ShardId=...,
    StartingPosition=...,
)
subscription = client.subscribe_to_shard(**subscription_params)
event_stream = subscription["EventStream"]
for event in event_stream:
    print(event)

Possible Solution

No response

Additional Information/Context

Worked around the issue with the following snippet, which is far from ideal for production code:

import botocore.eventstream

botocore.eventstream._MAX_PAYLOAD_LENGTH = 20 * 1024**2  # 20MB

SDK version used

1.35.77

Environment details (OS name and version, etc.)

AWS ECS container running image based off python:3.12.7-slim on AWS EC2 Linux/X86_64 (al2023-ami-ecs-hvm-2023.0.20240610-kernel-6.1-x86_64)

[khushail]

Hi @mjp9zn , thanks for reporting this. The internal specification for the protocol suggests there is a hard upper limit of 16MB for payload size. I have reached out to internal team for getting more context on this limit and get their insights. Also would file a ticket with Kinesis team to have this checked.

Will post updates soon.

Thanks.

[khushail]

Ticket created with Kinesis team - P195678195

EDIT: I have closed the ticket with Kinesis team. Looks like this issue is existent with all SDKs. Team is working on a fix.

[nateprewitt]

Hi @mjp9zn, I wanted to quickly follow up on this.

There was some internal discussion and this limit has been modified for some select services. This was initially a required constraint for the protocol which has since changed. We've reconciled the issue internally and you should see this fixed across AWS SDKs in future releases.

Boto3/Botocore has addressed this today with the release of 1.36.15. You should be able to update at your earliest convenience and no longer encounter this error. Please let us know if you have any other questions.