Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump amazon-kinesis-client to 2.2.10 #151

Open
benjben opened this issue Apr 30, 2020 · 4 comments
Open

Bump amazon-kinesis-client to 2.2.10 #151

benjben opened this issue Apr 30, 2020 · 4 comments
Assignees

Comments

@benjben
Copy link
Contributor

benjben commented Apr 30, 2020

We started to see weird errors in ES loader logs:

[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor - Shutting down record processor with shardId: shardId-000000000021 with reason TERMINATE
[RecordProcessor-0000] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask - Application exception. 
[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor - Shutting down record processor with shardId: shardId-000000000021 with reason TERMINATE
[RecordProcessor-0000] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask - Application exception. 

And we can't find a sensible explanation. Version 1.9.1 that we're currently using is 2 years old today. Time to bump!

@benjben benjben self-assigned this Apr 30, 2020
@chuwy
Copy link
Contributor

chuwy commented Apr 30, 2020

I think it's a much bigger workflow. In order to update the KCL lib we'd have to get rid of https://github.com/awslabs/amazon-kinesis-connectors, which is core of the ES (and S3) Loader. I'm considering to switch to something like https://github.com/laserdisc-io/fs2-aws which could make all streaming loaders very source (PubSub/Kinesis/NSQ) and sink (ES, S3) agnostic.

@benjben
Copy link
Contributor Author

benjben commented May 1, 2020

Indeed seems like removing amazon-kinesis-connectors would be a loss of time as we are already have the intention to merge the loaders and with a better technology. Using fs2 would be great !

@jbeemster
Copy link
Member

Getting this back online involved manually editing the KCL table to ignore the failed shards and moving it to the new shards... very painful recovery.

@szareiangm
Copy link
Contributor

Hey guys, sorry for being late. I think something similar to it happened for us, too. I think we manually overriden the amazon-kinesis-client version included in amazon-kinesis-connectors and it worked. Looks like, something in Kinesis backend has changed. I didn't contributed it back because it was too much mysterious and I didn't have proof to show one-to-one relation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants