Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS SQS Connection Errors #50

Open
nhairs opened this issue Feb 1, 2025 · 0 comments
Open

AWS SQS Connection Errors #50

nhairs opened this issue Feb 1, 2025 · 0 comments
Labels
bug Something isn't working open for work Issue can be picked up by anyone

Comments

@nhairs
Copy link
Owner

nhairs commented Feb 1, 2025

Need to make these less noisy if they are easily recovered

parsedmarcd-1  | [13:19:14][ERROR]: Uncaught exception
parsedmarcd-1  | Traceback (most recent call last):
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
parsedmarcd-1  |     conn = connection.create_connection(
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/util/connection.py", line 72, in create_connection
parsedmarcd-1  |     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/usr/local/lib/python3.12/socket.py", line 976, in getaddrinfo
parsedmarcd-1  |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  | socket.gaierror: [Errno -3] Try again
parsedmarcd-1  | 
parsedmarcd-1  | During handling of the above exception, another exception occurred:
parsedmarcd-1  | 
parsedmarcd-1  | Traceback (most recent call last):
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/httpsession.py", line 464, in send
parsedmarcd-1  |     urllib_response = conn.urlopen(
parsedmarcd-1  |                       ^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 802, in urlopen
parsedmarcd-1  |     retries = retries.increment(
parsedmarcd-1  |               ^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/util/retry.py", line 527, in increment
parsedmarcd-1  |     raise six.reraise(type(error), error, _stacktrace)
parsedmarcd-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/packages/six.py", line 770, in reraise
parsedmarcd-1  |     raise value
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 716, in urlopen
parsedmarcd-1  |     httplib_response = self._make_request(
parsedmarcd-1  |                        ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 404, in _make_request
parsedmarcd-1  |     self._validate_conn(conn)
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
parsedmarcd-1  |     conn.connect()
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connection.py", line 363, in connect
parsedmarcd-1  |     self.sock = conn = self._new_conn()
parsedmarcd-1  |                        ^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
parsedmarcd-1  |     raise NewConnectionError(
parsedmarcd-1  | urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f3ef593e630>: Failed to establish a new connection: [Errno -3] Try again
parsedmarcd-1  | 
parsedmarcd-1  | During handling of the above exception, another exception occurred:
parsedmarcd-1  | 
parsedmarcd-1  | Traceback (most recent call last):
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/parsedmarc/application.py", line 427, in main
parsedmarcd-1  |     job = self.source.get_job()
parsedmarcd-1  |           ^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/parsedmarc/source/aws.py", line 69, in get_job
parsedmarcd-1  |     self._sqs_queue.receive_messages(
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/boto3/resources/factory.py", line 581, in do_action
parsedmarcd-1  |     response = action(self, *args, **kwargs)
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/boto3/resources/action.py", line 88, in __call__
parsedmarcd-1  |     response = getattr(parent.meta.client, operation_name)(*args, **params)
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/client.py", line 569, in _api_call
parsedmarcd-1  |     return self._make_api_call(operation_name, kwargs)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/client.py", line 1005, in _make_api_call
parsedmarcd-1  |     http, parsed_response = self._make_request(
parsedmarcd-1  |                             ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/client.py", line 1029, in _make_request
parsedmarcd-1  |     return self._endpoint.make_request(operation_model, request_dict)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 119, in make_request
parsedmarcd-1  |     return self._send_request(request_dict, operation_model)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 200, in _send_request
parsedmarcd-1  |     while self._needs_retry(
parsedmarcd-1  |           ^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 360, in _needs_retry
parsedmarcd-1  |     responses = self._event_emitter.emit(
parsedmarcd-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/hooks.py", line 412, in emit
parsedmarcd-1  |     return self._emitter.emit(aliased_event_name, **kwargs)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/hooks.py", line 256, in emit
parsedmarcd-1  |     return self._emit(event_name, kwargs)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/hooks.py", line 239, in _emit
parsedmarcd-1  |     response = handler(**kwargs)
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 207, in __call__
parsedmarcd-1  |     if self._checker(**checker_kwargs):
parsedmarcd-1  |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 284, in __call__
parsedmarcd-1  |     should_retry = self._should_retry(
parsedmarcd-1  |                    ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 320, in _should_retry
parsedmarcd-1  |     return self._checker(attempt_number, response, caught_exception)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 363, in __call__
parsedmarcd-1  |     checker_response = checker(
parsedmarcd-1  |                        ^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 247, in __call__
parsedmarcd-1  |     return self._check_caught_exception(
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
parsedmarcd-1  |     raise caught_exception
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 279, in _do_get_response
parsedmarcd-1  |     http_response = self._send(request)
parsedmarcd-1  |                     ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 383, in _send
parsedmarcd-1  |     return self.http_session.send(request)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/httpsession.py", line 493, in send
parsedmarcd-1  |     raise EndpointConnectionError(endpoint_url=request.url, error=e)
parsedmarcd-1  | botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sqs.ap-southeast-2.amazonaws.com/"
parsedmarcd-1  | [13:19:48][ERROR]: Uncaught exception
parsedmarcd-1  | Traceback (most recent call last):
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
parsedmarcd-1  |     conn = connection.create_connection(
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/util/connection.py", line 72, in create_connection
parsedmarcd-1  |     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/usr/local/lib/python3.12/socket.py", line 976, in getaddrinfo
parsedmarcd-1  |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  | socket.gaierror: [Errno -3] Try again
parsedmarcd-1  | 
parsedmarcd-1  | During handling of the above exception, another exception occurred:
parsedmarcd-1  | 
parsedmarcd-1  | Traceback (most recent call last):
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/httpsession.py", line 464, in send
parsedmarcd-1  |     urllib_response = conn.urlopen(
parsedmarcd-1  |                       ^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 802, in urlopen
parsedmarcd-1  |     retries = retries.increment(
parsedmarcd-1  |               ^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/util/retry.py", line 527, in increment
parsedmarcd-1  |     raise six.reraise(type(error), error, _stacktrace)
parsedmarcd-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/packages/six.py", line 770, in reraise
parsedmarcd-1  |     raise value
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 716, in urlopen
parsedmarcd-1  |     httplib_response = self._make_request(
parsedmarcd-1  |                        ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 404, in _make_request
parsedmarcd-1  |     self._validate_conn(conn)
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
parsedmarcd-1  |     conn.connect()
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connection.py", line 363, in connect
parsedmarcd-1  |     self.sock = conn = self._new_conn()
parsedmarcd-1  |                        ^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
parsedmarcd-1  |     raise NewConnectionError(
parsedmarcd-1  | urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f3ef50f56a0>: Failed to establish a new connection: [Errno -3] Try again
parsedmarcd-1  | 
parsedmarcd-1  | During handling of the above exception, another exception occurred:
parsedmarcd-1  | 
parsedmarcd-1  | Traceback (most recent call last):
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/parsedmarc/application.py", line 427, in main
parsedmarcd-1  |     job = self.source.get_job()
parsedmarcd-1  |           ^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/parsedmarc/source/aws.py", line 69, in get_job
parsedmarcd-1  |     self._sqs_queue.receive_messages(
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/boto3/resources/factory.py", line 581, in do_action
parsedmarcd-1  |     response = action(self, *args, **kwargs)
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/boto3/resources/action.py", line 88, in __call__
parsedmarcd-1  |     response = getattr(parent.meta.client, operation_name)(*args, **params)
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/client.py", line 569, in _api_call
parsedmarcd-1  |     return self._make_api_call(operation_name, kwargs)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/client.py", line 1005, in _make_api_call
parsedmarcd-1  |     http, parsed_response = self._make_request(
parsedmarcd-1  |                             ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/client.py", line 1029, in _make_request
parsedmarcd-1  |     return self._endpoint.make_request(operation_model, request_dict)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 119, in make_request
parsedmarcd-1  |     return self._send_request(request_dict, operation_model)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 200, in _send_request
parsedmarcd-1  |     while self._needs_retry(
parsedmarcd-1  |           ^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 360, in _needs_retry
parsedmarcd-1  |     responses = self._event_emitter.emit(
parsedmarcd-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/hooks.py", line 412, in emit
parsedmarcd-1  |     return self._emitter.emit(aliased_event_name, **kwargs)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/hooks.py", line 256, in emit
parsedmarcd-1  |     return self._emit(event_name, kwargs)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/hooks.py", line 239, in _emit
parsedmarcd-1  |     response = handler(**kwargs)
parsedmarcd-1  |                ^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 207, in __call__
parsedmarcd-1  |     if self._checker(**checker_kwargs):
parsedmarcd-1  |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 284, in __call__
parsedmarcd-1  |     should_retry = self._should_retry(
parsedmarcd-1  |                    ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 320, in _should_retry
parsedmarcd-1  |     return self._checker(attempt_number, response, caught_exception)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 363, in __call__
parsedmarcd-1  |     checker_response = checker(
parsedmarcd-1  |                        ^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 247, in __call__
parsedmarcd-1  |     return self._check_caught_exception(
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
parsedmarcd-1  |     raise caught_exception
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 279, in _do_get_response
parsedmarcd-1  |     http_response = self._send(request)
parsedmarcd-1  |                     ^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/endpoint.py", line 383, in _send
parsedmarcd-1  |     return self.http_session.send(request)
parsedmarcd-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarcd-1  |   File "/home/appuser/.local/lib/python3.12/site-packages/botocore/httpsession.py", line 493, in send
parsedmarcd-1  |     raise EndpointConnectionError(endpoint_url=request.url, error=e)
parsedmarcd-1  | botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sqs.ap-southeast-2.amazonaws.com/"
@nhairs nhairs added bug Something isn't working open for work Issue can be picked up by anyone labels Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working open for work Issue can be picked up by anyone
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nhairs