Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mavlink: Trigger reconnection from receiver_loop if any recv error happens #306

Merged
merged 1 commit into from
Nov 29, 2023
Merged

mavlink: Trigger reconnection from receiver_loop if any recv error happens #306

merged 1 commit into from
Nov 29, 2023

Conversation

joaoantoniocardoso
Copy link
Collaborator

Fixes #305

@joaoantoniocardoso
Copy link
Collaborator Author

Tested. I killed the ardupilot manager on BlueOS, and we can see here that the error occurred and the task that tried the reconnection was the receiver:

2023-11-28T23:59:24.738986Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:25.740069Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:26.740951Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:27.741779Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:28.743703Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:29.744966Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:30.568338Z ERROR                                         MavReceiver ThreadId(03) receiver_loop: src/mavlink/manager.rs:94: Failed receiving message from Mavlink Connection: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
2023-11-28T23:59:30.746592Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:31.568675Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:31.572556Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:31.747955Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:32.572827Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:32.576659Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:32.749223Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:33.576867Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:33.580625Z  INFO                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:212: Successfully connected
2023-11-28T23:59:33.750502Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:34.750962Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:35.752510Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:36.081611Z ERROR                                         MavReceiver ThreadId(03) receiver_loop: src/mavlink/manager.rs:94: Failed receiving message from Mavlink Connection: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
2023-11-28T23:59:36.753488Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:37.081911Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:37.085629Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:37.753809Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:38.085818Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:38.089590Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:38.755189Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:39.089767Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:39.093509Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:39.756642Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:40.093702Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:40.097620Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:40.758044Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:41.097834Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:41.101626Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:41.759417Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:42.101815Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:42.105617Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:42.760833Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:43.105822Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:43.109576Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:43.762169Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:44.109769Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:44.113592Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:44.763659Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:45.113772Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:45.117495Z ERROR                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:216: Failed to connect, trying again in one second. Reason: Os {
    code: 111,
    kind: ConnectionRefused,
    message: "Connection refused",
}.
2023-11-28T23:59:45.765052Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:46.117649Z DEBUG                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:208: Connecting...
2023-11-28T23:59:46.123146Z  INFO                                         MavReceiver ThreadId(03) receiver_loop:connect{address="tcpout:192.168.2.2:5777"}: src/mavlink/manager.rs:212: Successfully connected
2023-11-28T23:59:46.766141Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:47.767342Z DEBUG                                     MavlinkCamera-1 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-28T23:59:48.768509Z DEBUG                                     MavlinkCamera-0 ThreadId(25) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent

@joaoantoniocardoso joaoantoniocardoso marked this pull request as ready for review November 29, 2023 00:02
@joaoantoniocardoso joaoantoniocardoso merged commit 14b88f8 into mavlink:master Nov 29, 2023
3 checks passed
@joaoantoniocardoso joaoantoniocardoso deleted the fix_mavlink_receiver_loop branch November 29, 2023 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrickelectric patrickelectric patrickelectric approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

mavlink: Failed receiving message from Mavlink Connection - "failed to fill whole buffer"
2 participants
@joaoantoniocardoso @patrickelectric