micro_ros_agent fails with eprosima::fastcdr::exception::NotEnoughMemoryException

[16stelter]

Hey all,

I've flashed the SD card of my Raspberry Pi 3B with the beta4 LeoOS and then installed the core2_firmware_ros2 as described. However, when trying to start the micro_ros_agent, it crashes with the following error message:

pi@leo:~$ ros2 run micro_ros_agent micro_ros_agent serial -D /dev/serial0 -b 460800 -v6
[1737130239.887182] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1737130239.891870] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 6
[1737130240.045756] info     | Root.cpp           | create_client            | create                 | client_key: 0x299CE561, session_id: 0x81
[1737130240.045950] info     | SessionManager.hpp | establish_session        | session established    | client_key: 0x299CE561, address: 0
[1737130240.046418] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x299CE561, len: 19, data: 
0000: 81 00 00 00 04 01 0B 00 00 00 58 52 43 45 01 00 01 0F 00
[1737130240.048828] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x299CE561, len: 40, data: 
0000: 81 80 00 00 01 07 20 00 00 0A 00 01 01 03 00 00 11 00 00 00 00 01 00 00 09 00 00 00 66 69 72 6D
0020: 77 61 72 65 00 61 FF 00
[1737130240.049351] warning  | FastDDSMiddleware.cpp | create_participant_by_bin | Overriding Micro XRCE-DDS Client DOMAIN_ID | domain_id: 0
[1737130240.211146] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x299CE561, len: 12, data: 
0000: 81 00 00 00 03 01 04 00 00 02 FF FE
[1737130240.211939] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x299CE561, len: 16, data: 
0000: 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00
[1737130240.353466] info     | ProxyClient.cpp    | create_participant       | participant created    | client_key: 0x299CE561, participant_id: 0x000(1)
[1737130240.354996] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x299CE561, len: 14, data: 
0000: 81 80 00 00 05 01 06 00 00 0A 00 01 00 00
[1737130240.355404] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x299CE561, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 01 00 00 00 80
[1737130240.415519] info     | Root.cpp           | delete_client            | delete                 | client_key: 0x299CE561
[1737130240.415970] info     | SessionManager.hpp | destroy_session          | session closed         | client_key: 0x299CE561, address: 0
[1737130240.423683] info     | Root.cpp           | create_client            | create                 | client_key: 0x355C609E, session_id: 0x81
[1737130240.423821] info     | SessionManager.hpp | establish_session        | session established    | client_key: 0x355C609E, address: 0
[1737130240.424070] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x355C609E, len: 19, data: 
0000: 81 00 00 00 04 01 0B 00 00 00 58 52 43 45 01 00 01 0F 00
[1737130240.426000] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x355C609E, len: 40, data: 
0000: 81 80 00 00 01 07 20 00 00 0A 00 01 01 03 00 00 11 00 00 00 00 01 00 00 09 00 00 00 66 69 72 6D
0020: 77 61 72 65 00 61 FF 00
[1737130240.426276] warning  | FastDDSMiddleware.cpp | create_participant_by_bin | Overriding Micro XRCE-DDS Client DOMAIN_ID | domain_id: 0
[1737130240.589444] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x355C609E, len: 12, data: 
0000: 81 00 00 00 03 01 04 00 00 02 FF FE
[1737130240.594435] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x355C609E, len: 16, data: 
0000: 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00
[1737130242.266356] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x355C609E, len: 16, data: 
0000: 80 00 00 00 02 01 08 00 00 0A FF FD 02 00 00 00
terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
  what():  Not enough memory in the buffer stream
[ros2run]: Aborted

The memory usage looks fine to me, with barely 50% used. I also want to note that during flashing of the firmware, I get another error, which does not seem to cause any obvious issues, though:

Write 703 chunks at address 0x8000000...
[====================] 100.0% address:0x802be00
Writing finished!
Read 703 chunks at address 0x8000000...
[====================] 100.0% address:0x802be00
Reading finished!
Verification FAILED: Data length does not match: 179940 bytes vs 179956 bytes.
Traceback (most recent call last):
  File "/opt/ros/humble/lib/leo_fw/flash", line 29, in <module>
    leo_fw.flash_firmware(
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/leo_fw/flash.py", line 182, in flash_firmware
    write_binary(firmware_path, reset_high=True)
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/leo_fw/flash.py", line 53, in write_binary
    subprocess.check_call(f"{base_cmd} -wv {binary_path}", shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'stm32loader -c rpi -f F4 -R -wv core2_firmware.bin' returned non-zero exit status 1.
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
    with os.fdopen(os.open(pr_filename,
FileNotFoundError: [Errno 2] No such file or directory: '/var/crash/_opt_ros_humble_lib_leo_fw_flash.1000.crash'

Original exception was:
Traceback (most recent call last):
  File "/opt/ros/humble/lib/leo_fw/flash", line 29, in <module>
    leo_fw.flash_firmware(
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/leo_fw/flash.py", line 182, in flash_firmware
    write_binary(firmware_path, reset_high=True)
  File "/opt/ros/humble/local/lib/python3.10/dist-packages/leo_fw/flash.py", line 53, in write_binary
    subprocess.check_call(f"{base_cmd} -wv {binary_path}", shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'stm32loader -c rpi -f F4 -R -wv core2_firmware.bin' returned non-zero exit status 1.
[ros2run]: Process exited with failure 1

Thanks for the help!

[16stelter]

After investigating this thoroughly, I can give some more details on what is happening. From what I understand, the OS already starts some sort of node that allows communication with the firmware so the command I first executed doesn't actually seem necessary. However, what is problematic is that if the firmware node crashes, for example when a ros2 jazzy device tries to communicate with the humble LeoOS, it never restarts, even when power cycling. If I then try to manually start a micro ros agent again, I get the same error message as described above. Nothing short of completely flashing the OS and the Core2 firmware again seems to recover from this.

For the second error message I mentioned, this only occurs when flashing from the LeoOS full image, on the lite image it works fine.

I've also found that using CycloneDDS makes it significantly harder to accidentally crash the firmware node, so I'll just be using that for now.

[bjsowa]

The micro ros agent is already started in a systemd user service. You can use these aliases to access the status and logs of these services.

What version of the firmware are you flashing?

[16stelter]

Heya, thanks for the aliases, that's really useful. Yea, I've figured out the part with the micro ros agent already, guess I misunderstood the readme, thanks!

For the firmware issue, I've tested it with the core2_firmware_ros2 version 1.3.0 and 1.2.0, both of them fail with the same error message. However, it only fails when LeoOS-ros2-beta4-2024-11-19-full is installed on the pi,
LeoOS-ros2-beta4-2024-11-19-lite works correctly. I have not tested the older versions.