Segmentation fault after connecting ROS 2 node to GNSS system

[behretv]

Hardware Information

GNSS system: ADMA-Micro
Target computer: NVIDIA Jetson Orin AGX Devkit (aarch64 architecture)

Detailed Description

The ROS node was initialized inside a Docker container and the connection to the GNSS system could be established, but immediately terminates with exit code -11.

The docker image can be built by using the following Dockerfile:

FROM ros:iron

SHELL ["/bin/bash", "-c", "-o", "pipefail"]

ENV DEBIAN_FRONTEND=noninteractive \ 
  force_color_prompt=yes \
  PIP_DISABLE_PIP_VERSION_CHECK=true \
  RMW_IMPLEMENTATION="rmw_cyclonedds_cpp" \
  ROS_PACKAGE_PATH="/opt/ros/${ROS_DISTRO}" \
  ROS_ROOT="/opt/ros/${ROS_DISTRO}"

RUN apt-get update -y && \
  apt-get upgrade -y && \
  apt-get install -y --no-install-recommends \
  python3-pip \
  vim-tiny \
  ros-"${ROS_DISTRO}"-ament-cmake-clang-format \
  ros-"${ROS_DISTRO}"-rmw-cyclonedds-cpp \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get clean

# Install ZED ROS wrapper
WORKDIR /tmp/colcon
RUN git config --global advice.detachedHead false \
  && source "${ROS_PACKAGE_PATH}/setup.bash" \
  && mkdir -p src/ \
  && git clone --depth 1 --branch ros2_v2.2.0 \
  --recursive https://github.com/GeneSysElektronik/adma_ros_driver.git \
  "src/adma_ros_driver" \
  && rosdep update \
  --include-eol-distros \
  --rosdistro="${ROS_DISTRO}" \
  && rosdep install \
  --from-paths src/ \
  --rosdistro "${ROS_DISTRO}" \
  --ignore-src -r -y  \
  && colcon build \
  --cmake-clean-first \
  --install-base "${ROS_ROOT}/install" \
  --merge-install \
  --ament-cmake-args " -Wno-dev" \
  && rm -rf \
  "build/" \
  "log/" \
  "src/" \
  "*.rosinstall" \
  /var/lib/apt/lists/* \
  && apt-get clean

RUN python3 -m pip install --no-cache-dir \
  pytest~=7.0

# Entry point source ROS
COPY scripts/entrypoint.sh /usr/local/bin/entrypoint
ENTRYPOINT ["entrypoint"]
WORKDIR /app

where the scripts/entrypoint.sh used inside the Dockerfile contains:

#!/bin/bash
source_if_exists() {
  # shellcheck disable=SC1090
  if [[ -f $1 ]]; then echo "Sourcing: $1" && source "$1"; fi
}
source_if_exists "${ROS_PACKAGE_PATH}/setup.bash"
source_if_exists "${ROS_ROOT}/install/setup.bash"
exec "$@"

the build process was executed with the following command:

docker buildx create --use --name=multiarch
docker buildx build \
      -t <image-name> \
      --provenance=false \
      --platform linux/amd64,linux/arm64/v8 \
      --file Dockerfile \
       .
docker buildx rm --force multiarch

The ROS node was launched using the following command:

local_root="$(git rev-parse --show-toplevel)"

docker run \
  -t \
  --rm \
  --workdir=/app \
  --network=host \
  -v "${local_root}/config/gensys.yml":"/opt/ros/iron/install/share/adma_ros2_driver/config/driver_config.yaml" \
  -v "${local_root}":/app \
  <image-name> \
  bash -c "ros2 launch adma_ros2_driver adma_driver.launch.py"

with the config/gensys.yml file:

/**:
  adma_ros2_driver:
    ros__parameters:
      destination_ip: '<ip-address>'
      destination_port: 11021
      use_performance_check: False
      protocol_version: 'v3.3.5'
      frame_ids:
        navsatfix: 'gnss_link'
        imu: 'imu_link'
        adma: 'adma'
        adma_status: 'adma_status'
        raw_data: 'data_raw'
        odometry_pose_id: 'adma' # header.frame_id (basicly could be a "name" of desired POI)
        odometry_twist_id: 'odometry' # odometry.child_frame_id
      odometry_yaw_offset: 0.0 # in degrees
      topic_pois: # those values can be mixed individually
        navsatfix: 1 # either 1-8 for desired POI or 0 if MRP(Measurement Reference Point) is required
        imu: 1
        velocity: 1
        odometry: 1
      mode: 0 # 0 = live (default) / 1 = rosbag replay of raw data
      time_mode: 0 # 0 = ADMA INS time / 1 = ROS system time 

[freak89dutch]

Thanks for the detailed information about the problem.
In your setup I see a "invalid" setup:
inside your Dockerfile you are cloning the ros2 branch with the v2.2.0 tag and later try to run the driver with the adma protocol version v3.3.5. This version is NOT included in the ros2 v2.2.0 (only in ROS1 v2.2.0 release). For ROS2 protocol v3.3.5 was introduced in the v2.3.0 release, in your desired ros2_v2.2.0 you can only use <= v3.3.4.
I think this could be the problem, since you are trying to setup the adma_ros2_driver with a protocol_version that is unknown and therefore the initialization process is not completing correct.

Could you try to test this suggestion by updating your branch/tag in the dockerfile and give me feedback if this helps?

[behretv]

I have replaced the branch tag with v2.3.0. Now I received the following error message during the build process:

...
#16 [linux/arm64 4/7] RUN git config --global advice.detachedHead false   && source "/opt/ros/iron/setup.bash"   && mkdir -p src/   && git clone --depth 1 --branch ros2_v2.3.0   --recursive https://github.com/GeneSysElektronik/adma_ros_driver.git   "src/adma_ros_driver"   && rosdep update   --include-eol-distros   --rosdistro="iron"   && rosdep install   --from-paths src/   --rosdistro "iron"   --ignore-src -r -y    && colcon build   --cmake-clean-first   --install-base "/opt/ros/iron/install"   --merge-install   --ament-cmake-args " -Wno-dev"   && rm -rf   "build/"   "log/"   "src/"   "*.rosinstall"   /var/lib/apt/lists/*   && apt-get clean
#16 3.915 Cloning into 'src/adma_ros_driver'...
#16 9.683 Warning: running 'rosdep update' as root is not recommended.
#16 9.684   You should run 'sudo rosdep fix-permissions' and invoke 'rosdep update' again without sudo.
#16 41.02 reading in sources list data from /etc/ros/rosdep/sources.list.d
#16 41.02 Hit https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/osx-homebrew.yaml
#16 41.02 Hit https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/base.yaml
#16 41.02 Hit https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/python.yaml
#16 41.02 Hit https://raw.githubusercontent.com/ros/rosdistro/master/rosdep/ruby.yaml
#16 41.02 Hit https://raw.githubusercontent.com/ros/rosdistro/master/releases/fuerte.yaml
#16 41.02 Query rosdistro index https://raw.githubusercontent.com/ros/rosdistro/master/index-v4.yaml
#16 41.02 Skip distro "ardent" different from requested "iron"
#16 41.02 Skip distro "bouncy" different from requested "iron"
#16 41.02 Skip distro "crystal" different from requested "iron"
#16 41.02 Skip distro "dashing" different from requested "iron"
#16 41.02 Skip distro "eloquent" different from requested "iron"
#16 41.02 Skip distro "foxy" different from requested "iron"
#16 41.02 Skip distro "galactic" different from requested "iron"
#16 41.02 Skip distro "groovy" different from requested "iron"
#16 41.02 Skip distro "humble" different from requested "iron"
#16 41.02 Skip distro "hydro" different from requested "iron"
#16 41.02 Skip distro "indigo" different from requested "iron"
#16 41.02 Add distro "iron"
#16 41.02 Skip distro "jade" different from requested "iron"
#16 41.02 Skip distro "jazzy" different from requested "iron"
#16 41.02 Skip distro "kinetic" different from requested "iron"
#16 41.02 Skip distro "lunar" different from requested "iron"
#16 41.02 Skip distro "melodic" different from requested "iron"
#16 41.02 Skip distro "noetic" different from requested "iron"
#16 41.02 Skip distro "rolling" different from requested "iron"
#16 41.02 updated cache in /root/.ros/rosdep/sources.cache
#16 48.87 ERROR: the following packages/stacks could not have their rosdep keys resolved
#16 48.87 to system dependencies:
#16 48.88 adma_ros2_driver: Cannot locate rosdep definition for [ament_cmake_clange_format]
#16 48.88 Continuing to install resolvable dependencies...
#16 48.88 #All required rosdeps installed successfully
#16 52.77 Starting >>> adma_ros_driver_msgs
#16 403.0 --- stderr: adma_ros_driver_msgs
#16 403.0 gmake[2]: *** [CMakeFiles/adma_ros_driver_msgs__rosidl_generator_py.dir/build.make:174: CMakeFiles/adma_ros_driver_msgs__rosidl_generator_py.dir/rosidl_generator_py/adma_ros_driver_msgs/msg/_error_warning_s.c.o] Segmentation fault (core dumped)
#16 403.0 gmake[2]: *** Waiting for unfinished jobs....
#16 403.0 gmake[1]: *** [CMakeFiles/Makefile2:506: CMakeFiles/adma_ros_driver_msgs__rosidl_generator_py.dir/all] Error 2
#16 403.0 gmake: *** [Makefile:146: all] Error 2
#16 403.0 ---
#16 403.0 Failed   <<< adma_ros_driver_msgs [5min 50s, exited with code 2]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 [Processing: adma_ros_driver_msgs]
#16 403.1 
#16 403.1 Summary: 0 packages finished [5min 52s]
#16 403.1   1 package failed: adma_ros_driver_msgs
#16 403.1   1 package had stderr output: adma_ros_driver_msgs
#16 403.1   3 packages not processed
#16 ERROR: process "/dev/.buildkit_qemu_emulator /bin/bash -c -o pipefail git config --global advice.detachedHead false   && source \"${ROS_PACKAGE_PATH}/setup.bash\"   && mkdir -p src/   && git clone --depth 1 --branch ros2_v2.3.0   --recursive https://github.com/GeneSysElektronik/adma_ros_driver.git   \"src/adma_ros_driver\"   && rosdep update   --include-eol-distros   --rosdistro=\"${ROS_DISTRO}\"   && rosdep install   --from-paths src/   --rosdistro \"${ROS_DISTRO}\"   --ignore-src -r -y    && colcon build   --cmake-clean-first   --install-base \"${ROS_ROOT}/install\"   --merge-install   --ament-cmake-args \" -Wno-dev\"   && rm -rf   \"build/\"   \"log/\"   \"src/\"   \"*.rosinstall\"   /var/lib/apt/lists/*   && apt-get clean" did not complete successfully: exit code: 2
------
 > [linux/arm64 4/7] RUN git config --global advice.detachedHead false   && source "/opt/ros/iron/setup.bash"   && mkdir -p src/   && git clone --depth 1 --branch ros2_v2.3.0   --recursive https://github.com/GeneSysElektronik/adma_ros_driver.git   "src/adma_ros_driver"   && rosdep update   --include-eol-distros   --rosdistro="iron"   && rosdep install   --from-paths src/   --rosdistro "iron"   --ignore-src -r -y    && colcon build   --cmake-clean-first   --install-base "/opt/ros/iron/install"   --merge-install   --ament-cmake-args " -Wno-dev"   && rm -rf   "build/"   "log/"   "src/"   "*.rosinstall"   /var/lib/apt/lists/*   && apt-get clean:
403.1 [Processing: adma_ros_driver_msgs]
403.1 [Processing: adma_ros_driver_msgs]
403.1 [Processing: adma_ros_driver_msgs]
403.1 [Processing: adma_ros_driver_msgs]
403.1 [Processing: adma_ros_driver_msgs]
403.1 
403.1 Summary: 0 packages finished [5min 52s]
403.1   1 package failed: adma_ros_driver_msgs
403.1   1 package had stderr output: adma_ros_driver_msgs
403.1   3 packages not processed
------
Dockerfile:24
...

[behretv]

I receive the same error for the ros2_master branch.

[freak89dutch]

Alright, thanks for your response.
Its a bit confusing that its crashing now while building the adma_ros_driver_msgs package...

I would need some time to find out the problem since 1. you are using a EOL ROS2 distribution (we are primary developing with Humble) and 2. you are working on an ARM device (with docker). Both facts can lead to problems that are previously not noticed. As soon as I have new informations I will keep you updated.

Meanwhile just a question to "exclude" 1: would it be able for you to "downgrade" to ros2 humble in your docker setup and try to build/run it (or even testing it with ros2 jammy) to check if these problems are also present in other ros2 distributions?

[behretv]

Thank you for your fast reply! ROS 2 humble would be even preferred for my setup. I will try to build the Docker image with different ROS versions and keep you updated.

[behretv]

I am able to build the docker image, however since I do not have a GNSS device for testing at the moment, would you be able to test the docker image? I have uploaded the docker image to Dockerhub:

docker pull hmcvlab/genesys-adma-ros:1.0.12

[freak89dutch]

Alright, thanks for your prepared work.
I have tested your container on my PC with some recorded gsdb data. I started the adma_ros2_driver inside the container and run our gsdb_replay tool on my local machine and everything was working (driver inside container can receive UDP data and publish the correct data to the ROS network).
So with this container you should be able to work. For me I take this information that possibly we have some incompatibility with ROS2 Iron. If you have any further feedback or problems feel free to ask. Thanks again for your cooperation!

[behretv]

Thanks for testing the Docker image. I will be able to test the image in the near future in combination with a GNSS system. I will provide feedback, if any issues occur.