新的 ort 多态部署库，以支持 CUDA、TensorRT 和 OpenVINO 的 Provider 选择

cv-rmvl · Jun 13, 2024 · aaed544 · aaed544
1 parent 1156bf3
commit aaed544
Show file tree

Hide file tree

Showing 37 changed files with 474 additions and 634 deletions.
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -195,7 +195,11 @@ foreach(m ${RMVL_EXTRA_MODULE})
   if("${m}" STREQUAL "types")
     continue()
   endif()
-  string(REGEX REPLACE "rmvl_" " " ${m}_modules ${RMVL_${mUP}_MODULES_BUILD})
+  if("${RMVL_${mUP}_MODULES_BUILD}" STREQUAL "")
+    set(${m}_modules "")
+  else()
+    string(REGEX REPLACE "rmvl_" " " ${m}_modules ${RMVL_${mUP}_MODULES_BUILD})
+  endif()
 endforeach()
 string(REGEX REPLACE "rmvl_" " " main_modules ${RMVL_MAIN_MODULES_BUILD})
 

diff --git a/README.md b/README.md
@@ -2,17 +2,17 @@
 
 **机器人控制与视觉库**
 
-| 构建配置 |            编译器/环境            |        Github Actions 工作流状态         |
-| :------: | :-------------------------------: | :--------------------------------------: |
-|  CMake   | GCC 7.5.0 x86_64-linux-gnu<br />GCC 9.4.0 x86_64-linux-gnu<br />GCC 12.3.0 x86_64-linux-gnu | [![1.x in Linux](https://github.com/cv-rmvl/rmvl/actions/workflows/linux-1.x.yml/badge.svg)](https://github.com/cv-rmvl/rmvl/actions/workflows/linux-1.x.yml) |
+| 构建配置 |                         编译器/环境                          |                  Github Actions 工作流状态                   |
+| :------: | :----------------------------------------------------------: | :----------------------------------------------------------: |
+|  CMake   | GCC 7.5.0 x86_64-linux-gnu<br />GCC 9.4.0 x86_64-linux-gnu<br />GCC 12.3.0 x86_64-linux-gnu<br />GCC 14.0.1 x86_64-linux-gnu | [![1.x in Linux](https://github.com/cv-rmvl/rmvl/actions/workflows/linux-1.x.yml/badge.svg)](https://github.com/cv-rmvl/rmvl/actions/workflows/linux-1.x.yml) |
 
-RMVL 最初是面向 RoboMaster 赛事的视觉库，因此称为 RoboMaster Vision Library，现计划并逐步完善与机器人相关的视觉、控制、通信的功能，旨在打造适用范围广、使用简洁、架构统一、功能强大的视觉控制一体库。
+RMVL 最初是面向 RoboMaster 赛事的视觉库，现在此之上逐步完善有关基础算法、机器视觉、通信的功能，旨在打造适用范围广、使用简洁、架构统一、功能强大的视觉控制一体库。
 
 ### 项目资源
 
-* 说明文档: <https://cv-rmvl.github.io>
+* 用户手册: <https://cv-rmvl.github.io>
 
-* 工单: <https://github.com/cv-rmvl/rmvl/issues>
+* 问题: <https://github.com/cv-rmvl/rmvl/issues>
 
 * VSCode 扩展插件
 

diff --git a/cmake/RMVLModule.cmake b/cmake/RMVLModule.cmake
@@ -100,9 +100,6 @@ macro(rmvl_add_module _name)
   if(NOT DEFINED BUILD_${the_module}_INIT)
     set(BUILD_${the_module}_INIT ON)
   endif()
-  if(DEFINED BUILD_${the_module})
-    unset(BUILD_${the_module} CACHE)
-  endif()
   option(BUILD_${the_module} "Include ${the_module} module into the RMVL build" ${BUILD_${the_module}_INIT}) # create option to enable/disable this module
 
   if(BUILD_${the_module})

diff --git a/doc/tutorials/introduction/linux/install.md b/doc/tutorials/introduction/linux/install.md
@@ -130,9 +130,7 @@ sudo apt install libeigen3-dev
 
 #### 1.4 onnxruntime{#install_onnxruntime}
 
-onnxruntime 库是目前数字识别所依赖的第三方库，如果有需要开启此功能，则需要安装 onnxruntime
-
-- 获取压缩包
+- 获取压缩包，可选择对应的发行版，也可以是 gpu、cuda12 或 cpu 版本，下面以 1.12.0 的 cpu 版本为例
   ```shell
   curl -SL https://github.com/microsoft/onnxruntime/releases/download/v1.12.0/onnxruntime-linux-x64-1.12.0.tgz -o onnxruntime-linux-x64-1.12.0.tgz
   ```
@@ -143,12 +141,9 @@ onnxruntime 库是目前数字识别所依赖的第三方库，如果有需要
 - 安装（复制头文件与库文件）
   ```shell
   sudo mkdir /usr/local/include/onnxruntime
-  sudo cp onnxruntime-linux-x64-1.12.0/include/* /usr/local/include/onnxruntime
-  sudo cp -r onnxruntime-linux-x64-1.12.0/lib /usr/local
-  ```
-- 移除中间文件
-  ```shell
-  rm -r onnxruntime-linux-x64-1.12.0 && rm onnxruntime-linux-x64-1.12.0.tgz
+  cd onnxruntime-linux-x64-1.12.0
+  sudo cp -r include/* /usr/local/include/onnxruntime
+  sudo cp -r lib /usr/local
   ```
 
 至此 onnxruntime 安装完成。

diff --git a/doc/tutorials/modules/algorithm/lsqnonlin.md b/doc/tutorials/modules/algorithm/lsqnonlin.md
@@ -72,7 +72,7 @@
 1. 选择一初始点 \f$\pmb x_0=(x_{1,0},x_{2,0},\cdots,x_{n,0})^T\f$；
 2. 算出 \f[\Delta\pmb x_0=-\pmb H_0^{-1}\pmb J_0^T\pmb\varphi(\pmb x_k)\tag{1-6}\f]
 3. 令 \f$\pmb x_1\f$ 为函数 \f$f(\pmb x)\f$ 的极小点的第 1 次近似，则有 \f[\pmb x_1=\pmb x_0+\Delta\pmb x_0\tag{1-7}\f]
-4. 以 \f$\pmb x_1\f$ 代替前面的 \f$\pmb x_0\f$，\f$\Delta\pmb x_1\f$ 代替 \f$\Delta\pmb x_0\f$，重复上述计算过程，直到 \f[\|\pmb\varphi(\pmb x_k)\|<\epsilon'\tag{1-8a}\f] 或 \f[\|\nabla f(\pmb x_k)\|<\epsilon"\tag{1-8b}\f] 为止。\f$\epsilon'\f$ 和 \f$\epsilon"\f$ 是预先给定的精度。
+4. 以 \f$\pmb x_1\f$ 代替前面的 \f$\pmb x_0\f$，\f$\Delta\pmb x_1\f$ 代替 \f$\Delta\pmb x_0\f$，重复上述计算过程，直到 \f[\|\pmb\varphi(\pmb x_k)\|<\epsilon'\tag{1-8a}\f] 或 \f[\|\nabla f(\pmb x_k)\|<\epsilon''\tag{1-8b}\f] 为止。\f$\epsilon'\f$ 和 \f$\epsilon''\f$ 是预先给定的精度。
 
 #### 1.3 改进
 
@@ -85,7 +85,7 @@
 
 #### 1.4 如何使用
 
-可参考 `rm::lsqnonlin` 函数，例如，我们需要拟合一个正弦函数，下面的 `obtain` 函数就是观测每一帧的数据。
+RMVL 提供了改进的 Gauss-Newton 迭代算法，可参考 `rm::lsqnonlin` 函数。例如，我们需要拟合一个正弦函数\f[y=A\sin(\omega t+\varphi_0)+b\f]其中，\f$A,\omega,\varphi_0,b\f$ 是待拟合的参数，不妨统一写为 \f$\pmb x=(A,\omega,\varphi_0,b)\f$，也就是说我们需要拟合的函数是\f[\green y=x_1\sin(x_2\green t+x_3)+x_4\f]其中 \f$t\f$ 和 \f$y\f$ 是可以观测到的数据，我们需要通过观测的数据来拟合 \f$\pmb x\f$ 的值。比方说，下面的 `obtain` 函数就可以观测每一帧的数据。
 
 ```cpp
 double obtain();

diff --git a/doc/tutorials/modules/tools/opcua.md b/doc/tutorials/modules/tools/opcua.md
@@ -97,8 +97,8 @@ int main()
 
 int main()
 {
-    // 创建 OPC UA 客户端，连接到 localhost:4840
-    rm::Client clt("opc.tcp://localhost:4840");
+    // 创建 OPC UA 客户端，连接到 127.0.0.1:4840
+    rm::Client clt("opc.tcp://127.0.0.1:4840");
 
     /* other code */
 }
@@ -141,7 +141,7 @@ int main()
 
 int main()
 {
-    rm::Client clt("opc.tcp://localhost:4840");
+    rm::Client clt("opc.tcp://127.0.0.1:4840");
 
     // 使用管道运算符 "|" 进行路径搜索，寻找待读取的变量
     auto node = rm::nodeObjectsFolder | clt.find("number");
@@ -210,7 +210,7 @@ int main()
 
 int main()
 {
-    rm::Client clt("opc.tcp://localhost:4840");
+    rm::Client clt("opc.tcp://127.0.0.1:4840");
 
     // 设置输入参数，1 和 2 是 Int32 类型的，因此可以直接隐式构造
     std::vector<rm::Variable> input = {1, 2};
@@ -287,7 +287,7 @@ int main()
 
 int main()
 {
-    rm::Client clt("opc.tcp://localhost:4840");
+    rm::Client clt("opc.tcp://127.0.0.1:4840");
 
     // 路径搜索寻找 C2
     auto node_c2 = rm::nodeObjectsFolder | clt.find("A") | clt.find("B1") | clt.find("C2");
@@ -369,7 +369,7 @@ int main()
 
 int main()
 {
-    rm::Client clt("opc.tcp://localhost:4840");
+    rm::Client clt("opc.tcp://127.0.0.1:4840");
     auto node = rm::nodeObjectsFolder | clt.find("number");
     for (int i = 0; i < 100; ++i)
     {
@@ -396,7 +396,7 @@ void onChange(UA_Client *, UA_UInt32, void *, UA_UInt32, void *, UA_DataValue *v
 
 int main()
 {
-    rm::Client clt("opc.tcp://localhost:4840");
+    rm::Client clt("opc.tcp://127.0.0.1:4840");
     auto node = rm::nodeObjectsFolder | clt.find("number");
     // 监视变量，这里的 onChange 同样可以写成无捕获列表的 lambda 表达式，因为存在隐式转换
     client.monitor(node_id, onChange, 5);

diff --git a/extra/detector/CMakeLists.txt b/extra/detector/CMakeLists.txt
@@ -3,7 +3,7 @@
 # ----------------------------------------------------------------------------
 rmvl_add_module(
   detector INTERFACE
-  DEPENDS imgproc group
+  DEPENDS group
 )
 
 # armor_detector

diff --git a/extra/detector/include/rmvl/detector/armor_detector.h b/extra/detector/include/rmvl/detector/armor_detector.h
@@ -28,7 +28,7 @@ namespace rm
 //! 装甲板识别模块
 class ArmorDetector final : public detector
 {
-    std::unique_ptr<OnnxRT> _ort;
+    std::unique_ptr<OnnxNet> _ort;
     std::unordered_map<int, RobotType> _robot_t;
 
 public:
@@ -37,16 +37,9 @@ class ArmorDetector final : public detector
 
     explicit ArmorDetector(const std::string &model)
     {
-        _ort = std::make_unique<OnnxRT>(model);
-        _robot_t[0] = RobotType::UNKNOWN;
-        _robot_t[1] = RobotType::HERO;
-        _robot_t[2] = RobotType::ENGINEER;
-        _robot_t[3] = RobotType::INFANTRY_3;
-        _robot_t[4] = RobotType::INFANTRY_4;
-        _robot_t[5] = RobotType::INFANTRY_5;
-        _robot_t[6] = RobotType::OUTPOST;
-        _robot_t[7] = RobotType::BASE;
-        _robot_t[8] = RobotType::SENTRY;
+        _ort = std::make_unique<ClassificationNet>(model);
+        for (int i = 0; i < 9; ++i)
+            _robot_t[i] = static_cast<RobotType>(i);
     }
 
     /**

diff --git a/extra/detector/include/rmvl/detector/detector.h b/extra/detector/include/rmvl/detector/detector.h
@@ -13,7 +13,7 @@
 
 #include "rmvl/group/group.h"
 
-#include "rmvl/imgproc/pretreat.h"
+#include "rmvl/core/pretreat.hpp"
 
 namespace rm
 {

diff --git a/extra/detector/include/rmvl/detector/gyro_detector.h b/extra/detector/include/rmvl/detector/gyro_detector.h
@@ -29,25 +29,18 @@ class GyroDetector final : public detector
 {
     int _armor_num; //!< 默认装甲板数目
 
-    std::unique_ptr<OnnxRT> _ort;
+    std::unique_ptr<OnnxNet> _ort;
     std::unordered_map<int, RobotType> _robot_t;
 
 public:
     GyroDetector(int armor_num) : _armor_num(armor_num) {}
     ~GyroDetector() = default;
 
-    explicit GyroDetector(const std::string &model, int armor_num) : _armor_num(armor_num)
+    GyroDetector(const std::string &model, int armor_num) : _armor_num(armor_num)
     {
-        _ort = std::make_unique<OnnxRT>(model);
-        _robot_t[0] = RobotType::UNKNOWN;
-        _robot_t[1] = RobotType::HERO;
-        _robot_t[2] = RobotType::ENGINEER;
-        _robot_t[3] = RobotType::INFANTRY_3;
-        _robot_t[4] = RobotType::INFANTRY_4;
-        _robot_t[5] = RobotType::INFANTRY_5;
-        _robot_t[6] = RobotType::OUTPOST;
-        _robot_t[7] = RobotType::BASE;
-        _robot_t[8] = RobotType::SENTRY;
+        _ort = std::make_unique<ClassificationNet>(model);
+        for (int i = 0; i < 9; ++i)
+            _robot_t[i] = static_cast<RobotType>(i);
     }
 
     /**

diff --git a/extra/detector/src/armor_detector/detect.cpp b/extra/detector/src/armor_detector/detect.cpp
@@ -29,7 +29,7 @@ DetectInfo ArmorDetector::detect(std::vector<group::ptr> &groups, cv::Mat &src,
     // 二值化处理图像
     PixChannel ch_minus = color == RED ? BLUE : RED;
     int thesh = color == RED ? para::armor_detector_param.GRAY_THRESHOLD_RED : para::armor_detector_param.GRAY_THRESHOLD_BLUE;
-    info.bin = rm::binary(src, color, ch_minus, thesh);
+    info.bin = binary(src, color, ch_minus, thesh);
 
     // 找到所有的灯条和装甲板
     find(info.bin, info.features, info.combos, info.rois);

diff --git a/extra/detector/src/armor_detector/find.cpp b/extra/detector/src/armor_detector/find.cpp
@@ -38,8 +38,11 @@ void ArmorDetector::find(cv::Mat &src, std::vector<feature::ptr> &features, std:
             for (const auto &armor : armors)
             {
                 cv::Mat roi = Armor::getNumberROI(src, armor);
-                auto type = _ort->inference({roi}, {para::armor_detector_param.MODEL_MEAN}, {para::armor_detector_param.MODEL_STD})[0];
-                armor->setType(_robot_t[type]);
+                PreprocessOptions preop;
+                preop.means = {para::armor_detector_param.MODEL_MEAN};
+                preop.stds = {para::armor_detector_param.MODEL_STD};
+                int idx = ClassificationNet::cast(_ort->inference({roi}, preop, {})).first;
+                armor->setType(_robot_t[idx]);
                 rois.emplace_back(roi);
             }
             // eraseFakeArmors(armors);

diff --git a/extra/detector/src/gyro_detector/find.cpp b/extra/detector/src/gyro_detector/find.cpp
@@ -9,8 +9,6 @@
  *
  */
 
-#include <iostream>
-
 #include <opencv2/imgproc.hpp>
 
 #include "rmvl/detector/gyro_detector.h"
@@ -40,8 +38,11 @@ void GyroDetector::find(cv::Mat &src, std::vector<feature::ptr> &features, std::
             for (const auto &armor : armors)
             {
                 cv::Mat roi = Armor::getNumberROI(src, armor);
-                auto type = _ort->inference({roi}, {para::gyro_detector_param.MODEL_MEAN}, {para::gyro_detector_param.MODEL_STD})[0];
-                armor->setType(_robot_t[type]);
+                PreprocessOptions preop;
+                preop.means = {para::gyro_detector_param.MODEL_MEAN};
+                preop.stds = {para::gyro_detector_param.MODEL_STD};
+                int idx = ClassificationNet::cast(_ort->inference({roi}, preop, {})).first;
+                armor->setType(_robot_t[idx]);
                 rois.emplace_back(roi);
             }
             // eraseFakeArmors(armors);

diff --git a/extra/detector/src/gyro_detector/match.cpp b/extra/detector/src/gyro_detector/match.cpp
@@ -140,8 +140,8 @@ void GyroDetector::match(std::vector<group::ptr> &groups, std::vector<combo::ptr
         {
             // 初始化代表装甲板集合
             std::unordered_set<combo::ptr> represent_set;
-            for (const auto &[represent, combo_set] : combo_maps)
-                represent_set.insert(represent);
+            for (const auto &p : combo_maps)
+                represent_set.insert(p.first); // represent
             // 距离最近的装甲板组匹配到相应的序列组中
             for (auto &p_group : groups)
             {
@@ -188,8 +188,8 @@ void GyroDetector::match(std::vector<group::ptr> &groups, std::vector<combo::ptr
         {
             // 初始化代表装甲板集合
             std::unordered_set<combo::ptr> represent_set;
-            for (const auto &[represent, combo_set] : combo_maps)
-                represent_set.insert(represent);
+            for (const auto &p : combo_maps)
+                represent_set.insert(p.first); // represent
             size_t before_size = groups.size();
             for (size_t i = 0; i < before_size; i++)
             {

diff --git a/include/rmvl/rmvl.hpp b/include/rmvl/rmvl.hpp
@@ -30,10 +30,6 @@
 #include "rmvl/light.hpp"
 #endif // HAVE_RMVL_LIGHT
 
-#ifdef HAVE_RMVL_IMGPROC
-#include "rmvl/imgproc.hpp"
-#endif // HAVE_RMVL_IMGPROC
-
 #ifdef HAVE_RMVL_OPCUA
 #include "rmvl/opcua.hpp"
 #endif // HAVE_RMVL_OPCUA

diff --git a/modules/camera/include/rmvl/camera/camutils.hpp b/modules/camera/include/rmvl/camera/camutils.hpp
@@ -80,7 +80,7 @@ struct CameraConfig
      * @param[in] modes 配置模式参数包
      */
     template <typename... Args>
-    inline void set(Args... modes) { [[maybe_unused]] int _[] = {(conf(modes), 0)...}; }
+    inline void set(Args... modes) { (..., conf(modes)); }
 
 private:
     inline void conf(TriggerChannel chn) { trigger_channel = chn; }

diff --git a/modules/core/CMakeLists.txt b/modules/core/CMakeLists.txt
@@ -16,7 +16,7 @@ rmvl_generate_para(core)
 rmvl_add_module(
   core
   EXTRA_HEADER ${EIGEN3_INCLUDE_DIR}
-  EXTERNAL ${CMAKE_THREAD_LIBS_INIT} opencv_core
+  EXTERNAL ${CMAKE_THREAD_LIBS_INIT} opencv_imgproc
 )
 
 rmvl_compile_definitions(

diff --git a/...s/imgproc/include/rmvl/imgproc/pretreat.h → modules/core/include/rmvl/core/pretreat.hpp b/...s/imgproc/include/rmvl/imgproc/pretreat.h → modules/core/include/rmvl/core/pretreat.hpp
@@ -1,11 +1,11 @@
 /**
- * @file pretreat.h
- * @author RoboMaster Vision Community
- * @brief Header of the image pretreating module
+ * @file pretreat.hpp
+ * @author zhaoxi ([email protected])
+ * @brief 图像预处理模块
  * @version 1.0
- * @date 2022-11-23
+ * @date 2024-06-05
  *
- * @copyright Copyright 2023 (c), RoboMaster Vision Community
+ * @copyright Copyright 2024 (c), zhaoxi
  *
  */
 
@@ -16,7 +16,15 @@
 namespace rm
 {
 
-//! @addtogroup imgproc
+//! @addtogroup core
+//! @{
+//! @defgroup core_pretreat 图像预处理模块
+//! @{
+//! @brief 提供了二值化等图像预处理功能
+//! @} core_pretreat
+//! @} core
+
+//! @addtogroup core_pretreat
 //! @{
 
 //! 像素通道枚举
@@ -52,6 +60,6 @@ cv::Mat binary(cv::Mat src, PixChannel ch1, PixChannel ch2, uint8_t threshold);
  */
 cv::Mat binary(cv::Mat src, uint8_t threshold);
 
-//! @} imgproc
+//! @} core_pretreat
 
 } // namespace rm
diff --git a/modules/imgproc/perf/perf_pretreat.cpp → modules/core/perf/perf_pretreat.cpp b/modules/imgproc/perf/perf_pretreat.cpp → modules/core/perf/perf_pretreat.cpp
@@ -1,19 +1,19 @@
 /**
  * @file perf_pretreat.cpp
- * @author RoboMaster Vision Community
- * @brief
+ * @author zhaoxi ([email protected])
+ * @brief 
  * @version 1.0
- * @date 2022-11-24
- *
- * @copyright Copyright 2023 (c), RoboMaster Vision Community
- *
+ * @date 2024-06-05
+ * 
+ * @copyright Copyright 2024 (c), zhaoxi
+ * 
  */
 
 #include <benchmark/benchmark.h>
 
 #include <opencv2/imgproc.hpp>
 
-#include "rmvl/imgproc/pretreat.h"
+#include "rmvl/core/pretreat.hpp"
 
 namespace rm_test
 {