chore: update main branch for 0.9.0-beta release

.github/workflows/licensechecker.yaml

@@ -0,0 +1,9 @@
+name: Check License Lines
+on: [pull_request]
+jobs:
+  check-license-lines:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@master
+      - name: Check License Lines
+        uses: kt3k/[email protected]

.github/workflows/poetry-test.yml

@@ -9,8 +9,12 @@ jobs:
   build-and-test-ros2:
     strategy:
       matrix:
-        ros_distro: [jazzy]
-    runs-on: ubuntu-latest
+        include:
+          - ros_distro: jazzy
+            ubuntu_version: "24.04"
+          - ros_distro: humble
+            ubuntu_version: "22.04"
+    runs-on: ubuntu-${{ matrix.ubuntu_version }}
 
     container:
       image: osrf/ros:${{ matrix.ros_distro }}-desktop-full
@@ -22,13 +26,7 @@ jobs:
         uses: snok/install-poetry@v1
 
       - name: Install python dependencies
-        run: poetry install --with gdino
-
-      - name: Install ROS 2 dependencies
-        shell: bash
-        run: |
-          apt-get update
-          apt-get install -y ros-${{ matrix.ros_distro }}-tf-transformations
+        run: poetry install --with gdino,nomad
 
       - name: Update rosdep
         shell: bash
@@ -38,6 +36,7 @@ jobs:
       - name: Install ROS 2 package dependencies
         shell: bash
         run: |
+          apt-get update
           rosdep install --from-paths src --ignore-src -r -y
 
       - name: Build the workspace

.gitignore

@@ -171,3 +171,4 @@ logs/
 !examples/imgs/*.md
 
 src/examples/*-demo
+artifact_database.pkl

.licenserc.json

@@ -0,0 +1,12 @@
+{
+  "**/*.py": [
+    "# Copyright ",
+    "#",
+    "# Licensed under the Apache License, Version 2.0 (the \"License\");"
+  ],
+  "**/*.{h,cpp,hpp}": [
+    "// Copyright ",
+    "//",
+    "// Licensed under the Apache License, Version 2.0 (the \"License\");"
+  ]
+}

CONTRIBUTING.md

@@ -15,11 +15,11 @@ These guidelines have been therefore strongly influenced by the [ROS2 Contributi
 
 - Adopt community best-practices whenever possible instead of ad-hoc processes
 
-  Think about the end-users experiance when developing and contributing. Features accessible to a larger amount of potential users, utilising widely available solutions are more likely to be accepted.
+  Think about the end-users experience when developing and contributing. Features accessible to a larger amount of potential users, utilising widely available solutions are more likely to be accepted.
 
 - Think about the community as a whole
 
-  Think about the bigger picture. There are developers building different robots with different constraints. The landscape of available AI models is rapidly changing, coming with different capabilities and constraints. RAI wants to accomodate requirements of the whole community.
+  Think about the bigger picture. There are developers building different robots with different constraints. The landscape of available AI models is rapidly changing, coming with different capabilities and constraints. RAI wants to accommodate requirements of the whole community.
 
 There are a number of ways you can contribute to the RAI project.
 
@@ -31,7 +31,7 @@ Some of the easiest ways to contribute to RAI involve engaging in community disc
 
 ### Setting up the development environment
 
-To get started install RAI from source code using these [instructions](/README.md).
+Set up your development environment following [the instructions](docs/developer_guide.md#developer-environment-setup).
 
 ### Starting the discussion
 
@@ -45,7 +45,7 @@ To submit a change begin by forking this repository and making the changes on th
 Once the changes are ready to be proposed create a pull request back to the repository.
 In order to maintain a linear and clear commit history please:
 
-- make sure that all committs have meaningful messages
+- make sure that all commits have meaningful messages
 - if batches of "cleanup" or similar commits are present - squash them together
 - rebase onto the main branch of repository before making the PR
 

README.md

@@ -4,13 +4,18 @@
 > **RAI is in beta phase now, expect friction. Early contributors are the most welcome!** \
 > **RAI is developing fast towards a glorious release in time for ROSCon 2024.**
 
-![rai-image](./docs/imgs/demos.png)
+RAI is a flexible AI agent framework to develop and deploy Gen AI features for your robots.
+
+---
 
 <div align="center">
 
+![rai-image](./docs/imgs/RAI_simple_diagram_medium.png)
+
+---
+
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 ![GitHub Release](https://img.shields.io/github/v/release/RobotecAI/rai)
-![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/RobotecAI/rai/poetry-test.yml)
 ![Contributors](https://img.shields.io/github/contributors/robotecai/rai)
 
 ![Static Badge](https://img.shields.io/badge/Ubuntu-24.04-orange)
@@ -24,6 +29,8 @@
 
 </div>
 
+---
+
 ## Overview
 
 The RAI framework aims to:
@@ -35,11 +42,10 @@ The RAI framework aims to:
 - Include ROS 2-oriented tooling for agents.
 - Support a comprehensive task/mission orchestrator.
 
-# Table of Contents
+## Table of Contents
 
 - [Features](#features)
-- [Quick Start](#quick-start)
-- [Developer guide](./docs/developer_guide.md)
+- [Setup](#setup)
 - [Usage examples (demos)](#planned-demos)
 - [Further documentation](#further-documentation)
 - [ROSCon 2024 Talk](#roscon-2024)
@@ -53,30 +59,27 @@ The RAI framework aims to:
 - [x] ROS 2 action calling and other interfaces. The Agent can dynamically list interfaces, check their message type, and publish.
 - [x] Integration with LangChain to abstract vendors and access convenient AI tools.
 - [x] Tasks in natural language to nav2 goals.
-- [ ] NoMaD integration.
-- [ ] OpenVLA integration.
+- [x] NoMaD integration.
+- [ ] Grounded SAM 2 integration.
 - [ ] Improved Human-Robot Interaction with voice and text.
 - [ ] SDK for RAI developers.
 - [ ] Support for at least 3 different AI vendors.
 - [ ] Additional tooling such as GroundingDino.
 - [ ] UI for configuration to select features and tools relevant for your deployment.
 
-# Quick Start
-
-Currently, RAI supports Ubuntu 24.04 with ROS 2 Jazzy and Python 3.12.
-The release will also support Ubuntu 22.04 with ROS 2 Humble (it should work or be close to working now).
+## Setup
 
 ### 1. Setting up the workspace:
 
 #### 1.1 Install poetry
 
-Install poetry (1.8+) with the following line, or
+Install poetry (1.8+) with the following line:
 
 ```bash
 curl -sSL https://install.python-poetry.org | python3 -
 ```
 
-by following the official [docs](https://python-poetry.org/docs/#installation)
+Alternatively, you can opt to do so by following the [official docs](https://python-poetry.org/docs/#installation).
 
 #### 1.2 Clone the repository:
 
@@ -108,76 +111,80 @@ source ./setup_shell.sh
 
 ### 3. Setting up vendors
 
-RAI is fully vendor-agnostic, however the beta development work currently utilizes OpenAI models. Setting the `OPENAI_API_KEY` environment variable will yield the best results.
+RAI aims to be vendor-agnostic. You can use the configuration in [config.toml](./config.toml) to set up your vendor of choice for most RAI modules.
 
-#### OpenAI
+> [!NOTE]  
+> Some of the RAI modules still are hardcoded to OpenAI models. An effort is underway to make them configurable via [config.toml](./config.toml) file.
 
-If you do not have a key, see how to generate one [here](https://platform.openai.com/docs/quickstart).
+If you do not have a vendor's key, follow the instructions below:
 
-```
-export OPENAI_API_KEY=""
-```
+**OpenAI:** [link](https://platform.openai.com/docs/quickstart).
+**AWS Bedrock:** [link](https://console.aws.amazon.com/bedrock/home?#/overview).
 
-#### Congratulations, your installation is now complete! Head to [Running example](./docs/developer_guide.md)
+Congratulations, your installation is now completed!
 
-# Running RAI
+## Running RAI
 
 ![rosbot-xl-example](./docs/imgs/rosbot-xl-example.gif)
 
-RAI is a sophisticated framework targeted at solving near general cases. As of now, we provide the following examples:
+You can start by running the following examples:
 
-1. Engage with your ROS 2 network through an intuitive Streamlit chat interface.
-2. Explore the O3DE Husarion ROSbot XL demo and assign tasks via natural language.
+1. Hello RAI: Interact directly with your ROS 2 environment through an intuitive Streamlit chat interface.
+2. Explore the O3DE Husarion ROSbot XL demo and have your robot do tasks defined with natural language.
 
-If you are more ambitious:
+### Hello RAI
 
-- Create your own robot description package and unleash it with the rai_whoami node.
-- Run Streamlit powered by your custom robot’s description package and effortlessly access your robot's documentation as well as identity and constitution.
-- Implement additional tools via LangChain's @tool and use them in your chat.
-
-## 1. Chat Interface
-
-Chat seamlessly with your setup, retrieve images from cameras, adjust parameters on the fly, and get comprehensive information about your topics.
+Chat seamlessly with your ROS 2 environment, retrieve images from cameras, adjust parameters, and get information about your ROS interfaces.
 
 ```bash
 streamlit run src/rai_hmi/rai_hmi/streamlit_hmi_node.py
 ```
 
 Remember to run this command in a sourced shell.
 
-## 2. O3DE Rosbot XL Demo
+### O3DE Rosbot XL Demo
 
 This demo provides a practical way to interact with and control a virtual Husarion ROSbot XL within a simulated environment.
 Using natural language commands, you can assign tasks to the robot, allowing it to perform a variety of actions.
 
 Given that this is a beta release, consider this demo as an opportunity to explore the framework's capabilities, provide feedback, and contribute.
 Try different commands, see how the robot responds, and use this experience to understand the potential and limitations of the system.
 
-Follow this guide: [husarion-rosbot-xl-demo](./docs/demos.md)
+Follow this guide: [husarion-rosbot-xl-demo](docs/demos.md)
+
+## What's next?
+
+Once you know your way around RAI, try the following challenges, with the aid the [developer guide](developer_guide.md):
+
+- Run RAI on your own robot and talk to it, asking questions about what is in its documentation (and others!).
+- Implement additional tools and use them in your interaction.
+- Try a complex, multi-step task for your robot, such as going to several points to perform observations!
+
+Soon you will have an opportunity to work with new RAI demos across several domains.
 
-## Planned demos
+### Planned demos
 
 | Application                                | Robot                          | Description                                                                                                                                      | Link                                                          |
 | ------------------------------------------ | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- |
 | Mission and obstacle reasoning in orchards | Autonomous tractor             | In a beautiful scene of a virtual orchard, RAI goes beyond obstacle detection to analyze best course of action for a given unexpected situation. | [🌾 demo](https://github.com/RobotecAI/rai-agriculture-demo)  |
-| Manipulation tasks with natural language   | Robot Arm (Franka Panda)       | Complete flexible manipulation tasks thanks to RAI and OpenVLA                                                                                   | [🦾 demo](https://github.com/RobotecAI/rai-manipulation-demo) |
+| Manipulation tasks with natural language   | Robot Arm (Franka Panda)       | Complete flexible manipulation tasks thanks to RAI and Grounded SAM 2                                                                            | [🦾 demo](https://github.com/RobotecAI/rai-manipulation-demo) |
 | Quadruped inspection demo                  | A robot dog (ANYbotics ANYmal) | Perform inspection in a warehouse environment, find and report anomalies                                                                         | link TBD                                                      |
 
-# Community
+## Community
 
-## RAI Q&A
+### RAI Q&A
 
 Please take a look at [Q&A](https://github.com/RobotecAI/rai/discussions/categories/q-a).
 
-## Developer Resources
+### Developer Resources
 
 See our [Developer Guide](docs/developer_guide.md).
 
-## Contributing
+### Contributing
 
 You are welcome to contribute to RAI! Please see our [Contribution Guide](CONTRIBUTING.md).
 
-## RAI release and talk
+### RAI release and talk
 
 RAI will be released on **October 15th**, right before [ROSCon 2024](https://roscon.ros.org/2024/).
 If you are going to the conference, come join us at RAI talk on October 23rd.

config.toml

@@ -0,0 +1,19 @@
+[vendor]
+name = "openai" # openai, aws, ollama
+
+[aws]
+simple_model = "anthropic.claude-3-haiku-20240307-v1:0"
+complex_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
+embeddings_model = "amazon.titan-embed-text-v1"
+region_name = "us-east-1"
+
+[openai]
+simple_model = "gpt-4o-mini"
+complex_model = "gpt-4o-2024-08-06"
+embeddings_model = "text-embedding-ada-002"
+
+[ollama]
+simple_model = "llama3.1"
+complex_model = "llama3.1:70b"
+embeddings_model = "llama3.1"
+base_url = "http://localhost:11434"

demos.repos

@@ -1,5 +1,5 @@
 repositories:
-  src/examples/rosbot-xl-demo:
+  src/examples/rai-rosbot-xl-demo:
     type: git
     url: https://github.com/RobotecAI/rai-rosbot-xl-demo.git
     version: development

docs/create_robots_whoami.md

@@ -1,20 +1,24 @@
-# robot_whoami
+# Your robot identity in RAI
 
-Certain parts of RAI rely on robot's whoami package. Your robot's whoami package serves as a configuration package for the rai_whoami node, which is responsible for the robot identity, self-understanding, ethical code, and documentation.
+RAI Agent needs to understand what kind of robot it is running on.
+This includes its looks, purpose, ethical code, equipment, capabilities and documentation.
+To configure RAI for your robot, provide contents for your robot's so called `whoami` package.
 
-## Creating robot's whoami (Franka Emika Panda)
+Your robot's `whoami` package serves as a configuration package for the `rai_whoami` node.
 
-1. Create a whoami package for panda
+## Example (Franka Emika Panda arm)
+
+1. Create a whoami package for Panda
 
 ```
 poetry run create_rai_ws --name panda --destination-directory src/examples
 ```
 
-2. Fill in the src/examples/panda_whoami/description folder with data:\
+2. Fill in the `src/examples/panda_whoami/description` folder with data:\
    2.1 Save [this image](https://robodk.com/robot/img/Franka-Emika-Panda-robot.png) into `src/examples/panda_whoami/description/images`\
    2.2 Save [this document](https://github.com/user-attachments/files/16417196/Franka.Emika.Panda.robot.-.RoboDK.pdf) in `src/examples/panda_whoami/description/documentation`
 
-3. Run the parse_whoami_package to build vector database and reason out the identity
+3. Run the `parse_whoami_package`. This will process the documentation, building it into a vector database, which is used by RAI agent to reason about its identity.
 
 ```
 poetry run parse_whoami_package src/examples/panda_whoami/description src/examples/panda_whoami/description
@@ -23,11 +27,11 @@ poetry run parse_whoami_package src/examples/panda_whoami/description src/exampl
 > [!IMPORTANT]  
 > For now, this works only if you have OPENAI_API_KEY variable exported.
 
-## Testing panda_whoami
+## Testing
 
-rai_whoami provides services for gathering information about current platform. Test the panda_whoami package by:
+You can test your new `panda_whoami` package by calling `rai_whoami` services:
 
-1. Building and sourcing the install
+2. Building and sourcing the install
 
 ```
 colcon build
@@ -42,9 +46,7 @@ ros2 run rai_whoami rai_whoami_node --ros-args -p robot_description_package:="pa
 ros2 service call /rai_whoami_identity_service std_srvs/srv/Trigger # ask for identity
 ros2 service call /rai_whoami_selfimages_service std_srvs/srv/Trigger # ask for images folder
 ros2 service call /rai_whoami_constitution_service std_srvs/srv/Trigger # ask for robot constitution
-ros2 service call /rai_whoami_documentation_service rai_interfaces/srv/VectorStoreRetrieval "query:  'maximum load'" # ask for panda's maximum load
+ros2 service call /rai_whoami_documentation_service rai_interfaces/srv/VectorStoreRetrieval "query:  'maximum load'" # ask for Panda's maximum load
 ```
 
-## Voila
-
-If rai_whoami responds to the service calls, panda_whoami package has been properly initialized.
+If your service calls succeed, your `panda_whoami` package has been properly initialized.