Dora the Mug Finder

Consists on the implementation of a deep neural network to classify objects collected from 3D models or RGB-D cameras
Explore the Wiki »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contributing
5. License
6. Contact
7. Acknowledgments

About The Project

This project was developed for Advanced Industrial Vision Systems class for the second report. The objective is to detect and extract objects from a point cloud and then pass it through a classifier that will tell what the object is and some information about it's physical characteristics. For example, it's a Mug and has a certain bounding box.

(back to top)

Getting Started

This project uses ROS Noetic to aid the use of the kinect camera and the object classification CNN is built based on PyTorch. To do object detection is used Open3D.

Prerequisites

To use this code, first add the next line in the bashrc or your shell configuration file:

#Dora The Mug FInder
export DORA=/home/andre/dora_the_mug_finder
export PYTHONPATH="$PYTHONPATH:${HOME}/catkin_ws/src/Dora_the_mug_finder_SAVI"

Replace the path of DORA to where all the datasets are stored in your computer. You can download them here.

Afterward, update the shell with the new configuration using:

source ~/.bashrc

If you use zsh, just change to .zshrc.

Inside DORA folder, there should be a structure similar to:

• models
• rgbd-dataset
• rgbd-scenes-v2
  • pc
  • imgs
• rosbag

Installation

To install the project, clone the repository inside the src folder of your catkin_ws, running the following lines:

git clone https://github.com/andrefdre/Dora_the_mug_finder_SAVI.git
cd ..
catkin_make

To install all the dependencies of this package, just run in your terminal:

roscd dora_the_mug_finder_bringup
cd ..
pip install -r requirements.txt

If you have/want to use a Kinect camera with this project, you can find here how to install all the dependencies needed.

(back to top)

Usage

Train Model

To Train the model, run the code:

rosrun dora_the_mug_finder_bringup model_train.py -fn <folder_name> -mn <model_name> -n_epochs 50 -batch_size 256 -c 0

Where the <folder_name> and <model_name> should be replaced by the names you want to give.

---

Run Object extractor and classifier

To run the detector with previous trained model run the code:

roslaunch dora_the_mug_finder_bringup dora_bringup.launch mn:=<model_name> fn:=<folder_name>

Where the <folder_name> and <model_name> should be replaced by a name for the model previously set while training. If you want to visualize extracted images run:

roslaunch dora_the_mug_finder_bringup dora_bringup.launch mn:=<model_name> fn:=<folder_name> visualize:=True

It's also possible to add the argument audio to initialize audio describing the objects, setting it to true:

roslaunch dora_the_mug_finder_bringup dora_bringup.launch mn:=<model_name> fn:=<folder_name> audio:=true

---

Run Kinect

It's also possible to use a kinect camera for processing in real time, by adding the kinect argument:

roslaunch dora_the_mug_finder_bringup dora_bringup.launch mn:=<model_name> fn:=<folder_name> kinect:=true

(back to top)

Contributing

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the GPL License. See LICENSE.txt for more information.

(back to top)

Contact

André Cardoso - [email protected]

Tatiana Resende - [email protected]

Fábio Sousa - [email protected]

Project Link: Dora the Mug Finder

(back to top)

Acknowledgments

• Professor Miguel Oliveira - [email protected]

(back to top)