From 79d590bdd3c2ed66e685ffae19239c595fa90494 Mon Sep 17 00:00:00 2001 From: Weixuan Xiao Date: Wed, 14 Aug 2024 15:46:32 +0200 Subject: [PATCH 1/6] Remove dark theme logo from README during release --- .github/workflows/do-release.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/do-release.yml b/.github/workflows/do-release.yml index 83d261de..c65b00a7 100644 --- a/.github/workflows/do-release.yml +++ b/.github/workflows/do-release.yml @@ -13,11 +13,16 @@ jobs: steps: - name: Checkout code uses: actions/checkout@v4.1.0 + + - name: Remove dark theme logo from README + run: sed -i 's/.*#gh-dark-mode-only.*//' README.md + - name: Setup PDM uses: pdm-project/setup-pdm@v3 with: python-version: "3.10" cache: false + - name: Build dist run: pdm build From e793da83b5ae3f3696752b181658d62a083f11f4 Mon Sep 17 00:00:00 2001 From: Weixuan Xiao Date: Wed, 14 Aug 2024 17:42:05 +0200 Subject: [PATCH 2/6] Add links to image classification and object detection tasks --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 65187bac..950edc88 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,8 @@ ## Supported computer vision tasks - **Facial Landmark Detection** ([Readme](https://github.com/Giskard-AI/giskard-vision/blob/main/giskard_vision/landmark_detection/Readme.md)) -- **Image Classification** 🔜 Coming soon! +- **Image Classification** ([Readme](https://github.com/Giskard-AI/giskard-vision/blob/main/giskard_vision/image_classification/Readme.md)) +- **Object Detection** ([Readme](https://github.com/Giskard-AI/giskard-vision/blob/main/giskard_vision/object_detection/Readme.md)) ## Install From f8fd37771b713254055800e29e05513d15cb9084 Mon Sep 17 00:00:00 2001 From: Weixuan Xiao Date: Thu, 15 Aug 2024 18:25:59 +0200 Subject: [PATCH 3/6] Add readme for image classification and object detection --- giskard_vision/image_classification/Readme.md | 20 +++++++++++++++++++ giskard_vision/object_detection/Readme.md | 19 ++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 giskard_vision/image_classification/Readme.md create mode 100644 giskard_vision/object_detection/Readme.md diff --git a/giskard_vision/image_classification/Readme.md b/giskard_vision/image_classification/Readme.md new file mode 100644 index 00000000..c7e7a442 --- /dev/null +++ b/giskard_vision/image_classification/Readme.md @@ -0,0 +1,20 @@ +# `image_classification` module + +This module contains model wrappers, dataloaders, tests and all the ingredients needed to evaluate your single-label image classification models. +In particular this module allows you to evaluate your model against the following criteria: + +- Performance on images with different basic image attributes. +- Performance on images with various metadata from the datasets. +- Robustness against image perturbations like blurring, resizing, recoloring (performed by `opencv`: https://github.com/opencv/opencv) + +## Wrapped Datasets + +- [geirhos_conflict_stimuli](https://www.tensorflow.org/datasets/catalog/geirhos_conflict_stimuli) through Tensorflow Datasets +- [CIFAR100](https://huggingface.co/datasets/uoft-cs/cifar100) through Hugging Face +- [Skin cancer](https://huggingface.co/datasets/marmal88/skin_cancer) through Hugging Face + + +## Supported Classification + +- [x] Multiclass and single label +- [ ] Multiclass and multi-label diff --git a/giskard_vision/object_detection/Readme.md b/giskard_vision/object_detection/Readme.md new file mode 100644 index 00000000..698e4d7a --- /dev/null +++ b/giskard_vision/object_detection/Readme.md @@ -0,0 +1,19 @@ +# `object_detection` module + +This module contains model wrappers, dataloaders, tests and all the ingredients needed to evaluate your object detection models. +In particular this module allows you to evaluate your model against the following criteria: + +- Performance on images with different basic image attributes. +- Performance on images with various metadata from the datasets. +- Robustness against image perturbations like blurring, resizing, recoloring (performed by `opencv`: https://github.com/opencv/opencv) + +## Wrapped Datasets + +- [Racoon](https://www.kaggle.com/datasets/debasisdotcom/racoon-detection/data) +- [300W](https://ibug.doc.ic.ac.uk/resources/300-W/), using the boundary box around all face landmarks +- [ffhq](https://github.com/DCGM/ffhq-features-dataset), using the boundary box around all face landmarks +- [Living room passes](https://huggingface.co/datasets/Nfiniteai/living-room-passes) through Hugging Face + +## Metric + +- [x] Intersection over Union (IoU) From 3c9780b82aea0d47bdb77a1b3b59a6f680720c7b Mon Sep 17 00:00:00 2001 From: Weixuan Xiao Date: Fri, 16 Aug 2024 18:37:54 +0200 Subject: [PATCH 4/6] Clarify the metric and the scan of supported sub-category in object detection --- giskard_vision/object_detection/Readme.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/giskard_vision/object_detection/Readme.md b/giskard_vision/object_detection/Readme.md index 698e4d7a..ffa5b356 100644 --- a/giskard_vision/object_detection/Readme.md +++ b/giskard_vision/object_detection/Readme.md @@ -14,6 +14,18 @@ In particular this module allows you to evaluate your model against the followin - [ffhq](https://github.com/DCGM/ffhq-features-dataset), using the boundary box around all face landmarks - [Living room passes](https://huggingface.co/datasets/Nfiniteai/living-room-passes) through Hugging Face -## Metric +## Scan and Metrics + +Once the model and dataloader (`dl`) are wrapped, you can scan the model with the scan API in Giskard vision core: + +```python +from giskard_vision.core.scanner import scan + +results = scan(model, dl) +``` + +It adapts the [scan API in Giskard Python library](https://github.com/Giskard-AI/giskard#2--scan-your-model-for-issues) to magically scan the vision model with the dataloader. The considered metric is: - [x] Intersection over Union (IoU) + +Currently, we only support one object both in the model prediction and ground truth, due to the constraint of the scan API. We will be working to remove such limit. From 9ba1e361f7df949f93087d61bcc079103363cbbf Mon Sep 17 00:00:00 2001 From: Weixuan Xiao Date: Fri, 16 Aug 2024 18:39:54 +0200 Subject: [PATCH 5/6] Add the scan code in landmark detection --- giskard_vision/landmark_detection/Readme.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/giskard_vision/landmark_detection/Readme.md b/giskard_vision/landmark_detection/Readme.md index ca296b9e..49eaea11 100644 --- a/giskard_vision/landmark_detection/Readme.md +++ b/giskard_vision/landmark_detection/Readme.md @@ -22,7 +22,17 @@ You can also check our publicly hosted versions of these datasets on S3: - https://poc-face-aligment.s3.eu-north-1.amazonaws.com/300W/300w.zip.004 - FFHQ (only meta data): https://poc-face-aligment.s3.eu-north-1.amazonaws.com/ffhq/json.zip -## Metrics +## Scan and Metrics + +Once the model and dataloader (`dl`) are wrapped, you can scan the model with the scan API in Giskard vision core: + +```python +from giskard_vision.core.scanner import scan + +results = scan(model, dl) +``` + +It adapts the [scan API in Giskard Python library](https://github.com/Giskard-AI/giskard#2--scan-your-model-for-issues) to magically scan the vision model with the dataloader. The considered metrics are: - [x] ME: Mean Euclidean distances - [x] NME: Normalised Mean Euclidean distances From 9acb2c78eebee043699f2f7e7d6813b625d9487d Mon Sep 17 00:00:00 2001 From: Weixuan Xiao Date: Fri, 16 Aug 2024 18:45:04 +0200 Subject: [PATCH 6/6] Clarify the scan of supported sub-category in image classification --- giskard_vision/image_classification/Readme.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/giskard_vision/image_classification/Readme.md b/giskard_vision/image_classification/Readme.md index c7e7a442..965a1694 100644 --- a/giskard_vision/image_classification/Readme.md +++ b/giskard_vision/image_classification/Readme.md @@ -14,7 +14,21 @@ In particular this module allows you to evaluate your model against the followin - [Skin cancer](https://huggingface.co/datasets/marmal88/skin_cancer) through Hugging Face -## Supported Classification +## Scan and Supported Classification + +Once the model and dataloader (`dl`) are wrapped, you can scan the model with the scan API in Giskard vision core: + +```python +from giskard_vision.core.scanner import scan + +results = scan(model, dl) +``` + +It adapts the [scan API in Giskard Python library](https://github.com/Giskard-AI/giskard#2--scan-your-model-for-issues) to magically scan the vision model with the dataloader. + +Currently, due to the constraint of the scan API, we support a subset of image classification tasks: - [x] Multiclass and single label - [ ] Multiclass and multi-label + +We will be working to remove such limit for the scan.