Fix `open-clip-torch` model inference #5395

jacobsela · 2025-01-16T01:30:39Z

What changes are proposed in this pull request?

Fix open clip zoo model.

How is this patch tested? If it is not, please explain why.

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("coco-2017", split="validation", max_samples=10, shuffle=True)
model = foz.load_zoo_model("open-clip-torch")

dataset.apply_model(model, label_field="predictions")

Summary by CodeRabbit

Refactor
- Updated preprocessing method for OpenClip model.
- Simplified image preprocessing during prediction.

These changes enhance the efficiency of the OpenClip model's image processing capabilities, leading to improved performance.

coderabbitai · 2025-01-16T01:30:46Z

Walkthrough

The pull request modifies the TorchOpenClipModel class in the fiftyone/utils/open_clip.py file. The changes primarily focus on adjusting the preprocessing attribute and method, specifically replacing self.preprocess with self._preprocess_aux. Additionally, the image preprocessing logic in the _predict_all method has been simplified by removing the unsqueezing operation. These updates enhance the control flow during image processing within the model.

Changes

File	Change Summary
`fiftyone/utils/open_clip.py`	- Replaced `self.preprocess` with `self._preprocess_aux` - Simplified image preprocessing in `_predict_all` by removing `unsqueeze(0)` - Updated context management for tensor operations using `contextlib.ExitStack`

Possibly related PRs

Bugfix: use device in all Torch models #5026: This PR modifies the TorchOpenClipModel class in fiftyone/utils/open_clip.py, specifically enhancing tensor device management in the _predict_all method, which is directly related to the changes made in the main PR regarding the same method.

Suggested reviewers

mwoodson1
danielgural
harpreetsahota204

Poem

🐰 In the world of code, we hop and play,
Preprocessing images in a brand new way.
With a flick of the paw, we simplify,
No more unsqueezing, just let them fly!
OpenClip shines, with elegance and grace,
A rabbit's joy in this coding space! 🎉

✨ Finishing Touches

📝 Generate Docstrings (Beta)

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

fiftyone/utils/open_clip.py (1)
92-92: Consider a more descriptive name for the preprocessing method.

The name _preprocess_aux is not very descriptive. Consider renaming it to better reflect its purpose, such as _preprocess_image or _clip_preprocess.
-            self._preprocess_aux,
+            self._preprocess_image,

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 985758b and 01cc875.

📒 Files selected for processing (1)

fiftyone/utils/open_clip.py (3 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (6)

GitHub Check: test / test-python (ubuntu-latest-m, 3.10)
GitHub Check: test / test-app
GitHub Check: build / build
GitHub Check: lint / eslint
GitHub Check: e2e / test-e2e
GitHub Check: build

🔇 Additional comments (2)

fiftyone/utils/open_clip.py (2)

Line range hint 138-152: Add tests for preprocessing changes.

The modifications to the preprocessing workflow could affect model predictions. Please add tests to verify:

The shape of preprocessed tensors

Model predictions with the new preprocessing flow

Batch processing behavior

Would you like me to help create test cases for these scenarios?

138-138: Verify tensor shape requirements.

The removal of .unsqueeze(0) suggests that self._preprocess now handles the batch dimension differently. Please verify that the tensor shape matches what the model expects.

Let's check the OpenCLIP source code to understand the preprocessing behavior:

✅ Verification successful

Tensor shape handling is correct

The removal of .unsqueeze(0) is valid as the batch dimension is properly handled by torch.stack(imgs) after preprocessing. The implementation correctly maintains the expected tensor shape for the model's encode_image method.
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:
#!/bin/bash
# Description: Check OpenCLIP's preprocessing implementation
rg -A 10 "create_model_and_transforms" $(fd -t f -e py . | grep -i "clip")
Length of output: 917

Script:
#!/bin/bash
# Search for transform or preprocessing related code
rg -A 5 "def _preprocess" fiftyone/utils/open_clip.py
Length of output: 55

Script:
#!/bin/bash
# Check the class definition and imports
cat fiftyone/utils/open_clip.py
Length of output: 4905

fiftyone/utils/open_clip.py

jacobsela · 2025-01-16T01:38:57Z

rebased to bugfix/zoo-clip-support-for-multi-gpu-setups

coderabbitai

Actionable comments posted: 5

🧹 Nitpick comments (18)

docs/source/getting_started/upgrading-mongodb.rst (3)
5-6: Add specific backup instructions.

While the documentation correctly emphasizes the importance of backups, it should provide specific instructions or link to MongoDB's backup documentation for users to follow.

Add a link to MongoDB's backup documentation or include specific backup commands:
 Voxel51 advises performing database backups of your mongodb
-data directory whenever performing a database upgrade.
+data directory whenever performing a database upgrade. For instructions
+on backing up your MongoDB data, see the
+`MongoDB backup documentation <https://www.mongodb.com/docs/manual/core/backups/>`_.
39-43: Specify minimum version requirements.

The upgrade instructions should specify why these particular versions are required for MongoDB 8 compatibility.

Add a note explaining the version requirements:
 2. Upgrade Fiftyone to 1.3.0+ and Fiftyone-db to 1.2.0+
+
+    .. note::
+
+        These versions are required as they include support for MongoDB 8
+        and necessary database migration tools.

     .. code-block:: shell
69-69: Add example MongoDB URI format.

The documentation should include an example of what the MongoDB URI looks like to help users identify it correctly.

Add an example URI format:
-        mongosh "$URI_FROM_ABOVE"
+        # Example URI format: mongodb://localhost:27017
+        mongosh "$URI_FROM_ABOVE"
docs/source/getting_started/install.rst (1)
211-216: Enhance the Ubuntu 24 upgrade note with more context.

The note should explain why Ubuntu 24 users specifically need to upgrade their MongoDB binaries.

Add more context to the note:
 .. note::

   If you are a FiftyOne 1.2.0 or lower user with an Ubuntu 24 operating system, 
   you will need to 
-  :ref:`upgrade your mongodb binaries <mongodb-7-to-8>`.
+  :ref:`upgrade your mongodb binaries <mongodb-7-to-8>` due to Ubuntu 24's
+  requirement for MongoDB 8.0 or higher.
fiftyone/utils/iou.py (2)
698-701: Simplify initialization of ious using a ternary operator

You can simplify the initialization of the ious variable by using a ternary operator, which makes the code more concise.

Apply this diff:
-    if sparse:
-        ious = defaultdict(list)
-    else:
-        ious = np.zeros((len(preds), len(gts)))
+    ious = defaultdict(list) if sparse else np.zeros((len(preds), len(gts)))
🧰 Tools

🪛 Ruff (0.8.2)

698-701: Use ternary operator ious = defaultdict(list) if sparse else np.zeros((len(preds), len(gts))) instead of if-else-block

Replace if-else-block with ious = defaultdict(list) if sparse else np.zeros((len(preds), len(gts)))

(SIM108)

709-709: Remove unused variable bbox_iou_fcn

The local variable bbox_iou_fcn is assigned but never used in the function _compute_dense_mask_ious. It's recommended to remove this unused variable to clean up the code.

Apply this diff:
-    bbox_iou_fcn = compute_bbox_iou
🧰 Tools

🪛 Ruff (0.8.2)

709-709: Local variable bbox_iou_fcn is assigned to but never used

Remove assignment to unused variable bbox_iou_fcn

(F841)
fiftyone/core/odm/embedded_document.py (1)
75-76: Simplify nested if statements

You can combine the nested if statements at lines 75-76 into a single condition for improved code readability.

Apply this diff:
-        if key is not None:
-            if isinstance(getattr(self.__class__, key, None), property):
+        if key is not None and isinstance(getattr(self.__class__, key, None), property):
🧰 Tools

🪛 Ruff (0.8.2)

75-76: Use a single if statement instead of nested if statements

(SIM102)
app/packages/core/src/plugins/SchemaIO/components/AutocompleteView.tsx (2)
130-136: Use strict equality operators

In line 130, consider using strict equality operators === and !== to avoid unintended type coercion.

Apply this diff:
-  const valuesOnlyIsDefault = valuesOnly == undefined || valuesOnly == null;
+  const valuesOnlyIsDefault = valuesOnly === undefined || valuesOnly === null;
Alternatively, since valuesOnly == null checks for both null and undefined, you could simplify the condition:
-  const valuesOnlyIsDefault = valuesOnly == undefined || valuesOnly == null;
+  const valuesOnlyIsDefault = valuesOnly == null;
77-77: Use strict equality operator in comparison

In isOptionEqualToValue, consider using === instead of == for a strict equality check between option and value.

Apply this diff:
-  return option == value;
+  return option === value;
tests/unittests/documents.py (2)

77-97: Consider adding more test cases for reserved attributes.

While the test covers the basic scenarios, consider adding test cases for:

Other reserved attributes besides has_mask

Attempting to modify multiple reserved attributes simultaneously

98-114: Add test cases for malformed exception messages.

The test covers standard patterns but should also verify behavior with:

Empty exception messages

Messages without quotes

Messages with multiple property patterns
app/packages/core/src/plugins/SchemaIO/components/DynamicIO.tsx (1)
39-44: LGTM! Enhanced schema fallback mechanism.

The additional fallback to computedSchema provides better robustness. Consider extracting the schema resolution logic to a separate function for better readability:
-      onChange(
-        path,
-        value,
-        schema ?? subSchema ?? computedSchema,
-        computedAncestors
-      );
+      const resolvedSchema = schema ?? subSchema ?? computedSchema;
+      onChange(path, value, resolvedSchema, computedAncestors);
app/packages/core/src/plugins/SchemaIO/components/DropdownView.tsx (1)
10-10: Consider using named import for better tree-shaking.
-import { AutocompleteView } from ".";
+import { AutocompleteView } from "./AutocompleteView";
fiftyone/core/odm/mixins.py (1)
1872-1878: LGTM! Clean implementation of nested path filtering.

The function effectively prevents operations on nested paths, enhancing the robustness of field operations.

Consider this minor optimization for better performance with large path lists:
def _remove_nested_paths(paths):
-    return [
-        path
-        for path in paths
-        if not any(path.startswith(p + ".") for p in paths)
-    ]
+    paths = sorted(paths, key=len)  # Sort by length to check shorter paths first
+    result = []
+    for path in paths:
+        if not any(path.startswith(p + ".") for p in result):
+            result.append(path)
+    return result
plugins/operators/__init__.py (2)
1788-1790: Track and fix the App bug preventing view setting.

The TODO comment indicates a known App bug that prevents setting the view after saving. Consider creating a tracking issue for this bug.

Would you like me to create a GitHub issue to track this App bug?

647-647: Optimize dictionary key iteration.

Replace key in dict.keys() with key in dict for better performance in these locations:
-for key in schema.keys():
+for key in schema:
Also applies to: 755-755, 907-907, 982-982

🧰 Tools

🪛 Ruff (0.8.2)

647-647: Use key in dict instead of key in dict.keys()

Remove .keys()

(SIM118)
fiftyone/operators/types.py (1)

1929-1929: Consider documenting the clone behavior.

Instead of suppressing the pylint warning, consider adding a docstring to the clone method explaining that it creates the tooltips attribute. This would make the code more maintainable and self-documenting.
fiftyone/core/dataset.py (1)
1939-1940: Consider using list comprehension for better readability.

The loop can be simplified using a more Pythonic list comprehension.
-        for list_field in list_fields:
-            pipeline.append({"$unwind": "$" + list_field})
+        pipeline.extend({"$unwind": "$" + list_field} for list_field in list_fields)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 01cc875 and c0dd032.

⛔ Files ignored due to path filters (4)

app/packages/app/src/pages/datasets/__generated__/DatasetPageQuery.graphql.ts is excluded by !**/__generated__/**, !**/__generated__/**
app/packages/relay/src/fragments/__generated__/datasetAppConfigFragment.graphql.ts is excluded by !**/__generated__/**, !**/__generated__/**
app/packages/relay/src/queries/__generated__/datasetQuery.graphql.ts is excluded by !**/__generated__/**, !**/__generated__/**
fiftyone/zoo/models/manifest-torch.json is excluded by !**/*.json

📒 Files selected for processing (42)

app/packages/core/src/components/Modal/ImaVidLooker.tsx (2 hunks)
app/packages/core/src/plugins/OperatorIO/utils/index.ts (1 hunks)
app/packages/core/src/plugins/SchemaIO/components/AutocompleteView.tsx (4 hunks)
app/packages/core/src/plugins/SchemaIO/components/DropdownView.tsx (2 hunks)
app/packages/core/src/plugins/SchemaIO/components/DynamicIO.tsx (1 hunks)
app/packages/core/src/plugins/SchemaIO/components/NativeModelEvaluationView/Evaluation.tsx (3 hunks)
app/packages/core/src/plugins/SchemaIO/index.tsx (2 hunks)
app/packages/looker/src/elements/imavid/index.ts (4 hunks)
app/packages/looker/src/lookers/imavid/constants.ts (0 hunks)
app/packages/looker/src/lookers/imavid/controller.ts (6 hunks)
app/packages/looker/src/util.ts (1 hunks)
app/packages/operators/src/components/OperatorExecutionButton/index.tsx (1 hunks)
app/packages/relay/src/fragments/datasetAppConfigFragment.ts (1 hunks)
app/packages/state/src/hooks/useCreateLooker.ts (4 hunks)
app/packages/state/src/recoil/selectors.ts (1 hunks)
app/packages/state/src/recoil/types.ts (1 hunks)
app/schema.graphql (1 hunks)
docs/source/getting_started/install.rst (1 hunks)
docs/source/getting_started/upgrading-mongodb.rst (1 hunks)
docs/source/release-notes.rst (1 hunks)
docs/source/user_guide/evaluation.rst (2 hunks)
fiftyone/constants.py (1 hunks)
fiftyone/core/dataset.py (1 hunks)
fiftyone/core/odm/__init__.py (1 hunks)
fiftyone/core/odm/dataset.py (2 hunks)
fiftyone/core/odm/embedded_document.py (3 hunks)
fiftyone/core/odm/mixins.py (4 hunks)
fiftyone/operators/types.py (2 hunks)
fiftyone/server/query.py (1 hunks)
fiftyone/utils/clip/zoo.py (1 hunks)
fiftyone/utils/eval/coco.py (1 hunks)
fiftyone/utils/eval/openimages.py (1 hunks)
fiftyone/utils/iou.py (9 hunks)
fiftyone/utils/open_clip.py (6 hunks)
fiftyone/utils/super_gradients.py (1 hunks)
fiftyone/utils/transformers.py (14 hunks)
fiftyone/utils/ultralytics.py (3 hunks)
package/graphql/setup.py (1 hunks)
plugins/operators/__init__.py (20 hunks)
requirements/common.txt (1 hunks)
setup.py (1 hunks)
tests/unittests/documents.py (2 hunks)

🔥 Files not summarized due to errors (1)

fiftyone/utils/clip/zoo.py: Error: Disallowed special token found: <|endoftext|>

💤 Files with no reviewable changes (1)

app/packages/looker/src/lookers/imavid/constants.ts

✅ Files skipped from review due to trivial changes (6)

requirements/common.txt
app/packages/core/src/plugins/OperatorIO/utils/index.ts
setup.py
package/graphql/setup.py
fiftyone/utils/eval/openimages.py
fiftyone/utils/eval/coco.py

🚧 Files skipped from review as they are similar to previous changes (1)

fiftyone/utils/open_clip.py

🧰 Additional context used

📓 Path-based instructions (14)

app/packages/relay/src/fragments/datasetAppConfigFragment.ts (1)