Fixing keras.ops.image.map_coordinates Failure with uint8 Input on TensorFlow Backend"

[NebulaNerdo]

Consider the following simple example

`import keras

image = keras.ops.ones((1, 1, 3), dtype='uint8')

coordinates = keras.ops.convert_to_tensor([-1., 0., 0.])[..., None, None]
interp = keras.ops.image.map_coordinates(image, coordinates, order=1, fill_mode='constant')`

that is expected to yield [[0]]. However, with KERAS_BACKEND=tensorflow this code snippet results in

2024-12-08 16:04:24.790791: W tensorflow/core/framework/op_kernel.cc:1841] OP_REQUIRES failed at gather_nd_op.cc:65 : INVALID_ARGUMENT: indices[0,0] = [-1, 0, 0] does not index into param shape [1,1,3], node name: GatherNd 2024-12-08 16:04:24.790814: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: INVALID_ARGUMENT: indices[0,0] = [-1, 0, 0] does not index into param shape [1,1,3], node name: GatherNd Traceback (most recent call last): File "<home>/tfmapc.py", line 11, in <module> interp = keras.ops.image.map_coordinates(image, coordinates, order=1, fill_mode='constant') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<home>/.env/lib/python3.12/site-packages/keras/src/ops/image.py", line 787, in map_coordinates return backend.image.map_coordinates( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<home>/.env/lib/python3.12/site-packages/keras/src/backend/tensorflow/image.py", line 485, in map_coordinates contribution = tf.cond(tf.reduce_all(validities), fast_path, slow_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<home>/.env/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "<home>/.env/lib/python3.12/site-packages/keras/src/backend/tensorflow/image.py", line 481, in slow_path tf.transpose(tf.gather_nd(input_arr, indices)), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__GatherNd_device_/job:localhost/replica:0/task:0/device:CPU:0}} indices[0,0] = [-1, 0, 0] does not index into param shape [1,1,3], node name: GatherNd [Op:GatherNd] name:

The problem does not occur if I change the dtype of image from uint8 to float32 or switch either to the jax or torch backends. Also changing the fill_mode from constant to nearest avoids the issue.

Keras version: 3.7.0

[sergiud]

Why do you plagiarize #20608?

[Mr-back007]

1. Convert the image to float32 before applying the operation:
   Operations like map_coordinates expect floating point inputs for precise interpolation. Since uint8 is an 8-bit unsigned integer type, the function can't operate correctly without the conversion.

Here's how to fix it:
`import tensorflow as tf
from tensorflow.keras.preprocessing import image

Example: Load an image (ensure it is uint8)

img = tf.image.decode_image(tf.io.read_file('image_path.jpg'))

Convert the image to float32 before applying map_coordinates

img_float = tf.cast(img, dtype=tf.float32)

Apply map_coordinates after conversion

from tensorflow.image import map_coordinates
output_image = map_coordinates(img_float, coordinates)
2. Make sure to scale the image properly: Depending on the operation you're applying, float32 images may need to be scaled between 0 and 1 (or possibly -1 and 1, depending on the model you are using). If you're dealing with images that have pixel values between 0 and 255, you might want to normalize the image after casting to float32.img_normalized = img_float / 255.0
`
3. Check if map_coordinates is being used properly:
Ensure that you're passing the correct shape and format of the coordinates to map_coordinates. Typically, the coordinates should have a shape of (n, 2) where n is the number of coordinates (pixels) you want to map.

Example:
`# Coordinates should be passed in the format of [y, x] (row, col)
coordinates = tf.constant([[50, 50], [100, 150]]) # Example coordinates

Apply map_coordinates on the normalized image

output_image = tf.image.map_coordinates(img_normalized, coordinates)
`
4. Debugging the issue:
If the issue persists, consider checking:

The TensorFlow version you are using (some older versions might have different behavior or bugs).
Ensure that the input image and the coordinates are correct.
Full Example:
`import tensorflow as tf
from tensorflow.keras.preprocessing import image

Load an image (ensure it is uint8)

img = tf.image.decode_image(tf.io.read_file('image_path.jpg'))

Convert the image to float32 before applying map_coordinates

img_float = tf.cast(img, dtype=tf.float32)

Normalize the image (if needed)

img_normalized = img_float / 255.0

Example coordinates (row, col) in the form of y, x

coordinates = tf.constant([[50, 50], [100, 150]]) # Example coordinates

Apply map_coordinates after conversion and scaling

from tensorflow.image import map_coordinates
output_image = tf.image.map_coordinates(img_normalized, coordinates)
`