ValueError: Dimension 2 in both shapes must be equal, but are 3 and 64 for 'Assign' (op: 'Assign') with input shapes: [3,3,3,64], [3,3,64,3].

[wangtong627]

i got this error while doing the testing,

Traceback (most recent call last):
File "main.py", line 57, in
model.upscale(p, save_intermediate=save, mode=mode, patch_size=patch_size, suffix=suffix)
File "C:\Users\Administrator\Desktop\Image-Super-Resolution-master\models.py", line 202, in upscale
model = self.create_model(img_dim_2, img_dim_1, load_weights=True)
File "C:\Users\Administrator\Desktop\Image-Super-Resolution-master\models.py", line 601, in create_model
if load_weights: model.load_weights(self.weight_path)
File "E:\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2656, in load_weights
f, self.layers, reshape=reshape)
File "E:\Anaconda3\lib\site-packages\keras\engine\topology.py", line 3382, in load_weights_from_hdf5_group
K.batch_set_value(weight_value_tuples)
File "E:\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2368, in batch_set_value
assign_op = x.assign(assign_placeholder)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 516, in assign
return state_ops.assign(self._variable, value, use_locking=use_locking)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\ops\state_ops.py", line 271, in assign
validate_shape=validate_shape)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 45, in assign
use_locking=use_locking, name=name)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
op_def=op_def)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2508, in create_op
set_shapes_for_outputs(ret)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1873, in set_shapes_for_outputs
shapes = shape_func(op)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1823, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 610, in call_cpp_shape_fn
debug_python_shape_fn, require_shape_fn)
File "E:\Anaconda3\lib\site-packages\tensorflow\python\framework\common_shapes.py", line 676, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Dimension 2 in both shapes must be equal, but are 3 and 64 for 'Assign' (op: 'Assign') with input shapes: [3,3,3,64], [3,3,64,3].

[ghost]

I have this too, in Python 2. Ubuntu 16.04, Tensorflow on SYCL.
Using TensorFlow backend.
Old Size : (200, 320, 3)
New Size : (400, 640, 3)
/usr/lib/python2.7/dist-packages/scipy/misc/pilutil.py:479: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it will be treated as np.int64 == np.dtype(int).type.
if issubdtype(ts, int):
/usr/lib/python2.7/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
elif issubdtype(type(size), float):
Number of patches = 248769, Patch Shape = (8, 8)
Saving intermediate image.
Traceback (most recent call last):
File "main.py", line 57, in
model.upscale(p, save_intermediate=save, mode=mode, patch_size=patch_size, suffix=suffix)
File "/home/stuntj/src/Image-Super-Resolution/models.py", line 202, in upscale
model = self.create_model(img_dim_2, img_dim_1, load_weights=True)
File "/home/stuntj/src/Image-Super-Resolution/models.py", line 601, in create_model
if load_weights: model.load_weights(self.weight_path)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2656, in load_weights
f, self.layers, reshape=reshape)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 3382, in load_weights_from_hdf5_group
K.batch_set_value(weight_value_tuples)
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2368, in batch_set_value
assign_op = x.assign(assign_placeholder)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 599, in assign
return state_ops.assign(self._variable, value, use_locking=use_locking)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 280, in assign
validate_shape=validate_shape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3263, in create_op
compute_device=compute_device)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3303, in _create_op_helper
set_shapes_for_outputs(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2494, in set_shapes_for_outputs
return _set_shapes_for_outputs(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2467, in _set_shapes_for_outputs
shapes = shape_func(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2397, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn
require_shape_fn)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Dimension 2 in both shapes must be equal, but are 3 and 64. Shapes are [3,3,3,64] and [3,3,64,3]. for 'Assign' (op: 'Assign') with input shapes: [3,3,3,64], [3,3,64,3].

[ghost]

"image_data_format": "channels_first" in .keras/keras.json does it for me.

[jaretburkett]

I will probably submit a pull request on this, although the dev could do it a lot easier than I could (fork, pull, edit, push, submit request, wait, etc), but a simple fix is to add this to the top of the main.py file

import keras
keras.backend.set_image_data_format('channels_first')

That way users don't need to adjust their keras globals that could screw other things up. It would end up looking like this.

import models
import argparse
import tensorflow as tf
# add
import keras
keras.backend.set_image_data_format('channels_first')
# end

parser = argparse.ArgumentParser(description="Up-Scales an image using Image Super Resolution Model")
# etc ...........

It would probably also need to be added to tests.py or just the models.py. Anywhere will work as long as it is called before and keras operations.

[titu1994]

You're right that it's an easy fix, but it's temporary. I will have to look into it closely to find all the places where the data format can cause this issue to fix it, and I currently do not have the time to do that.

[titu1994]

@jaretburkett since you have suggested the fix, it would be correct for you to add the PR. However if you don't want that, then I'll take care of it.

[titu1994]

This issue is due to the fact that when I first wrote this repo, I was using Theano with channels_first data format. However, recent updates were made to use Tensorflow backend instead.

The weights are from Keras 1.x, so they dont automatically adjust for the data format. I would suggest training your own models, or use rnsr or distilled_rnsr models, since they were trained on TF backend.

[jaretburkett]

I was going to write the fix and do a pull request, but thinking about it, if some models are trained with the Theano backend and some with the Tensor Flow backend, then it might be a little more involved on updating the models. This snippet would work for people specifically trying to run a Theano trained model on Tensorflow, but would break others trying to run a model trained on Tensorflow. Since all the models extend BaseSuperResolutionModel (from what I can tell), and it has code to correct the channel first on training and when before loading the model, so it would have to be done on the weights themself or on a per model basis. I looked up how one would accomplish this and it looks like @titu1994 also wrote a script to do this. It was the first thing I found. weight_convertion_tensorflow.py

It looks like you can make reshape=True when loading weights, and based on the keras source, it looks like it checks the backend version of the weights file and will reshape the ordering on load. I'll test it tonight if I have time and do a pull request if it does not present any issues

[titu1994]

@jaretburkett Thanks for the thorough reply. I'll look into the reshape flag in weight loading, but I think it's more for weights loaded in Keras 2.x. However, I'm happy to be proven wrong here.

As to the BaseSuperResolutionModel, the weight loading can be done in a hack way with a try catch. As in, use the default data format and attempt to load the weights, if it throws the above exception, try with the other data format. If that throws an error as well, then simply propogate that error to the user.

[jaretburkett]

I tried with the reshape parameter set to true, and it seems to work great. I set a batch test to test all of the models and it worked without issue on GPU with the latest keras and Tensorflow, But I did have an issue on DistilledResNetSR when running on the cpu when saving the images, but it was not consistent. It might have something to do with the by_name parameter being there. Threw an error sometimes, other times it didn't. I will try to test more tomorrow, but as of now, adding reshape=True seems to automatically adjust the weights for the shape of the layers.

if load_weights: model.load_weights(self.weight_path, reshape=True)