Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imresize spuriously causes NaNs with 4d image #763

Open
staticfloat opened this issue Oct 29, 2018 · 11 comments
Open

imresize spuriously causes NaNs with 4d image #763

staticfloat opened this issue Oct 29, 2018 · 11 comments
Labels
Milestone

Comments

@staticfloat
Copy link
Collaborator

With a 4-dimensional array:

m = randn(250,202,3,1)
imresize(m, (277, 224, 3, 1))

Running the imresize() multiple times shows NaN's randomly appearing. If I eliminate the fourth dimension, this works just fine:

imresize(m[:,:,:,1], (277, 224, 3))

I assume this is because imresize() is attempting to interpolate across the 3rd axis, and it's reaching out of bounds or something like that. If that is the case, it would be nice if I had a way to specify the dimensions across which I wanted to resize (which I am attempting to do by passing in an explicit tuple of sizes).

@staticfloat
Copy link
Collaborator Author

I should note the "4d image" I have is actually a tensor representing a 250 x 202 image with 3 channels, and there is only 1 within this current "batch".

@johnnychen94
Copy link
Member

johnnychen94 commented Oct 29, 2018

For more information, I tried this in two of my computers with

any(imresize(m, (277, 224, 3, 1)) .== NaN)

For MacOS I didn't observe NaN, i.e., get result as false

Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)

Interpolation Version: v0.10.5

For Linux: I get Segmentation fault on my machine (imresize(m[:,:,:,1], (277, 224, 3)) won't)

Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

Interpolation Version: v0.10.5
signal (11): Segmentation fault
in expression starting at no file:0
getindex at ./array.jl:732 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:290 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
macro expansion at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:274 [inlined]
interp_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:265 [inlined]
_getindex at /home/jc/.julia/packages/Interpolations/gioj0/src/Interpolations.jl:230 [inlined]
getindex at ./abstractarray.jl:905 [inlined]
BSplineInterpolation at /home/jc/.julia/packages/Interpolations/gioj0/src/b-splines/indexing.jl:8 [inlined]
imresize! at /home/jc/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:265
imresize! at /home/jc/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:240 [inlined]
imresize at /home/jc/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:214
unknown function (ip: 0x7f43aeb36f36)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1829
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:428
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f43a3e41c7f)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:787
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:117 [inlined]
#28 at ./task.jl:259
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2182
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1536 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:268
unknown function (ip: 0xffffffffffffffff)
Allocations: 40634301 (Pool: 40616850; Big: 17451); GC: 92
fish: “julia” terminated by signal SIGSEGV (Address boundary error)

@zygmuntszpak
Copy link
Member

With

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0 (2018-08-08 06:46 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

after a single execution of

m = randn(250,202,3,1)
imresize(m, (277, 224, 3, 1))

I get the following on Windows

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x15c43239 -- * at .\float.jl:399 [inlined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:230 [inlined]
getindex at .\abstractarray.jl:913 [inlined]
BSplineInterpolation at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\b-splines\indexing.j
l:8 [inlined]
imresize! at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:265
in expression starting at C:\Users\Spock\.julia\dev\Images\src\debug.jl:111
getindex at .\array.jl:732 [inlined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:290 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
macro expansion at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:274 [in
lined]
interp_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:265 [in
lined]
_getindex at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\Interpolations.jl:230 [inlined]
getindex at .\abstractarray.jl:913 [inlined]
BSplineInterpolation at C:\Users\Spock\.julia\packages\Interpolations\gioj0\src\b-splines\indexing.j
l:8 [inlined]
imresize! at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:265
imresize! at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:240 [inlined]
imresize at C:\Users\Spock\.julia\packages\ImageTransformations\CGe4u\src\resizing.jl:214
unknown function (ip: 0000000015C41BA5)
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
do_call at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:428
eval_stmt_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:363 [in
lined]
eval_body at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win64/build/src\
interpreter.c:799
unknown function (ip: FFFFFFFFFFFFFFFE)
unknown function (ip: 0000000012553C5F)
unknown function (ip: 0000000000000000)
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:831
jl_parse_eval_all at /home/Administrator/buildbot/worker/package_win64/build/src\ast.c:841
include_string at .\loading.jl:1002
include_string at C:\Users\Spock\.julia\packages\CodeTools\hB4Hy\src\eval.jl:30
unknown function (ip: 0000000015C0879A)
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
#117 at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:94
withpath at C:\Users\Spock\.julia\packages\CodeTools\hB4Hy\src\utils.jl:30
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1538 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
withpath at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:46 [inlined]
#116 at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:93 [inlined]
with_logstate at .\logging.jl:397
with_logger at .\logging.jl:493 [inlined]
#115 at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:92 [inlined]
hideprompt at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\repl.jl:85
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
macro expansion at C:\Users\Spock\.julia\packages\Atom\7rQ1O\src\eval.jl:91 [inlined]
#114 at .\task.jl:85
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1538 [inlined]
start_task at /home/Administrator/buildbot/worker/package_win64/build/src\task.c:268
Allocations: 42834304 (Pool: 42827894; Big: 6410); GC: 90

I'm not actually sure how to check which version of Interpolations I have installed, because ] st does not show it.

@johnnychen94
Copy link
Member

@zygmuntszpak It's listed in ~/.julia/environments/v1.0/Manifest.toml

@timholy
Copy link
Member

timholy commented Oct 31, 2018

Yikes!

If you launch julia with --check-bounds=yes, you get this:

julia> imresize(m, (277, 224, 3, 1))
ERROR: BoundsError: attempt to access 250×202×3×1 Array{Float64,4} at index [2, 2, 2, 0]
Stacktrace:
 [1] getindex at ./array.jl:732 [inlined]
 [2] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:290 [inlined]
 [3] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [4] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [5] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [6] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [7] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [8] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [9] macro expansion at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:274 [inlined]
 [10] interp_getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:265 [inlined]
 [11] _getindex at /home/tim/.julia/dev/Interpolations/src/Interpolations.jl:230 [inlined]
 [12] getindex at ./abstractarray.jl:905 [inlined]
 [13] BSplineInterpolation at /home/tim/.julia/dev/Interpolations/src/b-splines/indexing.jl:8 [inlined]
 [14] imresize!(::Array{Float64,4}, ::Interpolations.BSplineInterpolation{Float64,4,Array{Float64,4},Interpolations.BSpline{Interpolations.Linear},NTuple{4,Base.OneTo{Int64}}}) at /home/tim/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:265
 [15] imresize! at /home/tim/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:240 [inlined]
 [16] imresize(::Array{Float64,4}, ::NTuple{4,Int64}) at /home/tim/.julia/packages/ImageTransformations/CGe4u/src/resizing.jl:214
 [17] top-level scope at none:0

I haven't had time to look into whether it would be better to fix this in Interpolations or in ImageTransformations (or both). Presumably throwing an error when any dimension has size 1 would be undesirable, right?

@staticfloat
Copy link
Collaborator Author

If I ask imresize() to go from a size of (A, B, C, D) -> (A/2, B/2, C, D), I feel like it shouldn't matter what C and D are; if they're the same then imresize() should just be a no-op along those axes. (I realize that may not be how imresize() is implemented, but that would be nice, IMO)

@timholy
Copy link
Member

timholy commented Nov 3, 2018

I'm a bit short on time, but if anyone wants to tackle this the fix here (or rather, in ImageTransformations) is to use NoInterp for those dimensions.

@johnnychen94 johnnychen94 added this to the v1.0 milestone Jan 1, 2020
@johnnychen94
Copy link
Member

johnnychen94 commented Jan 1, 2020

It now throws a more informative error, but still, this needs to be fixed in ImageTransformation.jl

julia> m = randn(250,202,3,1);
julia> imresize(m, (277, 224, 3, 1))
ERROR: ArgumentError: size (250, 202, 3, 1) is inconsistent with BSpline(Interpolations.Linear()), use NoInterp along singleton dimensions
Stacktrace:
 [1] (::Interpolations.var"#err_singleton#11")(::Array{Float64,4}, ::Interpolations.BSpline{Interpolations.Linear}) at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:58
 [2] BSplineInterpolation at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:65 [inlined]
 [3] interpolate(::Type{Float64}, ::Type{Float64}, ::Array{Float64,4}, ::Interpolations.BSpline{Interpolations.Linear}) at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:133
 [4] interpolate at /home/jc/.julia/packages/Interpolations/0zZ6m/src/b-splines/b-splines.jl:152 [inlined]
 [5] imresize! at /home/jc/.julia/packages/ImageTransformations/7wC0C/src/resizing.jl:256 [inlined]
 [6] imresize(::Array{Float64,4}, ::NTuple{4,Int64}) at /home/jc/.julia/packages/ImageTransformations/7wC0C/src/resizing.jl:231
 [7] top-level scope at REPL[3]:1

@scimas
Copy link

scimas commented Apr 30, 2020

Is there any workaround for this?

@scimas
Copy link

scimas commented Apr 30, 2020

I was looking for functionality similar to PyTorch's nn.Upsample to use with Flux and although this isn't a complete replication of it, it works for my purposes fails to work with zygote.

using Interpolations

struct Upsampler
    ratio::Tuple{<:Real,<:Real}
end

function (up::Upsampler)(x)
    itp = interpolate(x, (BSpline(Linear()), BSpline(Linear()), NoInterp(), NoInterp()))
    ws = (size(x, 1) - 1) / (size(x, 1) * up.ratio[1] - 1)
    hs = (size(x, 2) - 1) / (size(x, 2) * up.ratio[2] - 1)
    itp(1:ws:size(x, 1), 1:hs:size(x, 2), 1:size(x, 3), 1:size(x, 4))
end

function Upsampler(r1, r2)
    r1 >= 1 || throw(DomainError(r1, "ratio must be >= 1"))
    r2 >= 1 || throw(DomainError(r2, "ratio must be >= 1"))
    Upsampler((r1, r2))
end

@functor Upsampler

up = Upsampler(2, 1.5);
x = rand(Float32, 4, 4, 3, 1);
out = up(x);
@assert size(out) == (8, 6, 3, 1)

It assumes the input is a 4d array with size (Width, Height, Channels, Batchsize), as expected by Flux, and performs linear interpolation.

In case someone stumbles upon this issue with a similar use case. Also let me know if any other better method is possible.

@johnnychen94
Copy link
Member

@scimas probably you want to subscribe FluxML/Flux.jl#1136

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants