LBFGSB returns parameters outside of bounds

[javier-garcia-tilburg]

The algorithm jaxopt.LBFGSB sometimes returns parameters outside of the specified bounds.

solver = LBFGSB(fun=objective_fn, jit=True, tol=1e-3,)

n_params = 41

res = solver.run(
    init_params=jnp.repeat(0.01, n_params),
    bounds=(jnp.repeat(0, n_params), jnp.repeat(1, n_params)),
    ...
)

print(res.params)

if any(res.params > 1):
    raise Exception("Parameters outside of bounds")

If you are interested in a minimal example to reproduce the bug, I can share it in private. Just tell me where to upload or send the file (~250 lines of code).

[mblondel]

CC @srvasude @emilyfertig

[emilyfertig]

Thanks for the report @javier-garcia-tilburg ! Is it significantly outside the bounds or only barely, such that it could be a numerics issue?

If it's not a numerics issue, I suspect it could be because zoom linesearch, the default, doesn't apply a max stepsize. L-BFGS-B has a default max stepsize of 1 for the linesearch, which is passed to the other linesearch methods and would keep the result in bounds. Could you try it with linesearch='hager-zhang' and linesearch='backtracking' and see if it has the same issue? (And if it's not too much trouble could you look at the step size returned by zoom_linesearch in your problem and confirm that it's > 1.?)

[mblondel]

CC @vroulet

[javier-garcia-tilburg]

@emilyfertig

Is it significantly outside the bounds or only barely, such that it could be a numerics issue?

I think it is significant; the boundaries are (0,1) and LBFGSB returns parameters close to 1.5. The minimal reproduction example produces a 1.11.

I suspect it could be because zoom linesearch [...] Could you try it with linesearch='hager-zhang' and linesearch='backtracking' and see if it has the same issue?

You are right, I'm using the default linesearch method, zoom. Regarding the other methods, backtracking does respect boundaries, but hager-zhang did not terminate in 29+ hours so I gave up.

could you look at the step size returned by zoom_linesearch in your problem and confirm that it's > 1.?

The information returned by .run() says that the (last?) step size is 1.0. During the optimization process, the variable new_stepsize seems to be 1.0 in about 95% of .update() calls, and 2.0 in the remaining 5%.

The variable new_stepsize was measured by adding a debug print after the following line.

jaxopt/jaxopt/_src/lbfgsb.py

Line 500 in d40b6d7

new_stepsize = ls_state.a_k

 jax.debug.print("new_stepsize {new_stepsize}", new_stepsize=new_stepsize)

[javier-garcia-tilburg]

Using max_stepsize=1 does not work, LBFGSB still returns values outside of bounds.

[emilyfertig]

Thanks, I'll work on a fix.

@javier-garcia-tilburg are you synced past d40b6d7? I would have expected that to fix the issue with Hager-Zhang not terminating.

[vroulet]

I'm working on revamping the zoom linesearch see #438. A max_stepsize will be incorporated. I'm fixing last bugs due to the optimizer potentially getting stuck too close to a solution and I hope to be done asap.

[javier-garcia-tilburg]

@emilyfertig

are you synced past d40b6d7

Nope, I am using jax 0.7

[kallel-mahdi]

Hi, thank you for the amazing effort put into this library.

For your information. Although i am using the current developement version, I am getting parameters that are out of bounds using LBFGS-B + Hager Zhang line search.

[vroulet]

Hello!

We have reimplemented the zoom linesearch to take a max_stepsize. Could you pull the most recent version and see if it still does not work? If not, could you send us a reproducible code such that we can investigate that a bit better?

[javier-garcia-tilburg]

@vroulet , I just tried the new version and I the problem is gone. The solution returned by LBFGSB (linesearch="zoom") is within bounds whether I leave max_stepsize=1 or not.

Thank you for your work and effort