Splitting LOO into its own package

[sethaxen]

PSIS is a useful algorithm in its own right whenever one is doing importance sampling, and so it makes sense for it to be in a lightweight package separate from LOO. Because this package is named ParetoSmooth.jl, this seems to be the sensible home for PSIS.

I propose the creation of a new package named LeaveOneOut.jl or LeaveOneOutCV.jl that would contain the (re)loo functionality currently in this package and have ParetoSmooth.jl as a dependency. This would also allow this package to be a more lightweight dependency.

Stand-alone loo/psis packages are already on the ArviZ.jl roadmap, and it would be nice to join efforts on these: arviz-devs/ArviZ.jl#129

[devmotion]

Isn't it possible to leverage some of the existing CV implementations, e.g. in MLJ?

(Edit: Just to be clear, this question is orthogonal to whether/how the package should be split)

[sethaxen]

I haven't looked at the CV implementations in MLJ, but LOO doesn't actually do CV. It does approximate CV using importance sampling (hence the PSIS implementation). I'd guess MLJ is doing something different.

[ParadaCarleton]

Isn't it possible to leverage some of the existing CV implementations, e.g. in MLJ?

(Edit: Just to be clear, this question is orthogonal to whether/how the package should be split)

I think we can piggyback off of them a bit; I'm actually making a PR right now that implements some MLJ interfaces. However, I think MLJ's evaluation functions do exact cross-validation, given an algorithm. Unless MLJ provides some way to do cross-validation, given some black-box refitting function? I'm very unfamiliar with MLJ, so I'd love to hear if this is possible.