Memory Leak for CUDNN API Handle

[lissahyacinth]

Issue

CUDNN is created as a lazy_static as a handle to CUDNN API (https://github.com/spearow/juice/blob/master/coaster-nn/src/frameworks/cuda/mod.rs#L15-L17)

Lazy_statics cannot be dropped, but CUDNN has a drop method that destroys the API handle, as encouraged in the CUDNN docs.
https://github.com/spearow/juice/blob/master/rcudnn/cudnn/src/cudnn.rs#L27-L32

Without this drop, we will have a minor memory leak on each GPU run.

Complications

The simplest way to resolve this issue would be to secure the lifetime of CUDNN to the CUDA backend, either with a check for CUDNN in the user environment, or assuming anyone using CUDA has CUDNN.

The current dependency chain for Juice is that Coaster-NN depends on Coaster, and so we cannot modify the initialise for CUDA in Coaster-NN, or load in Coaster-NN code into Coaster.

My initial suggestion would be to recreate CUDNN as a mutatable global (Rc may be suitable as there should never be multiple calls to CUDNN happening across threads), but this doesn't prevent the issue that we don't have any struct to tie it to without requesting users have a second backend call just to load CUDNN.

[drahnr]

Two cases where this is necessary:

1. regular exit when calling std::process::exit()
2. on panic

---

1. is more difficult, in C/C++ there are exit hooks but I am not aware of any in rust, but it would be interesting to know when drop() is called and when it is not
2. can be covered by creating a middleware panic handler which releases the cudnn handle, this can be done in the same lazy block https://doc.rust-lang.org/std/panic/index.html

[drahnr]

cross link: https://users.rust-lang.org/t/assure-drop-is-being-called/39423/3

[drahnr]

Solution to #71