Understand environment initialization on kernel startup

[joel99]

I've been running some deep networks (with CUDA 11.3) happily, via .py jupyter notebooks for a while. I then upgraded from cuda 11.3 -> 11.6 but the notebook constantly thinks I'm using cuda 11.3. Now, I realize I don't even know how cuda 11.3 was initially recognized. So I'm trying to understand where environment variables are being set that could cause this.

1. The cluster I'm using manages cuda via ModuleFiles. For example, my .bashrc and .bash_profile both call module load cuda-11.6 which sets environment variables like CUDA_PATH. In vscode shell, this variable is set correctly.

2. Jupyter notebook doesn't load env variables #8270 suggests .bashrc is never sourced i.e. I expect no environment variables to be set at all in this case. However, various test variables I put in .bashrc are set and printed via os.environ.

3. However, setting CUDA_PATH, either via module load or export CUDA_PATH=11.6 doesn't survive into the jupyter notebook. Specifically, it appears to be overridden back to 11.3. This suggests some Jupyter initialization is doing this. I attached the jupyter output.
   vscode_jupyter_debug.txt
   Note that for example CUDA_PATH is printed correctly early on.

4. Based on the logs, I think jupyter can only be inheriting cuda info from a conda environment (rlpyt) which has built in cudnn 8.2.0 (corresponding to cuda 11.3), so anything I've previously set with environment variables has been totally irrelevant. Confusingly, several commands are logged from both a new conda env (py10_2) and this old rlpyt). Now my question is -- why is it calling both these environments? How can I specify which one to initialize with? And why does jupyter seem to attempt (and fail) to initialize conda a few times before succeeding (in a different mode)?

[Cage89]

bump, having a similar problem

[lowrollr]

also experiencing this!