[CI] Upgrade Jenkins #2047
[CI] Upgrade Jenkins #2047
Conversation
With matplotlib > 3.4 MXNet fails in plotting with many notebooks (eg. chapter_appendix-mathematics-for-deep-learning/multivariable-calculus.md) throwing the following error: TypeError: no implementation found for 'numpy.column_stack' on types that implement __array_function__: [<class 'mxnet.numpy.ndarray'>, <class 'numpy.ndarray'>]
revise the call function in the module class.
| @@ -3,11 +3,10 @@ dependencies: | |||
| - pip | |||
| - pip: | |||
| - .. # d2l | |||
| - git+https://github.com/d2l-ai/d2l-book | |||
There was a problem hiding this comment.
Let's keep this dependency since we may keep upgrading it to enable new features
There was a problem hiding this comment.
For some reason when I setup build.yml with d2l-book dependency, I get an error with sphinx not being available. It is collected by pip but mxtheme requires sphinx and hence errors out.
There was a problem hiding this comment.
Then we may need to fix it if this is an issue with the current source of d2l-book
| @@ -16,7 +16,6 @@ stage("Build and Publish") { | |||
| sh label: "Build Environment", script: """set -ex | |||
| conda env update -n ${ENV_NAME} -f static/build.yml | |||
| conda activate ${ENV_NAME} | |||
| pip uninstall -y d2lbook | |||
There was a problem hiding this comment.
This is a workaround for the env cache (the d2lbook installation won't be triggered from build.yml since it's cached, which saves installation (>10 mins) every time a change is pushed) - in order to integrate the latest features of d2lbook we force to re-install it in this way. Maybe we can keep this for now and iterate over it after we have better ideas of env cache on Actions.
There was a problem hiding this comment.
I see, makes sense.
|
To accelerate the CI upgrading, I'm going to merge this PR but please address my comments in a separate PR (which won't block your integration). Thanks! |
This PR updates the infrastructure for d2l CI.
static/build.ymltensorflow==2.8.0andtensorflow-probability==0.16.0(Based on CUDA 11.2)torchandmxnetcuda version. (Based on CUDA 10.2)matplotlib==3.4.0; this is required to fix [MXNet] matplotlib >=3.5 raises TypeError with ax.plot_wireframe in MXNet ndarray #2046tensorflow==2.8.0bug in seq2seq; Thanks @cheungdavenmxnetwithnumpy>=1.20. Fixes issue number one mentioned in Pin the dependencies in setup.py #2044Changes made behind the scene:
The slave node now has multiple CUDA versions (10.2 and 11.2) for use with different frameworks.
LD_LIBRARY_PATHsets both of them in order and each library installs the framework accordingly.@astonzhang I suggest making a merge commit instead of squashing since we have multiple unrelated changes in this single PR and in case we need to revert only a particular change then we can easily do that.