[RFC] unbundle LLVM?

[h-vetinari]

Not sure how realistic this is, but the compilation of this package is spending a lot of time to recompile llvm/lib, which we're already doing in https://github.com/conda-forge/llvmdev-feedstock. There's other parts coming from MLIR, where I haven't check if we're building that separately somewhere.

Not sure if the surgery for this is feasible TBH, but thought it might be worth discussing. There's much more to (potentially) unbundle, see the upstream third_party folder... This also shows another potential problem, because tensorflow vendoring LLVM means that they're not necessarily in sync with (the ABI of) released versions.

CC @xhochy @isuruf @hmaarrfk

[h-vetinari]

because tensorflow vendoring LLVM means that they're not necessarily in sync with (the ABI of) released versions.

Though presumably we could build llvm prereleases under conda-forge/rc and use them as a build dep?

[ngam]

Definitely if we can pull it off, but from peeking inside yesterday it seemed it would require a lot of work --- there is no obvious on/off switch for this upstream. However, they don't seem to pull in something that outlandish or different from llvm/mlir. (Skimming over this btw, I believe we have all that's needed already built in conda-forge, both MLIR and LLVM, etc.)

[hmaarrfk]

I have very little knowledge of bazel, but it would be interesting if somebody could take it.

[ngam]

I have very little knowledge of bazel, but it would be interesting if somebody could take it.

That's precisely why I stopped looking. I do believe though we could tinker with the paths to get this fixed. It would require me a bit more work to even start making progress on this. Also for MacOS, this would save ~35% of the build time, I think. Not sure about the cuda builds --- I would assume maybe 20% or less.

I know we were talking about unbundling OneDNN at some point --- I don't see where that's being pulled in, so I am not sure how to go about that.

Either way, we likely need significant patching and/or some rigorous edits in the bazel setup, which is, again, relatively new to me but I am happy to dig deeper when time permits...

[hmaarrfk]

My guess is that we need to add it to:
https://github.com/tensorflow/tensorflow/blob/316bbe87c611667c0e0cb0aec525453f953f7835/third_party/systemlibs/syslibs_configure.bzl#L11