Install rust and cargo in manylinux image, prune unused packages.

[ScottTodd]

Progress on nod-ai/shark-ai#679

Tested:

cd base-docker-images
sudo docker buildx build --file dockerfiles/manylinux_x86_64.Dockerfile . --tag manylinux:latest

cd ../shark-ai
CACHE_DIR=~/.shark-ai-cache \
  OUTPUT_DIR=/tmp/wheelhouse \
  MANYLINUX_DOCKER_IMAGE=manylinux:latest \
  sudo -E ./shortfin/build_tools/build_linux_package.sh

# ******************** BUILD COMPLETE ********************
# + echo 'Generated binaries:'
# Generated binaries:
# + ls -l /tmp/wheelhouse
# total 40084
# -rw-r--r-- 1 root root 13691622 Dec 11 15:48 shortfin-3.0.0rc20241118-cp311-cp311-manylinux_2_28_x86_64.whl
# -rw-r--r-- 1 root root 13682296 Dec 11 15:50 shortfin-3.0.0rc20241118-cp312-cp312-manylinux_2_28_x86_64.whl
# -rw-r--r-- 1 root root 13666409 Dec 11 15:52 shortfin-3.0.0rc20241118-cp313-cp313-manylinux_2_28_x86_64.whl

Note that this image is in use without pinning in some repositories, but from what I can tell, all affected workflows are already failing and the code is unmaintained. For example: https://github.com/nod-ai/SRT/blob/373685f1cfff5dd6d934bf5858b6d58fc7a5bcdf/build_tools/pkgci/build_linux_packages.sh#L67.

[marbre]

I assumed we need get rustup via curl and use this to install Rust. However, if we get a working rustc this way, fine :) The one think to notice, SHORTFIN_ENABLE_TOKENIZERS defaults to OFF thus I don't know if your test did what you expected.

[ScottTodd]

The one think to notice, SHORTFIN_ENABLE_TOKENIZERS defaults to OFF thus I don't know if your test did what you expected.

Ah yeah, I'm testing with local changes to https://github.com/nod-ai/shark-ai too:

~/dev/projects/shark-ai (main)$ git diff
diff --git a/shortfin/build_tools/build_linux_package.sh b/shortfin/build_tools/build_linux_package.sh
index 91b944e..512721d 100755
--- a/shortfin/build_tools/build_linux_package.sh
+++ b/shortfin/build_tools/build_linux_package.sh
@@ -43,6 +43,7 @@ PYTHON_VERSIONS="${OVERRIDE_PYTHON_VERSIONS:-cp311-cp311 cp312-cp312 cp313-cp313
 OUTPUT_DIR="${OUTPUT_DIR:-${THIS_DIR}/wheelhouse}"
 CACHE_DIR="${CACHE_DIR:-}"
 SHORTFIN_ENABLE_TRACING="${SHORTFIN_ENABLE_TRACING:-ON}"
+SHORTFIN_ENABLE_TOKENIZERS="${SHORTFIN_ENABLE_TOKENIZERS:-ON}"
 
 function run_on_host() {
   echo "Running on host"
@@ -69,6 +70,7 @@ function run_on_host() {
     -e "OVERRIDE_PYTHON_VERSIONS=${PYTHON_VERSIONS}" \
     -e "OUTPUT_DIR=${OUTPUT_DIR}" \
     -e "SHORTFIN_ENABLE_TRACING=${SHORTFIN_ENABLE_TRACING}" \
+    -e "SHORTFIN_ENABLE_TOKENIZERS=${SHORTFIN_ENABLE_TOKENIZERS}" \
     ${extra_args} \
     "${MANYLINUX_DOCKER_IMAGE}" \
     -- ${THIS_DIR}/${SCRIPT_NAME}

[ScottTodd]

I assumed we need get rustup via curl and use this to install Rust. However, if we get a working rustc this way, fine :)

Yeah this is simpler and I trust it more. Found instructions at https://stackoverflow.com/questions/55089691/how-to-install-rust-toolset-on-rhel-in-ec2, and we're already using the EPEL (Extra Packages for Enterprise Linux) repo in this dockerfile