Bug: sam build fails to install onnxruntime==1.18.1

[zhemaituk]

Description:

onnxruntime==1.18.1 dependency fails to install in container and locally using sam build.

Steps to reproduce:

1. SAM init hello world example
2. Runtime: python3.12; Architectures: - arm64
3. Add to requirements.txt:

onnxruntime==1.18.1

4. sam build --use-container --debug

Observed result:

2024-08-14 11:08:27,171 | Mounting .../hello_world as /tmp/samcli/source:ro,delegated, inside runtime container                                                  
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
...
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
pip stdout: b'Collecting onnxruntime==1.18.1 (from -r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading onnxruntime-1.18.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (4.3 kB)\nCollecting coloredlogs (from onnxruntime==1.18.1->-r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\nCollecting flatbuffers (from onnxruntime==1.18.1->-r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)\nCollecting numpy<2.0,>=1.21.6 (from onnxruntime==1.18.1->-r /tmp/samcli/source/requirements.txt (line 1))\n  ...
Full dependency closure: {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), onnxruntime==1.18.1(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
initial compatible: {flatbuffers==24.3.25(wheel), mpmath==1.3.0(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
initial incompatible: {onnxruntime==1.18.1(wheel)}
Downloading missing wheels: {onnxruntime==1.18.1(wheel)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp312 --dest /tmp/samcli/scratch onnxruntime==1.18.1
pip stdout: b''
(from versions: none)\nERROR: No matching distribution found for onnxruntime==1.18.1\n\n[notice] A new release of pip is available: 24.0 -> 24.2\n[notice] To update, run: pip install --upgrade pip\n"
compatible wheels after second download pass: {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Final compatible: {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Final incompatible: {onnxruntime==1.18.1(wheel)}
Final missing wheels: {onnxruntime==1.18.1(wheel)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 71, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 265, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {onnxruntime==1.18.1(wheel)}

During handling of the above exception, another exception occurred:

...
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {onnxruntime==1.18.1(wheel)}

During handling of the above exception, another exception occurred:

...
aws_lambda_builders.actions.ActionFailedError: {onnxruntime==1.18.1(wheel)}

During handling of the above exception, another exception occurred:

...
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {onnxruntime==1.18.1(wheel)}

...

Build Failed
...                                                                                                                             
Error: PythonPipBuilder:ResolveDependencies - {onnxruntime==1.18.1(wheel)}

Expected result:

Build Successful

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

The same fails without --use-container as well, BUT only during first execution of sam build, second execution of sam build without cleaning .aws-sam folder is successful.

pip install onnxruntime==1.18.1 by itself works fine in both: container and locally.

OS: macOS 14.6.1 23G93
SAM CLI: version 1.121.0

{
  "version": "1.121.0",
  "system": {
    "python": "3.12.5",
    "os": "macOS-14.6.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "27.1.1",
    "aws_cdk": "Not available",
    "terraform": "1.9.4"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[hawflau]

@zhemaituk Thanks for raising the issue.

The package onnxruntime only provides Linux wheels for manylinux_2_27 and manylinux_2_28 and does not provide sdist. But Lambda Builders seems to not support these two manylinux tags. Marking this as a bug.

[fredbonin]

I have a similar problem with confluent_kafka==2.8.0.

Using sam build -u --debug, I get the error below

calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
pip stdout: b'Collecting confluent_kafka==2.8.0 (from -r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading confluent_kafka-2.8.0-cp313-cp313-manylinux_2_28_aarch64.whl.metadata (21 kB)\nDownloading confluent_kafka-2.8.0-cp313-cp313-manylinux_2_28_aarch64.whl (15.2 MB)\n   \xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81\xe2\x94\x81 15.2/15.2 MB 50.9 MB/s eta 0:00:00\nSaved /tmp/samcli/scratch/confluent_kafka-2.8.0-cp313-cp313-manylinux_2_28_aarch64.whl\nSuccessfully downloaded confluent_kafka\n'
pip stderr: b"WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.\nPlease see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.\nTo avoid this problem you can invoke Python with '-m pip' instead of running pip directly.\n\n[notice] A new release of pip is available: 24.2 -> 25.0\n[notice] To update, run: pip install --upgrade pip\n"
Full dependency closure: {confluent-kafka==2.8.0(wheel)}
initial compatible: set()
initial incompatible: {confluent-kafka==2.8.0(wheel)}
Downloading missing wheels: {confluent-kafka==2.8.0(wheel)}
...

From the logs, it says that it is incompatible but the file confluent_kafka-2.8.0-cp313-cp313-manylinux_2_28_aarch64.whl exists and works well with lambda. I was able to use it in a lambda using this command instead:

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.13:latest-arm64" /bin/sh -c "pip download -r requirements.txt --dest python/lib/python3.13/site-packages/; exit"

Could it be because the package name can be either confluent-kafka or confluent_kafka (I tried both and it did not work though)?