Bug: packaged SAM 1.98 fails build with ruby 3.2 while development version works correctly

[theist]

I've found and read #5530 but I can not reopen it and the closed issues have a COMMENT VISIBILITY WARNING so I've opened a new one

Description:

sam build installed from https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip fails to build indicating Bundler Failed: but under the same conditions a development copy of sam of the same version works correctly whe I do a samdev build

To add more mystery after samdev build works, sam build starts to work correctly until the .aws-sam directory is removed.

Steps to reproduce:

On a clean ubuntu 22.

• install ruby 3.2.2, install bundler
• install sam following AWS instructions for linux
• install samdev following the development instructions
• use a simple ruby3.2 lambda template
• sam build-> 💥
• samdev build -> 👌
  • after that sam build -> 👌
  • rm -fr .aws-sam
    • after that sam build -> 💥

I've created a Dockerfile with all the required parts at https://github.com/theist/sam-debug-weird-case

building this container and running it with docker run -ti /bin/bash and doing source /etc/profile.d/rbenv.sh and cd /lambda lets you to test the steps

Observed result:

root@df299b431131:/lambda# sam build --debug
2023-10-11 11:35:07,255 | Config file location: /lambda/samconfig.toml                                                                                                        
2023-10-11 11:35:07,257 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at '/lambda/samconfig.toml'...           
2023-10-11 11:35:07,258 | Configuration values successfully loaded.                                                                                                           
2023-10-11 11:35:07,259 | Configuration values are: {'stack_name': 'TestUpgradeRubyStaging', 'parameter_overrides': 'Environment=staging', 's3_bucket':                       
'bebanjo-aws-sam-cli-sourcebucket', 's3_prefix': 'test_upgrade_to_ruby_3_2_staging', 'cached': True, 'parallel': True}                                                        
2023-10-11 11:35:07,262 | Using SAM Template at /lambda/template.yaml                                                                                                         
2023-10-11 11:35:07,285 | Using config file: samconfig.toml, config environment: default                                                                                      
2023-10-11 11:35:07,286 | Expand command line arguments to:                                                                                                                   
2023-10-11 11:35:07,286 | --template_file=/lambda/template.yaml --parallel --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache --cached                   
--parameter_overrides={'Environment': 'staging'}                                                                                                                              
2023-10-11 11:35:07,342 | 'build' command is called                                                                                                                           
2023-10-11 11:35:07,342 | Starting Build use cache                                                                                                                            
2023-10-11 11:35:07,345 | Collected default values for parameters: {'Environment': 'staging'}                                                                                 
2023-10-11 11:35:07,360 | There is no customer defined id or cdk path defined for resource TestUpgradeRubyFunction, so we will use the resource logical id as the resource id 
2023-10-11 11:35:07,362 | 0 stacks found in the template                                                                                                                      
2023-10-11 11:35:07,362 | Collected default values for parameters: {'Environment': 'staging'}                                                                                 
2023-10-11 11:35:07,375 | There is no customer defined id or cdk path defined for resource TestUpgradeRubyFunction, so we will use the resource logical id as the resource id 
2023-10-11 11:35:07,376 | 1 resources found in the stack                                                                                                                      
2023-10-11 11:35:07,376 | Found Serverless function with name='TestUpgradeRubyFunction' and CodeUri='src/'                                                                    
2023-10-11 11:35:07,377 | --base-dir is not presented, adjusting uri src/ relative to /lambda/template.yaml                                                                   
2023-10-11 11:35:07,380 | 1 resources found in the stack                                                                                                                      
2023-10-11 11:35:07,380 | Found Serverless function with name='TestUpgradeRubyFunction' and CodeUri='src/'                                                                    
2023-10-11 11:35:07,381 | Instantiating build definitions                                                                                                                     
2023-10-11 11:35:07,383 | Same function build definition found, adding function (Previous: BuildDefinition(ruby3.2, /lambda/src, Zip, , 0ca1a330-5c9a-4364-9f42-c0aaef0708a0, 
{}, {}, x86_64, []), Current: BuildDefinition(ruby3.2, /lambda/src, Zip, , 30c62b7b-79b9-4ec7-b391-7e73232d0ba0, {}, {}, x86_64, []), Function:                               
Function(function_id='TestUpgradeRubyFunction', name='TestUpgradeRubyFunction', functionname='test_upgrade_to_ruby_3_2_staging', runtime='ruby3.2', memory=128, timeout=3,    
handler='filter.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/lambda/src', environment={'Variables': {'RAILS_ENV': 'staging'}}, rolearn=None,
layers=[], events=None, metadata={'SamResourceId': 'TestUpgradeRubyFunction'}, inlinecode=None, codesign_config_arn=None, architectures=None, function_url_config=None,       
function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='',                                
runtime_management_config=None))                                                                                                                                              
2023-10-11 11:35:07,387 | Async execution started                                                                                                                             
2023-10-11 11:35:07,388 | Invoking function functools.partial(<bound method ParallelBuildStrategy.build_single_function_definition of                                         
<samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x7f26d8f37750>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x7f26d8f9c250>)           
2023-10-11 11:35:07,389 | Running incremental build for runtime ruby3.2 for following resources (TestUpgradeRubyFunction)                                                     
2023-10-11 11:35:07,390 | Waiting for async results                                                                                                                           
2023-10-11 11:35:07,390 | Manifest file is changed (new hash: def7aa28dda73cf76958fb58dbaac709) or dependency folder (.aws-sam/deps/0ca1a330-5c9a-4364-9f42-c0aaef0708a0) is  
missing for (TestUpgradeRubyFunction), downloading dependencies and copying/building source                                                                                   
2023-10-11 11:35:07,391 | Building codeuri: /lambda/src runtime: ruby3.2 metadata: {} architecture: x86_64 functions: TestUpgradeRubyFunction                                 
2023-10-11 11:35:07,392 | Building to following folder /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                                         
2023-10-11 11:35:07,393 | Loading workflow module 'aws_lambda_builders.workflows'                                                                                             
2023-10-11 11:35:07,394 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'     
2023-10-11 11:35:07,396 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package',                     
application_framework=None)'                                                                                                                                                  
2023-10-11 11:35:07,397 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'       
2023-10-11 11:35:07,399 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'    
2023-10-11 11:35:07,400 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'      
2023-10-11 11:35:07,401 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'       
2023-10-11 11:35:07,403 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild',                     
application_framework=None)'                                                                                                                                                  
2023-10-11 11:35:07,405 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'       
2023-10-11 11:35:07,406 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'   
2023-10-11 11:35:07,407 | Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)' 
2023-10-11 11:35:07,408 | Found workflow 'RubyBundlerBuilder' to support capabilities 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 
2023-10-11 11:35:07,409 | Running workflow 'RubyBundlerBuilder'                                                                                                               
2023-10-11 11:35:07,409 |  Running RubyBundlerBuilder:CopySource                                                                                                              
2023-10-11 11:35:07,410 | Creating target folders at /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                                           
2023-10-11 11:35:07,411 | Copying directory metadata from source (/lambda/src) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction)                                
2023-10-11 11:35:07,411 | Copying source file (/lambda/src/Gemfile.lock) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction/Gemfile.lock)                         
2023-10-11 11:35:07,412 | Copying source file (/lambda/src/Gemfile) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction/Gemfile)                                   
2023-10-11 11:35:07,413 | Copying source file (/lambda/src/filter.rb) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction/filter.rb)                               
2023-10-11 11:35:07,414 | RubyBundlerBuilder:CopySource succeeded                                                                                                             
2023-10-11 11:35:07,414 |  Running RubyBundlerBuilder:RubyBundle                                                                                                              
2023-10-11 11:35:07,415 | Running bundle install in /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                                            
2023-10-11 11:35:07,415 | executing Bundler: ['bundle', 'install', '--without', 'development', 'test']                                                                        
2023-10-11 11:35:07,590 | RubyBundlerBuilder:RubyBundle failed                                                                                                                
Traceback (most recent call last):                                                                                                                                            
  File "aws_lambda_builders/workflows/ruby_bundler/actions.py", line 32, in execute                                                                                           
  File "aws_lambda_builders/workflows/ruby_bundler/bundler.py", line 69, in run                                                                                               
aws_lambda_builders.workflows.ruby_bundler.bundler.BundlerExecutionError: Bundler Failed:                                                                                     
                                                                                                                                                                              
During handling of the above exception, another exception occurred:                                                                                                           
                                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                            
  File "aws_lambda_builders/workflow.py", line 371, in run                                                                                                                    
  File "aws_lambda_builders/workflows/ruby_bundler/actions.py", line 34, in execute                                                                                           
aws_lambda_builders.actions.ActionFailedError: Bundler Failed:                                                                                                                
2023-10-11 11:35:07,593 | Exception raised during the execution                                                                                                               

Build Failed
2023-10-11 11:35:07,594 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                          
2023-10-11 11:35:07,599 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                          
2023-10-11 11:35:07,599 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'ddd9301b-b33f-4a9e-8ce7-e65ce8ef392c', 'installationId':                              
'c08609c1-769f-4c12-bb12-55a08165be20', 'sessionId': '70fb0591-3655-4841-9841-6462462daf45', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.3',               
'samcliVersion': '1.98.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType':      
'CFN', 'gitOrigin': None, 'projectName': '554e79e652b29f3a01dcf98e494b9928a1dc12cc83b83e730574c9b37ca6359f', 'initialCommit': None}, 'duration': 308, 'exitReason':           
'WorkflowFailedError', 'exitCode': 1}}]}                                                                                                                                      
2023-10-11 11:35:07,599 | Unable to find Click Context for getting session_id.                                                                                                
2023-10-11 11:35:07,603 | Sending Telemetry: {'metrics': [{'events': {'requestId': '6c41e59e-cfbc-4350-bfe3-15508edd89fd', 'installationId':                                  
'c08609c1-769f-4c12-bb12-55a08165be20', 'sessionId': '70fb0591-3655-4841-9841-6462462daf45', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.3',               
'samcliVersion': '1.98.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':  
'fd9ab8efedf34c6ea57fe86a09a0eb85', 'time_stamp': '2023-10-11 11:35:07.255', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml',        
'thread_id': '517fa62009d547d49b817d7dda0bf169', 'time_stamp': '2023-10-11 11:35:07.285', 'exception_name': None}, {'event_name': 'BuildFunctionRuntime', 'event_value':      
'ruby3.2', 'thread_id': 'c4ccb53c1b084ff4b8795f266635f986', 'time_stamp': '2023-10-11 11:35:07.381', 'exception_name': None}, {'event_name': 'BuildWorkflowUsed',             
'event_value': 'ruby-bundler', 'thread_id': 'b899112c7a4342cf8a9abf3d1b39c573', 'time_stamp': '2023-10-11 11:35:07.390', 'exception_name': None}, {'event_name':              
'BuildWorkflowUsed', 'event_value': 'ruby-bundler', 'thread_id': 'd98a2dd38c35413fabcabdd7d630d0c3', 'time_stamp': '2023-10-11 11:35:07.393', 'exception_name': None}]}}}]}   
2023-10-11 11:35:08,253 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                    
Error: RubyBundlerBuilder:RubyBundle - Bundler Failed: 
2023-10-11 11:35:08,258 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)      

with samdev

root@df299b431131:/lambda# rm -fr .aws-sam/
root@df299b431131:/lambda# samdev build --debug
2023-10-11 11:36:18 Attaching import module proxy for analyzing dynamic imports
2023-10-11 11:36:18,615 | Config file location: /lambda/samconfig.toml                                                                                                        
2023-10-11 11:36:18,618 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at '/lambda/samconfig.toml'...           
2023-10-11 11:36:18,619 | Configuration values successfully loaded.                                                                                                           
2023-10-11 11:36:18,620 | Configuration values are: {'stack_name': 'TestUpgradeRubyStaging', 'parameter_overrides': 'Environment=staging', 's3_bucket':                       
'bebanjo-aws-sam-cli-sourcebucket', 's3_prefix': 'test_upgrade_to_ruby_3_2_staging', 'cached': True, 'parallel': True}                                                        
2023-10-11 11:36:18,644 | Using SAM Template at /lambda/template.yaml                                                                                                         
2023-10-11 11:36:18,667 | Using config file: samconfig.toml, config environment: default                                                                                      
2023-10-11 11:36:18,667 | Expand command line arguments to:                                                                                                                   
2023-10-11 11:36:18,668 | --template_file=/lambda/template.yaml --parallel --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache --cached                   
--parameter_overrides={'Environment': 'staging'}                                                                                                                              
2023-10-11 11:36:18,669 | 'build' command is called                                                                                                                           
2023-10-11 11:36:18,669 | Starting Build use cache                                                                                                                            
2023-10-11 11:36:18,672 | Collected default values for parameters: {'Environment': 'staging'}                                                                                 
2023-10-11 11:36:18,689 | There is no customer defined id or cdk path defined for resource TestUpgradeRubyFunction, so we will use the resource logical id as the resource id 
2023-10-11 11:36:18,690 | 0 stacks found in the template                                                                                                                      
2023-10-11 11:36:18,691 | Collected default values for parameters: {'Environment': 'staging'}                                                                                 
2023-10-11 11:36:18,705 | There is no customer defined id or cdk path defined for resource TestUpgradeRubyFunction, so we will use the resource logical id as the resource id 
2023-10-11 11:36:18,705 | 1 resources found in the stack                                                                                                                      
2023-10-11 11:36:18,706 | Found Serverless function with name='TestUpgradeRubyFunction' and CodeUri='src/'                                                                    
2023-10-11 11:36:18,706 | --base-dir is not presented, adjusting uri src/ relative to /lambda/template.yaml                                                                   
2023-10-11 11:36:18,710 | 1 resources found in the stack                                                                                                                      
2023-10-11 11:36:18,710 | Found Serverless function with name='TestUpgradeRubyFunction' and CodeUri='src/'                                                                    
2023-10-11 11:36:18,711 | Instantiating build definitions                                                                                                                     
2023-10-11 11:36:18,712 | No previous build graph found, generating new one                                                                                                   
2023-10-11 11:36:18,712 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(ruby3.2, /lambda/src, Zip, ,                      
a582c81b-f82f-426e-a019-9992cc795a67, {}, {}, x86_64, []), Function: Function(function_id='TestUpgradeRubyFunction', name='TestUpgradeRubyFunction',                          
functionname='test_upgrade_to_ruby_3_2_staging', runtime='ruby3.2', memory=128, timeout=3, handler='filter.lambda_handler', imageuri=None, packagetype='Zip',                 
imageconfig=None, codeuri='/lambda/src', environment={'Variables': {'RAILS_ENV': 'staging'}}, rolearn=None, layers=[], events=None, metadata={'SamResourceId':                
'TestUpgradeRubyFunction'}, inlinecode=None, codesign_config_arn=None, architectures=None, function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip:     
('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None))                                                    
2023-10-11 11:36:18,716 | Async execution started                                                                                                                             
2023-10-11 11:36:18,716 | Invoking function functools.partial(<bound method ParallelBuildStrategy.build_single_function_definition of                                         
<samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x7f15cd27e110>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x7f15cd447d60>)           
2023-10-11 11:36:18,718 | Running incremental build for runtime ruby3.2 for following resources (TestUpgradeRubyFunction)                                                     
2023-10-11 11:36:18,718 | Waiting for async results                                                                                                                           
2023-10-11 11:36:18,719 | Manifest file is changed (new hash: def7aa28dda73cf76958fb58dbaac709) or dependency folder (.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67) is  
missing for (TestUpgradeRubyFunction), downloading dependencies and copying/building source                                                                                   
2023-10-11 11:36:18,720 | Building codeuri: /lambda/src runtime: ruby3.2 metadata: {} architecture: x86_64 functions: TestUpgradeRubyFunction                                 
2023-10-11 11:36:18,721 | Building to following folder /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                                         
2023-10-11 11:36:18,722 | Loading workflow module 'aws_lambda_builders.workflows'                                                                                             
2023-10-11 11:36:18,722 | Importing a package which was already defined in hidden imports name: aws_lambda_builders.workflows, package: None                                  
2023-10-11 11:36:18,724 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'     
2023-10-11 11:36:18,725 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package',                     
application_framework=None)'                                                                                                                                                  
2023-10-11 11:36:18,727 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'       
2023-10-11 11:36:18,729 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'    
2023-10-11 11:36:18,730 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'      
2023-10-11 11:36:18,732 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'       
2023-10-11 11:36:18,733 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild',                     
application_framework=None)'                                                                                                                                                  
2023-10-11 11:36:18,735 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'       
2023-10-11 11:36:18,736 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'   
2023-10-11 11:36:18,738 | Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)' 
2023-10-11 11:36:18,738 | Found workflow 'RubyBundlerBuilder' to support capabilities 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 
2023-10-11 11:36:18,739 | Running workflow 'RubyBundlerBuilder'                                                                                                               
2023-10-11 11:36:18,740 |  Running RubyBundlerBuilder:CopySource                                                                                                              
2023-10-11 11:36:18,740 | Creating target folders at /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                                           
2023-10-11 11:36:18,741 | Copying directory metadata from source (/lambda/src) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction)                                
2023-10-11 11:36:18,742 | Copying source file (/lambda/src/Gemfile.lock) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction/Gemfile.lock)                         
2023-10-11 11:36:18,742 | Copying source file (/lambda/src/Gemfile) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction/Gemfile)                                   
2023-10-11 11:36:18,743 | Copying source file (/lambda/src/filter.rb) to destination (/lambda/.aws-sam/build/TestUpgradeRubyFunction/filter.rb)                               
2023-10-11 11:36:18,744 | RubyBundlerBuilder:CopySource succeeded                                                                                                             
2023-10-11 11:36:18,744 |  Running RubyBundlerBuilder:RubyBundle                                                                                                              
2023-10-11 11:36:18,745 | Running bundle install in /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                                            
2023-10-11 11:36:18,745 | executing Bundler: ['bundle', 'install', '--without', 'development', 'test']                                                                        
2023-10-11 11:36:18,908 | RubyBundlerBuilder:RubyBundle succeeded                                                                                                             
2023-10-11 11:36:18,908 |  Running RubyBundlerBuilder:RubyBundleDeployment                                                                                                    
2023-10-11 11:36:18,909 | Running bundle install --deployment in /lambda/.aws-sam/build/TestUpgradeRubyFunction                                                               
2023-10-11 11:36:18,910 | executing Bundler: ['bundle', 'install', '--deployment', '--without', 'development', 'test']                                                        
2023-10-11 11:36:19,074 | RubyBundlerBuilder:RubyBundleDeployment succeeded                                                                                                   
2023-10-11 11:36:19,075 |  Running RubyBundlerBuilder:CleanUp                                                                                                                 
2023-10-11 11:36:19,075 | Clean up action: .aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67 does not exist and will be skipped.                                             
2023-10-11 11:36:19,076 | RubyBundlerBuilder:CleanUp succeeded                                                                                                                
2023-10-11 11:36:19,076 |  Running RubyBundlerBuilder:CopyDependencies                                                                                                        
2023-10-11 11:36:19,077 | Creating target folders at .aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/.bundle                                                               
2023-10-11 11:36:19,077 | Copying directory metadata from source (/lambda/.aws-sam/build/TestUpgradeRubyFunction/.bundle) to destination                                      
(.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/.bundle)                                                                                                                  
2023-10-11 11:36:19,078 | Copying source file (/lambda/.aws-sam/build/TestUpgradeRubyFunction/.bundle/config) to destination                                                  
(.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/.bundle/config)                                                                                                           
2023-10-11 11:36:19,079 | Creating target folders at .aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor                                                                
2023-10-11 11:36:19,080 | Copying directory metadata from source (/lambda/.aws-sam/build/TestUpgradeRubyFunction/vendor) to destination                                       
(.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor)                                                                                                                   
2023-10-11 11:36:19,081 | Creating target folders at .aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor/bundle                                                         
2023-10-11 11:36:19,081 | Copying directory metadata from source (/lambda/.aws-sam/build/TestUpgradeRubyFunction/vendor/bundle) to destination                                
(.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor/bundle)                                                                                                            
2023-10-11 11:36:19,082 | Creating target folders at .aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor/bundle/ruby                                                    
2023-10-11 11:36:19,082 | Copying directory metadata from source (/lambda/.aws-sam/build/TestUpgradeRubyFunction/vendor/bundle/ruby) to destination                           
(.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor/bundle/ruby)                                                                                                       
2023-10-11 11:36:19,083 | Creating target folders at .aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor/bundle/ruby/3.2.0                                              
2023-10-11 11:36:19,084 | Copying directory metadata from source (/lambda/.aws-sam/build/TestUpgradeRubyFunction/vendor/bundle/ruby/3.2.0) to destination                     
(.aws-sam/deps/a582c81b-f82f-426e-a019-9992cc795a67/vendor/bundle/ruby/3.2.0)                                                                                                 
2023-10-11 11:36:19,085 | RubyBundlerBuilder:CopyDependencies succeeded                                                                                                       
2023-10-11 11:36:19,085 | Async execution completed                                                                                                                           
2023-10-11 11:36:19,087 | There is no customer defined id or cdk path defined for resource TestUpgradeRubyFunction, so we will use the resource logical id as the resource id 
2023-10-11 11:36:19,088 | 1 resources found in the stack                                                                                                                      
2023-10-11 11:36:19,088 | Found Serverless function with name='TestUpgradeRubyFunction' and CodeUri='src/'                                                                    

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2023-10-11 11:36:19,091 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                          
2023-10-11 11:36:19,095 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                          
2023-10-11 11:36:19,096 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '0a19a08b-5021-4672-ae36-131fdbc1a702', 'installationId':                              
'c08609c1-769f-4c12-bb12-55a08165be20', 'sessionId': 'dbe79d70-4728-4666-bc66-ae1dcf335e53', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.10.12',              
'samcliVersion': '1.98.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'samdev build', 'metricSpecificAttributes': {'projectType':   
'CFN', 'gitOrigin': None, 'projectName': '554e79e652b29f3a01dcf98e494b9928a1dc12cc83b83e730574c9b37ca6359f', 'initialCommit': None}, 'duration': 424, 'exitReason': 'success',
'exitCode': 0}}]}                                                                                                                                                             
2023-10-11 11:36:19,096 | Unable to find Click Context for getting session_id.                                                                                                
2023-10-11 11:36:19,100 | Sending Telemetry: {'metrics': [{'events': {'requestId': '2ae9d53f-9408-474d-9a28-5233a619d3b8', 'installationId':                                  
'c08609c1-769f-4c12-bb12-55a08165be20', 'sessionId': 'dbe79d70-4728-4666-bc66-ae1dcf335e53', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.10.12',              
'samcliVersion': '1.98.0', 'commandName': 'samdev build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml',            
'thread_id': '4996ab2cb1f840468eb0cf416560245b', 'time_stamp': '2023-10-11 11:36:18.615', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value':    
'.toml', 'thread_id': '00a9c29301c746e38d05c2aed2e439f9', 'time_stamp': '2023-10-11 11:36:18.667', 'exception_name': None}, {'event_name': 'BuildFunctionRuntime',            
'event_value': 'ruby3.2', 'thread_id': '89efb4ad0043473cbeb19d71655f558a', 'time_stamp': '2023-10-11 11:36:18.711', 'exception_name': None}, {'event_name':                   
'BuildWorkflowUsed', 'event_value': 'ruby-bundler', 'thread_id': 'becceb493a1b479693ab4957989c9198', 'time_stamp': '2023-10-11 11:36:18.719', 'exception_name': None},        
{'event_name': 'BuildWorkflowUsed', 'event_value': 'ruby-bundler', 'thread_id': '042835d506594186943f3d502238f38c', 'time_stamp': '2023-10-11 11:36:18.722', 'exception_name':
None}]}}}]}                                                                                                                                                                   
2023-10-11 11:36:19,846 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                    
2023-10-11 11:36:19,850 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)    

Expected result:

packaged sam to work correctly as development version does

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Reproduced originally in debian 11

Reproduced in ubuntu container, with ruby 3.2.2 thru rbenv

sam installed from https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip

samdev installed thru sourcecode with venv and using make init

sam --version
SAM CLI, version 1.98.0

samdev --version
SAM CLI, version 1.98.0

This might be important as packaged sam features a different python version, I guess embedded and samdev uses OS python

sam:

{
  "version": "1.98.0",
  "system": {
    "python": "3.11.3",
    "os": "Linux-5.10.0-25-amd64-x86_64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

samdev

{
  "version": "1.98.0",
  "system": {
    "python": "3.10.12",
    "os": "Linux-5.10.0-25-amd64-x86_64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Thanks!

[lucashuy]

Thanks for opening this issue! I am able to reproduce this issue on Linux with the following versions of Ruby on both SAM CLI 1.98.0 and the development version:

dev-dsk-lucashuy-2a-0ec2379b % ruby --version
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-linux]

dev-dsk-lucashuy-2a-0ec2379b % bundle --version
Bundler version 2.2.3

dev-dsk-lucashuy-2a-0ec2379b % gem --version
3.2.3

However was not able to reproduce this using the latest version of everything:

dev-dsk-lucashuy-2a-0ec2379b % ruby --version
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

dev-dsk-lucashuy-2a-0ec2379b % bundle --version
Bundler version 2.4.10

dev-dsk-lucashuy-2a-0ec2379b % gem --version
3.4.10

Can you double check that the versions you have are up to date? Additionally, are you able to run sam build --use-container? This utilizes a Docker container to build the project in. The versions inside the container are the same as the versions I have in the second code block.

[theist]

Packaged sam with last ruby versions for me fails:

/lambda# ruby --version
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

/lambda# bundle --version
Bundler version 2.4.10

/lambda# gem --version
3.4.10

/lambda# sam --version
SAM CLI, version 1.98.0

/lambda# sam build    

Starting Build use cache                                                                                                                                                      
Manifest file is changed (new hash: def7aa28dda73cf76958fb58dbaac709) or dependency folder (.aws-sam/deps/9ae06e06-8c1c-4d78-9cf5-ec04764aee91) is missing for                
(TestUpgradeRubyFunction), downloading dependencies and copying/building source                                                                                               
Building codeuri: /lambda/src runtime: ruby3.2 metadata: {} architecture: x86_64 functions: TestUpgradeRubyFunction                                                           
 Running RubyBundlerBuilder:CopySource                                                                                                                                        
 Running RubyBundlerBuilder:RubyBundle                                                                                                                                        

Build Failed
Error: RubyBundlerBuilder:RubyBundle - Bundler Failed: 

It works correctly using --use-container

It also works correctly with samdev and the same versions

[lucashuy]

Hi, sorry about the delay in response. Using your provided Docker image, I installed a modified version of SAM CLI (based off the released 1.98.0) that fixes the Ruby error output. Interestingly, it looks like Ruby is not happy about the zlib library version that SAM CLI provides in this environment.

root@5d3a15a6840d:/lambda# ~/./me/dist/sam-beta build
Starting Build use cache
Manifest file is changed (new hash: def7aa28dda73cf76958fb58dbaac709) or dependency folder
(.aws-sam/deps/1c65cb81-137a-42b2-af4c-79678e0a76b7) is missing for (TestUpgradeRubyFunction),
downloading dependencies and copying/building source
Building codeuri: /lambda/src runtime: ruby3.2 metadata: {} architecture: x86_64 functions:
TestUpgradeRubyFunction
 Running RubyBundlerBuilder:CopySource
 Running RubyBundlerBuilder:RubyBundle
1

Build Failed
Error: RubyBundlerBuilder:RubyBundle - Bundler Failed: [DEPRECATED] The `--without` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local without 'development test'`, and stop using this flag
Don't run Bundler as root. Installing your bundle as root will break this application for all
non-root users on this machine.
--- ERROR REPORT TEMPLATE -------------------------------------------------------

LoadError: /root/me/dist/libz.so.1: version `ZLIB_1.2.9' not found (required by /root/.rbenv/versions/3.2.2/lib/ruby/3.2.0/x86_64-linux/zlib.so) - /root/.rbenv/versions/3.2.2/lib/ruby/3.2.0/x86_64-linux/zlib.so
  <internal:/root/.rbenv/versions/3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  <internal:/root/.rbenv/versions/3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  ...
  <more stack trace>
  ...

<ruby Github issue template output>

If I remove the library file that is included with the packaged version, it'll pick up the system installed version and build fine. The same happens if I just cp /lib/x86_64-linux-gnu/libz.so.1.2.11 ~/me/dist/libz.so.1 to replace the provided version with the version that the container has in /lib.

root@5d3a15a6840d:/lambda# ~/./me/dist/sam-beta build
Starting Build use cache
Manifest file is changed (new hash: def7aa28dda73cf76958fb58dbaac709) or dependency folder
(.aws-sam/deps/62e4fe44-f976-4bbe-9bd5-1b5a097d4739) is missing for (TestUpgradeRubyFunction),
downloading dependencies and copying/building source
Building codeuri: /lambda/src runtime: ruby3.2 metadata: {} architecture: x86_64 functions:
TestUpgradeRubyFunction
 Running RubyBundlerBuilder:CopySource
 Running RubyBundlerBuilder:RubyBundle
 Running RubyBundlerBuilder:RubyBundleDeployment
 Running RubyBundlerBuilder:CleanUp
 Running RubyBundlerBuilder:CopyDependencies

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided

Some more work will have to be done to see what we can do about the library we package (if we can remove it, or update or, or potentially something else). In the mean time, we can see about fixing the output caught by the workflow to capture the error message. As a workaround for now when using the packaged version of SAM CLI, using -u/--use-container to build the project is working fine for the time being.

[theist]

Thanks for digging into it! I found it quite weird and was afraid that was something only happening on my setup.

I guess we could try to use --use-container meanwhile. It is not just me, my company uses sam command line tool to develop quick lambdas in ruby and we were updating to 3.2 now that it is supported.

Do you think the next packaged version will get rid of the problem?

[sidhujus]

Hi, I looked into the issue a little bit as well and it appears to be that the zlib library that is coming with the ruby version provided in the image seems to have an issue which is causing the error. I tried replicating the error on my own desktop with the same versions and was unable to, but if i copy the zlib.so library located at /root/.rbenv/versions/3.2.2/lib/ruby/3.2.0/x86_64-linux in the container to my local ruby install I am able to replicate the error. If I then take my ruby installs version of that file and replace the one in the container with it then the build succeeds. All of this leads me to think that theres most likely something strange happening with ruby during its installation in this specific case.

There will need to be more work done to determine if this is caused by something unique happening during the ruby install on Ubuntu compared to other OS's or if its due to something else.

[sidhujus]

Closing for now as it seems to be a configuration issue rather than a bug within SAM CLI. Feel free to open another issue if you have any other questions

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[JamesKyburz]

I can't deploy a ruby hello world lambda on arm using sam and GitHub actions. I don't think it's a configuration issue.

Using the latest version of SAM cli, and also got version `ZLIB_1.2.9' not found

[theist]

@JamesKyburz according this most likely the devs won't notice unless the issue is reopened.

In my case I was forced to use --use-container and it worked ok for me that way, but I guess it won't solve yours in your env as it needs the env to be able to spin up containers. Hope it helps