XCode Command Line Tools 12.2 breaks gem installation due to inclusion of 11.0 SDK

[jcshort]

Bug report

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

Run brew audit jmeter as part of a version bump PR Homebrew/homebrew-core#66170

What happened (include command output)

Command output

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/byebug-11.1.3/ext/byebug

/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby -I /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0 -r ./siteconf20201203-67233-9ex9br.rb extconf.rb

creating Makefile
current directory: /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/byebug-11.1.3/ext/byebug

make "DESTDIR=" clean
current directory: /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/byebug-11.1.3/ext/byebug

make "DESTDIR="

make: *** No rule to make target /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/include/ruby-2.6.0/universal-darwin19/ruby/config.h', needed by breakpoint.o'.  Stop.
make failed, exit code 2
Gem files will remain installed in /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/byebug-11.1.3 for inspection.

Results logged to /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/extensions/universal-darwin-19/2.6.0/byebug-11.1.3/gem_make.out
An error occurred while installing byebug (11.1.3), and Bundler cannot continue.

Make sure that gem install byebug -v '11.1.3' --source 'https://rubygems.org/' succeeds before bundling.
In Gemfile:

byebug

What you expected to happen

Gems build against 10.15 SDK

Step-by-step reproduction instructions (by running brew install commands)

sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install
rm -rf /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/byebug-11.1.3
brew audit <formula>

byebug is of course not the only gem requiring compilation, but it's the first.

brew config output:

HOMEBREW_VERSION: 2.6.0-68-g5ad9496
ORIGIN: https://github.com/Homebrew/brew
HEAD: 5ad949672f1fd9ce900e1196a0ad2c020f869480
Last commit: 55 minutes ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 94f2606854535a6ad4c8009f0b7ac2b2d30cd820
Core tap last commit: 2 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.CeFH5FvPg6/org.macosforge.xquartz:0
HOMEBREW_EDITOR: /usr/bin/vim
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 12.0 build 1200
Git: 2.29.2 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 15.0.1
macOS: 10.15.7-x86_64
CLT: 12.2.0.0.1.1604628099
Xcode: 12.2
XQuartz: 2.7.11 => /opt/X11

brew doctor output:

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed with `brew diy`.
You should find replacements for the following formulae:
  gem-gemfury
  gem-fpm
  gem-mdl
  gem-rubocop

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  helm@2

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/redacted/.pyenv/shims/python3.9-config
  /Users/redacted/.pyenv/shims/python-config
  /Users/redacted/.pyenv/shims/python3-config
  /Users/redacted/.pyenv/shims/ansible-config
  /Users/redacted/.pyenv/shims/python3.8-config

Warning: Putting non-prefixed coreutils in your path can cause gmp builds to fail.

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  [email protected]

[jcshort]

Also of note: Command Line Tools 12.2 delivers 10.15 and 11.0 SDKs thusly:

/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
/Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk

and /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk symlinked to 11.0, which seems out of place on a 10.15 system.

Re-linking MacOSX.sdk to point at MacOSX10.15.sdk did not resolve the issue, only moving/deleting the MacOSX11.0.sdk directory entirely. Somewhere between homebrew and gem itself, SDK path discovery is doing the wrong thing

[jcshort]

and finally, relevant details from softwareupdate --history:

Command Line Tools for Xcode                       12.2       12/03/2020, 10:20:55

[MikeMcQuaid]

@jcshort Can we get the output from brew ruby -e 'puts MacOS.sdk_path'?

[MikeMcQuaid]

@jcshort And also the output after running export HOMEBREW_UPDATE_TO_TAG=1; brew update; brew config; brew ruby -e 'puts MacOS.sdk_path'

[mistydemeo]

Got it. The problem is not SDK selection; the problem is how Ruby finds its own headers.

The core error from Ruby's mkmf is

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/include/ruby-2.6.0/ruby/ruby.h:24:10: fatal error: 'ruby/config.h' file not found
#include "ruby/config.h"
         ^~~~~~~~~~~~~~~

The path for that is set inside mkmf.rb as '$(arch_hdrdir)/ruby/config.h', and arch_hdrdir is defined as RbConfig::CONFIG["rubyarchhdrdir"]. On 10.15 with an Xcode 2.2 SDK (Xcode or CLT), that value is /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/include/ruby-2.6.0/universal-darwin19. Unfortunately, as you pointed out, MacOSX.sdk is a symlink to the 11.0 SDK, and since the OS version is hardcoded in Ruby's include paths, no header exists at that path on 10.15. (macOS 11 has a header inside universal-darwin20.)

Arguably this is a bug in the CLT/Xcode packaging, but it's not likely to be fixed there. Unfortunately Ruby provides no way for us to override these variables from outside, and obviously we can't modify the system Ruby. This is probably a sign that we need to start using our vendored Ruby on macOS 10.15, even though it supplies a Ruby 2.6.

[leipert]

Experiencing this over at https://github.com/leipert/jetbrains-cask-bot as well:.
e.g.: https://github.com/leipert/jetbrains-cask-bot/runs/1495381102?check_suite_focus=true

I assume that the same will likely happen on any step in GH actions that does brew install-bundler-gems and isn't cached.

[mistydemeo]

A friend pointed out that a combination of a shim file and RUBYOPT can be used to monkey patch values in RbConfig. I'm investigating that as a potential fix. https://gist.github.com/tenderlove/6e0c2ca4f7edf8aded4f148ff9987bf4

[fxcoudert]

I am hitting this after updating the VM image of our Catalina nodes. This is currently blocking relaunch of CI on Catalina.

[fxcoudert]

Currently forcing HOMEBREW_FORCE_VENDOR_RUBY=1 on Catalina CI nodes, to work around the issue.

[mistydemeo]

I'm not sure if this is fixed yet. I'm still getting the system Ruby because the call to ruby_check_version_script is still considering the OS's Ruby to be new enough despite us having unset HOMEBREW_MACOS_SYSTEM_RUBY_NEW_ENOUGH: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/utils/ruby.sh#L9-L11

[mistydemeo]

Reopning. We've confirmed via brewsci/homebrew-bio#1219 that this is not fixed, unfortunately.

[fxcoudert]

Same on core: Homebrew/homebrew-core#66439

[mistydemeo]

I've merged the fix. If we want to ensure different kinds of fixes going forward we can reevaluate, but I wanted us to unblock the many users who are impacted by this.