Using curl to retrieve a HTTPS url that requires a self-signed certificate fails on macOS

[DUOLabs333]

Describe the bug

Let's say you have a website at https://testing.local, and you need to set your DNS server to w.x.y.z in the Preferences Pane to access it. curl https://testing.local will fail with curl: (35) OpenSSL/3.0.14: error:16000069:STORE routines::unregistered scheme.

Steps To Reproduce

Steps to reproduce the behavior:

1. Run nix-shell with

# shell.nix
let
  pkgs = import  <nixpkgs> {};
in
  pkgs.mkShell {
    packages = [ pkgs.git pkgs.curl ];
    # ...
  }

2. Run a server on a.b.c.d (it doesn't matter what it is)
3. Set up a DNS server such that foo.bar resolves to a.b.c.d
4. Add this DNS server to your DNS settings in the Preferences Pane
5. Run curl https://testing.local
6. See error

Expected behavior

There should be no error when running the command.

Screenshots

N/A

Additional context

Interestingly, curl https://a.b.c.d works as expected.

This seems to be a certificate issue: bypassing SSL certificate verification with -k fixes the issue. I am using a custom SSL certificate, but I'm not sure how to point NIX to it.

After some digging, I found that this has been brought up before: #283793

Notify maintainers

@lovek323

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 20.4.0, macOS 11.3.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/Users/system/.nix-defexpr/channels/nixpkgs`

---

Add a 👍 reaction to issues you find important.

[DUOLabs333]

I got around this issue by exporting the keychains in Keychain to a single PEM-encoded PEM file, and exporting the path to NIX_SSL_CERT_FILE and GIT_SSL_CAINFO (since git uses libcurl, at least by default):

rm ${keychainPath}; export PATH="${builtins.getEnv "PATH" }:$PATH"; security list-keychains | xargs -I{} sh -c 'security find-certificate -a -p "{}" >> ${keychainPath}; cat ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt >> ${keychainPath}'

Adjust as needed.

[vinylen]

I also experienced this with a x86 system, adding the -k flag solves it but, this has never been an issue in other distributions with curl for me. Is there something that needs to be looked at here? Or are we missing something critical in our NixOS systems for Curl?

I think this should be reopened.

❯ curl -V
curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 OpenSSL/3.0.14 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.61.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zst

- system: `"x86_64-linux"`
 - host os: `Linux 6.10.8, NixOS, 24.05 (Uakari), 24.05.20240907.68e7dce`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - nixpkgs: `/nix/store/32y945fdcfh1j5b8zv677vw7np9r7bbw-source`

EDIT: I might have been too hasty, i found this issue upstream, so it's not specific to NixOS.

If i modify the curl command to specifically use /etc/ssl/certs/ca-bundle.crt the error is more human readable:

CURL_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt curl https://a.b.x.d
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

EDIT 2:

Interestingly i don't get this error when i use the package "curlWithgnuTls":

nix-shell -p curlWithGnuTls
these 8 paths will be fetched (0.84 MiB download, 3.69 MiB unpacked):
  /nix/store/3iiwbpn2ll9z9k5dzm4a20hh4gf60glh-curl-8.7.1
  /nix/store/npj6a06hjihxswcc8iaqfxw4mhb0q8kc-curl-8.7.1-bin
  /nix/store/vx4rvsk3g35qgk434c71krfvrlfrkzvh-curl-8.7.1-dev
  /nix/store/ai64cb372lg4gqd8lp5p3y4n31d7xf8l-curl-8.7.1-man
  /nix/store/aakpck7mmlzf9m9sf1hkp3bjkm90kxnq-gmp-with-cxx-6.3.0-dev
  /nix/store/k8bifycm95cvr8fd3zd52j69kp9kf8cs-gnutls-3.8.5-bin
  /nix/store/ch2f11z7bwi856zclmly123dmwxa2v96-gnutls-3.8.5-dev
  /nix/store/1k7cfhsdmfnbp560x3vr8qs4x2rxyj85-nettle-3.9.1-dev
copying path '/nix/store/ai64cb372lg4gqd8lp5p3y4n31d7xf8l-curl-8.7.1-man' from 'https://cache.nixos.org'...
copying path '/nix/store/3iiwbpn2ll9z9k5dzm4a20hh4gf60glh-curl-8.7.1' from 'https://cache.nixos.org'...
copying path '/nix/store/k8bifycm95cvr8fd3zd52j69kp9kf8cs-gnutls-3.8.5-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/aakpck7mmlzf9m9sf1hkp3bjkm90kxnq-gmp-with-cxx-6.3.0-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/1k7cfhsdmfnbp560x3vr8qs4x2rxyj85-nettle-3.9.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/npj6a06hjihxswcc8iaqfxw4mhb0q8kc-curl-8.7.1-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/ch2f11z7bwi856zclmly123dmwxa2v96-gnutls-3.8.5-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/vx4rvsk3g35qgk434c71krfvrlfrkzvh-curl-8.7.1-dev' from 'https://cache.nixos.org'...

[nix-shell:/]$ curl https://a.b.x.d
curl: (60) server certificate verification failed. CAfile: none CRLfile: none
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[lf-]

I believe that this is caused by:

• Apple OpenSSL having some kind of undocumented patches that only affect openssl s_client and presumably curl when using OpenSSL that cause it to use the keychain. These don't affect openssl verify.
• Apple curl also supporting using SecureTransport, which also uses the keychain for certificates.

A fix here would be to possibly grab Apple's patches, assuming that they are open source (uh. hopefully.) and use them on macOS in nixpkgs, or to configure nixpkgs curl to use SecureTransport. Both of these seem okay to me at least.

[lf-]

diff --git a/pkgs/by-name/cu/curlMinimal/package.nix b/pkgs/by-name/cu/curlMinimal/package.nix
index 22aa1e4d64f3..6e3e22229620 100644
--- a/pkgs/by-name/cu/curlMinimal/package.nix
+++ b/pkgs/by-name/cu/curlMinimal/package.nix
@@ -24,6 +24,7 @@
 , scpSupport ? zlibSupport && !stdenv.hostPlatform.isSunOS && !stdenv.hostPlatform.isCygwin, libssh2
 , wolfsslSupport ? false, wolfssl
 , rustlsSupport ? false, rustls-ffi
+, secureTransportSupport ? false
 , zlibSupport ? true, zlib
 , zstdSupport ? false, zstd
 
@@ -46,6 +47,7 @@
 # files.
 
 assert !((lib.count (x: x) [ gnutlsSupport opensslSupport wolfsslSupport rustlsSupport ]) > 1);
+assert secureTransportSupport -> stdenv.hostPlatform.isDarwin;
 
 let
   openssl' = if http3Support then quictls else openssl;
@@ -149,6 +151,7 @@ stdenv.mkDerivation (finalAttrs: {
       (lib.withFeature http3Support "ngtcp2")
       (lib.withFeature rtmpSupport "librtmp")
       (lib.withFeature rustlsSupport "rustls")
+      (lib.withFeature secureTransportSupport "secure-transport")
       (lib.withFeature zstdSupport "zstd")
       (lib.withFeature pslSupport "libpsl")
       (lib.withFeatureAs brotliSupport "brotli" (lib.getDev brotli))
@@ -167,7 +170,7 @@ stdenv.mkDerivation (finalAttrs: {
       # Without this curl might detect /etc/ssl/cert.pem at build time on macOS, causing curl to ignore NIX_SSL_CERT_FILE.
       "--without-ca-bundle"
       "--without-ca-path"
-    ] ++ lib.optionals (!gnutlsSupport && !opensslSupport && !wolfsslSupport && !rustlsSupport) [
+    ] ++ lib.optionals (!gnutlsSupport && !opensslSupport && !wolfsslSupport && !rustlsSupport && !secureTransportSupport) [
       "--without-ssl"
     ] ++ lib.optionals (rustlsSupport && !stdenv.hostPlatform.isDarwin) [
       "--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt"

Some very lazy patching causes CURL_SSL_BACKEND=secure-transport to use the system keychain for certs, which is cute. Well, except for the part where it doesn't support TLS 1.3, according to the configure script's output. Incredible.

[OliverJAsh]

In case it helps: I have exactly the same issue except the error message is different:

$ curl "https://unsplash.localhost"
curl: (35) TLS connect error: error:80000002:system library::No such file or directory

[lf-]

@OliverJAsh that's this bug fix: #351920