BUG/MEDIUM: ssl: chosing correct certificate using RSA-PSS with TLSv1.3

The clienthello callback was written when TLSv1.3 was not yet out, and
signatures algorithm changed since then.

With TLSv1.2, the least significant byte was used to determine the
SignatureAlgorithm, which could be rsa(1), dsa(2), ecdsa(3).
https://datatracker.ietf.org/doc/html/rfc5246#section-7.4.1.4.1

This was used to chose which type of certificate to push to the client.

But TLSv1.3 changed that, and introduced new RSA-PSS algorithms that
does not have the least sinificant byte to 1.
https://datatracker.ietf.org/doc/html/rfc8446#section-4.2.3

This would result in chosing the wrong certificate when an RSA an ECDSA
ones are in the configuration for the same SNI or default entry.

This patch fixes the issue by parsing bothe hash and signature field to
check the RSA-PSS signature scheme.

This must fix issue haproxy#2852.

This must be backported in every stable versions. The code was moved
from ssl_sock.c to ssl_clienthello in recent versions.

src/ssl_clienthello.c

@@ -259,6 +259,7 @@ int ssl_sock_switchctx_cbk(SSL *ssl, int *al, void *arg)
 	if (SSL_client_hello_get0_ext(ssl, TLSEXT_TYPE_signature_algorithms, &extension_data, &extension_len)) {
 #endif
 		uint8_t sign;
+		uint8_t hash;
 		size_t len;
 		if (extension_len < 2)
 			goto abort;
@@ -269,7 +270,7 @@ int ssl_sock_switchctx_cbk(SSL *ssl, int *al, void *arg)
 		if (len % 2 != 0)
 			goto abort;
 		for (; len > 0; len -= 2) {
-			extension_data++; /* hash */
+			hash = *extension_data++; /* hash */
 			sign = *extension_data++;
 			switch (sign) {
 			case TLSEXT_signature_rsa:
@@ -278,6 +279,18 @@ int ssl_sock_switchctx_cbk(SSL *ssl, int *al, void *arg)
 			case TLSEXT_signature_ecdsa:
 				has_ecdsa_sig = 1;
 				break;
+			case 0x04:
+			case 0x05:
+			case 0x06:
+			case 0x09:
+			case 0x0a:
+			case 0x0b:
+				/* match the RSA-PSS sigalgs from TLSv1.3
+				 * https://datatracker.ietf.org/doc/html/rfc8446#section-4.2.3
+				 */
+				if (hash == 0x08)
+					has_rsa_sig = 1;
+				break;
 			default:
 				continue;
 			}