Add mTLS flags to install and enroll commands

[michel-laterman]

What does this PR do?

Add mTLS flags to install and enroll commands

Why is it important?

Certain proxy setups for customer's on-prem deployments require mTLS certificates for communication.

How to test

I created custom builds of the elastic-agent and fleet-server using a currently unreleased version of elastic-agent-libs with ability to specify ServerConfig.ClientAuth.
elastic-agent-libs has been updated to v0.7.4 which includes these changes

Create a ES cluster and enroll elastic-agent with fleet-server.
I used a self-signed CA to create all the certs:

sudo ./elastic-agent install --fleet-server-es=[REMOTE-ADDRESS] \
--fleet-server-service-token=[TOKEN] \
--fleet-server-policy=fleet-server-policy  \
--fleet-server-port=8220 \
--url=[URL] \
--certificate-authorities=/etc/ssl/certs/ca-certificates.crt,/tmp/ca.crt \
--fleet-server-es-ca=/etc/ssl/certs/ca-certificates.crt \
--fleet-server-cert=/tmp/fleet-server.crt \
--fleet-server-cert-key=/tmp/fleet-server.key \
--elastic-agent-cert=/tmp/fleet-server.crt \
--elastic-agent-cert-key=/tmp/fleet-server.key \
--fleet-server-es-cert=/tmp/fleet-server.crt \
--fleet-server-es-cert-key=/tmp/fleet-server.key \
--fleet-server-client-auth=required

Gather a diagnostics bundle and inspect fleet-server.yml

...
      server:
        host: 0.0.0.0
        port: 8220
        internalport: 8221
        tls:
            certificate_authorities:
                - /etc/ssl/certs/ca-certificates.crt
                - /tmp/ca.crt
            certificate: /tmp/fleet-server.crt
            key: '[redacted]'
            client_authentication: required

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation: Add agent mTLS flags ingest-docs#816
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[cmacknz]

Is there a way to test that an mTLS exchange between agent and Fleet server actually works?

We have a test fleet server you could configure to test the agent side of this, or just use a real fleet server if that is simpler: https://github.com/elastic/elastic-agent/tree/main/testing/fleetservertest

Example usage:

elastic-agent/testing/integration/proxy_url_test.go

Line 277 in bc3bbfd

action, err := fleetservertest.NewActionWithEmptyPolicyChange(

[michel-laterman]

@cmacknz, i'm trying to test that locally. However there are issues starting an agent + fleet-server instance with some of the changes I made, i'm going to set this as a draft and see if I can fix it.
I set the logs to debug and I can see:

{"log.level":"debug","@timestamp":"2024-01-05T21:23:21.589Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":798},"message":"Waiting for Elastic Agent to start: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/data/tmp/elastic-agent-control.sock: connect: connection refused\"","ecs.version":"1.6.0"}

[michel-laterman]

After changing the version of elastic-agent-libs to one that includes the changes in elastic/elastic-agent-libs#176, the agent is able to pass the cert + key to the output (for es) and mTLS to connect. The local config contains:

    server:
        ssl:
            certificate: <REDACTED>
            certificate_authorities:
                - /etc/ssl/certs/ca-certificates.crt
                - /tmp/ca.crt
            client_authentication: required
            key: <REDACTED>
    ssl:
        certificate: <REDACTED>
        certificate_authorities:
            - /etc/ssl/certs/ca-certificates.crt
            - /tmp/ca.crt
        key: <REDACTED>
        renegotiation: never

And fleet-server's config has:

        tls:
            enabled: null
            verificationmode: full
            versions: []
            ciphersuites: []
            cas:
                - /etc/ssl/certs/ca-certificates.crt
                - /tmp/ca.crt
            certificate:
                certificate: /tmp/fleet-server.crt
                key: '[redacted]'
            curvetypes: []
            clientauth: 4 # corresponds to required

However, I would expect client_auth to be set to none as no arg was passed for this example, so I still need to figure out what's going in there (but at least we know mtls is fully functional)

[AndersonQ]

@michel-laterman I tried to test it but I could not.

I don't understand how I'd install fleet-server under agent and also pass in --url=[URL]. Which URL did you pass?

[michel-laterman]

The --url is the same URL that you specify when adding a new fleet-server (it is listed as an arg when adding a fleet-server with advanced options).

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mtls-flags upstream/mtls-flags
git merge upstream/main
git push upstream mtls-flags

[elastic-sonarqube]

Quality Gate passed

The SonarQube Quality Gate passed, but some issues were introduced.

9 New issues
0 Security Hotspots
51.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube