[8.x](backport #6568) Fix Fleet Enrollment Handling for Containerized Agent

[mergify]

What does this PR do?

This PR introduces a fix for containerized Fleet-managed Elastic Agents to handle scenarios where:

• The Fleet url changes.
• The Fleet enrollment token changes.
• The agent is unenrolled from Kibana Fleet.

It achieves the above by enhancing the agent's logic inside the container cmd to:

• Verify enrollment conditions against stored state and only re-enroll when necessary.
• Validate API token validity with Fleet, ensuring correct authentication even when the configuration hasn't changed.

The PR also adds a Kubernetes integration tests to verify the proper behaviour of enrollment and re-enrollment under various scenarios, including:

• Re-deployment of agent with an updated fleet enrollment token.
• Re-deployment of agent that is unenrolled from Kibana Fleet with the same enrollment token.
• Re-deployment of agent across an older version and this one that ensures that no re-enrollment happens.

Key changes include:

• Introduction of shouldFleetEnroll to centralize logic for fleet enrollment decisions.
• Creation of PBKDF2-based enrollment token hashing to save in the agent state and be able to track token changes securely.
• Utilise the agent/{id}/acks path of Fleet server with empty events to check if the agent API token is still valid. More than happy to introduce a separate path just for this cause on Fleet server, although it will be the same the ACKs one with empty events (as of now at least)

You can easily see here in the CI run of the first commit in this PR that the Elastic Agent, before this PR, doesn't handle enrollment correctly and always resorts to using the Fleet token and URL stored in its state. This PR addresses that issue and ensures enrollment uses the correct configuration and token.

PS: the actual changes of this PR are this commit b6596d0 which is +305 -25 thus I consider this PR aligned with the team policies 🙂

Why is it important?

This fix ensures robust and predictable behavior for containerized Fleet-managed Elastic Agents and enhance user experience of managing elastic-agent in Kubernetes.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

The changes introduced in this PR are non-disruptive. They improve fleet enrollment handling and maintain backward compatibility.

How to test this PR locally

mage integration:auth
PLATFORMS=linux/arm64 EXTERNAL=true SNAPSHOT=true PACKAGES=docker mage -v package 
INSTANCE_PROVISIONER=kind STACK_PROVISIONER=stateful K8S_VERSION=v1.31.1 SNAPSHOT=true mage integration:kubernetes

Related issues

• Closes Elastic Agent doesn't update the enrollment token in Kubernetes Deployment statefile #3586

---

This is an automatic backport of pull request #6568 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
73.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube