Merge remote-tracking branch 'origin/main' into stephen/zstd_logs

Signed-off-by: Stephen Wakely <[email protected]>

.github/CODEOWNERS

@@ -547,6 +547,7 @@
 /pkg/tagger/                            @DataDog/container-platform
 /pkg/windowsdriver/                     @DataDog/windows-kernel-integrations
 /comp/core/workloadmeta/collectors/internal/cloudfoundry @DataDog/platform-integrations
+/comp/core/workloadmeta/collectors/internal/nvml @DataDog/ebpf-platform
 /pkg/sbom/                              @DataDog/container-integrations @DataDog/agent-security
 /pkg/internaltelemetry                  @DataDog/windows-kernel-integrations @DataDog/fleet
 /pkg/networkpath/                       @DataDog/network-device-monitoring @DataDog/Networks

.gitlab-ci.yml

@@ -1170,6 +1170,7 @@ workflow:
         - pkg/gpu/**/*
         - test/new-e2e/tests/gpu/**/*
         - pkg/collector/corechecks/gpu/**/*
+        - comp/core/workloadmeta/collectors/internal/nvml/**/*
       compare_to: main # TODO: use a variable, when this is supported https://gitlab.com/gitlab-org/gitlab/-/issues/369916
 
 # windows_docker_2022 configures the job to use the Windows Server 2022 runners.

.gitlab/e2e_install_packages/windows.yml

@@ -58,8 +58,6 @@
       - E2E_MSI_TEST: TestNPMInstallWithAddLocal
       - E2E_MSI_TEST: TestNPMUpgradeFromBeta
       - E2E_MSI_TEST: TestUpgradeFromV6
-      - E2E_MSI_TEST: TestFIPSAgentDoesNotInstallOverAgent
-      - E2E_MSI_TEST: TestAgentDoesNotInstallOverFIPSAgent
 
 new-e2e_windows_powershell_module_test:
   extends: .new_e2e_template
@@ -116,7 +114,13 @@ new-e2e-windows-agent-a7-x86_64-fips:
     - .new-e2e_agent_a7
   needs:
     - !reference [.needs_new_e2e_template]
+    - deploy_windows_testing-a7
     - deploy_windows_testing-a7-fips
+  parallel:
+    matrix:
+      - EXTRA_PARAMS: --run "TestFIPSAgent$"
+      - EXTRA_PARAMS: --run "TestFIPSAgentDoesNotInstallOverAgent$"
+      - EXTRA_PARAMS: --run "TestAgentDoesNotInstallOverFIPSAgent$"
   rules:
     - !reference [.on_deploy]
     - !reference [.on_e2e_or_windows_installer_changes]

CHANGELOG-DCA.rst

@@ -2,6 +2,51 @@
 Release Notes
 =============
 
+.. _Release Notes_7.61.0:
+
+7.61.0
+======
+
+.. _Release Notes_7.61.0_Prelude:
+
+Prelude
+-------
+
+Released on: 2025-01-13
+Pinned to datadog-agent v7.61.0: `CHANGELOG <https://github.com/DataDog/datadog-agent/blob/main/CHANGELOG.rst#7610>`_.
+
+.. _Release Notes_7.61.0_New Features:
+
+New Features
+------------
+
+- Implements the Kubernetes Admission Events webhooks. This new webhooks will emit Datadog Events
+  when receving Validation Admission requests. It will track deployments operations made by non-system
+  users.
+  The webhook is controlled by using the `admission_controller.kubernetes_admission_events.enabled` setting.
+
+- The cluster-agent now can collect pod disruption budgets from the cluster.
+
+
+.. _Release Notes_7.61.0_Enhancement Notes:
+
+Enhancement Notes
+-----------------
+
+- Cluster Agent: ``DatadogAgent`` custom resource, cluster Agent deployment, and node Agent daemonset manifests are now added to the flare archive when the Cluster Agent is deployed with the Datadog Operator (version 1.11.0+).
+
+- Cluster Agent: Don't overwrite the LD_PRELOAD environment variable if it's already set, append the path to Datadog's injection library instead.
+
+
+.. _Release Notes_7.61.0_Bug Fixes:
+
+Bug Fixes
+---------
+
+- The auto-instrumentation webhook no longer injects the default environment
+  variables when disabled.
+
+
 .. _Release Notes_7.60.1:
 
 7.60.1
@@ -148,7 +193,7 @@ Bug Fixes
 
 - Fixed an issue that prevented the Kubernetes autoscaler from evicting pods
   injected by the Admission Controller.
-  
+
 
 .. _Release Notes_7.57.1:
 
@@ -206,7 +251,7 @@ Bug Fixes
 
 - Library package versions for auto-instrumentation are now set to the latest major
   version of the library-package instead of `latest`.
-  
+
   * java:v1
   * dotnet:v2
   * python:v2

CHANGELOG.rst

@@ -2,6 +2,205 @@
 Release Notes
 =============
 
+.. _Release Notes_7.61.0:
+
+7.61.0
+======
+
+.. _Release Notes_7.61.0_Prelude:
+
+Prelude
+-------
+
+Release on: 2025-01-13
+
+- Please refer to the `7.61.0 tag on integrations-core <https://github.com/DataDog/integrations-core/blob/master/AGENT_CHANGELOG.md#datadog-agent-version-7610>`_ for the list of changes on the Core Checks
+
+
+.. _Release Notes_7.61.0_Upgrade Notes:
+
+Upgrade Notes
+-------------
+
+- Upgraded JMXFetch to `0.49.6 <https://github.com/DataDog/jmxfetch/releases/0.49.6>`_ which fixes a ``NullPointerException`` on
+  JBoss when user and password not set. See `0.49.6 <https://github.com/DataDog/jmxfetch/releases/tag/0.49.6>`_ for more details.
+
+- Windows containers were updated to use OpenJDK 11.0.25+9.
+
+
+.. _Release Notes_7.61.0_New Features:
+
+New Features
+------------
+
+- Add metrics origins for Nvidia Nim integration.
+
+- APM: New configuration apm_config.obfuscation.credit_cards.keep_values (DD_APM_OBFUSCATION_CREDIT_CARDS_KEEP_VALUES)
+  can be used to skip specific tag keys that are known to never contain credit card numbers. This is especially useful
+  in cases where a span tag value is a number that triggers false positives from the credit card obfuscator.
+
+- Add new metric, ``container.restarts``, which indicates the number of times a container has been restarted due to the restart policy.
+  For more details: https://docs.docker.com/engine/containers/start-containers-automatically/.
+
+- APM: Introducing the Error Tracking Standalone config option. Only span chunks
+  that contain errors or exception OpenTelemetry span events are taken into
+  consideration by sampling.
+
+- Add new windows images for LTSC 2019 and LTSC 2022:
+  - `datadog-agent:7-servercore-ltsc2019-amd64`
+  - `datadog-agent:7-servercore-ltsc2022-amd64`
+  - `datadog-agent:7-servercore-ltsc2019-jmx-amd64`
+  - `datadog-agent:7-servercore-ltsc2022-jmx-amd64`
+  - `datadog-agent:latest-servercore-ltsc2019-jmx`
+  - `datadog-agent:latest-servercore-ltsc2022-jmx`
+  - `datadog-agent:latest-servercore-ltsc2019`
+  - `datadog-agent:latest-servercore-ltsc2022`
+  - `datadog-agent:7.X.Y-ltsc2019`
+  - `datadog-agent:7.X.Y-ltsc2022`
+  - `datadog-agent:7.X.Y-ltsc2019-jmx`
+  - `datadog-agent:7.X.Y-ltsc2022-jmx`
+  - `datadog-agent:7.X.Y-servercore-ltsc2019`
+  - `datadog-agent:7.X.Y-servercore-ltsc2022`
+  - `datadog-agent:7.X.Y-servercore-ltsc2019-jmx`
+  - `datadog-agent:7.X.Y-servercore-ltsc2022-jmx`
+  - `datadog-agent:latest-ltsc2019`
+  - `datadog-agent:latest-ltsc2022`
+
+- [ha-agent] Add haagent component used for HA Agent feature.
+
+- Added support for collecting container image metadata when running on a CRI-O runtime.
+
+- USM now monitors TLS traffic encrypted with Go TLS by default.
+  To disable this feature, set the `service_monitoring_config.tls.go.enabled`
+  configuration option to false.
+
+- USM now monitors traffic encrypted with Istio mTLS by default.
+  To disable this feature, set the `service_monitoring_config.tls.istio.enabled` configuration option to false.
+
+- Introduced a new configuration variable `logs_config.http_protocol`, allowing users to enforce HTTP/1.1 for outgoing HTTP connections in the Datadog Agent. This provides better control over transport protocols and improves compatibility with systems that do not support HTTP/2.
+  By default, the log agent will now attempt to use HTTP/2 (unless a proxy is configured) and fall back to the best available protocol if HTTP/2 is not supported.
+
+- Added a new feature flag `enable_operation_and_resource_name_logic_v2` in DD_APM_FEATURES. Enabling this flag modifies the logic for computing operation and resource names from OTLP spans to produce shorter, more readable names and improve alignment with OpenTelemetry specifications.
+
+- Add support for PHP Single Step Instrumentation in Kubernetes (not enabled by default)
+
+
+.. _Release Notes_7.61.0_Enhancement Notes:
+
+Enhancement Notes
+-----------------
+
+- [ha-agent] Run HA enabled integrations only on leader Agent
+
+- [ha-agent] Add agent_group tag to datadog.agent.running metric
+
+- Add new host tag `provider_kind` from the value of `DD_PROVIDER_KIND` for Agents running in GCE.
+
+- Add ``query_timeout`` to customize the timeout for queries in the Oracle check.
+  Previously, this was fixed at 20,000 seconds.
+
+- Add ability to show Agent telemetry payloads to be sent by Agent
+  if the telemetry is enabled. One can run it with the following command:
+     `agent diagnose show-metadata agent-telemetry`. See
+  `docs <https://docs.datadoghq.com/data_security/agent/#telemetry-collection>` for more details.
+
+- Convert Prometheus style Counters and Histograms used in Agent telemetry
+  from monotonically increasing to non-monotonic values (reset on each scrape).
+  In addition de-accumulate Prometheus Histogram bucket values on each scrape.
+
+- Added support for more than 100 Aurora clusters in a user's account when using database autodiscovery
+
+- Adds some information about the SNMP autodiscovery status in the Agent status.
+
+- Adds a dedicated CRI-O Workloadmeta collector, enabling metadata collection
+  for containers running on a CRI-O runtime.
+
+- Enables a cache for SQL and MongoDB obfuscation. This cache is enabled by default but can be disabled by setting `apm_config.obfuscation.cache.enabled` to `false`.
+
+- Improved logging to add visibility for latency and transport protocol
+
+- Add a new configuration option ``log_level`` for commands where the logger is disabled by default.
+
+- Adds initial Windows support for TCP probes in Network Path.
+
+- Query Aurora instances per cluster to allow up to 100 instances per cluster
+  rather than 100 instances total.
+
+- The AWS Lambda Extension is now  able to read the full 128-bit trace ID
+  from the headers of the end-invocation HTTP request made by dd-trace or the
+  datadog-lambda-go library.
+
+- Standardized cluster check tagging across all environments, allowing DD_TAGS, DD_EXTRA_TAGS, DD_CLUSTER_CHECKS_EXTRA_TAGS, and DD_ORCHESTRATOR_EXPLORER_EXTRA_TAGS to apply to all cluster check data when operating on the Cluster Agent, Node Agent, or Cluster Checks Runner.
+
+
+.. _Release Notes_7.61.0_Deprecation Notes:
+
+Deprecation Notes
+-----------------
+
+- Deprecates the `apm_config.obfuscation.sql.cache` option in favor of `apm_config.obfuscation.cache`.
+
+- Remove deprecated config `otlp_config.metrics.instrumentation_library_metadata_as_tags`. Use `otlp_config.metrics.instrumentation_scope_metadata_as_tags` instead.
+
+- The remote tagger will attempt to connect to the core agent indefinitely until it is successful.
+  The ``remote_tagger_timeout_seconds`` configuration is removed, and the timeout is no longer configurable.
+
+- The remote tagger for the trace-agent and security-agent is now always enabled and can not be disabled
+  ``apm_config.remote_tagger``,  ``security_agent.remote_tagger``, and ``event_monitoring_config.remote_tagger`` config entries are removed.
+
+
+.. _Release Notes_7.61.0_Security Notes:
+
+Security Notes
+--------------
+
+- Fix CVE-2025-21613
+
+- Update ``golang.org/x/crypto`` to fix CVE-2024-45337.
+
+
+.. _Release Notes_7.61.0_Bug Fixes:
+
+Bug Fixes
+---------
+
+- Fix an issue where the remote workloadmeta was not receiving some unset
+  events for ECS containers, causing incorrect billing in CWS, CSPM, CSM Pro, CSM
+  Enterprise, and DevSecOps Enterprise Containers.
+
+- Corrects the method call for gauges to be Set instead of Add.
+
+- Fix Oracle execution plan collection failures caused by an out-of-range position column, which can occur if the execution plan is excessively large.
+
+- Fix excessive number of rows coming from active session history.
+
+- OTLP ingestion: Stop prefixing `http_server_duration`, `http_server_request_size` and `http_server_response_size` with `otelcol`.
+
+- Fixes the issue of disabled services producing an error message in the event log on start. Now produces an informational message.
+
+- Change `kubernetes.memory.working_set` and `kubernetes.memory.usage`
+  metrics to be of type gauge instead of rate.
+
+
+.. _Release Notes_7.61.0_Other Notes:
+
+Other Notes
+-----------
+
+- Add metric origins for Platform Integrations: Fly.io, Kepler, Octopus Deploy, and Scaphandre.
+
+- Extend Agent Telemetry to start reporting ``logs.sender_latency`` metric.
+
+- The `enable_receive_resource_spans_v2` flag now defaults to true in Converged Agent. This enables the refactored
+  version of the OTLP span receiver in trace agent, improves performance by 10%, and deprecates the following functionality:
+  - No longer checks for information about the resource in HTTP headers (ContainerID, Lang, LangVersion, Interpreter, LangVendor).
+  - No longer checks for resource-related values (container, env, hostname) in span attributes. This previous behavior did not follow the OTel spec.
+
+- Bumps the default value for `kube_cache_sync_timeout_seconds` from 5 to 10 seconds.
+
+- Added origin for new Milvus integration.
+
+
 .. _Release Notes_7.60.1:
 
 7.60.1
@@ -44,11 +243,11 @@ Upgrade Notes
 -------------
 
 - * Parameter ``peer_tags_aggregation`` (a.k.a. environment variable ``DD_APM_PEER_TAGS_AGGREGATION``) is now enabled by default. This means that aggregation of peer related tags (e.g., `peer.service`, `db.instance`, etc.) now happens in the Agent, which enables statistics for Inferred Entities. If you want to disable this feature, set `peer_tags_aggregation` to `false` in your Agent configuration.
-  
+
   * Parameter ``compute_stats_by_span_kind`` (a.k.a. environment variable ``DD_APM_COMPUTE_STATS_BY_SPAN_KIND``) is now enabled by default. This means spans with an eligible `span.kind` will have stats computed. If disabled, only top-level and measured spans will have stats computed. If you want to disable this feature, set `compute_stats_by_span_kind` to `false` in your Agent configuration.
-  
+
     Note: When using ``peer_tags_aggregation`` and ``compute_stats_by_span_kind``, a high cardinality of peer tags or APM resources can contribute to higher CPU and memory consumption. If enabling both causes the Agent to consume too many resources, try disabling `compute_stats_by_span_kind` first.
-  
+
   It is recommended that you update your tracing libraries according to the instructions `here <https://docs.datadoghq.com/tracing/guide/inferred-service-opt-in/?tab=java#apm-tracing-library-configuration>`_ and set ``DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED`` (or ``dd.trace.remove.integration-service-names.enabled``) to ``true``.
 
 - Upgraded JMXFetch to `0.49.5 <https://github.com/DataDog/jmxfetch/releases/0.49.5>`_ which adds support for ``UnloadedClassCount`` metric
@@ -63,15 +262,15 @@ New Features
 - `Inferred Service dependencies <https://docs.datadoghq.com/tracing/guide/inferred-service-opt-in/>`_ are now Generally Available (exiting Beta) and enabled by default. Inferred Services of all kinds now have trace metrics and are available in dependency maps. `apm_config.peer_tags_aggregation` and `apm_config.compute_stats_by_span_kind` both now default to `true` unless explicitly set to `false`.
 
 - Add `check_tag_cardinality` parameter config check.
-  
+
   By default `check_tag_cardinality` is not set which doesn't change the behavior of the checks.
   Once it is set in pod annotaions, it overrides the cardinality value provided in the base agent configuration.
   Example of usage:
   ```yaml
   ad.datadoghq.com/redis.checks: |
     {
       "redisdb": {
-        "check_tag_cardinality": "high", 
+        "check_tag_cardinality": "high",
         "instances": [
           {
             "host": "%%host%%",
@@ -100,7 +299,7 @@ Enhancement Notes
   based paths in Network Path.
   A cache of reverse DNS lookups is used to reduce the number of DNS
   queries. Additionally, reverse DNS lookups are now performed only
-  for private IPs and not for public IPs. 
+  for private IPs and not for public IPs.
 
 - Agent flare now includes system-probe telemetry data via ``system-probe/system_probe_telemetry.log``.
 
@@ -235,7 +434,7 @@ Enhancement Notes
   information about the Datadog Agent. This may include diagnostic
   logs and crash dumps of the Datadog Agent with obfuscated stack
   traces to support and further improve the Datadog Agent.
-  
+
   More details could be found in the
   `docs <https://docs.datadoghq.com/data_security/agent/#telemetry-collection>`_
 
@@ -247,10 +446,10 @@ Enhancement Notes
 
 - Agents are now built with Go ``1.22.8``.
 
-- While using the AWS Lambda Extension, when a Lambda Function is invoked by 
+- While using the AWS Lambda Extension, when a Lambda Function is invoked by
   a [properly instrumented][1] Step Function, the Lambda Function will create
-  its Trace and Parent IDs deterministically based on the Step Function's 
-  execution context. 
+  its Trace and Parent IDs deterministically based on the Step Function's
+  execution context.
   [1]: https://docs.datadoghq.com/serverless/step_functions/installation/?tab=custom "Install Serverless Monitoring for AWS Step Functions"
 
 - Updates default .NET library used for auto-instrumentation from v2 to v3
@@ -425,8 +624,8 @@ New Features
 - [oracle] Add the ``active_session_history`` configuration parameter to optionally ingest Oracle active session history samples instead of query sampling.
 
 - Added config option ``logs_config.tag_truncated_logs``.  When
-  enabled, file logs will come with a tag ``truncated:true`` if 
-  they were truncated by the Agent. 
+  enabled, file logs will come with a tag ``truncated:true`` if
+  they were truncated by the Agent.
 
 
 .. _Release Notes_7.58.0_Enhancement Notes:
@@ -480,7 +679,7 @@ Bug Fixes
 
 - Fixed issue with openSUSE 15 RC 6 where the eBPF tracer wouldn't start due to a failed validation of the ``tcp_sendpage`` probe.
 
-- Fixed a rare issue where short-lived containers could cause 
+- Fixed a rare issue where short-lived containers could cause
   logs to be sent with the wrong container ID.
 
 - Fix Windows Process Agent argument stripping to account for spaces in the executable path.