v2.7.0-rc.0

Deprecations

• Tempo serverless features are now deprecated and will be removed in an upcoming release #4017 @electron0zero

Breaking Changes

• Added maximum spans per span set to prevent queries from overwhelming read path. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #3646 (@andreasgerstmayr)
• Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
• Removed querier_forget_delay setting from the frontend. This configuration option did nothing. #3996 (@joe-elliott)
• Use Prometheus fast regexp for TraceQL regular expression matchers. #4329 (@joe-elliott)
  All regular expression matchers will now be fully anchored. span.foo =~ "bar" will now be evaluated as span.foo =~ "^bar$"

Changes

• [CHANGE] Disable gRPC compression in the querier and distributor for performance reasons #4429 (@carles-grafana)
  If you would like to re-enable it, we recommend 'snappy'. Use the following settings:

ingester_client:
    grpc_client_config:
        grpc_compression: "snappy"
metrics_generator_client:
    grpc_client_config:
        grpc_compression: "snappy"
querier:
    frontend_worker:
        grpc_client_config:
            grpc_compression: "snappy"

• [CHANGE] slo: include request cancellations within SLO [#4355] (#4355) (@electron0zero)
  request cancellations are exposed under result label in tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total with completed or canceled values to differentiate between completed and canceled requests.
• [CHANGE] update default config values to better align with production workloads #4340 (@electron0zero)
• [CHANGE] tempo-cli: add support for /api/v2/traces endpoint #4127 (@electron0zero)
  BREAKING CHANGE The tempo-cli now uses the /api/v2/traces endpoint by default,
  please use --v1 flag to use /api/traces endpoint, which was the default in previous versions.
• [CHANGE] TraceByID: don't allow concurrent_shards greater than query_shards. #4074 (@electron0zero)
• [CHANGE] BREAKING CHANGE The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• [CHANGE] BREAKING CHANGE Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger.
• [CHANGE] No longer send the final diff in GRPC streaming. Instead we rely on the streamed intermediate results. #4062 (@joe-elliott)
• [CHANGE] Update Go to 1.23.3 #4146 #4147 #4380 (@javiermolinar @mdisibio)
• [CHANGE] Return 422 for TRACE_TOO_LARGE queries #4160 (@zalegrala)
• [CHANGE] Tighten file permissions #4251 (@zalegrala)
• [CHANGE] Drop max live traces log message and rate limit trace too large. #4418 (@joe-elliott)
• [CHANGE] Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  BREAKING CHANGE After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

Features

• [FEATURE] tempo-cli: support dropping multiple traces in a single operation #4266 (@ndk)
• [FEATURE] Discarded span logging log_discarded_spans #3957 (@dastrobu)
• [FEATURE] TraceQL support for instrumentation scope #3967 (@ie-pham)
• [FEATURE] Export cost attribution usage metrics from distributor #4162 (@mdisibio)
• [FEATURE] TraceQL metrics: avg_over_time #4073 (@javiermolinar)
• [FEATURE] Limit tags and tag values search #4320 (@javiermolinar)

Enhancements

• [ENHANCEMENT] TraceQL: Add range condition for byte predicates #4198 (@ie-pham)
• [ENHANCEMENT] Add throughput and SLO metrics in the tags and tag values endpoints #4148 (@electron0zero)
• [ENHANCEMENT] BREAKING CHANGE Add maximum spans per span set. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• [ENHANCEMENT] Add query-frontend limit for max length of query expression #4397 (@electron0zero)
• [ENHANCEMENT] distributor: return trace id length when it is invalid #4407 (@carles-grafana)
• [ENHANCEMENT] Changed log level from INFO to DEBUG for the TempoDB Find operation using traceId to reduce excessive/unwanted logs in log search. #4179 (@Aki0x137)
• [ENHANCEMENT] Pushdown collection of results from generators in the querier #4119 (@electron0zero)
• [ENHANCEMENT] The span multiplier now also sources its value from the resource attributes. #4210 (@JunhoeKim)
• [ENHANCEMENT] TraceQL: Attribute iterators collect matched array values #3867 (@electron0zero, @stoewer)
• [ENHANCEMENT] Allow returning partial traces that exceed the MaxBytes limit for V2 #3941 (@javiermolinar)
• [ENHANCEMENT] Added new middleware to validate request query values #3993 (@javiermolinar)
• [ENHANCEMENT] Prevent massive allocations in the frontend if there is not sufficient pressure from the query pipeline. #3996 (@joe-elliott)
  BREAKING CHANGE Removed querier_forget_delay setting from the frontend. This configuration option did nothing.
• [ENHANCEMENT] Update metrics-generator config in Tempo distributed docker compose example to serve TraceQL metrics #4003 (@javiermolinar)
• [ENHANCEMENT] Reduce allocs related to marshalling dedicated columns repeatedly in the query frontend. #4007 (@joe-elliott)
• [ENHANCEMENT] Improve performance of TraceQL queries #4114 (@mdisibio)
• [ENHANCEMENT] Improve performance of TraceQL queries #4163 (@mdisibio)
• [ENHANCEMENT] Improve performance of some TraceQL queries using select() operation #4438 (@mdisibio)
• [ENHANCEMENT] Reduce memory usage of classic histograms in the span-metrics and service-graphs processors #4232 (@mdisibio)
• [ENHANCEMENT] Implement simple Fetch by key for cache items #4032 (@javiermolinar)
• [ENHANCEMENT] TraceQL metrics queries: add min_over_time #3975 (@javiermolinar)
• [ENHANCEMENT] TraceQL metrics queries: add max_over_time #4065 (@javiermolinar)
• [ENHANCEMENT] Write tenantindex as ...

v2.6.1

Bugfixes

• [BUGFIX] Bring back application-json content-type header. #4123 (@javiermolinar)

Changes

• [CHANGE] BREAKING CHANGE tempo-query is no longer a jaeger instance with grpcPlugin. Its now a standalone server. Serving a grpc api for jaeger on 0.0.0.0:7777 by default. #3840 (@frzifus)

Enhancements

• [ENHANCEMENT] Register gRPC health server to tempo-query #4178 (@frzifus)
• [ENHANCEMENT] Support Tempo on IBM s390x #4175 (@pavolloffay)
• [ENHANCEMENT] tempo-query: separate tls settings for server and client #4177 (@frzifus)
• [ENHANCEMENT] Speedup tempo-query trace search by allowing parallel queries #4159 (@pavolloffay)

v2.6.0

Breaking Changes

• autocomplete_filtering_enabled feature flag has been removed and always enabled.
• use_v2_sdk in the Azure backend has been removed. Azure v2 sdk has been promoted to GA/dfeault.
• Removed completedfilepath and blocksfilepath from the wal config.
• TraceQL metrics now bases historical requests off of a set of RF1 blocks written to the backend by the metrics-generators. Set flush_to_storage to true to run metrics queries on historical data.

  metrics_generator:
    processor:
      local_blocks:
        flush_to_storage: true
  

Changes

• [CHANGE] BREAKING CHANGE Remove autocomplete_filtering_enabled feature flag #3729 (@mapno)
• [CHANGE] Bump opentelemetry-collector to 0.102.1 #3784 (@debasishbsws)
• [CHANGE] Bump Jaeger query docker image to 1.57.0 #3652 (@iblancasa)
• [CHANGE] Update Go to 1.22.4 #3757 #3793 (@joe-elliott, @mapno)
• [CHANGE] Make vParquet4 the default block encoding #3810 (@ie-pham)
• [CHANGE] Azure v2 backend becomes the only and primary Azure backend #3875 (@zalegrala)
  BREAKING CHANGE The use_v2_sdk configuration option has been removed.

Features

• [FEATURE] TraceQL support for link scope and link:traceID and link:spanID #3741 (@stoewer)
• [FEATURE] TraceQL support for link attribute querying #3814 (@ie-pham)
• [FEATURE] TraceQL support for event scope and event:name intrinsic #3708 (@stoewer)
• [FEATURE] TraecQL support for event attributes #3708 (@ie-pham)
• [FEATURE] TraceQL support for event:timeSinceStart #3908 (@ie-pham)
• [FEATURE] Autocomplete support for events and links #3846 (@ie-pham)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno)
• [FEATURE] Add new compare() metrics function #3695 (@mdisibio)
• [FEATURE] Add new api /api/metrics/query for instant metrics queries #3859 (@mdisibio)
• [FEATURE] Add a q parameter to /api/v2/search/tags for tag name filtering #3822 (@joe-elliott)
• [FEATURE] Add exemplars to TraceQL metrics #3824 (@mapno)
• [FEATURE] Added new middleware to block matching urls #3963 (@javiermolinar)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno) #3995 (@mdisibio)
• [FEATURE] Add support for generating native histograms from metrics-generator #3789 (@zalegrala, @kvrhdn)

Enhancements

• [ENHANCEMENT] Implement arrays for traceql.Static with reused fields #3827 (@stoewer)
• [ENHANCEMENT] Tag value lookup use protobuf internally for improved latency #3731 (@mdisibio)
• [ENHANCEMENT] TraceQL metrics queries use protobuf internally for improved latency #3745 (@mdisibio)
• [ENHANCEMENT] TraceQL search and other endpoints use protobuf internally for improved latency and resource usage #3944 (@mdisibio)
• [ENHANCEMENT] Add local disk caching of metrics queries in local-blocks processor #3799 (@mdisibio)
• [ENHANCEMENT] Improve use of OTEL semantic conventions on the service graph #3711 (@zalegrala)
• [ENHANCEMENT] Performance improvement for rate() by () queries #3719 (@mapno)
• [ENHANCEMENT] Performance improvement for queries using trace-level intrinsics #3920 (@mdisibio)
• [ENHANCEMENT] Use multiple goroutines to unmarshal responses in parallel in the query frontend. #3713 (@joe-elliott)
• [ENHANCEMENT] Protect ingesters from panics by adding defer/recover to all read path methods. #3790 (@joe-elliott)
• [ENHANCEMENT] Added a boolean flag to enable or disable dualstack mode on Storage block config for S3 #3721 (@sid-jar, @mapno)
• [ENHANCEMENT] Add caching to query range queries #3796 (@mapno)
• [ENHANCEMENT] Only stream diffs on metrics queries #3808 (@joe-elliott)
• [ENHANCEMENT] Add data quality metric to measure traces without a root #3812 (@mapno)
• [ENHANCEMENT] Added an example for running Tempo vulture #3829 (@javiermolinar)
• [ENHANCEMENT] Add a new helper method to allow debugging e2e tests #3836 (@javiermolinar)
• [ENHANCEMENT] Refactor Tempo Vulture to reduce code complexity #3850 (@javiermolinar)
• [ENHANCEMENT] Self document makefile #3844 (@javiermolinar)
• [ENHANCEMENT] Added a Tempo CLI command to drop traces by id by rewriting blocks. #3856 (@joe-elliott)
• [ENHANCEMENT] Mixin, make recording rule range interval configurable and increase range interval in alert to support scrape interval of 1 minute #3851 (@jmichalek132)
• [ENHANCEMENT] Add vParquet4 support to the tempo-cli analyse blocks command #3868 (@stoewer)
• [ENHANCEMENT] Improve trace id lookup from Tempo Vulture by selecting a date range #3874 (@javiermolinar)
• [ENHANCEMENT] Add native histograms for internal metrics#3870 (@zalegrala)
• [ENHANCEMENT] Expose availability-zone as a cli flag in ingester #3881
• [ENHANCEMENT] Rename batches property of Trace to ResourceSpans to be OTEL compatible #3895
• [ENHANCEMENT] Reduce memory consumption of query-frontend#3888 (@joe-elliott)
• [ENHANCEMENT] Reduce log level verbosity for e2e tests#3900 (@javiermolinar)
• [ENHANCEMENT] Added new Traces api V2#3912 (@javiermolinar)
• [ENHANCEMENT] Update to the latest dskit #3915 (@andreasgerstmayr)
• [ENHANCEMENT] Reduce allocs building queriers sharded requests #3932 (@javiermolinar)
• [ENHANCEMENT] Allow compaction disablement per-tenant #3965 (@zalegrala)
• [ENHANCEMENT] Implement polling tenants concurrently #3647 (@zalegrala)
• [ENHANCEMENT] Reduce allocs of caching middleware #3976 (@joe-elliott)
• [ENHANCEMENT] BlockMeta improvements to reduce the size #3950 #3951 #3952(@zalegrala)
• [ENHANCEMENT] Add bytes and spans received to usage stats #3983 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix panic in certain metrics queries using rate() with by #3847 (@stoewer)
• [BUGFIX] Fix double appending the primary iterator on second pass with event iterator #3903 (@ie-pham)
• [BUGFIX] Fix metrics queries when grouping by attributes that may not exist #3734 (@mdisibio)
• [BUGFIX] Fix frontend parsing error on cached responses #3759 (@mdisibio)
• [BUGFIX] Fix autocomplete of a query using scoped instrinsics #3865 (@mdisibio)
• [BUGFIX] Fix metrics query histograms and quantiles on traceDuration #3879 (@mdisibio)
• [BUGFIX] Fix divide by 0 bug in query frontend exemplar calculations #3936 (@mdisibio)
• [BUGFIX] max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit #3618 (@kvrhdn)
• [BUGFIX] Fix http connection reuse on GCP and AWS by reading io.EOF through the http body. #3760 (@bmteller)
• [BUGFIX] Improved handling of complete blocks in localblocks processor after enabling flusing #3805 (@mapno)
• [BUGFIX] ...

v2.6.0-rc.1

• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno) #3995 (@mdisibio)
• [ENHANCEMENT] Add bytes and spans received to usage stats #3983 (@joe-elliott)

v2.6.0-rc.0

Breaking Changes

• autocomplete_filtering_enabled feature flag has been removed and always enabled.
• use_v2_sdk in the Azure backend has been removed. Azure v2 sdk has been promoted to GA/dfeault.
• Removed completedfilepath and blocksfilepath from the wal config.

Changes

• [CHANGE] BREAKING CHANGE Remove autocomplete_filtering_enabled feature flag #3729 (@mapno)
• [CHANGE] Bump opentelemetry-collector to 0.102.1 #3784 (@debasishbsws)
• [CHANGE] Bump Jaeger query docker image to 1.57.0 #3652 (@iblancasa)
• [CHANGE] Update Go to 1.22.4 #3757 #3793 (@joe-elliott, @mapno)
• [CHANGE] Make vParquet4 the default block encoding #3810 (@ie-pham)
• [CHANGE] Azure v2 backend becomes the only and primary Azure backend #3875 (@zalegrala)
  BREAKING CHANGE The use_v2_sdk configuration option has been removed.

Features

• [FEATURE] TraceQL support for link scope and link:traceID and link:spanID #3741 (@stoewer)
• [FEATURE] TraceQL support for link attribute querying #3814 (@ie-pham)
• [FEATURE] TraceQL support for event scope and event:name intrinsic #3708 (@stoewer)
• [FEATURE] TraecQL support for event attributes #3708 (@ie-pham)
• [FEATURE] TraceQL support for event:timeSinceStart #3908 (@ie-pham)
• [FEATURE] Autocomplete support for events and links #3846 (@ie-pham)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno)
• [FEATURE] Add new compare() metrics function #3695 (@mdisibio)
• [FEATURE] Add new api /api/metrics/query for instant metrics queries #3859 (@mdisibio)
• [FEATURE] Add a q parameter to /api/v2/search/tags for tag name filtering #3822 (@joe-elliott)
• [FEATURE] Add exemplars to TraceQL metrics #3824 (@mapno)
• [FEATURE] Added new middleware to block matching urls #3963 (@javiermolinar)

Enhancements

• [ENHANCEMENT] Implement arrays for traceql.Static with reused fields #3827 (@stoewer)
• [ENHANCEMENT] Tag value lookup use protobuf internally for improved latency #3731 (@mdisibio)
• [ENHANCEMENT] TraceQL metrics queries use protobuf internally for improved latency #3745 (@mdisibio)
• [ENHANCEMENT] TraceQL search and other endpoints use protobuf internally for improved latency and resource usage #3944 (@mdisibio)
• [ENHANCEMENT] Add local disk caching of metrics queries in local-blocks processor #3799 (@mdisibio)
• [ENHANCEMENT] Improve use of OTEL semantic conventions on the service graph #3711 (@zalegrala)
• [ENHANCEMENT] Performance improvement for rate() by () queries #3719 (@mapno)
• [ENHANCEMENT] Performance improvement for queries using trace-level intrinsics #3920 (@mdisibio)
• [ENHANCEMENT] Use multiple goroutines to unmarshal responses in parallel in the query frontend. #3713 (@joe-elliott)
• [ENHANCEMENT] Protect ingesters from panics by adding defer/recover to all read path methods. #3790 (@joe-elliott)
• [ENHANCEMENT] Added a boolean flag to enable or disable dualstack mode on Storage block config for S3 #3721 (@sid-jar, @mapno)
• [ENHANCEMENT] Add caching to query range queries #3796 (@mapno)
• [ENHANCEMENT] Only stream diffs on metrics queries #3808 (@joe-elliott)
• [ENHANCEMENT] Add data quality metric to measure traces without a root #3812 (@mapno)
• [ENHANCEMENT] Added an example for running Tempo vulture #3829 (@javiermolinar)
• [ENHANCEMENT] Add a new helper method to allow debugging e2e tests #3836 (@javiermolinar)
• [ENHANCEMENT] Refactor Tempo Vulture to reduce code complexity #3850 (@javiermolinar)
• [ENHANCEMENT] Self document makefile #3844 (@javiermolinar)
• [ENHANCEMENT] Added a Tempo CLI command to drop traces by id by rewriting blocks. #3856 (@joe-elliott)
• [ENHANCEMENT] Mixin, make recording rule range interval configurable and increase range interval in alert to support scrape interval of 1 minute #3851 (@jmichalek132)
• [ENHANCEMENT] Add vParquet4 support to the tempo-cli analyse blocks command #3868 (@stoewer)
• [ENHANCEMENT] Improve trace id lookup from Tempo Vulture by selecting a date range #3874 (@javiermolinar)
• [ENHANCEMENT] Add native histograms for internal metrics#3870 (@zalegrala)
• [ENHANCEMENT] Expose availability-zone as a cli flag in ingester #3881
• [ENHANCEMENT] Rename batches property of Trace to ResourceSpans to be OTEL compatible #3895
• [ENHANCEMENT] Reduce memory consumption of query-frontend#3888 (@joe-elliott)
• [ENHANCEMENT] Reduce log level verbosity for e2e tests#3900 (@javiermolinar)
• [ENHANCEMENT] Added new Traces api V2#3912 (@javiermolinar)
• [ENHANCEMENT] Update to the latest dskit #3915 (@andreasgerstmayr)
• [ENHANCEMENT] Reduce allocs building queriers sharded requests #3932 (@javiermolinar)
• [ENHANCEMENT] Allow compaction disablement per-tenant #3965 (@zalegrala)
• [ENHANCEMENT] Implement polling tenants concurrently #3647 (@zalegrala)
• [ENHANCEMENT] Reduce allocs of caching middleware #3976 (@joe-elliott)
• [ENHANCEMENT] BlockMeta improvements to reduce the size #3950 #3951 #3952(@zalegrala)

Bugfixes

• [BUGFIX] Fix panic in certain metrics queries using rate() with by #3847 (@stoewer)
• [BUGFIX] Fix double appending the primary iterator on second pass with event iterator #3903 (@ie-pham)
• [BUGFIX] Fix metrics queries when grouping by attributes that may not exist #3734 (@mdisibio)
• [BUGFIX] Fix frontend parsing error on cached responses #3759 (@mdisibio)
• [BUGFIX] Fix autocomplete of a query using scoped instrinsics #3865 (@mdisibio)
• [BUGFIX] Fix metrics query histograms and quantiles on traceDuration #3879 (@mdisibio)
• [BUGFIX] Fix divide by 0 bug in query frontend exemplar calculations #3936 (@mdisibio)
• [BUGFIX] max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit #3618 (@kvrhdn)
• [BUGFIX] Fix http connection reuse on GCP and AWS by reading io.EOF through the http body. #3760 (@bmteller)
• [BUGFIX] Improved handling of complete blocks in localblocks processor after enabling flusing #3805 (@mapno)
• [BUGFIX] Handle out of boundaries spans kinds #3861 (@javiermolinar)
• [BUGFIX] Maintain previous tenant blocklist on tenant errors #3860 (@zalegrala)
• [BUGFIX] Fix prefix handling in Azure backend Find() call #3875 (@zalegrala)
• [BUGFIX] BREAKING CHANGE Remove unused properties from the WAL configuration #3911 (@javiermolinar)
• [BUGFIX] Bring back OTEL receiver metrics #3917 (@javiermolinar)
• [BUGFIX] Correct block end time when the ingested traces are outside the ingestion slack #3954 (@javiermolinar)
• [BUGFIX] Fix race condition where a streaming response could be marshalled while being modi...

v2.5.0

Breaking Changes

• Ownership of /var/tempo is changing. Historically this has been owned by root:root, and with this change it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts impacts all users of the grafana/tempo Docker image.
  #2265

• In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
  #3663

• Remove trace by id hedging from the frontend. We consider this configuration an anti-pattern. Enabling it only caused additional issues with Tempo operation. The following have been removed:

  query_frontend:
      trace_by_id:
          hedge_requests_at: 2s
          hedge_requests_up_to: 2
  

  #3522

• Dropped meta-tag for tenant from trace by id multitenant. #3522

• The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers #3649

Changes

• [CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
• [CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala)
  BREAKING CHANGE Ownership of /var/tempo is changing. Historically this
  has been owned by root:root, and with this change it will now be owned by
  tempo:tempo with the UID/GID of 10001. The ingester and
  metrics-generator statefulsets may need to be chown'd in order to start
  properly. A jsonnet example of an init container is included with the PR.
  This impacts impacts all users of the grafana/tempo Docker image.
• [CHANGE] Remove vParquet encoding #3663 (@mdisibio)
  BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks
  have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
• [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
• [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
• [CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
• [CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
• [CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
• [CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)
• [CHANGE] Update Alpine image version to 3.20 #3710 (@joe-elliott)

Features

• [FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
• [FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
• [FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
• [FEATURE] Added gRPC streaming endpoints for Tempo APIs.
  • Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
  • Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
  • Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
  • Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
  • Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
• [FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)

Enhancements

• [ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
• [ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
• [ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer)
  Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary
  and InstrumentationLibrarySpan from the OTLP receivers
• [ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
• [ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
• [ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
• [ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
• [ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
• [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
• [ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
• [ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
• [ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
• [ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham)
  available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
• [ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)
• [ENHANCEMENT] TraceQL - Add support for trace:id and span:id #3670 (@ie-pham)
• [ENHANCEMENT] Add toggle to inject the tenant ID to generated metrics #3638 (@kvrhdn)

Bugfixes

• [BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
• [BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
• [BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
• [BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
• [BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID (@mdisibio)
• [BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
• [BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
• [BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
• [BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
• [BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
• [BUGFIX] Use os path separator to split blocks path. [#3552](https://github.co...

v2.5.0-rc.1

Changes

• [CHANGE] Update Alpine image version to 3.20 #3710 (@joe-elliott)

Enhancements

• [ENHANCEMENT] TraceQL - Add support for trace:id and span:id #3670 (@ie-pham)

Bugfixes

• [BUGFIX] Fix TraceQL queries involving non boolean operations between statics and attributes. #3698 (@joe-elliott)

v2.5.0-rc.0

Breaking Changes

• Ownership of /var/tempo is changing. Historically this has been owned by root:root, and with this change it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts impacts all users of the grafana/tempo Docker image.
  #2265

• In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
  #3663

• Remove trace by id hedging from the frontend. We consider this configuration an anti-pattern. Enabling it only caused additional issues with Tempo operation. The following have been removed:

  query_frontend:
      trace_by_id:
          hedge_requests_at: 2s
          hedge_requests_up_to: 2
  

  #3522

• Dropped meta-tag for tenant from trace by id multitenant. #3522

• The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers #3649

Changes

• [CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
• [CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala)
  BREAKING CHANGE Ownership of /var/tempo is changing. Historically this
  has been owned by root:root, and with this change it will now be owned by
  tempo:tempo with the UID/GID of 10001. The ingester and
  metrics-generator statefulsets may need to be chown'd in order to start
  properly. A jsonnet example of an init container is included with the PR.
  This impacts impacts all users of the grafana/tempo Docker image.
• [CHANGE] Remove vParquet encoding #3663 (@mdisibio)
  BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks
  have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
• [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
• [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
• [CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
• [CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
• [CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
• [CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)

Features

• [FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
• [FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
• [FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
• [FEATURE] Added gRPC streaming endpoints for Tempo APIs.
  • Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
  • Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
  • Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
  • Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
  • Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
• [FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)

Enhancements

• [ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
• [ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
• [ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer)
  Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary
  and InstrumentationLibrarySpan from the OTLP receivers
• [ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
• [ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
• [ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
• [ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
• [ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
• [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
• [ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
• [ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
• [ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
• [ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham)
  available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
• [ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
• [BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
• [BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
• [BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
• [BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID (@mdisibio)
• [BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
• [BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
• [BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
• [BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
• [BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
• [BUGFIX] Use os path separator to split blocks path. #3552 (@teyyubismayil)
• [BUGFIX] Correctly parse traceql queries with > 1024 character attribute names or static values. #3571 (@joe-elliott)
• [BUGFIX] Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants. [#3612](#3...

v2.4.2

• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)

v2.4.1

• [BUGFIX] Fix compaction/retention in AWS S3 and GCS when a prefix is configured. #3465 (@bpfoster)