Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve efficiency of ETW metrics exporter #134

Merged

Conversation

mattbodd
Copy link
Contributor

@mattbodd mattbodd commented Dec 3, 2024

Fixes #104
Design Decisions:

  • Follow .Net and C++ implementations and use helper methods to serialize fragments of metric points into a coherent, exportable buffer
  • Construct a new ResourceMetrics for each metric point and serialize each into an exportable buffer

Changes

Please provide a brief description of the changes here.

Merge requirement checklist

  • CONTRIBUTING guidelines followed
  • Unit tests added/updated (if applicable)
  • Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
  • Changes in public API reviewed (if applicable)

Copy link

codecov bot commented Dec 3, 2024

Codecov Report

Attention: Patch coverage is 75.63025% with 29 lines in your changes missing coverage. Please review.

Project coverage is 53.9%. Comparing base (8ecb2b2) to head (33889e3).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
opentelemetry-etw-metrics/src/exporter/mod.rs 75.6% 29 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main    #134     +/-   ##
=======================================
- Coverage   54.1%   53.9%   -0.2%     
=======================================
  Files         42      42             
  Lines       6282    6153    -129     
=======================================
- Hits        3400    3320     -80     
+ Misses      2882    2833     -49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mattbodd
Copy link
Contributor Author

Comparing the latest exporter benchmark results between the proposed export implementation and the previous export implementation, there is a ~26% improvement.

@mattbodd mattbodd marked this pull request as ready for review December 17, 2024 20:42
@mattbodd mattbodd requested a review from a team as a code owner December 17, 2024 20:42
@mattbodd
Copy link
Contributor Author

Comparing the latest exporter benchmark results between the proposed export implementation and the previous export implementation, there is a ~26% improvement.

With a more realistic ResourceMetrics, the improvement looks to be closer to 38%

@psandana
Copy link
Contributor

We noticed some tonic namespace usage along with types with Tonic prefix. Discussing with @cijothomas we got clarified OpenTelemetry is not using Tonic, they are just types coming from proto definitions.

Thus, this comment is not blocking as the change is unrelated to this PR, but we propose making those proto types more generic, thus we prevent future confusions regarding tonic usage or not.

…buffer in emit_export_metric_service_request
@mattbodd
Copy link
Contributor Author

We noticed some tonic namespace usage along with types with Tonic prefix. Discussing with @cijothomas we got clarified OpenTelemetry is not using Tonic, they are just types coming from proto definitions.

Thus, this comment is not blocking as the change is unrelated to this PR, but we propose making those proto types more generic, thus we prevent future confusions regarding tonic usage or not.

I borrowed the use pattern set in open-telemetry/opentelemetry-rust/opentelemetry-proto/src/transform/metrics.rs. @psandana or @cijothomas do you have suggestions for better generic names?


for _i in 0..iters {
runtime.block_on(async {
export(&exporter, &mut resource_metrics).await;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to micro benchmark just the serialization and export of one data point instead of 20000. It would keep the calculations simple when comparing two benchmark results.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is coming from - #134 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let me know if there's a preference between a micro benchmark and what Cijo suggested.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a blocker. Exporting Metrics is not in hot path, so it may not be worth perfecting this.

if let Some(proto_data) = proto_data {
match proto_data {
TonicMetricData::Histogram(hist) => {
for data_point in hist.data_points {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contents of each arm looks repetitive. Can this be factored into a function? It would help also to improve readability and maintainability.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is repetition that could probably be reduced and I'm open to suggestions. I don't see a clean way to abstract the repetitive logic into a function because specialization is needed to construct data points of different metric data types. We could probably add traits that allow for better abstraction, but I'm not sure I want to attempt that here.

@cijothomas cijothomas merged commit 434cc86 into open-telemetry:main Jan 25, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Etw-Metrics - Improvements required
6 participants