fix(gevent): resolve asyncio-gevent compatibility issues #11900

mabdinur · 2025-01-10T17:40:17Z

Description

In some unix operating system tests (ex: ubuntuu), ddtrace module unloading breaks setting asyncio event loops. This issue can not be reproduced on macos.

Reproduction

import asyncio
import time

loop = asyncio.get_event_loop()
loop_id = id(loop)
new_loop = asyncio.new_event_loop()
new_loop_id = id(new_loop)

print(f"old: {loop_id} new: {new_loop_id}")
import pdb; pdb.set_trace()
asyncio.set_event_loop(new_loop)

check = asyncio.get_event_loop()
check_id = id(check)
print(f"check: {id(check)}")


while check_id != new_loop_id:
    print("MISMATCH")
    print(f"check: {id(check)}")
    time.sleep(1)
    check = asyncio.get_event_loop()
    check_id = id(check)

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2025-01-10T17:58:43Z

CODEOWNERS have been resolved as:

ddtrace/internal/coverage/_native.c                                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/_native.pyi                                   @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_fallback.py               @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
.github/CODEOWNERS                                                      @DataDog/python-guild @DataDog/apm-core-python
ddtrace/contrib/pytest/plugin.py                                        @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/coverage.py                              @DataDog/ci-app-libraries
ddtrace/internal/coverage/code.py                                       @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/installer.py                                  @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation.py                            @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_10.py                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_11.py                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_12.py                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_8.py                      @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/multiprocessing_coverage.py                   @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/report.py                                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/threading_coverage.py                         @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/module.py                                              @DataDog/debugger-python @DataDog/apm-core-python
pyproject.toml                                                          @DataDog/python-guild
riotfile.py                                                             @DataDog/apm-python
setup.py                                                                @DataDog/python-guild
tests/coverage/test_coverage_multiprocessing.py                         @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/test_coverage_threading.py                               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/internal/test_module.py                                           @DataDog/debugger-python @DataDog/apm-core-python
ddtrace/internal/coverage/instrumentation_py3_9.py                      @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python

…11] (#10318)" This reverts commit 7f7ad1c.

…0291)" This reverts commit ac5e255.

…tiprocessing children (#9939)" This reverts commit 0214b2a.

… imports (#9609)" This reverts commit 4c843ab.

datadog-datadog-prod-us1 · 2025-01-10T18:14:53Z

ddtrace/internal/coverage/instrumentation_py3_11.py

+    replace = code.replace(
+        co_code=bytes(new_code),
+        co_consts=tuple(new_consts),
+        co_stacksize=code.co_stacksize + 4,  # TODO: Compute the value!


⚪ Code Quality Violation

Suggested change

co_stacksize=code.co_stacksize + 4, # TODO: Compute the value!

co_stacksize=code.co_stacksize + 4, # TODO(<owner>): Compute the value!

comments must have ownership (...read more)

When using TODO or FIXME, specify who write the annotation. It's a best practice to remind you who created the annotation and have potential context and information about this issue.

datadog-datadog-prod-us1 · 2025-01-10T18:14:53Z

ddtrace/internal/coverage/instrumentation_py3_9.py

+    replace = code.replace(
+        co_code=bytes(new_code),
+        co_consts=tuple(new_consts),
+        co_stacksize=code.co_stacksize + 4,  # TODO: Compute the value!


⚪ Code Quality Violation

Suggested change

co_stacksize=code.co_stacksize + 4, # TODO: Compute the value!

co_stacksize=code.co_stacksize + 4, # TODO(<owner>): Compute the value!

comments must have ownership (...read more)

When using TODO or FIXME, specify who write the annotation. It's a best practice to remind you who created the annotation and have potential context and information about this issue.

datadog-datadog-prod-us1 · 2025-01-10T18:14:53Z

ddtrace/internal/coverage/instrumentation_py3_fallback.py

+            lines.add(last_lineno)
+        except AttributeError:
+            # pseudo-instruction (e.g. label)
+            pass


🔴 Code Quality Violation

silent exception (...read more)

Using the pass statement in an exception block ignores the exception. Exceptions should never be ignored. Instead, the user must add code to notify an exception occurred and attempt to handle it or recover from it.

The exception to this rule is the use of StopIteration or StopAsyncIteration when implementing a custom iterator (as those errors are used to acknowledge the end of a successful iteration).

datadog-datadog-prod-us1 · 2025-01-10T18:14:53Z

ddtrace/internal/coverage/instrumentation_py3_fallback.py

+            if instr.name == "NOP":
+                continue


🔴 Code Quality Violation

too many nesting levels (...read more)

Avoid to nest too many loops together. Having too many loops make your code harder to understand.
Prefer to organize your code in functions and unit of code you can clearly understand.

Learn More

Computer Programming wikibooks - Minimize nesting

pr-commenter · 2025-01-10T18:32:35Z

Benchmarks

Benchmark execution time: 2025-01-10 19:54:36

Comparing candidate commit 822c0db in PR branch munir/2.11.0-with-reverts with baseline commit dc8591a in branch 2.11.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 154 metrics, 2 unstable metrics.

This reverts commit 83f8e3c.

datadog-dd-trace-py-rkomorn · 2025-01-10T18:37:58Z

Datadog Report

Branch report: munir/2.11.0-with-reverts
Commit report: 822c0db
Test service: dd-trace-py

✅ 0 Failed, 121709 Passed, 56649 Skipped, 4h 26m 41.92s Total duration (5h 29m 5.07s time saved)

…ule exec (#9809)" This reverts commit 6a280ee.

mabdinur added 4 commits January 10, 2025 13:12

Revert "fix: prevent module import callback side-effects [backport 2.…

6247b4e

…11] (#10318)" This reverts commit 7f7ad1c.

Revert "fix(internal): handle native module specs [backport 2.11] (#1…

941741a

…0291)" This reverts commit ac5e255.

Revert "chore(ci_visibility): fix potential hang when waiting for mul…

e6b76f8

…tiprocessing children (#9939)" This reverts commit 0214b2a.

Revert "chore(ci_visibility): collect module-level line execution and…

be9416a

… imports (#9609)" This reverts commit 4c843ab.

mabdinur force-pushed the munir/2.11.0-with-reverts branch from 4f357ab to be9416a Compare January 10, 2025 18:13

datadog-datadog-prod-us1 bot reviewed Jan 10, 2025

View reviewed changes

Revert "chore: restore module loader after import (#9638)"

7c10320

This reverts commit 83f8e3c.

Revert "chore(internal): avoid re-wrapping get_code() in calls of mod…

822c0db

…ule exec (#9809)" This reverts commit 6a280ee.

mabdinur closed this Jan 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(gevent): resolve asyncio-gevent compatibility issues #11900

fix(gevent): resolve asyncio-gevent compatibility issues #11900

mabdinur commented Jan 10, 2025

github-actions bot commented Jan 10, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot Jan 10, 2025

datadog-datadog-prod-us1 bot Jan 10, 2025

datadog-datadog-prod-us1 bot Jan 10, 2025

datadog-datadog-prod-us1 bot Jan 10, 2025

pr-commenter bot commented Jan 10, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn bot commented Jan 10, 2025 •

edited

Loading

	co_stacksize=code.co_stacksize + 4, # TODO: Compute the value!
	co_stacksize=code.co_stacksize + 4, # TODO(<owner>): Compute the value!

fix(gevent): resolve asyncio-gevent compatibility issues #11900

fix(gevent): resolve asyncio-gevent compatibility issues #11900

Conversation

mabdinur commented Jan 10, 2025

Description

Reproduction

Checklist

Reviewer Checklist

github-actions bot commented Jan 10, 2025 • edited Loading

datadog-datadog-prod-us1 bot Jan 10, 2025

Choose a reason for hiding this comment

⚪ Code Quality Violation

datadog-datadog-prod-us1 bot Jan 10, 2025

Choose a reason for hiding this comment

⚪ Code Quality Violation

datadog-datadog-prod-us1 bot Jan 10, 2025

Choose a reason for hiding this comment

🔴 Code Quality Violation

datadog-datadog-prod-us1 bot Jan 10, 2025

Choose a reason for hiding this comment

🔴 Code Quality Violation

Learn More

pr-commenter bot commented Jan 10, 2025 • edited Loading

Benchmarks

datadog-dd-trace-py-rkomorn bot commented Jan 10, 2025 • edited Loading

Datadog Report

github-actions bot commented Jan 10, 2025 •

edited

Loading

pr-commenter bot commented Jan 10, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn bot commented Jan 10, 2025 •

edited

Loading