Investigate the consequences of PEP-563

[sallner]

In PEP-563 a system for storing type hints is proposed and the implementation started in Python 3.7. The type hints are stored in __annotations__. It would be helpful to see, whether this could effect somehow the working principle of zope.annotation.

[jamadden]

I've looked at this a little. __annotations__ was introduced way back in PEP-3107 for Python 3. Functions and methods grew the possibility to have an __annotations__ attribute:

Python 3.4.8 (default, Mar 29 2018, 16:18:25)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
...     def thing(a:int) -> int:
...         return 1
...
>>> Foo.thing.__annotations__
{'a': <class 'int'>, 'return': <class 'int'>}

Python 3.6 accepted PEP-563 which let you put annotations on raw variables. Prior to that, only function/method arguments and return types could be annotated AIUI, and I can't find a way to get __annotations__ on a class. But with 3.6, you can now have __annotations__ showing up at class scope:

Python 3.6.2 (v3.6.2:5fd33b5926, Jul 16 2017, 20:11:06)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
...     a:int
...
>>> Foo.__annotations__
{'a': <class 'int'>}

PEP 563 only changes the values stored in the annotations dictionary into strings always. It's not enabled by default in 3.7 and requires a __future__ import:

Python 3.7.0rc1 (default, Jun 13 2018, 16:20:06)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import annotations
>>> class Foo:
...   a:int
...
>>> Foo.__annotations__
{'a': 'int'}

z.a.attribute.AttributeAnnotations uses __annotations__ and can thus conflict in Python 3.6 and above, leading to false sharing (and failure to persist):

>>> from zope.annotation import attribute
>>> foo = Foo()
>>> attr_ann = AttributeAnnotations(foo)
>>> len(attr_ann)
1
>>> attr_ann['baz'] = 'bar'
>>> foo.__annotations__
{'a': 'int', 'baz': 'bar'}
>>> Foo.__annotations__
{'a': 'int', 'baz': 'bar'}

I'm not sure how to avoid it. The obvious solution is to use vars directly instead of __getitem__, but that breaks in the presence of __slots__, or more generally when a descriptor is used.

[jamadden]

I guess we could add a check like if self.obj.__annotations__ is type(self.obj).__annotations__...

[mgedmin]

Should we rename the attribute to __zope_annotations__ and add backwards-compatibility code to migrate old persistent instances?

[freddrake]

On Thu, Jun 28, 2018 at 10:55 AM Marius Gedminas ***@***.***> wrote: Should we rename the attribute to __zope_annotations__ and add backwards-compatibility code to migrate old persistent instances?

This gets back to long-standing issue of whether 3rd-party libraries should define their own dunder names. We've lived with the ones we've already defined in the Zope ecosystem for a long time, so changing them as a rule is pretty non-sensical, but here's a case where it can bite. Guido's long maintained that this practice of the Zope community isn't supported. We should seriously consider selecting a non-dunder name when we do need to deal with something like specific like this, and not add any new Zope-specific dunder names. Perhaps (z.a) __annotations__ should become _zope_annotations? -Fred

…

-- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein

[icemac]

Closing this issue as we did not find any problems since the PEP was implemented.

[icemac]

Today I finally came across a problem: Using type annotations on a class which implements IAttributeAnnotatable creates __annotations__ on the class which is writeable and also appears on the instance. When using adapting an instance of such a class to IAnnotation the object on the class is changed!

So never do this:

@zope.interface.implementer(zope.annotation.interfaces.IAttributeAnnotatable)
class C:
    a: str | None = None

inst = C()
IAnnotations(inst)['foo'] = 'bar'

because then:

C.__annotations__ == {'a': str | None, 'foo': 'bar'}
C.__annotations__ is inst.__annotations__

Using Python 3.11 with current zope.annotation version.

[jamadden]

I was also just bitten by the same thing @icemac was. Type annotations on a class that's later adapted to IAnnotations via IAttributeAnnotatable using the default adapter is broken. This is really bad because adding a type annotation to any base class breaks annotations on all subclassess as well, and this can be extremely hard to track down. (I only found it because some object was getting location proxied because of how zope.annotation.factory.factory works.)

>>> from zope.configuration import xmlconfig
>>> import zope.annotation
>>> xmlconfig.file('configure.zcml', zope.annotation)
<zope.configuration.config.ConfigurationMachine object at 0x101e97350>
>>> from zope.annotation.interfaces import *
>>> from zope.interface import *
>>> @implementer(IAttributeAnnotatable)
... class C:
...    instance_attr:str
...
>>> inst = C()
>>> inst.instance_attr
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1                                                                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'C' object has no attribute 'instance_attr'
>>> C.__annotations__
{'instance_attr': <class 'str'>}
>>> IAnnotations(inst)['key'] = 42
>>> C.__annotations__ # The class value has been mutated!
{'instance_attr': <class 'str'>, 'key': 42}
>>> other_inst = C()
>>> IAnnotations(other_inst)['key'] # So annotations are shared between objects!
42
>>> class D(C):
...   pass
...
>>> d = D()
>>> IAnnotations(d)['key']
42

Earlier, I wrote:

we could add a check like if self.obj.__annotations__ is type(self.obj).__annotations__

That would appear to solve this problem. Are there any alternatives?

[d-maurer]

...
Earlier, I wrote:

we could add a check like if self.obj.__annotations__ is type(self.obj).__annotations__

That would appear to solve this problem. Are there any alternatives?

For the moment, inst.__annotations__ does not seem to have an important use for type hints: I expect that type checkers and optimizers need to determine the annotations for all base classes to get the complete picture as inst.__annotations__ reflects only the annotations from the nearest class defining __annotations__. Therefore, we might still abuse it for our purposes. However, things might change in the future.

As an alternative to your check we could use isinstance(self.obj.__annotations__, AttributeAnnotations) or directly look into inst.__dict__. Those, too, suffer from abusing the __annotations__ attribute.

I assume that using a different attribute (_zope_annotations has been proposed earlier) would be safer in the long run. We would need to provide migration code and it could use one of the above checks to determine whether inst.__annotations__ really needs to be migrated.

[jamadden]

For the moment, inst.__annotations__ does not seem to have an important use for type hints: I expect that type checkers and optimizers need to determine the annotations for all base classes to get the complete picture as inst.__annotations__ reflects only the annotations from the nearest class defining __annotations__.

I think that's a red herring and we can ignore it. The point of type checkers is that they're static, i.e., independent of the runtime state of the program, based only on what is declared in classes and annotations. Things happening to instances are pretty much ignored. E.g., mypy can't see/doesn't care that the instance in fact does have the desired attribute; it only cares about the (static) class:

$ cat foo.py
───────┬────────────────────────────────────────────────
       │ File: /tmp/foo.py
       │ Size: 89 B
───────┼────────────────────────────────────────────────
   1   │ class Foo:
   2   │     pass
   3   │
   4   │ foo = Foo()
   5   │ foo.thing = lambda: print('I am dynamic')
   6   │
   7   │ foo.thing()
───────┴───────────────────────────────────────────────
$ python foo.py
I am dynamic
$ mypy foo.py
/tmp/foo.py:5: error: "Foo" has no attribute "thing"  [attr-defined]
/tmp/foo.py:7: error: "Foo" has no attribute "thing"  [attr-defined]
Found 2 errors in 1 file (checked 1 source file)

As an alternative to your check we could use isinstance(self.obj.__annotations__, AttributeAnnotations)

isinstance is about three times slower than is. But that doesn't work anyway because self.obj.__annotations__ is either a OOBTree or a dict; and Class.__annotations__ is also just a dict. If they're both dict, you still have to do an identity check.

or directly look into inst.__dict__.

inst is not required to have a __dict__, slots are perfectly acceptable (and yes, I use that).

However, things might change in the future....We would need to provide migration code

To me, that's a (potential) problem for the future. I'm more interested in actually making this work now than dealing with a future-yet-to-be.

[d-maurer]

The problem might be bigger: a class can get an __annotations__ attribute without any type hints:
observed with Python 3.13.1

>>> class C: pass
... 
... C.__annotations__
... 
{}
>>> o = C()
>>> o.__annotations__
{}

[jamadden]

Wow, TIL! The descriptor for __annotations__ automatically creates __annotations__ as soon as it is accessed.

Interestingly, though, the "official" API to access annotations, inspect.get_annotations, doesn't provoke this behaviour.

>>> class C:
...     pass
>>> '__annotations__' in dir(C)
False
>>> import inspect
>>> inspect.get_annotations(C)
{}
>>> '__annotations__' in dir(C)
False
>>> C.__annotations__
{}
>>> '__annotations__' in dir(C)
True

[jamadden]

Ah, the auto-creation is new in 3.10, and I think is there to eliminate exactly the same kind of problem we're dealing with (false inheritance):

....As of Python 3.10, o.__annotations__ is guaranteed to always work on Python...classes,...Before Python 3.10, accessing __annotations__ on a class that defines no annotations but that has a parent class with annotations would return the parent’s __annotations__. In Python 3.10 and newer, the child class’s annotations will be an empty dict instead

[d-maurer]

Jason Madden wrote at 2024-12-10 05:18 -0800:

Ah, the auto-creation is new in 3.10, and I think is there to eliminate exactly the same kind of problem we're dealing with (false inheritance): > ....As of Python 3.10, `o.__annotations__` is guaranteed to always work on Python...classes,...Before Python 3.10, accessing `__annotations__` on a class that defines no annotations but that has a parent class with annotations would return the parent’s` __annotations__`. In Python 3.10 and newer, the child class�s annotations will be an empty dict instead

Thank you for the analysis! Unfortunately, this means that your check is not safe as shown by the following transscript ```pycon

>> class C1:

... x : int ...

>> class C2(C1): pass

...

>> o=C2() >> o.__annotations__

{'x': <class 'int'>}

>> _ is type(o).__annotations__

False

>> C2.__annotations__

{}

>> o.__annotations__

{}

>> o.__annotations__ is type(o).__annotations__

True ```

[jamadden]

Unfortunately, this means that your check is not safe

Right. On Python 3.10, you're supposed to use inspect.get_annotations. On earlier versions, you're supposed to use the dict directly: if isinstance(o, type): ann = o.__dict__.get('__annotations__', None).

[d-maurer]

***@***.*** wrote at 2024-12-10 15:02 +0100:

Jason Madden wrote at 2024-12-10 05:18 -0800: ... Unfortunately, this means that your check `inst.__annotations__ is not type(inst).__annotations__` is not safe ...

What about using a special annotation (key) to mark a storage used by `zope.annotation`? This would make us independent of the Python's peculiar `__annotations__` handling. It wont help us with migration. However, we could handle migration in the following way: If the used storage is a `BTree`, we know the attribute was not created by Python internally. Otherwise, we can assume that the annotations are likely not persistent and we can simple ignore the attribute.

[jamadden]

The official way to get annotations, inspect.get_annotations, flat-out refuses to work on instances. So I think the chances of __annotations__ on an instance meaning anything official anytime in the near future is close to nil.

>>> class C:
...     var:str
...
>>> inst = C()
>>> inspect.get_annotations(inst)
Traceback (most recent call last) 
...
TypeError: <__main__.C object at 0x10298a1d0> is not a module, class, or callable.