You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Hey guys, I'm using tortoise in production and I found a Postgres specific bug.
Postgres has a limitation where identifiers must be at most 63 character long. This means in practice that if you create a column name or alias which is longer than 63 bytes long, Postgres will truncate it.
This limit can be configured but it requires re-compiling Postgres which is not practical when using cloud-managed solutions.
Here's an example
SELECT 1 AS "my_very_very_very_long_unpractical_and_almost_comical_column_name_hehe";
This is a problem for tortoise which relies on columns aliases to resolve field names and build related models when doing joins, or even if any field has a very long name.
To Reproduce
This is a kind of comical example that abuses the length of field names. But note, in production models tend to have descriptive names and it is not uncommon to require many joins when executing a queryset with .select_related(), which generates very long aliases by joining field names using double underscores.
#file main.pyimportmodelsimporttortoiseasyncdefmain():
awaittortoise.Tortoise.init(
db_url="postgres://user:password@host/dbname",
modules={"models": ["models"]},
)
awaittortoise.Tortoise.generate_schemas()
author=awaitmodels.Author.create()
book=awaitmodels.Book.create(author_model_relation_with_long_name=author)
chapter=awaitmodels.Chapter.create(book_model_relation_with_long_name=book)
# Joins generate colums aliases with lenghth >63 charsqueryset=models.Chapter.all().select_related(
"book_model_relation_with_long_name",
"book_model_relation_with_long_name__author_model_relation_with_long_name",
)
print(queryset.sql(), end="\n\n")
# upon selecting, column aliases get truncated to 63 chars in Postgres response records# tortoise does not expect return aliases to be truncatedjoin=awaitqueryset.latest("id") # BOOM!tortoise.run_async(main())
For executing
python main.py
Which raises
IndexError: list index out of range
in
tortoise/backends/base/executor.py", line 120, in execute_select
obj = model._init_from_db(**{k.split(".")[1]: v for k, v in related_items})
Note that the issue is having an alias which is very long
SELECT
....
,"chapter__book_model_relation_with_long_name__author_model_relation_with_long_name"."id" "chapter__book_model_relation_with_long_name__author_model_relation_with_long_name.id"
...
FROM ...
and the returned alias from Postgres does not include the dot . that tortoise needed to split the field name from the model reference alias.
Expected behavior
I expect this not to raise an exception and work as any other select with joins.
Additional context
AS tortoise generates the SQL to be executed, it knows in which order the rows will be returned from asyncpg. Tortoise could probably use the column index of each row and map that index to the long alias name instead of relying in the alias returned by Postgres.
I'm not sure how hard this is to implement. If you guys could provide any comments I could give a try to solve this problem.
The text was updated successfully, but these errors were encountered:
Describe the bug
Hey guys, I'm using tortoise in production and I found a Postgres specific bug.
Postgres has a limitation where identifiers must be at most 63 character long. This means in practice that if you create a column name or alias which is longer than 63 bytes long, Postgres will truncate it.
This limit can be configured but it requires re-compiling Postgres which is not practical when using cloud-managed solutions.
Here's an example
returns
This is a problem for tortoise which relies on columns aliases to resolve field names and build related models when doing joins, or even if any field has a very long name.
To Reproduce
This is a kind of comical example that abuses the length of field names. But note, in production models tend to have descriptive names and it is not uncommon to require many joins when executing a queryset with
.select_related()
, which generates very long aliases by joining field names using double underscores.For executing
Which raises
in
Note that the issue is having an alias which is very long
and the returned alias from Postgres does not include the dot
.
that tortoise needed to split the field name from the model reference alias.Expected behavior
I expect this not to raise an exception and work as any other select with joins.
Additional context
AS tortoise generates the SQL to be executed, it knows in which order the rows will be returned from
asyncpg
. Tortoise could probably use the column index of each row and map that index to the long alias name instead of relying in the alias returned by Postgres.I'm not sure how hard this is to implement. If you guys could provide any comments I could give a try to solve this problem.
The text was updated successfully, but these errors were encountered: