fix: mysql generated schema syntax error #1845

waketzheng · 2025-01-08T14:16:51Z

Description

Fixes #1836

Motivation and Context

Remove ' IF NOT EXISTS' for INDEX creation sql if dialect is mysql.
Delete extra space in CREATE INDEX when INDEX_TYPE is empty.

How Has This Been Tested?

make ci

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added the changelog accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

codspeed-hq · 2025-01-08T14:18:40Z

CodSpeed Performance Report

Merging #1845 will not alter performance

_{Comparing waketzheng:fix-mysql-schema (b0c4f8a) with develop (e34c2c0)}

Summary

✅ 8 untouched benchmarks

coveralls · 2025-01-08T14:37:38Z

Pull Request Test Coverage Report for Build 12732215675

Details

4 of 4 (100.0%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.252%

Totals
Change from base Build 12714096970:	0.0%
Covered Lines:	6432
Relevant Lines:	7011

💛 - Coveralls

henadzit · 2025-01-08T22:01:27Z

tortoise/utils.py

+    if schema := generator.get_create_schema_sql(safe):  # pragma: nobranch
+        if client.schema_generator.DIALECT == "mysql":
+            # MySQL does not support 'IF NOT EXISTS' syntax for `INDEX`
+            schema = schema.replace(" INDEX IF NOT EXISTS", " INDEX")


This code is shared across all supported integrations, we shouldn't be really making hacks for particular databases here since we already use inheritance to customize the behavior of schema generation. Imagine if we start handling all edge cases for all databases here - the code will become unreadable pretty soon.

Looks like the issue is coming from https://github.com/tortoise/tortoise-orm/blob/develop/tortoise/indexes.py#L62. What is going to happen if we replace self.INDEX_CREATE_TEMPLATE with schema_generator.INDEX_CREATE_TEMPLATE?

I think it would be more correct to get rid of Index.get_sql altogether and just implement SQL generation for Index in schema_generator subclass, hence being able to customize it for each database.

@henadzit I add a class _db_mysql to the Index class.

@waketzheng what do you think about my other suggestions? We work quite a lot together on this project and it would help a lot if we communicate better. In this example it would help if you shared your thoughts on why you decided to approach this problem in a different way than you initially planned or than proposed by me.

I think it would be more correct to get rid of Index.get_sql altogether and just implement SQL generation for Index in schema_generator subclass, hence being able to customize it for each database.

What is going to happen if we replace self.INDEX_CREATE_TEMPLATE with schema_generator.INDEX_CREATE_TEMPLATE?

@henadzit I think word is limit. For me, code is much easy to understand than word. Many many times, I was able to get the point why sb code like that quickly without reading any comments.
This PR was expected to be a hotfix for the BUG. It may not be the best solution for the creation of Index. But keep the library work correctly is more important that make code better. I would rather fix bug soon and leave the refactor job later.
Why not replace self.INDEX_CREATE_TEMPLATE with schema_generator.INDEX_CREATE_TEMPLATE:

As the above picture showing, both tortoise/backends/mysql/schema_generator.py and tortoise.indexes.py have a INDEX_CREATE_TEMPLATE var, and they are different value. It mean that get rid of Index.get_sql altogether in schema_generator will cost much time to do the refactor. Let's keep small step forward, don't change too much in such a hotfix PR!

It isn't a hot fix PR since the bug hasn't gotten into any release.

I proposed the changes and you ignored my comments pretty much and made other changes. That isn't just about getting the point, it's simple politeness.

if you look at any successful open-source projects, you will see a lot of discussions and good communication.

Your changes at the moment aren't just a bug fix. You are introducing a new concept where the create index statements can be customized via class _db_*. The way it works in open-source and pretty much in any software development, other people will start following the pattern and extending the code base in a similar manner.

We need to be trying to improve the code base with each our change, otherwise it will decay pretty quickly. "I will do a hot fix and then we refactor it later" rarely works.

I'll give a shot at this issue to see if there is an approach where we can simplify things.

@henadzit

No matter whether it is a hot fix or not, we should keep the develop branch work fine.

Actually, I try your suggestion get rid of Index.get in schema_generator and find it hard to do that, so I push commits with another solution, I expected to get your response after compare these two solutions.

I read some PRs in poetry and fastapi, they did communicate a lot, but sometimes replied after several weeks. It is my pleasure that if we can communicate quickly and smoothly.

class _db_* is not a new concept, you can find it at https://github.com/tortoise/tortoise-orm/blob/develop/tortoise/fields/data.py#L90

Everyone have only 24 hours one day, we should focus on crucial things, "rarely" mean that there are jobs more important that this one.

I'll give a shot at this issue to see if there is an approach where we can simplify things.
OK, I'll pull your branch to local and add some tests.

henadzit · 2025-01-10T15:35:06Z

Here's my shot at this issue , where I'm removing Index.sql and relaying only on schema_generator to build correct SQL. Please have a look!

waketzheng · 2025-01-12T10:05:46Z

Here's my shot at this issue , where I'm removing Index.sql and relaying only on schema_generator to build correct SQL. Please have a look!

@henadzit My PR includes some useful tests. How about merge it and then your new PR can use them directly?

henadzit · 2025-01-13T15:51:12Z

tests/utils/test_describe_model.py



 class TestDescribeModels(test.TestCase):
    def test_describe_models_all_serializable(self):
        val = Tortoise.describe_models()
-        json.dumps(val)
+        json.dumps(val, default=dump_index)


This hides the issue that Index cannot be described in a serializable way, however, it doesn't fix it. I fixed it in my PR by implementing Index.describe.

henadzit · 2025-01-13T16:38:04Z

tests/testmodels.py

@@ -150,6 +151,9 @@ class ModelTestPydanticMetaBackwardRelations3(Model):
 class Node(Model):
    name = fields.CharField(max_length=10)

+    class Meta:
+        indexes = [Index(fields=("name",))]


It looks like you added this index because it exposes the issue with serializing Index in Model.describe. I was able to identify it only because I worked on the same issue in my PR, it wouldn't be clear to anyone reviewing this PR.

Even though this index exposes an issue, it isn't documented and completely not obvious why it is here. Someone can remove it later without knowing its purpose and the same regression can happen again. I added a dedicated class for testing indexes in my PR.

fix: mysql generated schema syntax error

2aa5e05

henadzit reviewed Jan 8, 2025

View reviewed changes

waketzheng added 5 commits January 9, 2025 14:31

tests: add test case for Index

b319db3

refactor: change index sql in its method

fe2fcbf

Remove extra space

7fd4742

refactor: use dialect special style for index

166cb3c

fix unittest error

e93fd7e

waketzheng added 2 commits January 12, 2025 16:15

Merge branch 'develop' into fix-mysql-schema

7db30bf

Only change get_sql

b0c4f8a

henadzit reviewed Jan 13, 2025

View reviewed changes

waketzheng closed this Jan 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: mysql generated schema syntax error #1845

fix: mysql generated schema syntax error #1845

waketzheng commented Jan 8, 2025 •

edited

Loading

codspeed-hq bot commented Jan 8, 2025 •

edited

Loading

coveralls commented Jan 8, 2025 •

edited

Loading

henadzit Jan 8, 2025 •

edited

Loading

waketzheng Jan 9, 2025

henadzit Jan 9, 2025

waketzheng Jan 9, 2025 •

edited

Loading

henadzit Jan 9, 2025 •

edited

Loading

waketzheng Jan 10, 2025

henadzit commented Jan 10, 2025 •

edited

Loading

waketzheng commented Jan 12, 2025

henadzit Jan 13, 2025

henadzit Jan 13, 2025

fix: mysql generated schema syntax error #1845

fix: mysql generated schema syntax error #1845

Conversation

waketzheng commented Jan 8, 2025 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Checklist:

codspeed-hq bot commented Jan 8, 2025 • edited Loading

CodSpeed Performance Report

Merging #1845 will not alter performance

Summary

coveralls commented Jan 8, 2025 • edited Loading

Pull Request Test Coverage Report for Build 12732215675

Details

💛 - Coveralls

henadzit Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

waketzheng Jan 9, 2025

Choose a reason for hiding this comment

henadzit Jan 9, 2025

Choose a reason for hiding this comment

waketzheng Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

henadzit Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

waketzheng Jan 10, 2025

Choose a reason for hiding this comment

henadzit commented Jan 10, 2025 • edited Loading

waketzheng commented Jan 12, 2025

henadzit Jan 13, 2025

Choose a reason for hiding this comment

henadzit Jan 13, 2025

Choose a reason for hiding this comment

waketzheng commented Jan 8, 2025 •

edited

Loading

codspeed-hq bot commented Jan 8, 2025 •

edited

Loading

coveralls commented Jan 8, 2025 •

edited

Loading

henadzit Jan 8, 2025 •

edited

Loading

waketzheng Jan 9, 2025 •

edited

Loading

henadzit Jan 9, 2025 •

edited

Loading

henadzit commented Jan 10, 2025 •

edited

Loading