Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix all config files #614

Merged
merged 18 commits into from
Oct 11, 2023
Merged

Fix all config files #614

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
e1a2cbc
rename field
b1ron Oct 6, 2023
3ce632b
fix main.go
b1ron Oct 6, 2023
8f22249
dbaas
b1ron Oct 6, 2023
f3e7599
more
b1ron Oct 6, 2023
2fa8ed1
more
b1ron Oct 6, 2023
29d9634
revert unnecessary renaming
b1ron Oct 10, 2023
b65baa0
stats
b1ron Oct 10, 2023
5ef68a0
tools
b1ron Oct 10, 2023
d5c3b52
python-example
b1ron Oct 10, 2023
dd3589d
ycsb
b1ron Oct 10, 2023
f6764db
calculation bug
b1ron Oct 10, 2023
e734496
keep UnexpectedSkip only
b1ron Oct 10, 2023
ea0f5a2
maintain the order
b1ron Oct 10, 2023
83eb03d
remove field and ignore problematic skips
b1ron Oct 10, 2023
d5c7170
allow verbose
b1ron Oct 10, 2023
8de3acd
kinda fixed
b1ron Oct 11, 2023
da7e3a0
remove verbose
b1ron Oct 11, 2023
7455714
sqlite
b1ron Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions internal/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,9 +194,9 @@ func (tc *TestConfig) Compare(results *TestResults) (*CompareResult, error) {
case Fail:
compareResult.ExpectedFail[test] = testResOutput
case Pass:
compareResult.UnexpectedPass[test] = testResOutput
compareResult.ExpectedPass[test] = testResOutput
case Skip:
compareResult.UnexpectedSkip[test] = testResOutput
compareResult.ExpectedSkip[test] = testResOutput
case Ignore:
fallthrough
case Unknown:
Expand Down
18 changes: 12 additions & 6 deletions internal/configload/stats.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@ import ic "github.com/FerretDB/dance/internal/config"

// stats represents the YAML representation of expected stats.
type stats struct {
Fail int `yaml:"fail"`
Skip int `yaml:"skip"`
Pass int `yaml:"pass"`
UnexpectedFail int `yaml:"unexpected_fail"`
UnexpectedSkip int `yaml:"unexpected_skip"`
UnexpectedPass int `yaml:"unexpected_pass"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are going back and forth there. Those fields should never be present in the YAML configuration file – we don't expect the unexpected. fail, skip, and pass are expected values. Unexpected values should not be in the file.

We split config and configload into separate packages exactly to have them separate. Why are we trying to blend them together again?

Copy link
Contributor Author

@b1ron b1ron Oct 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad for that confusion you are completely right. It's a temporary fix for when we removed our support for regex, as that logic accounted for Go tests that should be skipped. So, now they will be marked as UnexpectedSkip. All those other unexpected fields are redundant so I will remove them. I will also add a comment.

It's a little hacky but at least we have a working CI.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove UnexpectedSkip and all associated tests. Adding a comment does not change the fact that we expect unexpected things

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK cool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to remove all associated tests with unexpected skips but it didn't work as some tests rely on variables and helpers, etc. that are contained within other files.

I think I have a less tedious approach now.

Fail int `yaml:"fail"`
Skip int `yaml:"skip"`
Pass int `yaml:"pass"`
}

// convert converts stats to [*config.Stats].
Expand All @@ -30,8 +33,11 @@ func (s *stats) convert() *ic.Stats {
}

return &ic.Stats{
ExpectedFail: s.Fail,
ExpectedSkip: s.Skip,
ExpectedPass: s.Pass,
UnexpectedFail: s.UnexpectedFail,
UnexpectedSkip: s.UnexpectedSkip,
UnexpectedPass: s.UnexpectedPass,
ExpectedFail: s.Fail,
ExpectedSkip: s.Skip,
ExpectedPass: s.Pass,
}
}
12 changes: 6 additions & 6 deletions tests/dbaas_core-0.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -662,8 +662,8 @@ results:

postgresql:
stats:
expected_fail: 178
expected_pass: 67
fail: 178
pass: 67
fail:
include_fail:
- index
Expand All @@ -673,8 +673,8 @@ results:

sqlite:
stats:
expected_fail: 178
expected_pass: 67
fail: 178
pass: 67
fail:
include_fail:
- query
Expand All @@ -684,8 +684,8 @@ results:

mongodb:
stats:
expected_fail: 43
expected_pass: 202
fail: 43
pass: 202
fail:
- mongo/jstests/core/api/api_version_unstable_fields.js
- mongo/jstests/core/columnstore/columnstore_index.js
Expand Down
8 changes: 4 additions & 4 deletions tests/dbaas_core-1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -673,8 +673,8 @@ results:

sqlite:
stats:
expected_fail: 178
expected_pass: 66
fail: 178
pass: 66
fail:
include_fail:
- query
Expand All @@ -684,8 +684,8 @@ results:

mongodb:
stats:
expected_fail: 43
expected_pass: 201
fail: 43
pass: 201
fail:
- mongo/jstests/core/capped/capped_resize.js
- mongo/jstests/core/covered_query_with_sort.js
Expand Down
12 changes: 6 additions & 6 deletions tests/dbaas_core-2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -660,8 +660,8 @@ results:

postgresql:
stats:
expected_fail: 181
expected_pass: 62
fail: 181
pass: 62
fail:
include_fail:
- query
Expand All @@ -671,8 +671,8 @@ results:

sqlite:
stats:
expected_fail: 181
expected_pass: 62
fail: 181
pass: 62
fail:
include_fail:
- query
Expand All @@ -682,8 +682,8 @@ results:

mongodb:
stats:
expected_fail: 43
expected_pass: 200
fail: 43
pass: 200
fail:
- mongo/jstests/core/api/api_version_unstable_indexes.js
- mongo/jstests/core/clustered/clustered_collection_hint.js
Expand Down
12 changes: 6 additions & 6 deletions tests/dbaas_core-3.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -650,8 +650,8 @@ results:

postgresql:
stats:
expected_fail: 181
expected_pass: 57
fail: 181
pass: 57
fail:
include_fail:
- query
Expand All @@ -661,8 +661,8 @@ results:

sqlite:
stats:
expected_fail: 181
expected_pass: 57
fail: 181
pass: 57
fail:
include_fail:
- query
Expand All @@ -672,8 +672,8 @@ results:

mongodb:
stats:
expected_fail: 54
expected_pass: 184
fail: 54
pass: 184
fail:
- mongo/jstests/core/api/api_version_new_50_language_features.js
- mongo/jstests/core/api/apitest_db_profile_level.js
Expand Down
8 changes: 4 additions & 4 deletions tests/diff.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ args: ["-timeout=30s", "-shuffle=on", "./..."]
results:
ferretdb:
stats:
expected_fail: 95
expected_pass: 38
fail: 95
pass: 38

pass:
- regex: FerretDB$
Expand All @@ -16,8 +16,8 @@ results:

mongodb:
stats:
expected_fail: 95
expected_pass: 38
fail: 95
pass: 38

pass:
- regex: MongoDB$
Expand Down
6 changes: 3 additions & 3 deletions tests/dotnet-example-auth.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ args:
results:
postgresql:
stats:
expected_pass: 1
pass: 1

sqlite:
stats:
expected_pass: 1
pass: 1

mongodb:
stats:
# PLAIN is used in MongoDB to perform LDAP authentication.
expected_fail: 1
fail: 1
fail:
- dotnet-example
6 changes: 3 additions & 3 deletions tests/dotnet-example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ args:
results:
postgresql:
stats:
expected_pass: 1
pass: 1

sqlite:
stats:
expected_pass: 1
pass: 1

mongodb:
stats:
expected_pass: 1
pass: 1
2 changes: 1 addition & 1 deletion tests/enmeshed-runtime.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ args: [../enmeshed-runtime.sh]
results:
common:
stats:
expected_pass: 1
pass: 1

ferretdb:
stats:
Expand Down
6 changes: 3 additions & 3 deletions tests/java-example-auth.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ args: [../../java-example-auth.sh]
results:
postgresql:
stats:
expected_pass: 1
pass: 1

sqlite:
stats:
expected_pass: 1
pass: 1

mongodb:
stats:
# PLAIN is used in MongoDB to perform LDAP authentication.
expected_fail: 1
fail: 1
fail:
- java-example
6 changes: 3 additions & 3 deletions tests/java-example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ args: [../../java-example.sh]
results:
postgresql:
stats:
expected_pass: 1
pass: 1

sqlite:
stats:
expected_pass: 1
pass: 1

mongodb:
stats:
expected_pass: 1
pass: 1
18 changes: 9 additions & 9 deletions tests/mongo-go-driver.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,10 +170,10 @@ results:

postgresql:
stats:
unexpected_skip: 21
expected_fail: 10
expected_skip: 0
expected_pass: 633
unexpected_skip: 12
fail: 10
skip: 9
pass: 633
ignore:
# returns unknown result.
- go.mongodb.org/mongo-driver/mongo/integration/TestClientStress
Expand All @@ -190,9 +190,9 @@ results:
sqlite:
stats:
unexpected_skip: 21
expected_fail: 11
expected_skip: 0
expected_pass: 632
fail: 11
skip: 0
pass: 632
ignore:
# returns unknown result.
- go.mongodb.org/mongo-driver/mongo/integration/TestClientStress
Expand All @@ -211,8 +211,8 @@ results:
mongodb:
stats:
unexpected_skip: 352
expected_skip: 83
expected_pass: 2259
skip: 83
pass: 2259
skip:
# execution failed: test must be skipped: "the \"modifyCollection\" operation is not supported"
- go.mongodb.org/mongo-driver/mongo/integration/unified/TestUnifiedSpec/collection-management/modifyCollection-pre_and_post_images.json/modifyCollection_to_changeStreamPreAndPostImages_enabled
Expand Down
6 changes: 3 additions & 3 deletions tests/mongo-tools.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ args: ["-timeout=20m", "-shuffle=on", "./..."]
results:
postgresql:
stats:
expected_pass: 6
pass: 6

sqlite:
stats:
expected_pass: 6
pass: 6

mongodb:
stats:
expected_pass: 6
pass: 6
12 changes: 6 additions & 6 deletions tests/mongo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,8 @@ results:

postgresql:
stats:
expected_fail: 62
expected_pass: 39
fail: 62
pass: 39
fail:
- mongo/jstests/core/write/update/updatej.js
include_fail:
Expand All @@ -156,8 +156,8 @@ results:

sqlite:
stats:
expected_fail: 61
expected_pass: 40
fail: 61
pass: 40
fail:
include_fail:
- aggregation
Expand All @@ -168,8 +168,8 @@ results:

mongodb:
stats:
expected_fail: 8
expected_pass: 93
fail: 8
pass: 93
fail:
# both tests invoke the MongoRunner and also fail on resmoke.py with exit code 253.
# they both seem to use a key file with incorrect permissions.
Expand Down
6 changes: 3 additions & 3 deletions tests/python-example-auth.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ args: [../python-example-auth.sh]
results:
postgresql:
stats:
expected_pass: 1
pass: 1

sqlite:
stats:
expected_pass: 1
pass: 1

mongodb:
stats:
# PLAIN is used in MongoDB to perform LDAP authentication.
expected_fail: 1
fail: 1
fail:
- python-example
6 changes: 3 additions & 3 deletions tests/python-example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ args: [../python-example.sh]
results:
postgresql:
stats:
expected_pass: 1
pass: 1

sqlite:
stats:
expected_pass: 1
pass: 1

mongodb:
stats:
expected_pass: 1
pass: 1
Loading