Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test uses assertDictEqual which (in this case) is unreliable #335

Open
alanbchristie opened this issue Aug 18, 2022 · 0 comments
Open

Test uses assertDictEqual which (in this case) is unreliable #335

alanbchristie opened this issue Aug 18, 2022 · 0 comments
Assignees
Labels
bug Something isn't working High priority

Comments

@alanbchristie
Copy link
Collaborator

The do_full_scan() method (line 721 of test_api.py in the latest main version) executes this test...

self.assertDictEqual(
    json.loads(json.dumps(response.json(), sort_keys=True)),
    json.loads(json.dumps(test_data_set[get_type], sort_keys=True)),
)

But this test is proving to be fragile. Although the top-level keys are correct one entry in the dictionary (results) is a list with two entries. The order of the list has now changed. But the entries in the list are identical. Here's the current failure (seen in fragalysis-stack)...

tests_1     | ======================================================================
[461](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:462)
tests_1     | FAIL: test_secure (tests.test_api.APIUrlsTestCase)
[462](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:463)
tests_1     | ----------------------------------------------------------------------
[463](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:464)
tests_1     | Traceback (most recent call last):
[464](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:465)
tests_1     |   File "/code/tests/test_api.py", line 723, in test_secure
[465](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:466)
tests_1     |     self.do_full_scan(self.user_two, self.secret_target_data)
[466](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:467)
tests_1     |   File "/code/tests/test_api.py", line 718, in do_full_scan
[467](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:468)
tests_1     |     json.loads(json.dumps(test_data_set[get_type], sort_keys=True)),
[468](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:469)
tests_1     | AssertionError: {'cou[86 chars]id': 1, 'metadata': None, 'project_id': [1], '[432 chars]ne}]} != {'cou[86 chars]id': 2, 'metadata': None, 'project_id': [2], '[432 chars]ne}]}
[469](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:470)
tests_1     |   {'count': 2,
[470](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:471)
tests_1     |    'next': None,
[471](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:472)
tests_1     |    'previous': None,
[472](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:473)
tests_1     |    'results': [{'default_squonk_project': None,
[473](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:474)
tests_1     | +               'id': 2,
[474](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:475)
tests_1     | +               'metadata': None,
[475](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:476)
tests_1     | +               'project_id': [2],
[476](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:477)
tests_1     | +               'protein_set': [2],
[477](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:478)
tests_1     | +               'sequences': [{'chain': '', 'sequence': ''}],
[478](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:479)
tests_1     | +               'template_protein': '/media/secret_pdb.pdb',
[479](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:480)
tests_1     | +               'title': 'SECRET_TARGET',
[480](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:481)
tests_1     | +               'upload_status': None,
[481](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:482)
tests_1     | +               'zip_archive': None},
[482](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:483)
tests_1     | +              {'default_squonk_project': None,
[483](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:484)
tests_1     |                 'id': 1,
[484](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:485)
tests_1     |                 'metadata': None,
[485](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:486)
tests_1     |                 'project_id': [1],
[486](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:487)
tests_1     |                 'protein_set': [1],
[487](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:488)
tests_1     |                 'sequences': [{'chain': '', 'sequence': ''}],
[488](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:489)
tests_1     |                 'template_protein': '/media/my_pdb.pdb',
[489](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:490)
tests_1     |                 'title': 'DUMMY_TARGET',
[490](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:491)
tests_1     |                 'upload_status': None,
[491](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:492)
tests_1     | -               'zip_archive': None},
[492](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:493)
tests_1     | -              {'default_squonk_project': None,
[493](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:494)
tests_1     | -               'id': 2,
[494](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:495)
tests_1     | -               'metadata': None,
[495](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:496)
tests_1     | -               'project_id': [2],
[496](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:497)
tests_1     | -               'protein_set': [2],
[497](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:498)
tests_1     | -               'sequences': [{'chain': '', 'sequence': ''}],
[498](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:499)
tests_1     | -               'template_protein': '/media/secret_pdb.pdb',
[499](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:500)
tests_1     | -               'title': 'SECRET_TARGET',
[500](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:501)
tests_1     | -               'upload_status': None,
[501](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:502)
tests_1     |                 'zip_archive': None}]}
[502](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:503)
tests_1     | 
[503](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:504)
tests_1     | ----------------------------------------------------------------------
[504](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:505)
tests_1     | Ran 12 tests in 8.297s
[505](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:506)
tests_1     | 
[506](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:507)
tests_1     | FAILED (failures=1)
[507](https://github.com/xchem/fragalysis-backend/runs/7714315052?check_suite_focus=true#step:7:508)

The backend tests are OK, when run in the stack repo they fail.

Is the test a valid test (i.e. is the order of results important)? My guess is unlikely. Regardless, the test needs to be changed. This test is running succes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working High priority
Projects
None yet
Development

No branches or pull requests

1 participant