New implementation of parser WIP #199

vaibhavchellani · 2018-06-04T08:07:17Z

Fixes #193

Checklist

My branch is up-to-date with upstream/develop branch.
Everything works and tested for Python 3.5.2 and above.

Description

We are now able to parse the objects defined at variable locations , able to identify collections and present a more accurate conversion of open api spec to hydra spec

Test Logs

@xadahiya @Shekharrajak @kkoci @pcalcina @chrizandr

vaibhavchellani · 2018-06-04T08:11:51Z

there is excessive logging in this presently for debugging purposes , will remove once approved .

vaibhavchellani · 2018-06-06T12:18:41Z

@xadahiya @chrizandr i am stuck with test_app.py on this , unable to get through that ,please review and let me know if you found any solution or recommend changes

xadahiya · 2018-06-06T12:40:07Z

@vaibhavchellani Tests are working for me(test_app, test_auth) only one test if failing. See this

vaibhavchellani · 2018-06-06T12:43:09Z

you tried with the doc that was created by the parser ? @xadahiya

xadahiya · 2018-06-06T12:50:21Z

Yeah, I'm using hydrus.samples.hydra_doc_sample. Only CRUD tests are failing. Rest everything works fine.

vaibhavchellani · 2018-06-07T05:31:05Z

hey @xadahiya thanks , however these are my test results , and i am unable to debug these .

xadahiya · 2018-06-07T05:50:08Z

Show me your test_app file

vaibhavchellani · 2018-06-07T05:52:52Z

https://github.com/vaibhavchellani/hydrus/blob/test/hydrus/tests/test_app.py

xadahiya · 2018-06-07T06:02:40Z

I'm doing the exact same thing see https://www.diffchecker.com/yvyAJW9p . Probably your hydrus installation is outdated. Do one thing cd to main hydrus directory and run pip install -e .. This should fix the problem.

vaibhavchellani · 2018-06-07T06:07:53Z

okay yes , that did it , now ill try to debug and will let you know

xadahiya · 2018-06-07T06:09:18Z

See test_crud you'll probably get errors in that. Also, please try to finish this in a day. I'll review your code then. Good luck :)

vaibhavchellani · 2018-06-09T07:31:05Z

xadahiya · 2018-06-09T10:43:57Z

@vaibhavchellani Can you update the code to follow mypy format like the rest of Hydrus code.
https://mypy.readthedocs.io/en/latest/

xadahiya · 2018-06-09T17:25:19Z

Add pyyaml to requirements.txt

xadahiya · 2018-06-09T17:38:18Z

Also, remove redundant print statements rest everything looks fine for now.

xadahiya · 2018-06-10T06:52:03Z

@vaibhavchellani You haven't added any documentation to the wiki yet like we discussed in the call. Add a page expaining the parser structure to https://github.com/HTTP-APIs/hydra-ecosystem-wiki/.

Also, finish the things I've mentioned above as soon as you can. We need to merge soon!

xadahiya · 2018-06-10T07:05:52Z

@vaibhavchellani How are you tests passing? Mine are still failing.
These are the files I'm using https://gist.github.com/xadahiya/fad7037ca85975c85a23b2431d911024
Logs from test_app.py

(hydra) xadahiya@xa-Inspiron-7560:~/hydra/hydrus/hydrus/tests$ python -m unittest -v test_app
Creating a temporary database...
going for create doc
Classes and properties added successfully.
Setting up Hydrus utilities... 
Creating utilities context... 
Setup done, running tests...
test_Collections_GET (test_app.ViewsTestCase)
Test GET on collection endpoints. ... ok
test_Collections_PUT (test_app.ViewsTestCase)
Test insert data to the collection. ... ok
test_Endpoints_Contexts (test_app.ViewsTestCase)
Test all endpoints contexts are generated properly. ... ok
test_EntryPoint_context (test_app.ViewsTestCase)
Test for the EntryPoint context. ... ok
test_Index (test_app.ViewsTestCase)
Test for the index. ... ok
test_Vocab (test_app.ViewsTestCase)
Test the vocab. ... ok
test_bad_objects (test_app.ViewsTestCase)
Checks if bad objects are added or not. ... ok
test_bad_requests (test_app.ViewsTestCase)
Checks if bad requests are handled or not. ... ok
test_endpointClass_DELETE (test_app.ViewsTestCase)
Check non collection Class DELETE. ... ok
test_endpointClass_GET (test_app.ViewsTestCase)
Check non collection Class GET. ... [2018-06-10 12:29:12,875] ERROR in app: Exception on /demoapi/Pet [GET]
Traceback (most recent call last):
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/xadahiya/hydra/hydrus/hydrus/app.py", line 353, in get
    abort(endpoint_['status'])
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/werkzeug/exceptions.py", line 707, in abort
    return _aborter(status, *args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/werkzeug/exceptions.py", line 686, in __call__
    raise LookupError('no exception for %r' % code)
LookupError: no exception for 200
FAIL
test_endpointClass_POST (test_app.ViewsTestCase)
Check non collection Class POST. ... [2018-06-10 12:29:12,881] ERROR in app: Exception on /demoapi/Pet [POST]
Traceback (most recent call last):
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/xadahiya/hydra/hydrus/hydrus/app.py", line 449, in post
    abort(endpoint_['status'])
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/werkzeug/exceptions.py", line 707, in abort
    return _aborter(status, *args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/werkzeug/exceptions.py", line 686, in __call__
    raise LookupError('no exception for %r' % code)
LookupError: no exception for 200
FAIL
test_endpointClass_PUT (test_app.ViewsTestCase)
Check non collection Class PUT. ... [2018-06-10 12:29:12,887] ERROR in app: Exception on /demoapi/Pet [PUT]
Traceback (most recent call last):
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/xadahiya/hydra/hydrus/hydrus/app.py", line 416, in put
    abort(endpoint_['status'])
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/werkzeug/exceptions.py", line 707, in abort
    return _aborter(status, *args, **kwargs)
  File "/home/xadahiya/miniconda3/envs/hydra/lib/python3.6/site-packages/werkzeug/exceptions.py", line 686, in __call__
    raise LookupError('no exception for %r' % code)
LookupError: no exception for 200
FAIL
test_object_DELETE (test_app.ViewsTestCase)
Test DELETE of a given object using ID. ... ok
test_object_POST (test_app.ViewsTestCase)
Test replace of a given object using ID. ... ok
test_object_PUT_at_id (test_app.ViewsTestCase)
Create object in collection using PUT at specific ID. ... ok

======================================================================
FAIL: test_endpointClass_GET (test_app.ViewsTestCase)
Check non collection Class GET.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_app.py", line 271, in test_endpointClass_GET
    assert response_get.status_code in [200, 404]
AssertionError

======================================================================
FAIL: test_endpointClass_POST (test_app.ViewsTestCase)
Check non collection Class POST.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_app.py", line 245, in test_endpointClass_POST
    assert put_response.status_code == 201
AssertionError

======================================================================
FAIL: test_endpointClass_PUT (test_app.ViewsTestCase)
Check non collection Class PUT.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_app.py", line 231, in test_endpointClass_PUT
    assert put_response.status_code == 201
AssertionError

----------------------------------------------------------------------
Ran 15 tests in 0.463s

FAILED (failures=3)

Logs from test_crud.py

(hydra) xadahiya@xa-Inspiron-7560:~/hydra/hydrus/hydrus/tests$ python -m unittest -v test_crud
Creating a temporary datatbsse...
Classes and properties added successfully.
Setup done, running tests...
test_delete (test_crud.TestCRUD)
Test CRUD delete. ... ERROR
test_delete_id (test_crud.TestCRUD)
Test CRUD delete when wrong/undefined ID is given. ... ERROR
test_delete_type (test_crud.TestCRUD)
Test CRUD delete when wrong/undefined class is given. ... ERROR
test_get (test_crud.TestCRUD)
Test CRUD get. ... ERROR
test_get_id (test_crud.TestCRUD)
Test CRUD get when wrong/undefined ID is given. ... FAIL
test_get_type (test_crud.TestCRUD)
Test CRUD get when wrong/undefined class is given. ... ok
test_insert (test_crud.TestCRUD)
Test CRUD insert. ... ERROR
test_insert_id (test_crud.TestCRUD)
Test CRUD insert when used ID is given. ... ERROR
test_insert_type (test_crud.TestCRUD)
Test CRUD insert when wrong/undefined class is given. ... ERROR
test_update (test_crud.TestCRUD)
Test CRUD update. ... ERROR

======================================================================
ERROR: test_delete (test_crud.TestCRUD)
Test CRUD delete.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 86, in test_delete
    insert_response = crud.insert(object_=object_, id_=id_, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

======================================================================
ERROR: test_delete_id (test_crud.TestCRUD)
Test CRUD delete when wrong/undefined ID is given.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 136, in test_delete_id
    insert_response = crud.insert(object_=object_, id_=id_, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

======================================================================
ERROR: test_delete_type (test_crud.TestCRUD)
Test CRUD delete when wrong/undefined class is given.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 122, in test_delete_type
    insert_response = crud.insert(object_=object_, id_=id_, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

======================================================================
ERROR: test_get (test_crud.TestCRUD)
Test CRUD get.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 64, in test_get
    response = crud.insert(object_=object_, id_=id_, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

======================================================================
ERROR: test_insert (test_crud.TestCRUD)
Test CRUD insert.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 57, in test_insert
    response = crud.insert(object_=object_, id_=1, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

======================================================================
ERROR: test_insert_id (test_crud.TestCRUD)
Test CRUD insert when used ID is given.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 164, in test_insert_id
    insert_response = crud.insert(object_=object_, id_=id_, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 166, in test_insert_id
    response_code, message = e.get_HTTP()
AttributeError: 'TypeError' object has no attribute 'get_HTTP'

======================================================================
ERROR: test_insert_type (test_crud.TestCRUD)
Test CRUD insert when wrong/undefined class is given.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 150, in test_insert_type
    object_["@type"] = "otherClass"
TypeError: 'NoneType' object does not support item assignment

======================================================================
ERROR: test_update (test_crud.TestCRUD)
Test CRUD update.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 74, in test_update
    insert_response = crud.insert(object_=object_, id_=id_, session=self.session)
  File "/home/xadahiya/hydra/hydrus/hydrus/data/crud.py", line 111, in insert
    RDFClass.name == object_["@type"]).one()
TypeError: 'NoneType' object is not subscriptable

======================================================================
FAIL: test_get_id (test_crud.TestCRUD)
Test CRUD get when wrong/undefined ID is given.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/xadahiya/hydra/hydrus/hydrus/tests/test_crud.py", line 105, in test_get_id
    assert 404 == response_code
AssertionError

----------------------------------------------------------------------
Ran 10 tests in 0.059s

FAILED (failures=1, errors=8)

vaibhavchellani · 2018-06-10T07:58:06Z

hey @xadahiya , yes i was unable to resolve this , i was able to take it to the one failure , the flock doc also had the same failure so i left it at that point as you suggested .

xadahiya · 2018-06-10T08:26:17Z

That's fine. Please finish the rest

vaibhavchellani · 2018-06-10T11:17:06Z

@xadahiya done ! let me know if any more changes are to be made .

xadahiya · 2018-06-10T14:09:36Z

@vaibhavchellani You need to add anything before merging?

vaibhavchellani · 2018-06-10T14:10:33Z

I dont think so @xadahiya

vaibhavchellani added 11 commits June 1, 2018 00:33

parsed paths and got classes

fa064a7

pep 8 alignment

46b4129

added sample

6bcf6c2

new working parser

9fc8d5c

fixed implementation errors

e2a615f

fixed bugs

863c980

fixed architecture

703b44b

added more error handling and logs

3e86860

new sample

af6872c

created get_ops

8e82683

fixed bugs with get_ops

c01f88e

vaibhavchellani added 8 commits June 4, 2018 13:47

added docstrings

54ee5f1

required functionality added

c1a61df

fixed bug

1bf32b7

added protocol to id

4c4415f

added trycatch replacement

13c5bfb

parsed responses

655c458

fixed status bug

3174a6f

removed schema from responses

3134e46

added modified test

cb19846

vaibhavchellani added 2 commits June 9, 2018 12:52

refactoed status represntation

8a45a98

changed tests to like they were before

4960fff

vaibhavchellani added 3 commits June 10, 2018 01:20

added test

5174f55

removed redundant print statements

87206b2

Added test file

907805e

mypy compliant

d97bee1

xadahiya merged commit 9d1bef6 into HTTP-APIs:develop Jun 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New implementation of parser WIP #199

New implementation of parser WIP #199

vaibhavchellani commented Jun 4, 2018 •

edited

Loading

vaibhavchellani commented Jun 4, 2018

vaibhavchellani commented Jun 6, 2018

xadahiya commented Jun 6, 2018 •

edited

Loading

vaibhavchellani commented Jun 6, 2018

xadahiya commented Jun 6, 2018

vaibhavchellani commented Jun 7, 2018

xadahiya commented Jun 7, 2018

vaibhavchellani commented Jun 7, 2018

xadahiya commented Jun 7, 2018

vaibhavchellani commented Jun 7, 2018

xadahiya commented Jun 7, 2018

vaibhavchellani commented Jun 9, 2018

xadahiya commented Jun 9, 2018

xadahiya commented Jun 9, 2018

xadahiya commented Jun 9, 2018

xadahiya commented Jun 10, 2018

xadahiya commented Jun 10, 2018

vaibhavchellani commented Jun 10, 2018

xadahiya commented Jun 10, 2018

vaibhavchellani commented Jun 10, 2018

xadahiya commented Jun 10, 2018

vaibhavchellani commented Jun 10, 2018

New implementation of parser WIP #199

New implementation of parser WIP #199

Conversation

vaibhavchellani commented Jun 4, 2018 • edited Loading

Checklist

Description

Test Logs

vaibhavchellani commented Jun 4, 2018

vaibhavchellani commented Jun 6, 2018

xadahiya commented Jun 6, 2018 • edited Loading

vaibhavchellani commented Jun 6, 2018

xadahiya commented Jun 6, 2018

vaibhavchellani commented Jun 7, 2018

xadahiya commented Jun 7, 2018

vaibhavchellani commented Jun 7, 2018

xadahiya commented Jun 7, 2018

vaibhavchellani commented Jun 7, 2018

xadahiya commented Jun 7, 2018

vaibhavchellani commented Jun 9, 2018

xadahiya commented Jun 9, 2018

xadahiya commented Jun 9, 2018

xadahiya commented Jun 9, 2018

xadahiya commented Jun 10, 2018

xadahiya commented Jun 10, 2018

vaibhavchellani commented Jun 10, 2018

xadahiya commented Jun 10, 2018

vaibhavchellani commented Jun 10, 2018

xadahiya commented Jun 10, 2018

vaibhavchellani commented Jun 10, 2018

vaibhavchellani commented Jun 4, 2018 •

edited

Loading

xadahiya commented Jun 6, 2018 •

edited

Loading