Feature/1693 api v3 delete recreate #2055

dk1844 · 2022-04-28T11:55:14Z

usedIn functionality both for .../{name}/used-in and .../{name}/{version}/used-in exists
- for depencies of: schema->dataset, schema->mapping tables, mapping tables -> dataset, propdef->dataset (all based on version-less used-in)
disable for entities via DELETE .../{name}, enable for entities via PUT .../{name}
- enable/disable is done for the whole entity (all versions at once), changes only possible on enabled entities (!isDisabled check).
UsedIn normalization, empty - V2 had it mixed (sometimes None, sometimes Some(Seq.empty) -> now it all Nones. + unittest
all of the above reflected in integTests.

Partially implements #1693

Release notes suggestion

Both /api-v3/{datasets|schemas|mapping-tables|property-defintions}/{name}/used-in and /api-v3/{datasets|schemas|mapping-tables|property-defintions}/{name}/{version}/used-in REST API V3 added, the former is used for disable requests to check the dependencies. UsedIn normalization introduced.

…e contours, empty DatasetControllerV3 to test

- login allowed at /api/login & /api-v3/login - v2/v3 BaseRestApiTest distinquished

…st cases added

…der and no content -> IT updated

… IT updated

…version, includes a sample CR)

…st and post-import + IT to prove correct behavior

…mport + IT to prove correct behavior - fix with common location processing with segment stripping (+normalization)

… GET datasets/{name}/{version}

…/audit-trail added

…ion} now works for # or 'latest' (IT = regression test)

…{name}/{version} and /{name}/latest - improved

…- supports latest for as version-expression, impl for datasets improved by actual existence checking + IT test cases for non-existing/non-latest queries

…properties - supports latest for as version-expression; get impl is unvalidated, put impl checks validity - login is now common, under /api/login for both v2 and v3 (did not work previously)

…properties - supports latest for as version-expression; get impl is unvalidated, put impl checks validity - extended for different validation cases - login is now common, under /api/login for both v2 and v3 (did not work previously)

…ture -> CompletableFuture (mistake reverted)

… added. IT mostly adjusted, but there are todos - DatasetServiceV3 introduced to carry difference in behavior to DatasetService. original entity validation has been divided into create-validation and regular-entity validation. - buildfix for VersionedModelServiceTest

…sion}` in proper validations

…asets/dsName/version/rules, GET datasets/dsName/version/rules/# + IT

- v2 ignores this added information - needs

…api-v3-schema-etc

…defaults

…ts + IT

@benedeki

…ion`. Small updates, thanks @benedeki

… with `MappingTableControllerV3.withMappingTableToResponse`

…3-delete-recreate

% Conflicts: % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/controllers/SchemaController.scala % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/controllers/v3/VersionedModelControllerV3.scala % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/SchemaService.scala % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/VersionedModelService.scala % rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/v3/DatasetControllerV3IntegrationSuite.scala

…3-delete-recreate % Conflicts: % rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/v3/DatasetControllerV3IntegrationSuite.scala

% Conflicts: % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/controllers/v3/PropertyDefinitionControllerV3.scala % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/v3/MappingTableServiceV3.scala % rest-api/src/main/scala/za/co/absa/enceladus/rest_api/services/v3/SchemaServiceV3.scala % rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/v3/MappingTableControllerV3IntegrationSuite.scala % rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/v3/PropertyDefinitionControllerV3IntegrationSuite.scala % rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/v3/SchemaControllerV3IntegrationSuite.scala

Zejnilovic

Tested the disabling and enabling + used-in on name and on name+version.

I am curious. When entity is disable and somebody knows its name he can still "use it" which seems a tad weird. Maybe we should come up with a definition of disabled. Like

disallow all PUT/POST calls except for PUT on /{name} to enable.
validate should return error disabled so it cannot be run in spark-jobs.
maybe also:
- disallow export
- disallow audit-trail
- on GET /{name} print name, version and disabled true/false (in general not just here)
- on GET /{name}/{version} just print basic info and say it is disabled

Happy to discuss more

...src/main/scala/za/co/absa/enceladus/rest_api/controllers/v3/VersionedModelControllerV3.scala

Zejnilovic · 2022-05-12T08:42:36Z

Tested the disabling and enabling + used-in on name and on name+version.

I am curious. When entity is disable and somebody knows its name he can still "use it" which seems a tad weird. Maybe we should come up with a definition of disabled. Like

disallow all PUT/POST calls except for PUT on /{name} to enable.

validate should return error disabled so it cannot be run in spark-jobs.

maybe also:

disallow export

disallow audit-trail

on GET /{name} print name, version and disabled true/false (in general not just here)

on GET /{name}/{version} just print basic info and say it is disabled

Happy to discuss more

We agreed on:

disallow all PUT/POST calls except for PUT on /{name} to enable.
validate should return error disabled so it cannot be run in spark-jobs.
on GET /{name} print name, version and status

…bled` information - mainly on GET ...{/name}

dk1844 · 2022-05-12T11:51:09Z

We agreed on:

disallow all PUT/POST calls except for PUT on /{name} to enable.

validate should return error disabled so it cannot be run in spark-jobs.
It turned out a bit bigger, so there is another PR to this branch Feature/1693 api v3 disabled fails validation #2066

on GET /{name} print name, version and status

implemented for V3 API by adding the disabled field to that "summary"

…tion. IT updated.

data-model/src/main/scala/za/co/absa/enceladus/model/versionedModel/NamedVersion.scala

… an error message (`UsedIn` wrapped in `EntityInUseException`)

…, cleanup

sonarqubecloud · 2022-05-18T09:31:28Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
18 Code Smells

No Coverage information
4.9% Duplication

AdrianOlosutean

Nice job, looks good to me. Code reviewed

Zejnilovic

Reviewed, built, run.

AdrianOlosutean · 2022-05-30T11:18:22Z

rest-api/src/main/scala/za/co/absa/enceladus/rest_api/controllers/v3/DatasetControllerV3.scala

@@ -57,14 +58,12 @@ class DatasetControllerV3 @Autowired()(datasetService: DatasetServiceV3)
        case Some((entity, validation)) =>


Shouln't this endpoint have the same validation?

dk1844 added 30 commits March 7, 2022 15:35

#1693 API v3: VersionedModel v3 init - VersionedModelControllerV3 bas…

f1798b2

…e contours, empty DatasetControllerV3 to test

#1693 API v3: some basic VersionedModelControllerV3 integration test

0f62851

- login allowed at /api/login & /api-v3/login - v2/v3 BaseRestApiTest distinquished

#1693 API v3: VersionedModelControllerV3 integration test cont'd - te…

c01cfda

…st cases added

#1693 API v3: VersionedModelControllerV3 - post to yield location hea…

4fd08e2

…der and no content -> IT updated

#1693 API v3: VersionedModelControllerV3 - put to yield no content ->…

5c8e005

… IT updated

#1693 API v3: VersionedModelControllerV3 - IT for /ds/export (latest …

458bfed

…version, includes a sample CR)

#1693 API v3: VersionedModelControllerV3 - location header for put/po…

a7a6cf9

…st and post-import + IT to prove correct behavior

#1693 API v3: VersionedModelControllerV3 - location header for post-i…

af489b1

…mport + IT to prove correct behavior - fix with common location processing with segment stripping (+normalization)

#1693 API v3: VersionedModelControllerV3 -IT for GET datasets/{name},…

21f8008

… GET datasets/{name}/{version}

#1693 API v3: VersionedModelControllerV3 - IT for GET datasets/{name}…

6f63117

…/audit-trail added

#1693 API v3: VersionedModelControllerV3 - GET /datasets/{name}/{vers…

76e87ce

…ion} now works for # or 'latest' (IT = regression test)

#1693 API v3: VersionedModelControllerV3 - common implementation of /…

d62cd90

…{name}/{version} and /{name}/latest - improved

#1693 API v3: VersionedModelControllerV3 - /{name}/{version}/used-in …

fc4c359

…- supports latest for as version-expression, impl for datasets improved by actual existence checking + IT test cases for non-existing/non-latest queries

#1693 API v3: VersionedModelControllerV3 - GET/PUT /{name}/{version}/…

e6c8d8c

…properties - supports latest for as version-expression; get impl is unvalidated, put impl checks validity - login is now common, under /api/login for both v2 and v3 (did not work previously)

#1693 Fix: (v2) VersionedModelController.getLatestVersionNumber(): Fu…

219360a

…ture -> CompletableFuture (mistake reverted)

#1693 VersionedModelControllerV3 - IT extended for `POST /{name}/{ver…

101b37a

…sion}` in proper validations

#1693 VersionedModelControllerV3 - conformance rule mgmt GET+POST dat…

b934958

…asets/dsName/version/rules, GET datasets/dsName/version/rules/# + IT

#1693 PR review update - typos etc

51ad013

#1693 Validation-warnings as payload to caller in V3

ba5edce

- v2 ignores this added information - needs

#1693 Validation-warnings as payload to caller in V3 - IT covered

662119b

#1693 API v3 impl for MTs, no tests yet

b0fd955

#1693 import validation check added to IT.

d480445

Merge branch 'feature/1693-api-v3-versioned-model' into feature/1693-…

bffbf55

…api-v3-schema-etc

Merge branch 'develop-ver-3.0' into feature/1693-api-v3-versioned-model

3542a43

Merge branch 'feature/1693-api-v3-versioned-model' into feature/1693-…

51d68a8

…api-v3-schema-etc

#1693 api-v3 mapping table specific - IntegTest for GET/POST/PUT .../…

e1c904e

…defaults

#1693 api-v3 mapping table specific - MT now validates if schema exis…

489ae12

…ts + IT

#1693 schema validation for DSs + IntegTest update

d6ef401

dk1844 requested a review from Zejnilovic May 9, 2022 08:13

dk1844 added 5 commits May 9, 2022 10:37

#1693 NamedLatestVersion generalized into a multipurpose `NamedVers…

db1227b

…ion`. Small updates, thanks @benedeki

#1693 PRReview /{name}/{version}/defaults processing generalization…

56ba9ed

… with `MappingTableControllerV3.withMappingTableToResponse`

Merge branch 'feature/1693-api-v3-schema-etc' into feature/1693-api-v…

28c7334

…3-delete-recreate

Merge branch 'feature/1693-api-v3-schema-etc' into feature/1693-api-v…

67f163e

…3-delete-recreate % Conflicts: % rest-api/src/test/scala/za/co/absa/enceladus/rest_api/integration/controllers/v3/DatasetControllerV3IntegrationSuite.scala

Base automatically changed from feature/1693-api-v3-schema-etc to develop-ver-3.0 May 10, 2022 09:28

dk1844 dismissed AdrianOlosutean’s stale review via 67f163e May 10, 2022 09:28

Zejnilovic reviewed May 11, 2022

View reviewed changes

#1692 Review update: API v3 summary (NamedVersion) now contains `disa…

0ff6851

…bled` information - mainly on GET ...{/name}

#1692 Review update: UsedIn.{isEmpty|nonEmpty} omitted from serializa…

4574b09

…tion. IT updated.

Zejnilovic reviewed May 13, 2022

View reviewed changes

data-model/src/main/scala/za/co/absa/enceladus/model/versionedModel/NamedVersion.scala Show resolved Hide resolved

This was referenced May 16, 2022

V3 API enable should check dependencies #2065

Open

Feature/1693 api v3 disabled fails validation #2066

Merged

dk1844 added 5 commits May 16, 2022 16:40

Merge branch 'develop-ver-3.0' into feature/1693-api-v3-delete-recreate

982e813

#1693 VersionedSummaryV2 Bson support fix

006feb0

#1693 VersionedSummaryV2 support fix - tests & impl

2905892

#1693 disable fail due to nonEmpty used in now carries a wrapper with…

e593df1

… an error message (`UsedIn` wrapped in `EntityInUseException`)

#1693 Future {throw x} replaced with Future.failed(x) in rest_api…

34292c4

…, cleanup

AdrianOlosutean approved these changes May 18, 2022

View reviewed changes

Zejnilovic approved these changes May 19, 2022

View reviewed changes

Zejnilovic added the PR:tested Only for PR - PR was tested by a tester (person) label May 19, 2022

dk1844 merged commit 0c9bf73 into develop-ver-3.0 May 24, 2022

dk1844 deleted the feature/1693-api-v3-delete-recreate branch May 24, 2022 09:44

AdrianOlosutean reviewed May 30, 2022

View reviewed changes

dk1844 linked an issue Jun 7, 2022 that may be closed by this pull request

Experimental API 3 - Versioned Model #1693

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/1693 api v3 delete recreate #2055

Feature/1693 api v3 delete recreate #2055

dk1844 commented Apr 28, 2022 •

edited

Loading

Zejnilovic left a comment

Zejnilovic commented May 12, 2022

dk1844 commented May 12, 2022 •

edited

Loading

sonarqubecloud bot commented May 18, 2022

AdrianOlosutean left a comment

Zejnilovic left a comment

AdrianOlosutean May 30, 2022

		@@ -57,14 +58,12 @@ class DatasetControllerV3 @Autowired()(datasetService: DatasetServiceV3)
		case Some((entity, validation)) =>

Feature/1693 api v3 delete recreate #2055

Feature/1693 api v3 delete recreate #2055

Conversation

dk1844 commented Apr 28, 2022 • edited Loading

Release notes suggestion

Zejnilovic left a comment

Choose a reason for hiding this comment

Zejnilovic commented May 12, 2022

dk1844 commented May 12, 2022 • edited Loading

sonarqubecloud bot commented May 18, 2022

AdrianOlosutean left a comment

Choose a reason for hiding this comment

Zejnilovic left a comment

Choose a reason for hiding this comment

AdrianOlosutean May 30, 2022

Choose a reason for hiding this comment

dk1844 commented Apr 28, 2022 •

edited

Loading

dk1844 commented May 12, 2022 •

edited

Loading