#118: DB cleanup and IT tests addition to CI

[lsulak]

Preparation for #118 - I don't really want to do all the DB setup for tests manually, it's too much effort. I used Flyway to automate it. It's not ideal - the actual data insertions and cleanups after each test - but it's better than it was and we can improve it on the next increment.

What was done:

• DB files (SQL and DDL) were moved and renamed / polished to be compatible with Flyway tool; it seems like a lot of changes and files, but really it was mostly small changes
• I created .github/workflows/integration_tests.yml - but there are many things to improve in the future:
  • test coverage
  • better documentation perhaps (Update README files for Slick and Doobie INT tests #116)
  • better test isolation - test data init and cleanup (Integration tests: Better Data Initialization and Cleanup #121)

--

Relates to #118
Relates to #116
Relates to #121
Relates to #19

[github-actions]

JaCoCo code coverage report - scala 2.13.12

Total Project Coverage	27.94%	🍏

There is no coverage information present for the Files changed

[github-actions]

JaCoCo code coverage report - scala 2.12.17

Total Project Coverage	27.05%	🍏

There is no coverage information present for the Files changed

[salamonpavel]

Good job, this was definitely missing. Let's address this also in other projects.

[benedeki]

I think it adds to the global dependencies, doesn't it? I think flyway dependencies shouldn't be a generic dependency.

[lsulak]

I believe this is what @salamonpavel was dealing with in the past and there was no way how to circumvent this (I know :( ) by extracting / moving elsewhere. Anyway, it's part of the build, i.e. general dependency: https://github.com/AbsaOSS/atum-service/blob/master/build.sbt#L53-L59

(remember, Flyway is temporary anyway)

[Zejnilovic]

I agree with @benedeki

lazy val modulNeedingFlyaway = project
  .enablePlugins(FlywayPlugin)
  .settings(
    flywayUrl := FlywayConfiguration.flywayUrl
    flywayUser := FlywayConfiguration.flywayUser
    flywayPassword := FlywayConfiguration.flywayPassword
    flywayLocations := FlywayConfiguration.flywayLocations
    flywaySqlMigrationSuffixes := FlywayConfiguration.flywaySqlMigrationSuffixes
    libraryDependencies ++= flywayDependencies
  )

and then run it like sbt modulNeedingFlyaway/flywayMigrate

Are we saying there is an issue with this @lsulak ?

[benedeki]

Can we redefine the flywayMigrate to behave like sbt modulNeedingFlyaway/flywayMigrate?

[Zejnilovic]

Like this addCommandAlias("flywayMigrate", "modulNeedingFlyaway/flywayMigrate")?

[lsulak]

Let me try, thanks @Zejnilovic. To be fair I haven't been playing with this one as my primary objective here was to actually have an IT suite embedded in our CI & I borrowed what was done in Atum Service - but I agree that we can improve this mechanism in build.sbt

[benedeki]

That's why it got approved, not blocking the whole thing 😉

[lsulak]

tested, it works very nicely, thank you Sasa! #124

[benedeki]

Well, except the Flyway dependency I think it's OK. That one can be addressed later.