Code

#374 Incremental Ingestion#487

Code

Merged

yruslan merged 55 commits intomainAbsaOSS/pramen:mainfrom feature/374-incremental-ingestionAbsaOSS/pramen:feature/374-incremental-ingestionCopy head branch name to clipboardOct 29, 2024

+8,152-2,173

Commits

Commits on Sep 30, 2024

• #374 Add a table in bookeeping for storing offsets.

  yruslan

  committed
  f5211b4

  Copy full SHA for f5211b4

• #374 Add bookeeping interfaces for offset management.

  yruslan

  committed
  d6d1a13

  Copy full SHA for d6d1a13

• #374 Implement offset management DB operations.

  yruslan

  committed
  9b09f15

  Copy full SHA for 9b09f15

• #374 Improve offset management DB operations and add test suites.

  yruslan

  committed
  5bda4c5

  Copy full SHA for 5bda4c5

• #374 Bump up the minor version number since breaking changes are to be introduced.

  yruslan

  committed
  45fec57

  Copy full SHA for 45fec57

• #374 Add the notion of 'batchId' and 'getCurrentBatch' for the metastore.

  yruslan

  committed
  9d79f40

  Copy full SHA for 9d79f40

• #374 Remove parenthesis of several get methods.

  yruslan

  committed
  86df4b5

  Copy full SHA for 86df4b5

• #374 Add interfaces for sources to fetch data based on offsets.

  yruslan

  committed
  71201ac

  Copy full SHA for 71201ac

• #374 Add table reader interfaces for incremental processing.

  yruslan

  committed
  533bd93

  Copy full SHA for 533bd93

• #374 Add an end to end test for incremental processing.

  yruslan

  committed
  27ab3cd

  Copy full SHA for 27ab3cd

• Fixup

  yruslan

  committed
  7329801

  Copy full SHA for 7329801

• #374 Add initial support for incremental transformers.

  yruslan

  committed
  ba63e16

  Copy full SHA for ba63e16

• #374 Make normal transformers compatible with incremental ingestion, and add more useful methods to metastore interfaces.

  yruslan

  committed
  0232e9c

  Copy full SHA for 0232e9c

• #374 Add support for reruns for incremental ingestion with offsets and info dates + offsets.

  yruslan

  committed
  239e1f1

  Copy full SHA for 239e1f1

• #374 Add support for historical runs, and for re-committing uncommitted offsets.

  yruslan

  committed
  307c0f1

  Copy full SHA for 307c0f1

• #374 Implement the offset type: 'datetime' for incremental ingestion.

  yruslan

  committed
  a8a77dc

  Copy full SHA for a8a77dc

• Update Jacoco.

  yruslan

  committed
  d6fa73c

  Copy full SHA for d6fa73c

• #374 Add integrations tests missing from the last fixup.

  yruslan

  committed
  fe246e5

  Copy full SHA for fe246e5

• #374 Another fixup from compile warnings.

  yruslan

  committed
  91fb48a

  Copy full SHA for 91fb48a

• #374 Fixed Spark 2.4.8 support in integration tests.

  Show description for 1f1d7a9
  yruslan

  committed
  1f1d7a9

  Copy full SHA for 1f1d7a9

• Update Jacoco report version

  yruslan

  committed
  edd90b3

  Copy full SHA for edd90b3

• Make Jacoco take into account integration tests by including them in 'sbt test'.

  yruslan

  committed
  78a0675

  Copy full SHA for 78a0675

• #374 Add offset configuration for JDBC sources.

  yruslan

  committed
  567973a

  Copy full SHA for 567973a

• Update Scala, Spark versions for sbt builds.

  yruslan

  committed
  3f59329

  Copy full SHA for 3f59329

• #374 Fix a corner case of running incremental ingestion out of order of offsets.

  yruslan

  committed
  537b85d

  Copy full SHA for 537b85d

• #374 Simplify the incremental ingestion interface for data sources.

  yruslan

  committed
  83b3c7a

  Copy full SHA for 83b3c7a

• #374 Implement offset queries generation in SQL generators.

  yruslan

  committed
  8fe53f1

  Copy full SHA for 8fe53f1

• #374 Add implementation for incremental ingestion from JDBC, and added first test suite for it.

  yruslan

  committed
  5b8983c

  Copy full SHA for 5b8983c

• #374 Add server timezone to JDBC reader configuration to accommodate for timezone differences in timestamp offsets.

  yruslan

  committed
  8b4bfb1

  Copy full SHA for 8b4bfb1

• #374 Fix the logic of incremental ingestion when information date is present.

  yruslan

  committed
  8b3ed71

  Copy full SHA for 8b3ed71

• #374 Improve email notifications for incremental operations.

  yruslan

  committed
  858138f

  Copy full SHA for 858138f

• #374 Calculate throughput based on appended records for incremental jobs.

  yruslan

  committed
  7da2627

  Copy full SHA for 7da2627

• #374 Fix the way retrospective updates are determined.

  yruslan

  committed
  6f13121

  Copy full SHA for 6f13121

• #374 Add number of appended records to journal.

  yruslan

  committed
  451f556

  Copy full SHA for 451f556

• Improve performance of querying timestamp fields for PostgreSQL and MySQL when the table has an index for that field.

  yruslan

  committed
  1ec4946

  Copy full SHA for 1ec4946

• Fix the check for retrospective updates.

  yruslan

  committed
  aeac279

  Copy full SHA for aeac279

• #421 Allow transient non-cached jobs not return record count.

  yruslan

  committed
  64e47b1

  Copy full SHA for 64e47b1

• #374 Improve the description of the getCurrentBatch() metastore interface.

  yruslan

  committed
  f861bf9

  Copy full SHA for f861bf9

Commits on Oct 1, 2024

• #374 Update README with the new feature.

  yruslan

  committed
  f3704b8

  Copy full SHA for f3704b8

• #374 Add more tests for SQL generation related to offsets.

  yruslan

  committed
  f8d9d6e

  Copy full SHA for f8d9d6e

Commits on Oct 4, 2024

• #374 Implement offset management based on inclusive intervals.

  yruslan

  committed
  7bce016

  Copy full SHA for 7bce016

• #374 Fix new incremental ingestion and integration tests to match inclusive intervals.

  yruslan

  committed
  99b3689

  Copy full SHA for 99b3689

Commits on Oct 7, 2024

• #374 Fix new incremental ingestion and integration tests to match inclusive intervals.

  yruslan

  committed
  fdea270

  Copy full SHA for fdea270

• Fix a timing dependency of a unit test.

  yruslan

  committed
  745315b

  Copy full SHA for 745315b

Commits on Oct 8, 2024

• #374 Remove minimum values for offset types.

  Show description for 30ee28b
  yruslan

  committed
  30ee28b

  Copy full SHA for 30ee28b

Commits on Oct 9, 2024

• #374 Fix a scenario when uncommitted offsets are not properly handled.

  Show description for d44a8db
  yruslan

  committed
  d44a8db

  Copy full SHA for d44a8db

• #374 Refactor validation for the incremental ingestion job.

  yruslan

  committed
  b3f91b1

  Copy full SHA for b3f91b1

Commits on Oct 24, 2024

• Add unit tests for the incremental scheduling strategy

  yruslan

  committed
  7df79cd

  Copy full SHA for 7df79cd

• Remove the nasty 'var' and possible error related to it.

  yruslan

  committed
  9e86134

  Copy full SHA for 9e86134

• Update pramen/api/src/main/scala/za/co/absa/pramen/api/Source.scala

  Show description for f6676f2
  

  yruslan

  and

  kevinwallimann

  authored
  f6676f2

  Copy full SHA for f6676f2

• Update pramen/core/src/test/scala/za/co/absa/pramen/core/tests/utils/SparkUtilsSuite.scala

  Show description for d5f3267
  

  yruslan

  and

  kevinwallimann

  authored
  d5f3267

  Copy full SHA for d5f3267

• Apply suggestions from code review

  Show description for f3a8b35
  

  yruslan

  and

  kevinwallimann

  authored
  f3a8b35

  Copy full SHA for f3a8b35

Commits on Oct 25, 2024

• Fix PR suggestions.

  yruslan

  committed
  df88ce5

  Copy full SHA for df88ce5

• Fix imports removed by IDE.

  yruslan

  committed
  aecd5e5

  Copy full SHA for aecd5e5

• Fix more PR suggestions regarding splitting complex identifiers.

  yruslan

  committed
  ba5e5de

  Copy full SHA for ba5e5de