Remove the limit of the incremental ingestion of files so that files can arrive out of order

[yruslan]

Background

Currently, incremental ingestion requires a monotonically increasing field. But for file-based ingestion we can be smarter and ingest new files by comparing file names in metastore for that day with files at the landing area.

Feature

Remove the limit of the incremental ingestion of files so that files can arrive out of order.

Example [Optional]

--

Proposed Solution [Optional]

--