Support compressed file sources #5245
Labels
enhancement
New feature or request
good first issue
Good for newcomers
help wanted
Extra attention is needed
Is your feature request related to a problem? Please describe.
In order to develop ingestion pipelines for AWS infrastructure component logs (ALB, cloudfront...), I am testing my pipelines with files coming from this infra, which are compressed gzip files.
However, data-prepper does not support compression in the file source plugin, adding an extra step of decompressing the files manually.
In general this is a feature that would be welcomed as compressed log files are a common occurrence.
Describe the solution you'd like
the
file
source should have acompression
field, set tonone
by default. When the plugin is started, the input stream that reads the file comes from theDecompressionEngine
that corresponds to thecompression
field value.Describe alternatives you've considered (Optional)
N/A
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: