Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support compressed file sources #5245

Open
joelmarty opened this issue Dec 5, 2024 · 2 comments · May be fixed by #5255
Open

Support compressed file sources #5245

joelmarty opened this issue Dec 5, 2024 · 2 comments · May be fixed by #5255
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@joelmarty
Copy link
Contributor

Is your feature request related to a problem? Please describe.
In order to develop ingestion pipelines for AWS infrastructure component logs (ALB, cloudfront...), I am testing my pipelines with files coming from this infra, which are compressed gzip files.

However, data-prepper does not support compression in the file source plugin, adding an extra step of decompressing the files manually.

In general this is a feature that would be welcomed as compressed log files are a common occurrence.

Describe the solution you'd like
the file source should have a compression field, set to none by default. When the plugin is started, the input stream that reads the file comes from the DecompressionEngine that corresponds to the compression field value.

Describe alternatives you've considered (Optional)
N/A

Additional context
Add any other context or screenshots about the feature request here.

@dlvenable dlvenable added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed and removed untriaged labels Dec 10, 2024
@dlvenable
Copy link
Member

Data Prepper already has some code that the file source could use for this. The s3 source is a good working example.

@joelmarty , Would you be able to add this feature to the file source?

@joelmarty joelmarty linked a pull request Dec 11, 2024 that will close this issue
4 tasks
@joelmarty
Copy link
Contributor Author

@dlvenable Yes, I was preparing the PR and getting approval from my employer for this contribution. Please review 🙏🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
Development

Successfully merging a pull request may close this issue.

2 participants