[Filebeat] aws-s3 - include_s3_metadata config does not work

[andrewkroh]

For non-user defined headers, the include_s3_metadata feature no longer works. For example:

include_s3_metadata:
  - last-modified    # Not working (built-in aws metadata).
  - x-amz-meta-owner # Worker (user-defined metadata).

The test case¹ that verified the feature works was accidentally removed at some point. When the AWS SDK was upgraded, and the struct tags that it depended on were removed, then it no longer functioned as designed.

Footnotes

1. https://github.com/elastic/beats/blob/c25fca8bdf816fb924b56b25dfd3488e0f4ff51f/x-pack/filebeat/input/awss3/collector_test.go#L494-L507 ↩

[botelastic]

This issue doesn't have a Team:<team> label.

[andrewkroh]

Unfortunately, without the struct tags that were in older versions of the AWS SDK (example shown below), I think we will need a mapping of header names to struct fields in order to retain the same behavior. There is a function in the SDK called awsRestxml_deserializeOpHttpBindingsGetObjectOutput that has the headers to struct field info.

// Please also see https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/GetObjectOutput
type GetObjectOutput struct {
	_ struct{} `type:"structure" payload:"Body"`

	AcceptRanges *string `location:"header" locationName:"accept-ranges" type:"string"`

Or we can try to access the raw HTTP response and read the headers before they are parsed by the SDK. I think this is possible, but not sure it is officially documented anywhere (which could mean that it could be removed in the future).