Add registry to Filebeat's diagnostic

[belimawr]

Proposed commit message

This commit adds Filebeat's registry folder to the Elastic-Agent
diagnostics. It's called registry.tar.gz and includes all registry files
on ${path.home}/registry.

The registry is first archived into a temporary tar file. The
temporary file is created by calling os.CreateTemp and will use the
OS's temporary folder. Then it's gziped in memory and returned to
Elastic-Agent, finally the temporary file is removed from the disk.

If the final gziped file is more than 20mb, it is skipped due to its large
size.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None.

Author's Checklist

How to test this PR locally

The easiest way to test it locally is to package the Elastic-Agent with your local build of Beats by setting EXTERNAL=false when calling mage build in the Elastic-Agent folder.

Then, deploy the Elastic-Agent, let it ingest some logs and request the diagnostics. Extract the diagnostics and look for registry.tar.gz in components/filestram-* folders.

You can also run the integration tests:

go test -tags integration -count=1 -run=TestEmptyegistryIsInDiagnostics -v ./tests/integration
go test -tags integration -count=1 -run=TestFilestreamRegistryIsInDiagnostics -v ./tests/integration

Related issues

• Closes Add Filebeat registry to the diagnostics bundle for the control V2 protocol #33238

## Use cases
## Screenshots

Logs

Full log entries

{"log.level":"debug","@timestamp":"2024-11-27T11:00:35.812-0500","log.logger":"diagnostics","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.gzipRegistry","file.name":"beater/diagnostics.go","file.line":54},"message":"temporary file '/tmp/filebeat-registry-640175774.tar' created","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-27T11:00:35.812-0500","log.logger":"diagnostics","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.tarFolder","file.name":"beater/diagnostics.go","file.line":116},"message":"starting to walk '/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestRegistryIsInDiagnostics3276586231/data/registry'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-27T11:00:35.812-0500","log.logger":"diagnostics","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.tarFolder.func1","file.name":"beater/diagnostics.go","file.line":140},"message":"adding '/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestRegistryIsInDiagnostics3276586231/data/registry/filebeat/log.json' to the tar archive","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-27T11:00:35.812-0500","log.logger":"diagnostics","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.tarFolder.func1","file.name":"beater/diagnostics.go","file.line":140},"message":"adding '/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestRegistryIsInDiagnostics3276586231/data/registry/filebeat/meta.json' to the tar archive","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-27T11:00:35.812-0500","log.logger":"diagnostics","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.gzipRegistry","file.name":"beater/diagnostics.go","file.line":63},"message":"finished gziping Filebeat's registry","service.name":"filebeat","ecs.version":"1.6.0"}

Only the message field:

"temporary file '/tmp/filebeat-registry-640175774.tar' created"
"starting to walk '/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestRegistryIsInDiagnostics3276586231/data/registry'"
"adding '/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestRegistryIsInDiagnostics3276586231/data/registry/filebeat/log.json' to the tar archive"
"adding '/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestRegistryIsInDiagnostics3276586231/data/registry/filebeat/meta.json' to the tar archive"
"finished gziping Filebeat's registry"

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

1. Is there a limit to how big the contents of the ${path.home}/registry directory can get? If there isn't a limit in Filebeat itself, I think the diagnostics collection here needs a size limit of some sort. Otherwise if we accidentally create a huge registry diagnostics won't work at all.
2. Can a more restrictive pattern be used instead of just taking everything in the directory? Perhaps you can only take files with permissions that match the filebeat user and group? This is to stop arbitrary data from being uploaded to Fleet and then decompressed later by an unsuspecting user or engineer.

[belimawr]

1. Is there a limit to how big the contents of the ${path.home}/registry directory can get? If there isn't a limit in Filebeat itself, I think the diagnostics collection here needs a size limit of some sort. Otherwise if we accidentally create a huge registry diagnostics won't work at all.
2. Can a more restrictive pattern be used instead of just taking everything in the directory? Perhaps you can only take files with permissions that match the filebeat user and group? This is to stop arbitrary data from being uploaded to Fleet and then decompressed later by an unsuspecting user or engineer.

1. No there isn't, I've been thinking about it, I believe gRPC has a limit, so in my mind that was a kind of a guardrail. A limit for the payload (gziped "file") is easy to add. It's hard for me, at the moment, to gauge what would be an ideal limit for a high load environment like Kubernetes. The log file is caped at 10Mb, however the checkpoint is unbound. Given compression, I'd say at least 20Mb to be on the safe side.
2. Yes, that can be implemented, all files that make up the registry are well known. The registry itself is in the data folder of Elastic-Agent, so there should be no extra data there. The only reason I can think of to have some files there we do not want is if some malicious actor adds something there. Anyways, I agree it's an nice extra security feature.

I'll implement those changes.

[cmacknz]

The log file is caped at 10Mb, however the checkpoint is unbound. Given compression, I'd say at least 20Mb to be on the safe side.

The Fleet server upload API used to get diagnostics as an action probably has a limit, I'm not sure what it is by default off the top of my head. I would expect it is ~100 MB so if we go with 20 MB we need to make sure we don't push ourselves past that in real use cases.

The few compressed gzipped diagnostics I have laying around are all <5 MB now for reference.

[cmacknz]

The Fleet settings aren't super clear on what the upload file size limit is, maybe there isn't one? The files are chunked as part of the implementation. https://github.com/elastic/fleet-server/blob/4132a505f0c759e332a890fdf701b8ff238cf91a/fleet-server.reference.yml#L200-L214

[belimawr]

All review comments have been addressed.

The failure from TestRegistryIsInDiagnostics has been fixed
The failure from TestFilestreamMetadataUpdatedOnRename is addressed by #42213
The failure from TestTranslateGUIDWithLDAP is unrelated to this PR and I cannot reproduce it

[cmacknz]

Did you test this with a real agent uploading diagnostics that include the registry to Fleet? If not, can you? I want to make sure this actually works end to end before merging it.

[belimawr]

Did you test this with a real agent uploading diagnostics that include the registry to Fleet? If not, can you? I want to make sure this actually works end to end before merging it.

Yes, I have tested it end-to-end multiple times and even demoed it after the On-Week. The only thing I haven't manually tested is the limit when the registry is too large. I can try testing this at some before merging.