Beat log rotation error while running under Agent on Windows

[andrewkroh]

Filebeat running as a subprocess of Agent (with monitoring enabled) is unable rotate its log file. I don't know what would cause the os.Stat call to fail. It fails intermittently based on the attached log. I don't think it affects functionality because the logger can still continue, but the old file(s) might not get purged immediately. They will get purged on the next successful purge.

• Version: 8.4.3
• Operating System: Windows
• Username: NT AUTHORITY\SYSTEM (S-1-5-18)

Error from Agent log:

filebeat stderr: "2022-10-31 09:00:00.0004948 -0400 EDT m=+422052.451213201 write error: error file rotating files reason: time interval: failed on C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-90167b\\logs\\default\\filebeat-20221031-964.ndjson during rotation: CreateFile C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-90167b\\logs\\default\\filebeat-20221031-964.ndjson: Access is denied.\n"

agent-8.4.3-log-filtered.txt

Related code:

• https://github.com/uber-go/zap/blob/ed5598a9e26d75aae01f4846ce31b3fe785c57c7/zapcore/entry.go#L258
• https://github.com/elastic/elastic-agent-libs/blob/dbea8c3e504f4074608ab860d9bd7bc478d525ae/file/rotator.go#L223
• https://github.com/elastic/elastic-agent-libs/blob/dbea8c3e504f4074608ab860d9bd7bc478d525ae/file/rotator.go#L325 - This os.Stat call on the files to be deleted fails with "Access is denied."

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

I have seen this in Windows agent logs before but don't have a great idea of what causes it. Did you reproduce this yourself, or is this something you saw investigating user problem?

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[cmacknz]

I suspect this happens because we attempt to delete the file while it is still being written (or at least still open).

We see similar behavior in agent when uninstalling it, occasionally some of the log files get left behind. elastic/elastic-agent#3342

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!