[8.14](backport #39243) [Auditbeat][add_session_metadata processor] Fix more potential enrichment failures

[mergify]

Proposed commit message

Fix two more cases that could cause unenriched processes in the add_session_metadata processor.

It was possible for auditd events to arrive before the ebpf event added processes to the process DB, now the enrichment will wait for the process to be inserted into the DB, if it's not already before enrichment is run on it. Also stop attempting to enrich failed syscall events, and modifying the DB based on these.

Changes:

• With the ebpf backend, when an event is processed wait for a process to be added to the DB before enriching, if it's not already in the DB before the event is received.
• Do not enrich failed syscall auditd events. Since failed syscalls don't actually cause a process to be created, they should not be enriched, or inserted to the process
• Remove scrapeAncestors from DB. The intention of this was to fill in missed processes, but now processes should not be missed with epbf, and ineffective with procfs, as the process will most likely already be ended. This was causing DB inconsistancies when run on failed syscall events, and I haven't ever seen any cases where it's helpful now.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. Setup Auditbeat with the session view processors and connected to ES/Kibana.
2. Run some commands in a session. Compile commands, such as building Beats, caused many unenriched processes before this fix, so it's good to try that.
3. In Kibana, on the host page, apply this filter not process.entry_leader.entity_id : * and process.pid: * and not auditd.result: "fail"
4. There shouldn't be any events shown.

Related issues

• Closes [Security-Linux Platform Integrations] Auditbeat missing Session View Process fields  #38955

---

This is an automatic backport of pull request #39243 done by [Mergify](https://mergify.com).

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-05-01T21:53:43.295+0000

• Duration: 55 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	472
Skipped	60
Total	532

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)