[libbeat] Introduce Apache Pulsar to beats as an output plugin #37552

dao-jun · 2024-01-05T11:59:27Z

Proposed commit message

Introduce Apache Pulsar to beats as an output plugin, and users could send data to Pulsar.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service · 2024-01-05T11:59:32Z

💚 CLA has been signed

mergify · 2024-01-05T12:00:06Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @dao-jun? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-01-05T12:04:56Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-05T11:59:46.038+0000
Duration: 5 min 1 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/dao-jun return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-06T10:14:45Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-06T10:08:44.631+0000
Duration: 6 min 57 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/dao-jun return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…ar_output

elasticmachine · 2024-01-06T10:29:22Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-06T10:22:42.539+0000
Duration: 7 min 36 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/dao-jun return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-06T10:40:17Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-06T10:34:47.749+0000
Duration: 5 min 25 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/dao-jun return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-06T12:32:30Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-06T12:25:54.670+0000
Duration: 7 min 31 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/dao-jun return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-08T11:04:35Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-08T10:59:47.610+0000
Duration: 5 min 44 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/beats.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/dao-jun return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/dao-jun : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-01-10T23:52:19Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b dev/pulsar_output upstream/dev/pulsar_output
git merge upstream/main
git push upstream dev/pulsar_output

cmacknz · 2024-01-11T20:48:29Z

/test

libbeat/outputs/pulsar/docs/pulsar.asciidoc

# Conflicts: # go.sum

dao-jun · 2024-05-16T08:51:34Z

@ycombinator Resolved, but there is an error

What does it mean?

ycombinator · 2024-05-16T21:41:30Z

@ycombinator Resolved, but there is an error What does it mean?

It means the NOTICE.txt file needs to be regenerated. For that, please run make update and the commit the changes. Thanks!

dao-jun · 2024-05-17T04:36:54Z

@ycombinator Resolved, but there is an error What does it mean?

It means the NOTICE.txt file needs to be regenerated. For that, please run make update and the commit the changes. Thanks!

Oh, yes, I remember now, since this PR pending for a long time, I forget I need to update the NOTICE.txt file

dao-jun · 2024-05-17T09:37:04Z

@ycombinator It seems the ci checks passed, can we keep going on the PR?

ycombinator · 2024-05-17T20:43:47Z

buildkite test this

mergify · 2024-05-21T17:41:21Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b dev/pulsar_output upstream/dev/pulsar_output
git merge upstream/main
git push upstream dev/pulsar_output

mergify · 2024-10-17T14:04:03Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

# Conflicts: # NOTICE.txt # go.mod # go.sum # libbeat/publisher/includes/includes.go

dao-jun · 2024-12-06T11:17:02Z

@ycombinator @pierrehilbert @cmacknz Can you please help trigger the tests and merge the PR?
It has blocked for a loooong time

cmacknz · 2024-12-06T16:13:43Z

/test

mergify · 2025-02-19T14:09:52Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b dev/pulsar_output upstream/dev/pulsar_output
git merge upstream/main
git push upstream dev/pulsar_output

flexitrev · 2025-02-25T22:25:33Z

@dao-jun thank you for this excellent PR. We actually plan on using Otel exporters for new beat outputs and it looks like you are the maintainer for that code as well https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/pulsarexporter

In future releases of the Elastic Distribution of the Open Telemetry Collector (EDOT) we plan on introducing the ability to run beats as OTel receivers. https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-otel-command

dao-jun · 2025-02-26T03:46:28Z

@dao-jun thank you for this excellent PR. We actually plan on using Otel exporters for new beat outputs and it looks like you are the maintainer for that code as well https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/pulsarexporter

In future releases of the Elastic Distribution of the Open Telemetry Collector (EDOT) we plan on introducing the ability to run beats as OTel receivers. https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-otel-command

Does it mean that we don't need this PR anymore?

Add pulsar output.

8475cc4

dao-jun requested review from a team as code owners January 5, 2024 11:59

dao-jun requested review from belimawr and fearful-symmetry January 5, 2024 11:59

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 5, 2024

mergify bot assigned dao-jun Jan 5, 2024

dao-jun changed the title ~~[draft] Add pulsar output.~~ [draft][libbeat] Introduce Apache Pulsar to beats as an output plugin Jan 5, 2024

pierrehilbert added the Team:Elastic-Agent Label for the Agent team label Jan 5, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 5, 2024

cmacknz requested a review from faec January 5, 2024 19:49

Merge branch 'main' into dev/pulsar_output

342713d

dao-jun added 2 commits January 6, 2024 18:22

make update & make fmt

0748969

Merge remote-tracking branch 'origin/dev/pulsar_output' into dev/puls…

8d67fae

…ar_output

update go.sum

26b961d

Add tests and doc

b2a3ed8

dao-jun added 2 commits January 8, 2024 18:58

Merge branch 'main' into dev/pulsar_output

f159206

Update branch

f559f94

cmacknz reviewed Jan 11, 2024

View reviewed changes

libbeat/outputs/pulsar/docs/pulsar.asciidoc Show resolved Hide resolved

Merge branch 'main' into dev/pulsar_output

8505608

# Conflicts: # go.sum

fix lint

b33b434

dao-jun added 2 commits May 17, 2024 12:34

Update NOTICE.txt

d509d82

Merge branch 'main' into dev/pulsar_output

c39a104

Merge branch 'main' into dev/pulsar_output

c15fa1c

pierrehilbert mentioned this pull request Jun 26, 2024

filebeat can support pulsar input/output #40007

Open

pierrehilbert marked this pull request as draft October 17, 2024 14:03

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 17, 2024

dao-jun added 2 commits December 6, 2024 10:55

Merge branch 'main' into dev/pulsar_output

de508fa

# Conflicts: # NOTICE.txt # go.mod # go.sum # libbeat/publisher/includes/includes.go

rebase master branch

63ab7e0

dao-jun marked this pull request as ready for review December 6, 2024 03:23

dao-jun added 6 commits December 6, 2024 11:33

fix build error

bf42439

fix build error

c110a33

Add pulsar integration test

7546dac

Add pulsar integration test

9c35a31

Add pulsar integration test

461aebd

fix lint check

1550a65

flexitrev self-requested a review February 19, 2025 14:09

flexitrev removed their request for review February 24, 2025 13:19

[libbeat] Introduce Apache Pulsar to beats as an output plugin #37552

Are you sure you want to change the base?

[libbeat] Introduce Apache Pulsar to beats as an output plugin #37552

Conversation

dao-jun commented Jan 5, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

cla-checker-service bot commented Jan 5, 2024 • edited Loading

mergify bot commented Jan 5, 2024

elasticmachine commented Jan 5, 2024

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

elasticmachine commented Jan 6, 2024

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

elasticmachine commented Jan 6, 2024

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

elasticmachine commented Jan 6, 2024

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

elasticmachine commented Jan 6, 2024

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

elasticmachine commented Jan 8, 2024

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

mergify bot commented Jan 10, 2024

cmacknz commented Jan 11, 2024

dao-jun commented May 16, 2024 • edited Loading

ycombinator commented May 16, 2024

dao-jun commented May 17, 2024

dao-jun commented May 17, 2024

ycombinator commented May 17, 2024

mergify bot commented May 21, 2024

mergify bot commented Oct 17, 2024

dao-jun commented Dec 6, 2024

cmacknz commented Dec 6, 2024

mergify bot commented Feb 19, 2025

flexitrev commented Feb 25, 2025

dao-jun commented Feb 26, 2025

dao-jun commented Jan 5, 2024 •

edited

Loading

cla-checker-service bot commented Jan 5, 2024 •

edited

Loading

`Load a resource file from a library`

`Error signal`

`Load a resource file from a library`

`Error signal`

`Load a resource file from a library`

`Error signal`

`Load a resource file from a library`

`Error signal`

`Load a resource file from a library`

`Error signal`

`Load a resource file from a library`

`Error signal`

dao-jun commented May 16, 2024 •

edited

Loading