Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update geoip config and use extensions #3975

Merged

Conversation

asifsmohammed
Copy link
Collaborator

@asifsmohammed asifsmohammed commented Jan 18, 2024

Description

This PR updates geoip processor configuration and also uses extensions created in #3944

Old config:

- geoip:
        aws:
          region: us-east-1
          sts_role_arn: "arn:aws:iam::524438387957:role/s3-sqs-role"
        keys:
          - key:
              source: "message"
              target: "target1"
              attributes: ["city_name","country_name"]
        service_type:
          maxmind:
            database_path:
              - url: 
            load_type: "cache"
            cache_size: 8192
            cache_refresh_schedule: P30D

New pipeline config

geoip:
  entries:
    - source: "message"
       target: "target1"
       fields: ["city_name","country_name"]

Data prepper config

extensions:
  geoip_service:
    maxmind:
      database_paths: ["path1", "path2"]
      database_refresh_interval: "P10D"
      cache_size: 2048
      aws:
        region: "us-east-1"
        sts_role_arn: "arn:aws:iam::123456789:role/data-prepper-execution-role"

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Asif Sohail Mohammed <[email protected]>
Copy link
Member

@dlvenable dlvenable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this is looking good.

*/
public GeoIPProcessorService(GeoIPProcessorConfig geoIPProcessorConfig, String tempPath) {
public GeoIPProcessorService(final GeoIpServiceConfig geoIpServiceConfig) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is going to be provided by the extension in the end, right?

It would be good to move this into the .extension package. That could happen in a follow-on PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's provided by extension supplier. I will move it to right package.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QUES: do we know if this will potentially be used in other pipeline plugins? Need to catch up on justification why we put it in extensions

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it will be used by other plugins. But the extension will allow Data Prepper to run a single GeoIP "service" which any number of processors can share.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By "number of processors" are we expecting future processor plugin to depend upon this? It would be good to have examples but not blocking this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean any number of geoip processors.

pipeline-a:
  processor:
    - geoip:
    - geoip:
pipeline-b:
  processor:
    - geoip:

All of these will share the same underlying objects and resources from Maxmind.

@@ -42,13 +41,13 @@ public HttpDBDownloadService(String prefixDir) {
* Initialisation of Download through Url
* @param urlList urlList
*/
public void initiateDownload(List<DatabasePathURLConfig> urlList) throws Exception {
public void initiateDownload(List<String> urlList) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire package (databasedownload) could probably be put under the .extension package as well - .extension.databasedownload.

Again, this could be a follow-on PR.

@asifsmohammed asifsmohammed merged commit b7970a0 into opensearch-project:main Jan 22, 2024
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants