[BUG][opensearch sink] Config option 'insecure' not honored when 'cert' is configured

[janhoy]

Describe the bug
The insecure option should override cert, not the other way around.

To Reproduce
Steps to reproduce the behavior:

1. Setup an Opensearch with demo certificates
2. Generate a TLS certificate pem file for the configuration, which does not match the host name of opensearch
3. Configure prepper with opensearch exporter having both cert and insecure in config:

   sink:
     opensearch:
       hosts: ["https://localhost:9200"]
       cert: path/to/wrong/hostname/cert.pem
       insecure: true

4. Observe that prepper cannot connect to Opensearch, it complains about hostname validation, even if insecure is set.
5. Remove the cert line and try again
6. Now prepper can connect to your opensearch

Expected behavior
The documentation says the following for the two settings:

cert (Optional) : CA certificate that is pem encoded. Accepts both .pem or .crt. This enables the client to trust the CA that has signed the certificate that the OpenSearch cluster is using. Default is null.
insecure (Optional): A boolean flag to turn off SSL certificate verification. If set to true, CA certificate verification will be turned off and insecure HTTP requests will be sent. Default to false.

These two are by definition mutual exclusive. But the documentation does not talk about their relationship or the fact that cert will override insecure. The principle of least surprise would be that insecure: true overrides the presence of cert, not the other way around. You'd also expect a warning in the log whenever this is the case.

Screenshots
Code that disregards insecure flag:

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/ConnectionConfiguration.java

Lines 276 to 283 in 956a89a

	final String certPath = pluginSetting.getStringOrDefault(CERT_PATH, null);
	final boolean insecure = pluginSetting.getBooleanOrDefault(INSECURE, false);
	if (certPath != null) {
	builder = builder.withCert(certPath);
	} else {
	//We will set insecure flag only if certPath is null
	builder = builder.withInsecure(insecure);
	}

Environment (please complete the following information):

• Kubernetes
• Opensearch helm chart
• Data prepper helm chart
• Opensearch demo certificates

Additional context
The k8s service has a different name than the CN in auto generated demo certificates. Since the Data prepper is configured to talk to the servicename, there is a host name verification error when using the pem cert from opensearch.

[kkondaka]

Sounds like a bug and easy to fix.

[janhoy]

Yes, I found that the opensearch source already behaves that way, making insecure override cert. See my Pull Request for a fix.