More docs, add definition fo elasticsearch client config

docs/migrating.md

@@ -84,10 +84,10 @@ All heading are config options are defined using their original `parsedmarc` nam
 
 `save_forensic`: not supported - no comparable option.
 
-`save_smtp_tls`: not supported - [GitHub Issue #5](https://github.com/nhairs/parsedmarc-fork/issues/5)
+`save_smtp_tls`: not supported - [GitHub Issue #5](https://github.com/nhairs/parsedmarc-fork/issues/5).
 `strip_attachment_payloads`: moved to `parser.strip_attachment_payloads`
 
-`output`: not supported - [GitHub Issue #24](https://github.com/nhairs/parsedmarc-fork/issues/24)
+`output`: not supported - [GitHub Issue #24](https://github.com/nhairs/parsedmarc-fork/issues/24).
 `aggregate_json_filename`: not supported - see `output` above.
 
 `forensic_json_filename`: not supported - see `output` above.
@@ -96,9 +96,9 @@ All heading are config options are defined using their original `parsedmarc` nam
 
 `offline`: moved to `parser.offline`
 
-`always_use_local_files`: not supported - [GitHub Issue #10](https://github.com/nhairs/parsedmarc-fork/issues/10)
+`always_use_local_files`: not supported - [GitHub Issue #10](https://github.com/nhairs/parsedmarc-fork/issues/10).
 
-`local_reverse_dns_map_path`: not supported - [GitHub Issue #10](https://github.com/nhairs/parsedmarc-fork/issues/10)
+`local_reverse_dns_map_path`: not supported - [GitHub Issue #10](https://github.com/nhairs/parsedmarc-fork/issues/10).
 
 `nameservers`: moved to `parser.nameservers.[]`
 
@@ -112,54 +112,206 @@ All heading are config options are defined using their original `parsedmarc` nam
 
 `n_procs`: not supported - no comparable option.
 
+
 #### `mailbox`
 
-Use the `.email:MailboxConnectionSource` Source.
+Use a `.email:MailboxConnectionSource` Source.
+
+`reports_folder`: no changes.
+
+`archive_folder`: no changes.
+
+`watch`: not supported - no comparable option.
+
+`delete`: not supported - use `mode: "delete"` instead.
+
+`test`: not supported - use `mode: "test"` instead.
+
+`batch_size`: not supported - no comparable option.
+
+`check_timeout`: not supported - no comparable option.
+
 
 #### `imap`
 
-Use the `.email:Imap` Source.
+Use a `.email:Imap` Source.
+
+`host`: no changes.
+
+`port`: now optional and will select the appropriate default port based on SSL/TLS settings.
+
+`ssl`: no changes.
+
+`skip_certificate_verification`: moved to `verify_ssl`.
+
+`user`: moved to `username`
+
+`password`: no changes.
+
 
 #### `msgraph`
 
-Use the `.email.MicosoftGraph` Source.
+Use a `.email.MicosoftGraph` Source.
+
+`auth_method`: no changes.
+
+`user`: moved to `username`.
+
+`password`: no changes.
+
+`client_id`: no changes.
+
+`client_secret`: no changes.
+
+`tenant_id`: no changes.
+
+`mailbox`: no changes.
+
+`token_file`: no changes.
+
+`allow_unencrypted_storage`: no changes.
+
 
 #### `elasticsearch`
 
-Use the `.elasticsearch:Elasticsearch` Sink.
+Use a `.elasticsearch:Elasticsearch` Sink.
+
+`hosts`: moved to `client.hosts`.
+
+`user`: moved to `client.username`.
+
+`password`: moved to `client.password`.
+
+`apiKey`: moved to `client.api_key`.
+
+`ssl`: moved to `client.use_ssl`.
+
+`timeout`: moved to `client.timeout`.
+
+`cert_path`: moved to `client.ssl_cert_path`.
+
+`index_suffix`: moved to `client.index_suffix`.
+
+`index_prefix`: not supported - [GitHub Issue #12](https://github.com/nhairs/parsedmarc-fork/issues/12).
+
+`monthly_indexes`: moved to `client.monthly_indexes`.
+
+`number_of_shards`: moved to `client.number_of_shards`.
+
+`number_of_replicas`: moved to `client.number_of_replicas`.
+
 
 #### `opensearch`
 
-Not supported - [GitHub issue #6](https://github.com/nhairs/parsedmarc-fork/issues/6)
+Not supported - [GitHub issue #6](https://github.com/nhairs/parsedmarc-fork/issues/6).
+
 
 #### `splunk_hec`
 
-Not supported - [GitHub Issue #27](https://github.com/nhairs/parsedmarc-fork/issues/27)
+Not supported - [GitHub Issue #27](https://github.com/nhairs/parsedmarc-fork/issues/27).
+
 
 #### `kafka`
 
-Use the `.kafka:Kafka` Sink.
+Use a `.kafka:Kafka` Sink.
+
+`hosts`: moved to `client.hosts`.
+
+`user`: moved to `client.username`.
+
+`password`: moved to `client.password`.
+
+`ssl`: moved to `client.ssl`.
+
+`skip_certificate_verification`: moved to `client.skip_certificate_verification`.
+
+`aggregate_topic`: moved to `aggregate_report_topic`.
+
+`forensic_topic`: moved to `forensic_report_topic`.
+
 
 #### `smtp`
 
-Not supported - [GitHub Issue #29](https://github.com/nhairs/parsedmarc-fork/issues/29)
+Not supported - [GitHub Issue #29](https://github.com/nhairs/parsedmarc-fork/issues/29).
+
 
 #### `s3`
 
-Use the `.aws:S3` Sink.
+Use a `.aws:S3` Sink.
+
+`bucket`: no changes.
+
+`path`: moved to `path_prefix`.
+
+`region_name`: moved to `client.region_name`.
+
+`endpoint_url`: moved to `client.endpoint_url`.
+
+`access_key_id`: moved to `client.aws_access_key_id`.
+
+`secret_access_key`: moved to `client.aws_secret_access_key`.
+
 
 #### `syslog`
 
-Use the `.syslog:Syslog` Sink.
+Use a `.syslog:Syslog` Sink.
+
+`server`: moved to `syslog_host`.
+
+`port`: moved to `syslog_port`.
+
 
 #### `gmail_api`
 
-Use the `.email:Google` Source.
+Use a `.email:Google` Source.
+
+`credentials_file`: no changes.
+
+`token_file`: no changes.
+
+`include_spam_trash`: no changes.
+
+`scopes`: no changes.
+
+`oauth2_port`: no changes.
+
+`paginate_messages`: not supported - [GitHub Issue #14](https://github.com/nhairs/parsedmarc-fork/issues/14).
+
 
 #### `log_analytics`
 
-Use the `.azure:LogAnalytics` Sink.
+Use a `.azure:LogAnalytics` Sink.
+
+`client_id`: no changes.
+
+`client_secret`: no changes.
+
+`tenant_id`: no changes.
+
+`dce`: moved to `data_collection_endpoint`.
+
+`dcr_immutable_id`: moved to `data_collection_rule_id`.
+
+`dcr_aggreate_stream`: moved to `aggregate_report_stream`.
+
+`dcr_forensic_stream`: moved to `forensic_report_stream`.
+
+`dcr_smtp_tls_stream`: not supported - [GitHub Issue #5](https://github.com/nhairs/parsedmarc-fork/issues/5).
+
 
 #### `gelf`
 
-Not supported - [GitHub Issue #13](https://github.com/nhairs/parsedmarc-fork/issues/13)
+Not supported - [GitHub Issue #13](https://github.com/nhairs/parsedmarc-fork/issues/13).
+
+
+#### `webhook`
+
+Use a `.webhook:JsonWebhook` Sink.
+
+`aggregate_url`: moved to `dmarc_aggregate_url`.
+
+`forensic_url`: moved to `dmarc_forensic_url`.
+
+`smtp_tls_url`: not supported - [GitHub Issue #5](https://github.com/nhairs/parsedmarc-fork/issues/5).
+
+`timeout`: moved to `http_timeout`.

src/parsedmarc/sink/elasticsearch.py

@@ -4,7 +4,10 @@
 from __future__ import annotations
 
 # Standard Library
-from typing import Any, Dict, Literal
+from typing import List, Literal
+
+# Installed
+from pydantic import BaseModel
 
 # Local
 from ..const import AppState
@@ -29,7 +32,7 @@ def setup(self) -> None:
         self._state = AppState.SETTING_UP
 
         try:
-            self.client = ElasticsearchClient(**self.config.client)
+            self.client = ElasticsearchClient(**dict(self.config.client))
             self.client.migrate_indexes()
 
         except:
@@ -68,6 +71,19 @@ def process_forensic_report(self, report: ForensicReport) -> None:
 class ElasticsearchConfig(BaseConfig):
     """Elasticsearch Config"""
 
-    # As per https://elasticsearch-py.readthedocs.io/en/v8.13.0/api/elasticsearch.html
-    client: Dict[str, Any]
+    client: ElasticsearchClientConfig
     on_duplicate: Literal["discard"] = "discard"  # TODO: implement update logic and add
+
+
+class ElasticsearchClientConfig(BaseModel):
+    hosts: str | List[str]
+    use_ssl: bool = False
+    ssl_cert_path: str | None = None
+    username: str | None = None
+    password: str | None = None
+    api_key: str | None = None
+    timeout: float = 60.0
+    index_suffix: str | None = None
+    monthly_index: bool = True
+    number_of_shards: int = 1
+    number_of_replicas: int = 0

tests/test_source_sink_config.py

@@ -96,7 +96,7 @@ def test_source_init(class_: Type[Source], config: Dict[str, Any]):
     "class_, config",
     [
         # ElasticSearch
-        (parsedmarc.sink.elasticsearch.Elasticsearch, {"client": {"foo": "bar"}}),
+        (parsedmarc.sink.elasticsearch.Elasticsearch, {"client": {"hosts": "foo"}}),
         # Util
         (parsedmarc.sink.util.Noop, {}),
         (parsedmarc.sink.util.Console, {}),