Merge branch 'develop' of github.com:Altinity/clickhouse-sink-connect…

…or into 965-ddl-translation-error-modify-column-datatype-with-default-null
Altinity · Jan 20, 2025 · b970bf8 · b970bf8
2 parents 5c9001a + 5908bed
commit b970bf8
Show file tree

Hide file tree

Showing 11 changed files with 185 additions and 52 deletions.
diff --git a/README.md b/README.md
@@ -79,7 +79,7 @@ First two are good tutorials on MySQL and PostgreSQL respectively.
 
 ## Roadmap 
 
-[2024 Roadmap](https://github.com/Altinity/clickhouse-sink-connector/issues/401)
+[2025 Roadmap](https://github.com/Altinity/clickhouse-sink-connector/issues/401)
 
 ## Help
 

diff --git a/doc/Monitoring.md b/doc/Monitoring.md
@@ -96,6 +96,20 @@ The default username/password is `admin/admin`
 ![](img/Grafana_dashboard.png)
 ![](img/Grafana_dashboard_2.png)
 
+**Setup Alerts for Lag** \
+To setup alerts for lag, you can use the Lag(Seconds) widget in Grafana.
+This widget displays the lag in seconds between the last record received by the sink connector and the time the current time in ClickHouse.
+Click on the three dots on the right, click on More and then click on "Nw Alert Rule".
+![](img/grafana_alert_1.png)
+
+This will open a new window where you can define the alert rule. as shown below.
+
+![](img/grafana_alert_2.png)
+
+The alert threshold can be tested in the Expresions tab as shown below.
+![](img/grafana_alert_3.png)
+
+
 
 **Memory**
 

diff --git a/doc/architecture.md b/doc/architecture.md
@@ -5,9 +5,9 @@ using [Debezium](debezium) into a common log format and then applies those
 transactions to tables in ClickHouse. 
 
 There are two modes of operation. 
-* Lightweight Sink Connector - Combines extract and apply operations 
+* **Lightweight Sink Connector** - Combines extract and apply operations 
   into a single process. 
-* Kafka Sink Connector - Separates extract and apply operations into separate
+* **Kafka Sink Connector** - Separates extract and apply operations into separate
   processes, using a Kafka-compatible event stream for transport between them. 
 
 Debezium offers change data capture on a number of database types. The 

diff --git a/doc/configuration.md b/doc/configuration.md
@@ -14,7 +14,7 @@
 | clickhouse.server.password                                | ClickHouse password                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | clickhouse.server.port                                    | ClickHouse port, For TLS(use the correct port `8443` or `443`                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
 | snapshot.mode                                             | "initial" -> Data that already exists in source database will be replicated. "schema_only" -> Replicate data that is added/modified after the connector is started.\<br/> MySQL: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-snapshot-mode \ <br/>PostgreSQL: https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-snapshot-mode  <br/> MongoDB: initial, never. https://debezium.io/documentation/reference/stable/connectors/mongodb.html |
-| connector.class                                           | MySQL -> "io.debezium.connector.mysql.MySqlConnector" <br/> PostgreSQL -> <br/> Mongo ->   <br/>                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| connector.class                                           | MySQL -> `io.debezium.connector.mysql.MySqlConnector` <br/> PostgreSQL -> `io.debezium.connector.postgresql.PostgresConnector <br/> Mongo -> `io.debezium.connector.mongodb.MongoDbConnector`  <br/>                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | offset.storage.file.filename                              | Offset storage file(This stores the offsets of the source database) MySQL: mysql binlog file and position, gtid set. Make sure this file is durable and its not persisted in temp directories.                                                                                                                                                                                                                                                                                                                                       |
 | database.history.file.filename                            | Database History: Make sure this file is durable and its not persisted in temp directories.                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | schema.history.internal.file.filename                     | Schema History: Make sure this file is durable and its not persisted in temp directories.                                                                                                                                                                                                                                                                                                                                                                                                                                            |

diff --git a/doc/img/grafana_alert_1.png b/doc/img/grafana_alert_1.png
diff --git a/doc/img/grafana_alert_2.png b/doc/img/grafana_alert_2.png
diff --git a/doc/img/grafana_alert_3.png b/doc/img/grafana_alert_3.png
diff --git a/doc/quickstart.md b/doc/quickstart.md
@@ -30,13 +30,17 @@ sudo apt install clickhouse-client
 
 Use Docker Compose to start containers.
 Set the `CLICKHOUSE_SINK_CONNECTOR_LT_IMAGE` to the latest release from the Releases page.
-or run `./getLatestTag.sh` which will set the environment variable 
+or run `./getLatestTag.sh` which will print the environment variable 
+that need to be exported.
 
 ```
 cd sink-connector-lightweight/docker
 ./getLatestTag.sh
 ```
-
+Example:
+```
+ export CLICKHOUSE_SINK_CONNECTOR_LT_IMAGE=altinity/clickhouse-sink-connector:2.5.0-lt
+```
 ```
 docker compose -f docker-compose-mysql.yml up --renew-anon-volumes
 ```
-Original file line number
+Diff line change
@@ Expand Up @@
     ## Roadmap
-    [2024 Roadmap](https://github.com/Altinity/clickhouse-sink-connector/issues/401)
+    [2025 Roadmap](https://github.com/Altinity/clickhouse-sink-connector/issues/401)
     ## Help
@@ Expand Down @@