Lab 8

app_python/app.py

@@ -1,26 +1,32 @@
 import pytz
-from flask import Flask
+from flask import Flask, Response
 from datetime import datetime
+from prometheus_client import Counter, Gauge, generate_latest
 
 app = Flask(__name__)
 
 # Global variable with timezone
 TIME_ZONE = ""
 CONFIG_FILE = "config.txt"
 
+# Prometheus metrics
+REQUEST_COUNT = Counter('app_request_count', 'Total number of requests')
+REQUEST_TIME = Gauge('app_request_time_seconds', 'Time spent processing requests')
+ERROR_COUNT = Counter('app_error_count', 'Total number of errors')
+
 
 def get_timezone(file=CONFIG_FILE) -> pytz.timezone:
     """
-    Loading timezone frome configuration file
+    Loading timezone from configuration file
     """
     try:
         with open(file, "r") as config_file:
             time_zone = config_file.read().strip()
 
         return pytz.timezone(time_zone)
     except Exception:
-
         # Default value
+        ERROR_COUNT.inc()  # Increment error counter
         return pytz.timezone("Europe/Moscow")
 
 
@@ -31,13 +37,27 @@ def current_time() -> str:
     """
     global TIME_ZONE
 
+    # Start tracking request time
+    start_time = datetime.now()
+
     # Set up TIME_ZONE if it was not
     if TIME_ZONE == "":
         TIME_ZONE = get_timezone()
 
     time = datetime.now(TIME_ZONE)
+    REQUEST_COUNT.inc()  # Increment request counter
+    REQUEST_TIME.set((datetime.now() - start_time).total_seconds())  # Set request time
+
     return f"Current time in {TIME_ZONE} is {time.strftime('%H:%M:%S')}"
 
 
+@app.route("/metrics")
+def metrics() -> Response:
+    """
+    Expose Prometheus metrics.
+    """
+    return Response(generate_latest(), mimetype="text/plain")
+
+
 if __name__ == "__main__":
-    app.run(host="0.0.0.0", port=5000, debug=True)
+    app.run(host="0.0.0.0", port=5000, debug=True)

app_python/requirements.txt

@@ -6,3 +6,4 @@ Jinja2==3.1.5
 MarkupSafe==3.0.2
 pytz==2024.2
 Werkzeug==3.1.3
+prometheus_client==0.21.1

monitoring/LOGGING.md

@@ -31,6 +31,6 @@ The same `loki` network for all components `Promtail` service. It runs on 3100 p
 
 ## Results demonstrarion
 
-![Screenshot](img/screenshot.png)
+![Screenshot](img/lab7.png)
 
 On the screenshot above you may see dashboard with all containers logs. My application on C++ does not write logs, so I had no ability to show them. The simplest way to demonstrate its log aggregation was to manually write in its *.log file. I saved there line from app_python log (look on diference in time).

monitoring/METRICS.md

@@ -0,0 +1,76 @@
+# Metrics
+
+This report provides results of the lab with demonstration of screenshots.
+
+## Task 1
+
+In this step service `prometheus` was introduced into `docker-compose.yml`. Process of configuration is similar to the previous lab (check services `grafana` and `promtail`). The result you may see on the screenshot.
+
+![Loki and Prometheus](img/loki_and_prometheus_monitoring.png)
+![Loki's metric](img/loki_metric.png)
+![Prometheus' metric](img/prometheus_metric.png)
+
+## Task 2
+
+Dashboards from examples (![Loki](https://grafana.com/grafana/dashboards/13407-loki2-0-global-metrics/) and ![Prometheus](https://grafana.com/grafana/dashboards/3662-prometheus-2-0-overview/)) was used. The process was simple (just importing, using UI, which is intuitive). Metrics for other services was introdiced. There are results:
+
+**Loki**:
+
+![Loki](img/loki_overview_1.png)
+![Loki](img/loki_overview_2.png)
+![Loki](img/loki_overview_3.png)
+
+**Prometheus:**
+![Prometheus](img/prometheus_overview_1.png)
+![Prometheus](img/prometheus_overview_2.png)
+![Prometheus](img/prometheus_overview_3.png)
+![Prometheus](img/prometheus_overview_4.png)
+![Prometheus](img/prometheus_overview_5.png)
+![Prometheus](img/prometheus_overview_6.png)
+
+**Other services' metrics:**
+
+![Grafana's metric](img/grafana_metric.png)
+![Promtail's metric](img/promtail_metric.png)
+
+Log rotation mechanism realised by introducing
+
+```yaml
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "10m"  
+        max-file: "3"
+```
+
+for each component in `docker-compose.yml`.
+
+For memory limits was used
+
+```yaml
+    mem_limit: 512m
+```
+
+Finally, all services was gathered (python_app from bonus task too)
+
+![All services](img/all_services.png)
+
+## Bonus task
+
+**Metrics:**
+
+![Python app's metric](img/python_app_metric.png)
+
+**Health check:**
+
+Was introduced
+
+```yaml
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:<port>/<url's tail>"]
+      interval: <interval>s
+      timeout: <timeout>s
+      retries: <retries>
+      ```
+
+for each service.

monitoring/docker-compose.yml

@@ -11,18 +11,34 @@ services:
     command: -config.file=/etc/loki/local-config.yaml
     networks:
       - loki
+    mem_limit: 512m
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:3100/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "10m"  
+        max-file: "3"
 
   promtail:
     image: grafana/promtail:latest
     volumes:
       - /var/lib/docker/containers:/var/lib/docker/containers:ro
+    ports:
+      - "9080:9080"
     command: |
       -config.file=/etc/promtail/config.yml
     entrypoint:
       - sh
       - -c
       - |
         cat <<EOF > /etc/promtail/config.yml
+        server:
+          http_listen_port: 9080
+          
         client:
           url: http://loki:3100/api/prom/push
 
@@ -36,9 +52,19 @@ services:
                   __path__: /var/lib/docker/containers/*/*log
         EOF
         /usr/bin/promtail -config.file=/etc/promtail/config.yml
-    
     networks:
       - loki
+    mem_limit: 512m
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:9080/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "10m"  
+        max-file: "3"
 
   grafana:
     environment:
@@ -69,4 +95,81 @@ services:
     ports:
       - "3000:3000"
     networks:
-      - loki
+      - loki
+    mem_limit: 512m
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "10m"  
+        max-file: "3" 
+
+  prometheus:
+    image: prom/prometheus:latest
+    ports:
+      - "9090:9090"
+    entrypoint:
+      - sh
+      - -c
+      - |
+        cat <<EOF > /etc/prometheus/prometheus.yml
+        global:
+          scrape_interval: 15s
+
+        scrape_configs:
+          - job_name: 'loki'
+            static_configs:
+              - targets: ['loki:3100']
+
+          - job_name: 'prometheus'
+            static_configs:
+              - targets: ['localhost:9090']
+
+          - job_name: 'grafana'
+            static_configs:
+              - targets: ['grafana:3000']
+
+          - job_name: 'promtail'
+            static_configs:
+              - targets: ['promtail:9080']
+
+          - job_name: 'python_app'
+            static_configs:
+              - targets: ['python_app:5000']
+        EOF
+        /bin/prometheus --config.file=/etc/prometheus/prometheus.yml
+    networks:
+      - loki
+    mem_limit: 512m
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:9090/-/healthy"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "10m"  
+        max-file: "3"
+
+  python_app:
+    image: voronm1522/devops:python-app
+    ports:
+      - "5000:5000"
+    networks:
+      - loki
+    mem_limit: 512m
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:5000"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "10m"
+        max-file: "3"