Optionally add prometheus metrics endpoint (#7)

Using prometheus-fastapi-instrumentor.
Set the environment variable DIALS_REST_ENABLE_METRICS=1 to enable the
/metrics endpoint.

The JWT_SECRET environment variable now has a DIALS_REST_ prefix.

README.md

@@ -1,4 +1,4 @@
-A RESTful API to a (limited) subset of DIALS functionality.
+# A RESTful API to a (limited) subset of DIALS functionality.
 
 ```
 mamba env create -f environment.yml
@@ -8,15 +8,14 @@ mamba install dials uvicorn[standard] -y
 pip install -e .
 ```
 
-Generate a JWT secret and store it in the `JWT_SECRET` environment variable:
+Generate a JWT secret and store it in the `DIALS_REST_JWT_SECRET` environment variable:
 ```
-$ export JWT_SECRET=`openssl rand -hex 32`
+$ export DIALS_REST_JWT_SECRET=`openssl rand -hex 32`
 ```
 
 Create a new access token:
 ```
-$ create-access-token --expiry 2022-12-31
-eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzI0NDQ4MDB9.8J_5yadgK3UrErs1AOXKxjlvkzc-GCNA6Eg-v9obpvU
+$ export TOKEN=`create-access-token --expiry 2023-06-01`
 ```
 
 Start the app:
@@ -27,7 +26,7 @@ $ uvicorn dials_rest.main:app --reload
 <!-- curl -X 'POST' 'http://127.0.0.1:8001/export_bitmap/' -H 'accept: application/json' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2ODU1Nzc2MDB9.i6ipplAzjhfBDAZFRsw3UTXYWbQnzZ02YDUSnpvz4j0' -H 'Content-Type: application/json' -d '{ -->
 In another terminal:
 ```
-curl -X 'POST' 'http://127.0.0.1:8000/export_bitmap/' -H 'accept: application/json' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzI0NDQ4MDB9.8J_5yadgK3UrErs1AOXKxjlvkzc-GCNA6Eg-v9obpvU' -H 'Content-Type: application/json' -d '{
+curl -X 'POST' 'http://127.0.0.1:8000/export_bitmap/' -H 'accept: application/json' -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{
   "filename": "/dls/i24/data/2022/cm31109-5/myoglobin_dithionite/myglobin_3_00001.cbf",
   "image_index": 1,
   "format": "png",
@@ -42,17 +41,37 @@ curl -X 'POST' 'http://127.0.0.1:8000/export_bitmap/' -H 'accept: application/js
 }' -o image.png
 ```
 
+## Docker/podman
 To build with docker/podman:
 ```
 $ podman build -t dials-rest --format=docker .
 ```
 
 To create an access token:
 ```
-$ podman run -e JWT_SECRET=$JWT_SECRET -it dials-rest /env/bin/create-access-token
+$ podman run -e DIALS_REST_JWT_SECRET=$DIALS_REST_JWT_SECRET -it dials-rest /env/bin/create-access-token
 ```
 
 To run the server:
 ```
-$ podman run -e JWT_SECRET=$JWT_SECRET -p 127.0.0.1:8081:80 dials-rest
+$ podman run -e DIALS_REST_JWT_SECRET=$DIALS_REST_JWT_SECRET -p 127.0.0.1:8081:80 dials-rest
+```
+
+
+## Monitoring
+Before starting the DIALS REST server export the environment variable `export DIALS_REST_ENABLE_METRICS=1` to enable a `/metrics` endpoint in [prometheus](https://prometheus.io/) format.
+
+Next add a simple `prometheus.yml` config file that tells prometheus where to scrape metrics from:
+```
+$ cat > prometheus.yaml << EOF
+scrape_configs:
+  - job_name: "dials-rest"
+    scrape_interval: 15s
+    static_configs:
+      - targets: ["localhost:8000"]
+EOF
+```
+Run prometheus via podman (or equivalently docker) exposed on port 9091:
+```
+$ podman run --network=host -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --web.listen-address="localhost:9091" --config.file=/etc/prometheus/prometheus.yml
 ```

environment-dev.yml

@@ -8,6 +8,7 @@ dependencies:
   - dials-data
   - fastapi
   - httpx
+  - prometheus-fastapi-instrumentator
   - pytest
   - python=3.10
   - python-dateutil

environment.yml

@@ -6,6 +6,7 @@ channels:
 dependencies:
   - dials>=3.14
   - fastapi
+  - prometheus-fastapi-instrumentator
   - python=3.10
   - python-dateutil
   - python-jose

src/dials_rest/main.py

@@ -13,6 +13,15 @@
 app.include_router(find_spots.router)
 app.include_router(image.router)
 
+if settings.enable_metrics:
+    from prometheus_fastapi_instrumentator import Instrumentator
+
+    instrumentator = Instrumentator(
+        excluded_handlers=["/metrics"],
+    )
+    instrumentator.instrument(app)
+    instrumentator.expose(app)
+
 
 @app.get("/")
 async def root():

src/dials_rest/settings.py

@@ -2,16 +2,21 @@
 
 from functools import lru_cache
 
-from pydantic import BaseSettings
+from pydantic import BaseSettings, Field
 
 
 class Settings(BaseSettings):
     jwt_secret: str
+    enable_metrics: bool = Field(
+        default=False,
+        description="Expose metrics in prometheus format on the /metrics endpoint",
+    )
 
     @staticmethod
     @lru_cache
     def get():
         return Settings()
 
     class Config:
+        env_prefix = "DIALS_REST_"
         secrets_dir = "/opt/secrets"

tests/conftest.py

@@ -16,15 +16,15 @@ def dials_data():
 
 @pytest.fixture
 def client(monkeypatch):
-    monkeypatch.setenv("JWT_SECRET", JWT_SECRET)
+    monkeypatch.setenv("DIALS_REST_JWT_SECRET", JWT_SECRET)
     from dials_rest.main import app
 
     return TestClient(app)
 
 
 @pytest.fixture
 def access_token(monkeypatch):
-    monkeypatch.setenv("JWT_SECRET", JWT_SECRET)
+    monkeypatch.setenv("DIALS_REST_JWT_SECRET", JWT_SECRET)
 
     from dials_rest.auth import create_access_token