Monitor Business Apps with OpenTelemetry and VMware Aria Operations for Applications (Wavefront)

A project which demonstrates usage of OpenTelemetry K8S Operator and VMware Aria Operations for Applications to monitor Application without modifing its source code.

This is a Proof-of-Concept. OpenTelemetry Collector and Operator APIs are subjects to change. Use with caution!

The purpose is to gather "RED" metrics to monitor App health (See Google SRE Book) and App traces to visualize app architecure and potential errors.

How it works

1. OTel Operator injects OTel Auto-Instrumentation in Application Pod (sidecar)
2. OTel Auto-Instrumentation sends metrics and traces to Wavefront Proxy (OTL Protocol over gRPC or HTTP)
3. Wavefront Proxy with built-in OTel Collector capabilities buffers and sends metrics and traces to VMware Aria Operations for Applications tenant.

Installation

Prerequisites

• Kubernetes >= 1.19
• VMware Aria Operations for Applications API KEY + TENANT NAME (Get Trial Code)

Install Wavefront Proxy (OTel Collector Service)

Install Wavefront Proxy Collector which gather metrics from Kubernetes, deployed middleware and acts as OTel Collector Service.

The installation is done through our Operator

kubectl apply -f https://github.com/wavefrontHQ/observability-for-kubernetes/releases/download/v2.17.1/wavefront-operator.yaml
kubectl wait pods -l app.kubernetes.io/component=controller-manager -n observability-system  --for=condition=Ready

Create a secret with your API KEY

export YOUR_WAVEFRONT_TOKEN=aaaaaa-bbbb-cccc-9ddddd7a-eeeeeee
kubectl create -n observability-system secret generic wavefront-secret --from-literal token=${YOUR_WAVEFRONT_TOKEN}

Then create a Wavefront CRD with the OTLP protocol activated and the proper URL to your tenant

kubectl apply -f - <<EOF
apiVersion: wavefront.com/v1alpha1
kind: Wavefront
metadata:
  name: wavefront
  namespace: observability-system
spec:
  clusterName: dbrice-gke
  wavefrontUrl: https://longboard.wavefront.com/
  dataCollection:
    metrics:
      enable: true
  dataExport:
    wavefrontProxy:
      enable: true
      otlp:
        grpcPort: 4317
        httpPort: 4318
        resourceAttrsOnMetricsIncluded: true
EOF

Install cert-manager (prerequisite for OTel operator)

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.8/cert-manager.yaml
kubectl wait pods -l app=webhook -n cert-manager --for=condition=Ready

Install opentelemetry-operator

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
kubectl wait pods -l control-plane=controller-manager -n opentelemetry-operator-system  --for=condition=Ready

Create OpenTelemetry auto-instrumentation injection

This example deploys an OpenTelementry auto-instrumentation for Java, Python and NodeJS. Others Languages in Roadmap

The endpoint is the wavefront-proxy service over gRPC port (4317)

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://wavefront-proxy.observability-system:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://wavefront-proxy.observability-system:4318
  dotnet:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://wavefront-proxy.observability-system:4318
EOF

Demo with Spring App and RabbitMQ

The auto-instrument capability is used on a simple Rabbit MQ Hello World tutorial. The Java source code is not altered.

Thanks to OpenTelemetry the different message from sender to receiver via rabbitMQ are tracked with a unique ID and make possible to generate a visualization of the application architecture.

Deploy Rabbit MQ

Deploy RabbitMQ from Helm Chart provided by Bitnami

username and password are soft-coded in the rabbitMQ tutorial.

helm repo add bitnami https://charts.bitnami.com/bitnami  && helm repo update

helm install rabbitserver bitnami/rabbitmq \
    --set persistence.enabled=false \
    --set metrics.enabled=true \
    --set auth.username=tutorial \
    --set auth.password=tutorial \
    --set nameOverride=rabbitserver

kubectl wait pods -l app.kubernetes.io/instance=rabbitserver  --for=condition=Ready

Deploy the Java Apps

The Apps are deployed with the core K8S API (Pods, Deployments, Jobs, ...).

An annotation is added in order to trigger Auto-instrumentation.

instrumentation.opentelemetry.io/inject-java: "default/my-instrumentation"

An env variable is added to make visualization nicer.

env:
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "application=rabbitApp"

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: rabbitmq-tutorials
    run: sender
  name: sender
  annotations:
    instrumentation.opentelemetry.io/inject-java: "default/my-instrumentation"
spec:
  containers:
  - image: docker.io/bdekany/rabbitmq-tutorials:spring-amqp
    name: sender
    env:
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: "application=rabbitApp"
    command: ["java"]
    args: ["-jar", "rabbitmq-tutorials.jar", "--spring.profiles.active=hello-world,sender,remote"]
  dnsPolicy: ClusterFirst
  restartPolicy: Always
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: rabbitmq-tutorials
    run: receiver
  name: receiver
  annotations:
    instrumentation.opentelemetry.io/inject-java: "default/my-instrumentation"
spec:
  containers:
  - image: docker.io/bdekany/rabbitmq-tutorials:spring-amqp
    name: receiver
    env:
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: "application=rabbitApp"
    command: ["java"]
    args: ["-jar", "rabbitmq-tutorials.jar", "--spring.profiles.active=hello-world,receiver,remote"]
  dnsPolicy: ClusterFirst
  restartPolicy: Always
EOF

Check the logs of both containers.

kubectl logs -l app=rabbitmq-tutorials -f

[otel.javaagent 2023-08-09 12:16:57:010 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.28.0
 __                 __   ___                  
|__)_ |_ |_ .|_|\/|/  \   |    |_ _  _. _ | _ 
| \(_||_)|_)||_|  |\_\/   | |_||_(_)| |(_||_) 
                                              
Ready ... running for 10000ms
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'
 [x] Sent 'Hello World!'
 [x] Received 'Hello World!'
 [x] Sent 'Hello World!'

Build your own docker image

git clone https://github.com/rabbitmq/rabbitmq-tutorials
cd rabbitmq-tutorials/spring-amqp
cp ../../Dockerfile .
docker build .