Skip to content

Commit

Permalink
plugin added OpenTelemetry plugin and example (#1092)
Browse files Browse the repository at this point in the history 
* added OpenTelemetry plugin and example

---------

Co-authored-by: zilto <tjean@DESKTOP-V6JDCS2>
  • Loading branch information
@zilto
zilto and zilto authored Aug 14, 2024
1 parent 3571554 commit 3b2253b
Show file tree
Hide file tree
Showing 7 changed files with 569 additions and 0 deletions.
41 changes: 41 additions & 0 deletions examples/LLM_Workflows/observability_openllmetry/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Monitor Hamilton with OpenTelemetry, OpenLLMetry and Traceloop

In this simple example, you'll learn how to use the `OpenTelemetryTracer` to emit traces of your Hamilton code using the OpenTelemetry format, in particular LLM applications.

![Traceloop screenshot](image.png)

[OpenTelemetry](https://opentelemetry.io/) is an open-source cross-language tool that allows to instrument, generate, collect, and export telemetry data (metrics, logs, traces), and constitute an industry-recognized standard. Learn more about it in this [Awesome OpenTelemetry repository](https://github.com/magsther/awesome-opentelemetry)

[OpenLLMetry](https://github.com/traceloop/openllmetry) is an open-source Python library that automatically instruments with OpenTelemetry components of your LLM stack including LLM providers (OpenAI, Anthropic, HuggingFace, Cohere, etc.), vector databases (Weaviate, Qdrant, Chroma, etc.), and frameworks ([Burr](https://github.com/dagworks-inc/burr), Haystack, LangChain, LlamaIndex). In concrete terms, it means you automatically get detailed traces of API calls, retrieval operations, or text transformations for example.

One thing to note, OpenTelemetry is a middleware; it doesn't provide a destination to store data nor a dashboard. For this example, we'll use the tool [Traceloop](https://www.traceloop.com/), which is built by the developers of OpenLLMetry. It has a generous free-tier and can be conveniently set up in a few lines of code for this demo.

## Set up
Having access to a [Traceloop account](https://www.traceloop.com/) and an API key is a pre-requisite.

1. Create a virtual environment and activate it
```bash
python -m venv venv && . venv/bin/active
```

2. Install requirements.
```bash
pip install -r requirements.txt
```

3. Set environment variables for your API keys `OPENAI_API_KEY` and `TRACELOOP_API_KEY`

4. Execute the code
```bash
python run.py
```

5. Explore results on Traceloop (or your OpenTelemetry destination).

### Without Traceloop

For this example to work without Traceloop, you will need to set up your own [OpenTelemetry destination](https://opentelemetry.io/ecosystem/vendors/). We suggest using [Jaeger](https://www.jaegertracing.io/docs/1.47/getting-started/) and included Python code to route telemetry to it in `run.py`.

## Should I still use the Hamilton UI?

Absolutely! OpenTelemetry focsues on collecting telemetry about the internals of code and external API calls. It's a standard amongst web services. There's no conflict between the OpenTelemetry tracer and the tracking for the Hamilton UI. In fact, the Hamilton UI captures a superset of what OpenTelemetry allows, tailored to the Hamilton framework such as: visualizations, data lineage, summary statistics, and more utilities to improve your development experience. In the not too distant future, the Hamilton UI could ingest OpenTelemetry data 😉 (contributions welcomed!)
318 changes: 318 additions & 0 deletions examples/LLM_Workflows/observability_openllmetry/notebook.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,318 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Monitor Hamilton with OpenTelemetry, OpenLLMetry and Traceloop\n",
"\n",
"\n",
"It will showcase how the [OpenTelemetry](https://opentelemetry.io/) plugin for Hamilton can trace node and using [OpenLLMetry](https://github.com/traceloop/openllmetry) allows to automatically trace popular LLM components. This example uses [Traceloop](https://www.traceloop.com/) as a destination for telemetry, but other [open-source options are available](https://opentelemetry.io/ecosystem/vendors/) (Jaeger, Elastic, Clickhouse, Grafana, etc.)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from hamilton import driver\n",
"from hamilton.plugins import h_opentelemetry\n",
"from traceloop.sdk import Traceloop\n",
"\n",
"%load_ext hamilton.plugins.jupyter_magic"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define a dataflow\n",
"The next cell creates a Python module named `llm_dataflow` and defines a dataflow that creates an OpenAI client and queries the chat completions endpoint to center an HTML `<div>` tag.\n",
"\n",
"You'll notice that nothing special is added to enable tracing"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 2.43.0 (0)\n",
" -->\n",
"<!-- Title: %3 Pages: 1 -->\n",
"<svg width=\"282pt\" height=\"167pt\"\n",
" viewBox=\"0.00 0.00 282.00 167.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 163)\">\n",
"<title>%3</title>\n",
"<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-163 278,-163 278,4 -4,4\"/>\n",
"<g id=\"clust1\" class=\"cluster\">\n",
"<title>cluster__legend</title>\n",
"<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-74 8,-151 104,-151 104,-74 8,-74\"/>\n",
"<text text-anchor=\"middle\" x=\"56\" y=\"-135.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
"</g>\n",
"<!-- universal_truth -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>universal_truth</title>\n",
"<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M262,-64C262,-64 145,-64 145,-64 139,-64 133,-58 133,-52 133,-52 133,-12 133,-12 133,-6 139,0 145,0 145,0 262,0 262,0 268,0 274,-6 274,-12 274,-12 274,-52 274,-52 274,-58 268,-64 262,-64\"/>\n",
"<text text-anchor=\"start\" x=\"144\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">universal_truth</text>\n",
"<text text-anchor=\"start\" x=\"194\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
"</g>\n",
"<!-- llm_client -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>llm_client</title>\n",
"<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M92,-64C92,-64 20,-64 20,-64 14,-64 8,-58 8,-52 8,-52 8,-12 8,-12 8,-6 14,0 20,0 20,0 92,0 92,0 98,0 104,-6 104,-12 104,-12 104,-52 104,-52 104,-58 98,-64 92,-64\"/>\n",
"<text text-anchor=\"start\" x=\"19\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">llm_client</text>\n",
"<text text-anchor=\"start\" x=\"30\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">OpenAI</text>\n",
"</g>\n",
"<!-- llm_client&#45;&gt;universal_truth -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>llm_client&#45;&gt;universal_truth</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M104.24,-32C110.09,-32 116.19,-32 122.36,-32\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"122.78,-35.5 132.78,-32 122.78,-28.5 122.78,-35.5\"/>\n",
"</g>\n",
"<!-- function -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>function</title>\n",
"<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-119.5C84,-119.5 28,-119.5 28,-119.5 22,-119.5 16,-113.5 16,-107.5 16,-107.5 16,-94.5 16,-94.5 16,-88.5 22,-82.5 28,-82.5 28,-82.5 84,-82.5 84,-82.5 90,-82.5 96,-88.5 96,-94.5 96,-94.5 96,-107.5 96,-107.5 96,-113.5 90,-119.5 84,-119.5\"/>\n",
"<text text-anchor=\"middle\" x=\"56\" y=\"-97.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x7f78c554f050>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%cell_to_module llm_dataflow -d\n",
"import openai\n",
"\n",
"def llm_client() -> openai.OpenAI:\n",
" return openai.OpenAI()\n",
"\n",
"def universal_truth(llm_client: openai.OpenAI) -> str:\n",
" response = llm_client.chat.completions.create(\n",
" model=\"gpt-4o-mini\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a benevolent all-knowning being\"},\n",
" {\"role\": \"user\", \"content\": \"Please center my HTML <div> tag\"},\n",
" ],\n",
" )\n",
" return str(response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build the dataflow\n",
"The Hamilton `Builder` object takes the previously defined dataflow via `.with_modules(llm_dataflow)`. Then, we pass the `OpenTelemetryTracer` object via `.with_adapters()` to trace dataflow execution. Make sure to call `Traceloop.init()` before creating the `OpenTelemetryTracer`.\n",
"\n",
"You'll need to set your `OPENAI_API_KEY` and your `TRACELOOP_API_KEY` to run the cell."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"os.environ[\"OPENAI_API_KEY\"] = ...\n",
"os.environ[\"TRACELOOP_API_KEY\"] = ..."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32mTraceloop syncing configuration and prompts\u001b[39m\n",
"\u001b[32mTraceloop exporting traces to https://api.traceloop.com authenticating with bearer token\n",
"\u001b[39m\n"
]
}
],
"source": [
"Traceloop.init()\n",
"\n",
"dr = (\n",
" driver.Builder()\n",
" .with_modules(llm_dataflow)\n",
" .with_adapters(h_opentelemetry.OpenTelemetryTracer())\n",
" .build()\n",
")\n",
"\n",
"# If you wanted to use another OpenTelemetry destination such as the\n",
"# open-source Jaeger, setup the container locally and use the following code\n",
"\n",
"# from opentelemetry import trace\n",
"# from opentelemetry.sdk.trace import TracerProvider\n",
"# from opentelemetry.sdk.trace.export import SimpleSpanProcessor\n",
"# from opentelemetry.exporter.jaeger import JaegerExporter\n",
"\n",
"# jaeger_exporter = JaegerExporter(agent_host_name='localhost', agent_port=5775)\n",
"# span_processor = SimpleSpanProcessor(jaeger_exporter)\n",
"# provider = TracerProvider(active_span_processor=span_processor)\n",
"# trace.set_tracer_provider(provider)\n",
"\n",
"results = dr.execute([\"universal_truth\"])"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"To center an HTML `<div>` element, you can use CSS. There are several methods to achieve this, depending on whether you want to center it horizontally, vertically, or both. Here are a few common methods:\n",
"\n",
"### Centering Horizontally\n",
"\n",
"One of the simplest ways to center a `<div>` horizontally is to set its width, and then use `margin: auto;`. Here's an example:\n",
"\n",
"```html\n",
"<!DOCTYPE html>\n",
"<html lang=\"en\">\n",
"<head>\n",
" <meta charset=\"UTF-8\">\n",
" <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
" <title>Center Div</title>\n",
" <style>\n",
" .centered-div {\n",
" width: 50%; /* Set a width */\n",
" margin: 0 auto; /* Center horizontally */\n",
" background-color: lightblue; /* Optional: Background color */\n",
" padding: 20px; /* Optional: Padding */\n",
" text-align: center; /* Optional: Center text */\n",
" }\n",
" </style>\n",
"</head>\n",
"<body>\n",
" <div class=\"centered-div\">\n",
" This div is centered horizontally!\n",
" </div>\n",
"</body>\n",
"</html>\n",
"```\n",
"\n",
"### Centering Vertically and Horizontally\n",
"\n",
"To center a `<div>` both vertically and horizontally, you can use Flexbox or CSS Grid. Here's an example using Flexbox:\n",
"\n",
"```html\n",
"<!DOCTYPE html>\n",
"<html lang=\"en\">\n",
"<head>\n",
" <meta charset=\"UTF-8\">\n",
" <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
" <title>Center Div</title>\n",
" <style>\n",
" body {\n",
" display: flex;\n",
" justify-content: center; /* Center horizontally */\n",
" align-items: center; /* Center vertically */\n",
" height: 100vh; /* Full viewport height */\n",
" margin: 0; /* Remove default margin */\n",
" }\n",
"\n",
" .centered-div {\n",
" width: 50%; /* Set a width */\n",
" background-color: lightblue; /* Optional: Background color */\n",
" padding: 20px; /* Optional: Padding */\n",
" text-align: center; /* Optional: Center text */\n",
" }\n",
" </style>\n",
"</head>\n",
"<body>\n",
" <div class=\"centered-div\">\n",
" This div is centered both vertically and horizontally!\n",
" </div>\n",
"</body>\n",
"</html>\n",
"```\n",
"\n",
"### Centering with CSS Grid\n",
"\n",
"Here's another way to center using CSS Grid:\n",
"\n",
"```html\n",
"<!DOCTYPE html>\n",
"<html lang=\"en\">\n",
"<head>\n",
" <meta charset=\"UTF-8\">\n",
" <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\n",
" <title>Center Div</title>\n",
" <style>\n",
" body {\n",
" display: grid;\n",
" height: 100vh; /* Full viewport height */\n",
" place-items: center; /* Center both vertically and horizontally */\n",
" margin: 0; /* Remove default margin */\n",
" }\n",
"\n",
" .centered-div {\n",
" width: 50%; /* Set a width */\n",
" background-color: lightblue; /* Optional: Background color */\n",
" padding: 20px; /* Optional: Padding */\n",
" text-align: center; /* Optional: Center text */\n",
" }\n",
" </style>\n",
"</head>\n",
"<body>\n",
" <div class=\"centered-div\">\n",
" This div is centered using CSS Grid!\n",
" </div>\n",
"</body>\n",
"</html>\n",
"```\n",
"\n",
"Choose the method that best fits your layout needs!\n"
]
}
],
"source": [
"print(results[\"universal_truth\"])"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
4 changes: 4 additions & 0 deletions examples/LLM_Workflows/observability_openllmetry/requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
openai
opentelemetry-instrumentation-openai
sf-hamilton
traceloop-sdk
Loading

0 comments on commit 3b2253b

Please sign in to comment.