Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge main #2294

Closed
wants to merge 14 commits into from
+1,556 −596
Closed

merge main #2294

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
df9be77
fix: "Loading Pods View" and "failed to get pod details" state on UI …
adarsh0728 Dec 9, 2024
a8288ae
chore: clippy error fixes (#2271)
BulkBeing Dec 10, 2024
1c0989a
fix: Numaflow serving sink (#2103)
yhl25 Dec 10, 2024
8bed236
feat: mvtx gauge metrics support (#2259)
veds-g Dec 10, 2024
059a585
feat: counter metrics visualizer for mono-vertex (#2256)
adarsh0728 Dec 10, 2024
b2e480b
chore: make Message cheap to clone (#2272)
yhl25 Dec 11, 2024
10b4978
docs: updated CHANGELOG.md (#2276)
github-actions[bot] Dec 12, 2024
a32ebbc
chore: add container type to server info log (#2286)
kohlisid Dec 13, 2024
883650c
chore: always publish watermark to sink OT (#2288)
whynowy Dec 15, 2024
d75998f
feat: pipeline gauge metrics (#2284)
veds-g Dec 15, 2024
d944615
chore(deps): bump nanoid from 3.3.7 to 3.3.8 in /ui (#2289)
dependabot[bot] Dec 16, 2024
c05c96a
chore(deps): bump golang.org/x/crypto from 0.27.0 to 0.31.0 (#2290)
dependabot[bot] Dec 16, 2024
fc14696
fix: set max decode size of proto message, add mvtx metrics (#2283)
yhl25 Dec 16, 2024
7450e16
chore: e2e tests for metrics APIs (#2277)
adarsh0728 Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Changelog

## v1.4.2 (2024-12-11)

* [c9dc38f4](https://github.com/numaproj/numaflow/commit/c9dc38f4cce2b5db598536a7539f2a35febcf1ca) Update manifests to v1.4.2
* [fea792b3](https://github.com/numaproj/numaflow/commit/fea792b36bd342adcdcdd96768b6fdd68921bfd2) fix: set max decode size of proto message (#2275)

### Contributors

* Sidhant Kohli

## v1.4.1 (2024-12-05)

* [346f2a73](https://github.com/numaproj/numaflow/commit/346f2a7321d158fa9ce9392cfdcc76d671d6f577) Update manifests to v1.4.1
119 changes: 100 additions & 19 deletions config/advanced-install/namespaced-numaflow-server.yaml
View file
Original file line number Diff line number Diff line change
@@ -143,6 +143,71 @@ data:
# example for local prometheus service
# url: http://prometheus-operated.monitoring.svc.cluster.local:9090
patterns:
- name: vertex_gauge
object: vertex
title: Vertex Pending Messages
description: This query is the total number of pending messages for the vertex
expr: |
sum($metric_name{$filters}) by ($dimension, period)
params:
- name: start_time
required: false
- name: end_time
required: false
metrics:
- metric_name: vertex_pending_messages
required_filters:
- namespace
- pipeline
- vertex
dimensions:
- name: pod
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: pod
required: false
- name: period
required: false
- name: vertex
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: period
required: false

- name: mono_vertex_gauge
object: mono-vertex
title: Pending Messages Lag
description: This query is the total number of pending messages for the mono vertex
expr: |
sum($metric_name{$filters}) by ($dimension, period)
params:
- name: start_time
required: false
- name: end_time
required: false
metrics:
- metric_name: monovtx_pending
required_filters:
- namespace
- mvtx_name
dimensions:
- name: pod
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: pod
required: false
- name: period
required: false
- name: mono-vertex
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: period
required: false

- name: mono_vertex_histogram
object: mono-vertex
title: Processing Time Latency
@@ -164,28 +229,22 @@ data:
- namespace
- mvtx_name
dimensions:
- name: mono-vertex
- name: pod
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: pod
required: false
- metric_name: monovtx_sink_time_bucket
required_filters:
- namespace
- mvtx_name
dimensions:
- name: mono-vertex
# expr: optional expression for prometheus query
# overrides the default expression
# Add histogram metrics similar to the pattern above
#- metric_name: monovtx_sink_time_bucket
# required_filters:
# - namespace
# - mvtx_name
# dimensions:
# - name: pod
# #expr: optional
# filters:
# - name: pod
# required: false
# - name: mono-vertex
# #expr: optional
- name: pod
filters:
- name: pod
required: false

- name: vertex_throughput
object: vertex
title: Vertex Throughput and Message Rates
@@ -206,8 +265,30 @@ data:
- vertex
dimensions:
- name: vertex
# expr: optional expression for prometheus query
# overrides the default expression
- name: pod
filters:
- name: pod
required: false

- name: mono_vertex_throughput
object: mono-vertex
title: Mono-Vertex Throughput and Message Rates
description: This pattern measures the throughput of a mono-vertex in messages per second across different dimensions
expr: sum(rate($metric_name{$filters}[$duration])) by ($dimension)
params:
- name: duration
required: true
- name: start_time
required: false
- name: end_time
required: false
metrics:
- metric_name: monovtx_read_total
required_filters:
- namespace
- mvtx_name
dimensions:
- name: mono-vertex
- name: pod
filters:
- name: pod
119 changes: 100 additions & 19 deletions config/advanced-install/numaflow-server.yaml
View file
Original file line number Diff line number Diff line change
@@ -150,6 +150,71 @@ data:
# example for local prometheus service
# url: http://prometheus-operated.monitoring.svc.cluster.local:9090
patterns:
- name: vertex_gauge
object: vertex
title: Vertex Pending Messages
description: This query is the total number of pending messages for the vertex
expr: |
sum($metric_name{$filters}) by ($dimension, period)
params:
- name: start_time
required: false
- name: end_time
required: false
metrics:
- metric_name: vertex_pending_messages
required_filters:
- namespace
- pipeline
- vertex
dimensions:
- name: pod
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: pod
required: false
- name: period
required: false
- name: vertex
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: period
required: false

- name: mono_vertex_gauge
object: mono-vertex
title: Pending Messages Lag
description: This query is the total number of pending messages for the mono vertex
expr: |
sum($metric_name{$filters}) by ($dimension, period)
params:
- name: start_time
required: false
- name: end_time
required: false
metrics:
- metric_name: monovtx_pending
required_filters:
- namespace
- mvtx_name
dimensions:
- name: pod
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: pod
required: false
- name: period
required: false
- name: mono-vertex
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: period
required: false

- name: mono_vertex_histogram
object: mono-vertex
title: Processing Time Latency
@@ -171,28 +236,22 @@ data:
- namespace
- mvtx_name
dimensions:
- name: mono-vertex
- name: pod
# expr: optional expression for prometheus query
# overrides the default expression
filters:
- name: pod
required: false
- metric_name: monovtx_sink_time_bucket
required_filters:
- namespace
- mvtx_name
dimensions:
- name: mono-vertex
# expr: optional expression for prometheus query
# overrides the default expression
# Add histogram metrics similar to the pattern above
#- metric_name: monovtx_sink_time_bucket
# required_filters:
# - namespace
# - mvtx_name
# dimensions:
# - name: pod
# #expr: optional
# filters:
# - name: pod
# required: false
# - name: mono-vertex
# #expr: optional
- name: pod
filters:
- name: pod
required: false

- name: vertex_throughput
object: vertex
title: Vertex Throughput and Message Rates
@@ -213,8 +272,30 @@ data:
- vertex
dimensions:
- name: vertex
# expr: optional expression for prometheus query
# overrides the default expression
- name: pod
filters:
- name: pod
required: false

- name: mono_vertex_throughput
object: mono-vertex
title: Mono-Vertex Throughput and Message Rates
description: This pattern measures the throughput of a mono-vertex in messages per second across different dimensions
expr: sum(rate($metric_name{$filters}[$duration])) by ($dimension)
params:
- name: duration
required: true
- name: start_time
required: false
- name: end_time
required: false
metrics:
- metric_name: monovtx_read_total
required_filters:
- namespace
- mvtx_name
dimensions:
- name: mono-vertex
- name: pod
filters:
- name: pod
Loading