(Appears on: +MonoVertexSpec, Templates)
@@ -5052,6 +5055,192 @@+ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+Image string
+ |
+
++ + | + +
+
+PullPolicy
+
+Kubernetes core/v1.PullPolicy
+ |
+
++ + | + +
+
+Env
+
+\[\]Kubernetes core/v1.EnvVar
+ |
+
++ + | + +
+
+DefaultResources
+
+Kubernetes core/v1.ResourceRequirements
+ |
+
++ + | + +
+ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+Image string
+ |
+
++ + | + +
+
+PullPolicy
+
+Kubernetes core/v1.PullPolicy
+ |
+
++ + | + +
+
+Env
+
+\[\]Kubernetes core/v1.EnvVar
+ |
+
++ + | + +
+
+DefaultResources
+
+Kubernetes core/v1.ResourceRequirements
+ |
+
++ + | + +
-Specifies the number of retries before marking this job failed. More -info: -https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy -Numaflow defaults to 20 +Specifies the number of retries before marking this job failed. More +info: +https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy +Numaflow defaults to 20 +
+ + + + + + + + + +string alias)
+
+
++ +(Appears on: +GSSAPI) +
+ ++ +
+ +KRB5AuthType describes the kerberos auth type +
+ + + ++ +(Appears on: +AbstractSink) +
+ ++ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+brokers \[\]string
+ |
+
++ + | + +
+
+topic string
+ |
+
++ + | + +
+
+tls
+TLS
+ |
+
+
+
+(Optional)
+ + +TLS user to configure TLS connection for kafka broker TLS.enable=true +default for TLS. + + + |
+
+
+
+config string
+ |
+
++ +(Optional) + | + +
+
+sasl
+ SASL
+ |
+
+
+
+(Optional)
+ + +SASL user to configure SASL connection for kafka broker SASL.enable=true +default for SASL. + + + |
+
+
+ +(Appears on: +Source) +
+ ++ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+brokers \[\]string
+ |
+
++ + | + +
+
+topic string
+ |
+
++ + | + +
+
+consumerGroup string
+ |
+
++ + | + +
+
+tls
+TLS
+ |
+
+
+
+(Optional)
+ + +TLS user to configure TLS connection for kafka broker TLS.enable=true +default for TLS. + + + |
+
+
+
+config string
+ |
+
++ +(Optional) + | + +
+
+sasl
+ SASL
+ |
+
+
+
+(Optional)
+ + +SASL user to configure SASL connection for kafka broker SASL.enable=true +default for SASL. + + + |
+
+
+ +(Appears on: +PipelineSpec) +
+ ++ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+deleteGracePeriodSeconds int32
+ |
+
+
+
+(Optional)
+ + +DeleteGracePeriodSeconds used to delete pipeline gracefully + + + |
+
+
+
+desiredPhase
+ PipelinePhase
+
+ |
+
+
+
+(Optional)
+ + +DesiredPhase used to bring the pipeline from current phase to desired +phase + + + |
+
+
+
+pauseGracePeriodSeconds int32
+ |
+
+
+
+(Optional)
+ + +PauseGracePeriodSeconds used to pause pipeline gracefully + + + |
+
+
+ +(Appears on: +AbstractSink) +
+ ++ +
+ +string alias)
+
+
++ +(Appears on: +TagConditions) +
+ ++ +
+ ++ +(Appears on: +AbstractPodTemplate) +
+ ++ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+annotations map\[string\]string
+ |
+
++ + | + +
+
+labels map\[string\]string
+ |
+
++ + | + +
+ +
+ +| + +Field + | + ++ +Description + | + +||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+
+metadata
+
+Kubernetes meta/v1.ObjectMeta
+ |
+
+
+
+Refer to the Kubernetes API documentation for the fields of the
+metadata field.
+ |
+
+||||||||||||||||||||||
+
+spec
+ MonoVertexSpec
+
+ |
+
+
+
+ +
|
+
+||||||||||||||||||||||
+
+status
+
+MonoVertexStatus
+ |
+
++ +(Optional) + | + +
+ +(Appears on: +MonoVertexSpec) +
+ ++ +
+ +| + +Field + | + ++ +Description + | + +
|---|---|
+
+readBatchSize uint64
+ |
+
+
+
+(Optional)
+ + +Read batch size from the source. + + + |
+
+
+
+readTimeout
+
+Kubernetes meta/v1.Duration
+ |
+
+
+
+(Optional)
+ + +Read timeout duration from the source. |
@@ -6944,9 +7990,9 @@
string alias)
+MonoVertexPhase (string alias)
(Appears on: -GSSAPI) +MonoVertexStatus)
-
- -KRB5AuthType describes the kerberos auth type -
- -(Appears on: -AbstractSink) +MonoVertex)
@@ -7007,11 +8048,12 @@
brokers \[\]string
+replicas int32
topic string
+source
+ Source
tls
-TLS
+sink
+ Sink
- -TLS user to configure TLS connection for kafka broker TLS.enable=true -default for TLS. -
-config string
+AbstractPodTemplate
+
+AbstractPodTemplate
+
+(Members of AbstractPodTemplate are embedded into this
+type.)
+
sasl
- SASL
+containerTemplate
+
+ContainerTemplate
-SASL user to configure SASL connection for kafka broker SASL.enable=true -default for SASL. +Container template for the main numa container.
- -(Appears on: -Source) -
- -- -
- -| - -Field - | - -- -Description - | - -
|---|---|
-brokers \[\]string
+volumes
+
+\[\]Kubernetes core/v1.Volume
|
+(Optional) |
-topic string
+limits
+
+MonoVertexLimits
|
+(Optional)
+ + +Limits define the limitations such as buffer read batch size for all the +vertices of a pipeline, will override pipeline level settings + + |
@@ -7156,11 +8171,18 @@
-consumerGroup string
+scale
+ Scale
|
+(Optional)
+ + +Settings for autoscaling + + |
@@ -7169,8 +8191,9 @@
-tls
-TLS
+initContainers
+
+\[\]Kubernetes core/v1.Container
|
@@ -7178,8 +8201,8 @@ (Optional)-TLS user to configure TLS connection for kafka broker TLS.enable=true -default for TLS. +List of customized init containers belonging to the pod. More info: +https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
@@ -7190,12 +8213,19 @@
-config string
+sidecars
+
+\[\]Kubernetes core/v1.Container
|
(Optional)
+ + +List of customized sidecar containers belonging to the pod. + + |
@@ -7204,8 +8234,9 @@
-sasl
- SASL
+daemonTemplate
+ DaemonTemplate
+
|
@@ -7213,8 +8244,7 @@ (Optional)-SASL user to configure SASL connection for kafka broker SASL.enable=true -default for SASL. +Template for the daemon service deployment. |
@@ -7225,15 +8255,15 @@
(Appears on: -PipelineSpec) +MonoVertex)
@@ -7266,15 +8296,15 @@
deleteGracePeriodSeconds int32
+Status
+ Status
-DeleteGracePeriodSeconds used to delete pipeline gracefully
+(Members of Status are embedded into this type.)
desiredPhase
- PipelinePhase
-
+phase
+
+MonoVertexPhase
- -DesiredPhase used to bring the pipeline from current phase to desired -phase -
-pauseGracePeriodSeconds int32
+replicas uint32
- -PauseGracePeriodSeconds used to pause pipeline gracefully -
-- -(Appears on: -AbstractSink) -
++
selector string
+string alias)
-
++
+reason string
+
+
+message string
+
| + |
-Field
-
+lastUpdated
+
+Kubernetes meta/v1.Time
+ |
-+ | -Description - + |
|---|---|---|---|
-annotations map\[string\]string
+lastScaledAt
+
+Kubernetes meta/v1.Time
|
@@ -7412,7 +8412,7 @@
labels map\[string\]string
+observedGeneration int64
(Appears on: -AbstractVertex) +AbstractVertex, +MonoVertexSpec) @@ -10342,7 +11343,8 @@ (Appears on: -AbstractVertex) +AbstractVertex, +MonoVertexSpec) @@ -10532,7 +11534,8 @@ (Appears on: -AbstractVertex) +AbstractVertex, +MonoVertexSpec) @@ -10701,6 +11704,7 @@
(Appears on:
InterStepBufferServiceStatus,
+MonoVertexStatus,
PipelineStatus,
VertexStatus)
diff --git a/search/search_index.json b/search/search_index.json
index 75ed11e249..6c6a7fa11f 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Numaflow \u00b6 Welcome to Numaflow! A Kubernetes-native, serverless platform for running scalable and reliable event-driven applications. Numaflow decouples event sources and sinks from the processing logic, allowing each component to independently auto-scale based on demand. With out-of-the-box sources and sinks, and built-in observability, developers can focus on their processing logic without worrying about event consumption, writing boilerplate code, or operational complexities. Each step of the pipeline can be written in any programming language, offering unparalleled flexibility in using the best programming language for each step and ease of using the languages you are most familiar with. Use Cases \u00b6 Event driven applications: Process events as they happen, e.g., updating inventory and sending customer notifications in e-commerce. Real time analytics: Analyze data instantly, e.g., social media analytics, observability data processing. Inference on streaming data: Perform real-time predictions, e.g., anomaly detection. Workflows running in a streaming manner. Learn more in our User Guide . Key Features \u00b6 Kubernetes-native: If you know Kubernetes, you already know how to use Numaflow. Serverless: Focus on your code and let the system scale up and down based on demand. Language agnostic: Use your favorite programming language. Exactly-Once semantics: No input element is duplicated or lost even as pods are rescheduled or restarted. Auto-scaling with back-pressure: Each vertex automatically scales from zero to whatever is needed. Data Integrity Guarantees \u00b6 Minimally provide at-least-once semantics Provide exactly-once semantics for unbounded and near real-time data sources Preserving order is not required Roadmap \u00b6 Map Streaming (1.3) Demo \u00b6 Getting Started \u00b6 For set-up information and running your first Numaflow pipeline, please see our getting started guide .","title":"Home"},{"location":"#numaflow","text":"Welcome to Numaflow! A Kubernetes-native, serverless platform for running scalable and reliable event-driven applications. Numaflow decouples event sources and sinks from the processing logic, allowing each component to independently auto-scale based on demand. With out-of-the-box sources and sinks, and built-in observability, developers can focus on their processing logic without worrying about event consumption, writing boilerplate code, or operational complexities. Each step of the pipeline can be written in any programming language, offering unparalleled flexibility in using the best programming language for each step and ease of using the languages you are most familiar with.","title":"Numaflow"},{"location":"#use-cases","text":"Event driven applications: Process events as they happen, e.g., updating inventory and sending customer notifications in e-commerce. Real time analytics: Analyze data instantly, e.g., social media analytics, observability data processing. Inference on streaming data: Perform real-time predictions, e.g., anomaly detection. Workflows running in a streaming manner. Learn more in our User Guide .","title":"Use Cases"},{"location":"#key-features","text":"Kubernetes-native: If you know Kubernetes, you already know how to use Numaflow. Serverless: Focus on your code and let the system scale up and down based on demand. Language agnostic: Use your favorite programming language. Exactly-Once semantics: No input element is duplicated or lost even as pods are rescheduled or restarted. Auto-scaling with back-pressure: Each vertex automatically scales from zero to whatever is needed.","title":"Key Features"},{"location":"#data-integrity-guarantees","text":"Minimally provide at-least-once semantics Provide exactly-once semantics for unbounded and near real-time data sources Preserving order is not required","title":"Data Integrity Guarantees"},{"location":"#roadmap","text":"Map Streaming (1.3)","title":"Roadmap"},{"location":"#demo","text":"","title":"Demo"},{"location":"#getting-started","text":"For set-up information and running your first Numaflow pipeline, please see our getting started guide .","title":"Getting Started"},{"location":"APIs/","text":"Packages: numaflow.numaproj.io/v1alpha1 numaflow.numaproj.io/v1alpha1 Resource Types: AbstractPodTemplate ( Appears on: AbstractVertex , DaemonTemplate , JetStreamBufferService , JobTemplate , NativeRedis , SideInputsManagerTemplate , VertexTemplate ) AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on. Field Description metadata Metadata (Optional) Metadata sets the pods\u2019s metadata, i.e. annotations and labels nodeSelector map\\[string\\]string (Optional) NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node\u2019s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ tolerations \\[\\]Kubernetes core/v1.Toleration (Optional) If specified, the pod\u2019s tolerations. securityContext Kubernetes core/v1.PodSecurityContext (Optional) SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. imagePullSecrets \\[\\]Kubernetes core/v1.LocalObjectReference (Optional) ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod priorityClassName string (Optional) If specified, indicates the Redis pod\u2019s priority. \u201csystem-node-critical\u201d and \u201csystem-cluster-critical\u201d are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ priority int32 (Optional) The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ affinity Kubernetes core/v1.Affinity (Optional) The pod\u2019s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ serviceAccountName string (Optional) ServiceAccountName applied to the pod runtimeClassName string (Optional) RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the \u201clegacy\u201d RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class automountServiceAccountToken bool (Optional) AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. dnsPolicy Kubernetes core/v1.DNSPolicy (Optional) Set DNS policy for the pod. Defaults to \u201cClusterFirst\u201d. Valid values are \u2018ClusterFirstWithHostNet\u2019, \u2018ClusterFirst\u2019, \u2018Default\u2019 or \u2018None\u2019. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to \u2018ClusterFirstWithHostNet\u2019. dnsConfig Kubernetes core/v1.PodDNSConfig (Optional) Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. AbstractSink ( Appears on: Sink ) Field Description log Log (Optional) Log sink is used to write the data to the log. kafka KafkaSink (Optional) Kafka sink is used to write the data to the Kafka. blackhole Blackhole (Optional) Blackhole sink is used to write the data to the blackhole sink, which is a sink that discards all the data written to it. udsink UDSink (Optional) UDSink sink is used to write the data to the user-defined sink. AbstractVertex ( Appears on: PipelineSpec , VertexSpec ) Field Description name string source Source (Optional) sink Sink (Optional) udf UDF (Optional) containerTemplate ContainerTemplate (Optional) Container template for the main numa container. initContainerTemplate ContainerTemplate (Optional) Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user. AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) volumes \\[\\]Kubernetes core/v1.Volume (Optional) limits VertexLimits (Optional) Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings scale Scale (Optional) Settings for autoscaling initContainers \\[\\]Kubernetes core/v1.Container (Optional) List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ sidecars \\[\\]Kubernetes core/v1.Container (Optional) List of customized sidecar containers belonging to the pod. partitions int32 (Optional) Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only. sideInputs \\[\\]string (Optional) Names of the side inputs used in this vertex. sideInputsContainerTemplate ContainerTemplate (Optional) Container template for the side inputs watcher container. Authorization ( Appears on: HTTPSource , ServingSource ) Field Description token Kubernetes core/v1.SecretKeySelector (Optional) A secret selector which contains bearer token To use this, the client needs to add \u201cAuthorization: Bearer \u201d in the header BasicAuth ( Appears on: NatsAuth ) BasicAuth represents the basic authentication approach which contains a user name and a password. Field Description user Kubernetes core/v1.SecretKeySelector (Optional) Secret for auth user password Kubernetes core/v1.SecretKeySelector (Optional) Secret for auth password Blackhole ( Appears on: AbstractSink ) Blackhole is a sink to emulate /dev/null BufferFullWritingStrategy ( string alias) ( Appears on: Edge ) BufferServiceConfig ( Appears on: InterStepBufferServiceStatus ) Field Description redis RedisConfig jetstream JetStreamConfig CombinedEdge ( Appears on: VertexSpec ) CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It\u2019s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc. Field Description Edge Edge (Members of Edge are embedded into this type.) fromVertexType VertexType From vertex type. fromVertexPartitionCount int32 (Optional) The number of partitions of the from vertex, if not provided, the default value is set to \u201c1\u201d. fromVertexLimits VertexLimits (Optional) toVertexType VertexType To vertex type. toVertexPartitionCount int32 (Optional) The number of partitions of the to vertex, if not provided, the default value is set to \u201c1\u201d. toVertexLimits VertexLimits (Optional) ConditionType ( string alias) ConditionType is a valid value of Condition.Type Container ( Appears on: SideInput , UDF , UDSink , UDSource , UDTransformer ) Container is used to define the container properties for user-defined functions, sinks, etc. Field Description image string (Optional) command \\[\\]string (Optional) args \\[\\]string (Optional) env \\[\\]Kubernetes core/v1.EnvVar (Optional) envFrom \\[\\]Kubernetes core/v1.EnvFromSource (Optional) volumeMounts \\[\\]Kubernetes core/v1.VolumeMount (Optional) resources Kubernetes core/v1.ResourceRequirements (Optional) securityContext Kubernetes core/v1.SecurityContext (Optional) imagePullPolicy Kubernetes core/v1.PullPolicy (Optional) ContainerTemplate ( Appears on: AbstractVertex , DaemonTemplate , JetStreamBufferService , JobTemplate , NativeRedis , SideInputsManagerTemplate , VertexTemplate ) ContainerTemplate defines customized spec for a container Field Description resources Kubernetes core/v1.ResourceRequirements (Optional) imagePullPolicy Kubernetes core/v1.PullPolicy (Optional) securityContext Kubernetes core/v1.SecurityContext (Optional) env \\[\\]Kubernetes core/v1.EnvVar (Optional) envFrom \\[\\]Kubernetes core/v1.EnvFromSource (Optional) DaemonTemplate ( Appears on: Templates ) Field Description AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) replicas int32 (Optional) Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller containerTemplate ContainerTemplate (Optional) initContainerTemplate ContainerTemplate (Optional) Edge ( Appears on: CombinedEdge , PipelineSpec ) Field Description from string to string conditions ForwardConditions (Optional) Conditional forwarding, only allowed when \u201cFrom\u201d is a Source or UDF. onFull BufferFullWritingStrategy (Optional) OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to \u201cretryUntilSuccess\u201d FixedWindow ( Appears on: Window ) FixedWindow describes a fixed window Field Description length Kubernetes meta/v1.Duration Length is the duration of the fixed window. streaming bool (Optional) Streaming should be set to true if the reduce udf is streaming. ForwardConditions ( Appears on: Edge ) Field Description tags TagConditions Tags used to specify tags for conditional forwarding Function ( Appears on: UDF ) Field Description name string args \\[\\]string (Optional) kwargs map\\[string\\]string (Optional) GSSAPI ( Appears on: SASL ) GSSAPI represents a SASL GSSAPI config Field Description serviceName string realm string usernameSecret Kubernetes core/v1.SecretKeySelector UsernameSecret refers to the secret that contains the username authType KRB5AuthType valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH passwordSecret Kubernetes core/v1.SecretKeySelector (Optional) PasswordSecret refers to the secret that contains the password keytabSecret Kubernetes core/v1.SecretKeySelector (Optional) KeytabSecret refers to the secret that contains the keytab kerberosConfigSecret Kubernetes core/v1.SecretKeySelector (Optional) KerberosConfigSecret refers to the secret that contains the kerberos config GeneratorSource ( Appears on: Source ) Field Description rpu int64 (Optional) duration Kubernetes meta/v1.Duration (Optional) msgSize int32 (Optional) Size of each generated message keyCount int32 (Optional) KeyCount is the number of unique keys in the payload value uint64 (Optional) Value is an optional uint64 value to be written in to the payload jitter Kubernetes meta/v1.Duration (Optional) Jitter is the jitter for the message generation, used to simulate out of order messages for example if the jitter is 10s, then the message\u2019s event time will be delayed by a random time between 0 and 10s which will result in the message being out of order by 0 to 10s valueBlob string (Optional) ValueBlob is an optional string which is the base64 encoding of direct payload to send. This is useful for attaching a GeneratorSource to a true pipeline to test load behavior with true messages without requiring additional work to generate messages through the external source if present, the Value and MsgSize fields will be ignored. GetDaemonDeploymentReq Field Description ISBSvcType ISBSvcType Image string PullPolicy Kubernetes core/v1.PullPolicy Env \\[\\]Kubernetes core/v1.EnvVar DefaultResources Kubernetes core/v1.ResourceRequirements GetJetStreamServiceSpecReq Field Description Labels map\\[string\\]string ClusterPort int32 ClientPort int32 MonitorPort int32 MetricsPort int32 GetJetStreamStatefulSetSpecReq Field Description ServiceName string Labels map\\[string\\]string NatsImage string MetricsExporterImage string ConfigReloaderImage string ClusterPort int32 ClientPort int32 MonitorPort int32 MetricsPort int32 ServerAuthSecretName string ServerEncryptionSecretName string ConfigMapName string PvcNameIfNeeded string StartCommand string DefaultResources Kubernetes core/v1.ResourceRequirements GetRedisServiceSpecReq Field Description Labels map\\[string\\]string RedisContainerPort int32 SentinelContainerPort int32 GetRedisStatefulSetSpecReq Field Description ServiceName string Labels map\\[string\\]string RedisImage string SentinelImage string MetricsExporterImage string InitContainerImage string RedisContainerPort int32 SentinelContainerPort int32 RedisMetricsContainerPort int32 CredentialSecretName string TLSEnabled bool PvcNameIfNeeded string ConfConfigMapName string ScriptsConfigMapName string HealthConfigMapName string DefaultResources Kubernetes core/v1.ResourceRequirements GetSideInputDeploymentReq Field Description ISBSvcType ISBSvcType Image string PullPolicy Kubernetes core/v1.PullPolicy Env \\[\\]Kubernetes core/v1.EnvVar DefaultResources Kubernetes core/v1.ResourceRequirements GetVertexPodSpecReq Field Description ISBSvcType ISBSvcType Image string PullPolicy Kubernetes core/v1.PullPolicy Env \\[\\]Kubernetes core/v1.EnvVar SideInputsStoreName string ServingSourceStreamName string PipelineSpec PipelineSpec DefaultResources Kubernetes core/v1.ResourceRequirements GroupBy ( Appears on: UDF ) GroupBy indicates it is a reducer UDF Field Description window Window Window describes the windowing strategy. keyed bool (Optional) allowedLateness Kubernetes meta/v1.Duration (Optional) AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness). storage PBQStorage Storage is used to define the PBQ storage for a reduce vertex. HTTPSource ( Appears on: Source ) Field Description auth Authorization (Optional) service bool (Optional) Whether to create a ClusterIP Service ISBSvcPhase ( string alias) ( Appears on: InterStepBufferServiceStatus ) ISBSvcType ( string alias) ( Appears on: GetDaemonDeploymentReq , GetSideInputDeploymentReq , GetVertexPodSpecReq , InterStepBufferServiceStatus ) IdleSource ( Appears on: Watermark ) Field Description threshold Kubernetes meta/v1.Duration Threshold is the duration after which a source is marked as Idle due to lack of data. Ex: If watermark found to be idle after the Threshold duration then the watermark is progressed by IncrementBy . stepInterval Kubernetes meta/v1.Duration (Optional) StepInterval is the duration between the subsequent increment of the watermark as long the source remains Idle. The default value is 0s which means that once we detect idle source, we will be incrementing the watermark by IncrementBy for time we detect that we source is empty (in other words, this will be a very frequent update). incrementBy Kubernetes meta/v1.Duration IncrementBy is the duration to be added to the current watermark to progress the watermark when source is idling. InterStepBufferService Field Description metadata Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field. spec InterStepBufferServiceSpec redis RedisBufferService jetstream JetStreamBufferService status InterStepBufferServiceStatus (Optional) InterStepBufferServiceSpec ( Appears on: InterStepBufferService ) Field Description redis RedisBufferService jetstream JetStreamBufferService InterStepBufferServiceStatus ( Appears on: InterStepBufferService ) Field Description Status Status (Members of Status are embedded into this type.) phase ISBSvcPhase message string config BufferServiceConfig type ISBSvcType observedGeneration int64 JetStreamBufferService ( Appears on: InterStepBufferServiceSpec ) Field Description version string JetStream version, such as \u201c2.7.1\u201d replicas int32 JetStream StatefulSet size containerTemplate ContainerTemplate (Optional) ContainerTemplate contains customized spec for NATS container reloaderContainerTemplate ContainerTemplate (Optional) ReloaderContainerTemplate contains customized spec for config reloader container metricsContainerTemplate ContainerTemplate (Optional) MetricsContainerTemplate contains customized spec for metrics container persistence PersistenceStrategy (Optional) AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) settings string (Optional) Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream . For limits, only \u201cmax_payload\u201d is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only \u201cmax_memory_store\u201d and \u201cmax_file_store\u201d are supported for configuration, do not set \u201cstore_dir\u201d as it has been hardcoded. startArgs \\[\\]string (Optional) Optional arguments to start nats-server. For example, \u201c-D\u201d to enable debugging output, \u201c-DV\u201d to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. bufferConfig string (Optional) Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, \u201cstream\u201d, \u201cconsumer\u201d, \u201cotBucket\u201d and \u201cprocBucket\u201d. Available fields under \u201cstream\u201d include \u201cretention\u201d (e.g. interest, limits, workerQueue), \u201cmaxMsgs\u201d, \u201cmaxAge\u201d (e.g. 72h), \u201creplicas\u201d (1, 3, 5), \u201cduplicates\u201d (e.g. 5m). Available fields under \u201cconsumer\u201d include \u201cackWait\u201d (e.g. 60s) Available fields under \u201cotBucket\u201d include \u201cmaxValueSize\u201d, \u201chistory\u201d, \u201cttl\u201d (e.g. 72h), \u201cmaxBytes\u201d, \u201creplicas\u201d (1, 3, 5). Available fields under \u201cprocBucket\u201d include \u201cmaxValueSize\u201d, \u201chistory\u201d, \u201cttl\u201d (e.g. 72h), \u201cmaxBytes\u201d, \u201creplicas\u201d (1, 3, 5). encryption bool (Optional) Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages. tls bool (Optional) Whether enable TLS, defaults to false Enabling TLS might impact the performance JetStreamConfig ( Appears on: BufferServiceConfig ) Field Description url string JetStream (NATS) URL auth NatsAuth streamConfig string (Optional) tlsEnabled bool TLS enabled or not JetStreamSource ( Appears on: Source ) Field Description url string URL to connect to NATS cluster, multiple urls could be separated by comma. stream string Stream represents the name of the stream. tls TLS (Optional) TLS configuration for the nats client. auth NatsAuth (Optional) Auth information JobTemplate ( Appears on: Templates ) Field Description AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) containerTemplate ContainerTemplate (Optional) ttlSecondsAfterFinished int32 (Optional) ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won\u2019t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 backoffLimit int32 (Optional) Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 KRB5AuthType ( string alias) ( Appears on: GSSAPI ) KRB5AuthType describes the kerberos auth type KafkaSink ( Appears on: AbstractSink ) Field Description brokers \\[\\]string topic string tls TLS (Optional) TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. config string (Optional) sasl SASL (Optional) SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. KafkaSource ( Appears on: Source ) Field Description brokers \\[\\]string topic string consumerGroup string tls TLS (Optional) TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. config string (Optional) sasl SASL (Optional) SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. Lifecycle ( Appears on: PipelineSpec ) Field Description deleteGracePeriodSeconds int32 (Optional) DeleteGracePeriodSeconds used to delete pipeline gracefully desiredPhase PipelinePhase (Optional) DesiredPhase used to bring the pipeline from current phase to desired phase pauseGracePeriodSeconds int32 (Optional) PauseGracePeriodSeconds used to pause pipeline gracefully Log ( Appears on: AbstractSink ) LogicOperator ( string alias) ( Appears on: TagConditions ) Metadata ( Appears on: AbstractPodTemplate ) Field Description annotations map\\[string\\]string labels map\\[string\\]string NativeRedis ( Appears on: RedisBufferService ) Field Description version string Redis version, such as \u201c6.0.16\u201d replicas int32 Redis StatefulSet size redisContainerTemplate ContainerTemplate (Optional) RedisContainerTemplate contains customized spec for Redis container sentinelContainerTemplate ContainerTemplate (Optional) SentinelContainerTemplate contains customized spec for Redis container metricsContainerTemplate ContainerTemplate (Optional) MetricsContainerTemplate contains customized spec for metrics container initContainerTemplate ContainerTemplate (Optional) persistence PersistenceStrategy (Optional) AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) settings RedisSettings (Optional) Redis configuration, if not specified, global settings in numaflow-controller-config will be used. NatsAuth ( Appears on: JetStreamConfig , JetStreamSource , NatsSource ) NatsAuth defines how to authenticate the nats access Field Description basic BasicAuth (Optional) Basic auth which contains a username and a password token Kubernetes core/v1.SecretKeySelector (Optional) Token auth nkey Kubernetes core/v1.SecretKeySelector (Optional) NKey auth NatsSource ( Appears on: Source ) Field Description url string URL to connect to NATS cluster, multiple urls could be separated by comma. subject string Subject holds the name of the subject onto which messages are published. queue string Queue is used for queue subscription. tls TLS (Optional) TLS configuration for the nats client. auth NatsAuth (Optional) Auth information NoStore ( Appears on: PBQStorage ) NoStore means there will be no persistence storage and there will be data loss during pod restarts. Use this option only if you do not care about correctness (e.g., approx statistics pipeline like sampling rate, etc.). PBQStorage ( Appears on: GroupBy ) PBQStorage defines the persistence configuration for a vertex. Field Description persistentVolumeClaim PersistenceStrategy (Optional) emptyDir Kubernetes core/v1.EmptyDirVolumeSource (Optional) no_store NoStore (Optional) PersistenceStrategy ( Appears on: JetStreamBufferService , NativeRedis , PBQStorage ) PersistenceStrategy defines the strategy of persistence Field Description storageClassName string (Optional) Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 accessMode Kubernetes core/v1.PersistentVolumeAccessMode (Optional) Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes volumeSize k8s.io/apimachinery/pkg/api/resource.Quantity Volume size, e.g. 50Gi Pipeline Field Description metadata Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field. spec PipelineSpec interStepBufferServiceName string (Optional) vertices \\[\\]AbstractVertex edges \\[\\]Edge Edges define the relationships between vertices lifecycle Lifecycle (Optional) Lifecycle define the Lifecycle properties limits PipelineLimits (Optional) Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex\u2019s settings watermark Watermark (Optional) Watermark enables watermark progression across the entire pipeline. templates Templates (Optional) Templates are used to customize additional kubernetes resources required for the Pipeline sideInputs \\[\\]SideInput (Optional) SideInputs defines the Side Inputs of a pipeline. status PipelineStatus (Optional) PipelineLimits ( Appears on: PipelineSpec ) Field Description readBatchSize uint64 (Optional) Read batch size for all the vertices in the pipeline, can be overridden by the vertex\u2019s limit settings. bufferMaxLength uint64 (Optional) BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits. bufferUsageLimit uint32 (Optional) BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits. readTimeout Kubernetes meta/v1.Duration (Optional) Read timeout for all the vertices in the pipeline, can be overridden by the vertex\u2019s limit settings PipelinePhase ( string alias) ( Appears on: Lifecycle , PipelineStatus ) PipelineSpec ( Appears on: GetVertexPodSpecReq , Pipeline ) Field Description interStepBufferServiceName string (Optional) vertices \\[\\]AbstractVertex edges \\[\\]Edge Edges define the relationships between vertices lifecycle Lifecycle (Optional) Lifecycle define the Lifecycle properties limits PipelineLimits (Optional) Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex\u2019s settings watermark Watermark (Optional) Watermark enables watermark progression across the entire pipeline. templates Templates (Optional) Templates are used to customize additional kubernetes resources required for the Pipeline sideInputs \\[\\]SideInput (Optional) SideInputs defines the Side Inputs of a pipeline. PipelineStatus ( Appears on: Pipeline ) Field Description Status Status (Members of Status are embedded into this type.) phase PipelinePhase message string lastUpdated Kubernetes meta/v1.Time vertexCount uint32 sourceCount uint32 sinkCount uint32 udfCount uint32 mapUDFCount uint32 reduceUDFCount uint32 observedGeneration int64 RedisBufferService ( Appears on: InterStepBufferServiceSpec ) Field Description native NativeRedis Native brings up a native Redis service external RedisConfig External holds an External Redis config RedisConfig ( Appears on: BufferServiceConfig , RedisBufferService ) Field Description url string (Optional) Redis URL sentinelUrl string (Optional) Sentinel URL, will be ignored if Redis URL is provided masterName string (Optional) Only required when Sentinel is used user string (Optional) Redis user password Kubernetes core/v1.SecretKeySelector (Optional) Redis password secret selector sentinelPassword Kubernetes core/v1.SecretKeySelector (Optional) Sentinel password secret selector RedisSettings ( Appears on: NativeRedis ) Field Description redis string (Optional) Redis settings shared by both master and slaves, will override the global settings from controller config master string (Optional) Special settings for Redis master node, will override the global settings from controller config replica string (Optional) Special settings for Redis replica nodes, will override the global settings from controller config sentinel string (Optional) Sentinel settings, will override the global settings from controller config SASL ( Appears on: KafkaSink , KafkaSource ) Field Description mechanism SASLType SASL mechanism to use gssapi GSSAPI (Optional) GSSAPI contains the kerberos config plain SASLPlain (Optional) SASLPlain contains the sasl plain config scramsha256 SASLPlain (Optional) SASLSCRAMSHA256 contains the sasl plain config scramsha512 SASLPlain (Optional) SASLSCRAMSHA512 contains the sasl plain config SASLPlain ( Appears on: SASL ) Field Description userSecret Kubernetes core/v1.SecretKeySelector UserSecret refers to the secret that contains the user passwordSecret Kubernetes core/v1.SecretKeySelector (Optional) PasswordSecret refers to the secret that contains the password handshake bool SASLType ( string alias) ( Appears on: SASL ) SASLType describes the SASL type Scale ( Appears on: AbstractVertex ) Scale defines the parameters for autoscaling. Field Description disabled bool (Optional) Whether to disable autoscaling. Set to \u201ctrue\u201d when using Kubernetes HPA or any other 3rd party autoscaling strategies. min int32 (Optional) Minimum replicas. max int32 (Optional) Maximum replicas. lookbackSeconds uint32 (Optional) Lookback seconds to calculate the average pending messages and processing rate. cooldownSeconds uint32 (Optional) Deprecated: Use scaleUpCooldownSeconds and scaleDownCooldownSeconds instead. Cooldown seconds after a scaling operation before another one. zeroReplicaSleepSeconds uint32 (Optional) After scaling down the source vertex to 0, sleep how many seconds before scaling the source vertex back up to peek. targetProcessingSeconds uint32 (Optional) TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It\u2019s only effective for source vertices. targetBufferAvailability uint32 (Optional) TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. replicasPerScale uint32 (Optional) ReplicasPerScale defines maximum replicas can be scaled up or down at once. The is use to prevent too aggressive scaling operations scaleUpCooldownSeconds uint32 (Optional) ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set. scaleDownCooldownSeconds uint32 (Optional) ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set. ServingSource ( Appears on: Source ) ServingSource is the HTTP endpoint for Numaflow. Field Description auth Authorization (Optional) service bool (Optional) Whether to create a ClusterIP Service msgIDHeaderKey string The header key from which the message id will be extracted store ServingStore Persistent store for the callbacks for serving and tracking ServingStore ( Appears on: ServingSource ) ServingStore to track and store data and metadata for tracking and serving. Field Description url string URL of the persistent store to write the callbacks ttl Kubernetes meta/v1.Duration (Optional) TTL for the data in the store and tracker SessionWindow ( Appears on: Window ) SessionWindow describes a session window Field Description timeout Kubernetes meta/v1.Duration Timeout is the duration of inactivity after which a session window closes. SideInput ( Appears on: PipelineSpec ) SideInput defines information of a Side Input Field Description name string container Container volumes \\[\\]Kubernetes core/v1.Volume (Optional) trigger SideInputTrigger SideInputTrigger ( Appears on: SideInput ) Field Description schedule string The schedule to trigger the retrievement of the side input data. It supports cron format, for example, \u201c0 30 \\* \\* \\* \\*\u201d. Or interval based format, such as \u201c@hourly\u201d, \u201c@every 1h30m\u201d, etc. timezone string (Optional) SideInputsManagerTemplate ( Appears on: Templates ) Field Description AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) containerTemplate ContainerTemplate (Optional) Template for the side inputs manager numa container initContainerTemplate ContainerTemplate (Optional) Template for the side inputs manager init container Sink ( Appears on: AbstractVertex ) Field Description AbstractSink AbstractSink (Members of AbstractSink are embedded into this type.) fallback AbstractSink (Optional) Fallback sink can be imagined as DLQ for primary Sink. The writes to Fallback sink will only be initiated if the ud-sink response field sets it. SlidingWindow ( Appears on: Window ) SlidingWindow describes a sliding window Field Description length Kubernetes meta/v1.Duration Length is the duration of the sliding window. slide Kubernetes meta/v1.Duration Slide is the slide parameter that controls the frequency at which the sliding window is created. streaming bool (Optional) Streaming should be set to true if the reduce udf is streaming. Source ( Appears on: AbstractVertex ) Field Description generator GeneratorSource (Optional) kafka KafkaSource (Optional) http HTTPSource (Optional) nats NatsSource (Optional) transformer UDTransformer (Optional) udsource UDSource (Optional) jetstream JetStreamSource (Optional) serving ServingSource (Optional) Status ( Appears on: InterStepBufferServiceStatus , PipelineStatus , VertexStatus ) Status is a common structure which can be used for Status field. Field Description conditions \\[\\]Kubernetes meta/v1.Condition (Optional) Conditions are the latest available observations of a resource\u2019s current state. TLS ( Appears on: JetStreamSource , KafkaSink , KafkaSource , NatsSource ) Field Description insecureSkipVerify bool (Optional) caCertSecret Kubernetes core/v1.SecretKeySelector (Optional) CACertSecret refers to the secret that contains the CA cert certSecret Kubernetes core/v1.SecretKeySelector (Optional) CertSecret refers to the secret that contains the cert keySecret Kubernetes core/v1.SecretKeySelector (Optional) KeySecret refers to the secret that contains the key TagConditions ( Appears on: ForwardConditions ) Field Description operator LogicOperator (Optional) Operator specifies the type of operation that should be used for conditional forwarding value could be \u201cand\u201d, \u201cor\u201d, \u201cnot\u201d values \\[\\]string Values tag values for conditional forwarding Templates ( Appears on: PipelineSpec ) Field Description daemon DaemonTemplate (Optional) DaemonTemplate is used to customize the Daemon Deployment. job JobTemplate (Optional) JobTemplate is used to customize Jobs. sideInputsManager SideInputsManagerTemplate (Optional) SideInputsManagerTemplate is used to customize the Side Inputs Manager. vertex VertexTemplate (Optional) VertexTemplate is used to customize the vertices of the pipeline. Transformer ( Appears on: UDTransformer ) Field Description name string args \\[\\]string (Optional) kwargs map\\[string\\]string (Optional) UDF ( Appears on: AbstractVertex ) Field Description container Container (Optional) builtin Function (Optional) groupBy GroupBy (Optional) UDSink ( Appears on: AbstractSink ) Field Description container Container UDSource ( Appears on: Source ) Field Description container Container UDTransformer ( Appears on: Source ) Field Description container Container (Optional) builtin Transformer (Optional) Vertex ( Appears on: VertexInstance ) Field Description metadata Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field. spec VertexSpec AbstractVertex AbstractVertex (Members of AbstractVertex are embedded into this type.) pipelineName string interStepBufferServiceName string (Optional) replicas int32 (Optional) fromEdges \\[\\]CombinedEdge (Optional) toEdges \\[\\]CombinedEdge (Optional) watermark Watermark (Optional) Watermark indicates watermark progression in the vertex, it\u2019s populated from the pipeline watermark settings. status VertexStatus (Optional) VertexInstance VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index. Field Description vertex Vertex hostname string replica int32 VertexLimits ( Appears on: AbstractVertex , CombinedEdge ) Field Description readBatchSize uint64 (Optional) Read batch size from the source or buffer. It overrides the settings from pipeline limits. readTimeout Kubernetes meta/v1.Duration (Optional) Read timeout duration from the source or buffer It overrides the settings from pipeline limits. bufferMaxLength uint64 (Optional) BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. bufferUsageLimit uint32 (Optional) BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. VertexPhase ( string alias) ( Appears on: VertexStatus ) VertexSpec ( Appears on: Vertex ) Field Description AbstractVertex AbstractVertex (Members of AbstractVertex are embedded into this type.) pipelineName string interStepBufferServiceName string (Optional) replicas int32 (Optional) fromEdges \\[\\]CombinedEdge (Optional) toEdges \\[\\]CombinedEdge (Optional) watermark Watermark (Optional) Watermark indicates watermark progression in the vertex, it\u2019s populated from the pipeline watermark settings. VertexStatus ( Appears on: Vertex ) Field Description Status Status (Members of Status are embedded into this type.) phase VertexPhase replicas uint32 selector string reason string message string lastScaledAt Kubernetes meta/v1.Time observedGeneration int64 VertexTemplate ( Appears on: Templates ) Field Description AbstractPodTemplate AbstractPodTemplate (Members of AbstractPodTemplate are embedded into this type.) (Optional) containerTemplate ContainerTemplate (Optional) Template for the vertex numa container initContainerTemplate ContainerTemplate (Optional) Template for the vertex init container VertexType ( string alias) ( Appears on: CombinedEdge ) Watermark ( Appears on: PipelineSpec , VertexSpec ) Field Description disabled bool (Optional) Disabled toggles the watermark propagation, defaults to false. maxDelay Kubernetes meta/v1.Duration (Optional) Maximum delay allowed for watermark calculation, defaults to \u201c0s\u201d, which means no delay. idleSource IdleSource (Optional) IdleSource defines the idle watermark properties, it could be configured in case source is idling. Window ( Appears on: GroupBy ) Window describes windowing strategy Field Description fixed FixedWindow (Optional) sliding SlidingWindow (Optional) session SessionWindow (Optional) Generated with gen-crd-api-reference-docs .","title":"APIs"},{"location":"quick-start/","text":"Quick Start \u00b6 In this page, we will guide you through the steps to: Install Numaflow. Create and run a simple pipeline. Create and run an advanced pipeline. Before you begin: prerequisites \u00b6 To try Numaflow, you will first need to setup using one of the following options to run container images: Docker Desktop podman Then use one of the following options to create a local Kubernete Cluster: Docker Desktop Kubernetes k3d kind minikube You will also need kubectl to manage the cluster. Follow these steps to install kubectl . In case you need a refresher, all the kubectl commands used in this quick start guide can be found in the kubectl Cheat Sheet . Installing Numaflow \u00b6 Once you have completed all the prerequisites, run the following command lines to install Numaflow and start the Inter-Step Buffer Service that handles communication between vertices. kubectl create ns numaflow-system kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/stable/config/install.yaml kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/0-isbsvc-jetstream.yaml Creating a simple pipeline \u00b6 As an example, we will create a simple pipeline that contains a source vertex to generate messages, a processing vertex that echos the messages, and a sink vertex that logs the messages. Run the command below to create a simple pipeline. kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/1-simple-pipeline.yaml To view a list of pipelines you've created, run: kubectl get pipeline # or \"pl\" as a short name This should create a response like the following, with AGE indicating the time elapsed since the creation of your simple pipeline. NAME PHASE MESSAGE VERTICES AGE simple-pipeline Running 3 9s To inspect the status of the pipeline, use kubectl get pods . Note that the pod names will be different from the sample response: # Wait for pods to be ready kubectl get pods NAME READY STATUS RESTARTS AGE isbsvc-default-js-0 3 /3 Running 0 19s isbsvc-default-js-1 3 /3 Running 0 19s isbsvc-default-js-2 3 /3 Running 0 19s simple-pipeline-daemon-78b798fb98-qf4t4 1 /1 Running 0 10s simple-pipeline-out-0-xc0pf 1 /1 Running 0 10s simple-pipeline-cat-0-kqrhy 2 /2 Running 0 10s simple-pipeline-in-0-rhpjm 1 /1 Running 0 11s Now you can watch the log for the output vertex. Run the command below and remember to replace xxxxx with the appropriate pod name above. kubectl logs -f simple-pipeline-out-0-xxxxx This should generate an output like the sample below: 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"VT+G+/W7Dhc=\" , \"Createdts\" :1661471977707552597 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"0TaH+/W7Dhc=\" , \"Createdts\" :1661471977707615953 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"EEGH+/W7Dhc=\" , \"Createdts\" :1661471977707618576 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"WESH+/W7Dhc=\" , \"Createdts\" :1661471977707619416 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"YEaH+/W7Dhc=\" , \"Createdts\" :1661471977707619936 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"qfomN/a7Dhc=\" , \"Createdts\" :1661471978707942057 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"aUcnN/a7Dhc=\" , \"Createdts\" :1661471978707961705 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"iUonN/a7Dhc=\" , \"Createdts\" :1661471978707962505 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"mkwnN/a7Dhc=\" , \"Createdts\" :1661471978707963034 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"jk4nN/a7Dhc=\" , \"Createdts\" :1661471978707963534 } Numaflow also comes with a built-in user interface. NOTE : Please install the metrics server if your local Kubernetes cluster does not bring it by default (e.g., Kind). You can install it by running the below command. kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml kubectl patch -n kube-system deployment metrics-server --type = json -p '[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/args/-\",\"value\":\"--kubelet-insecure-tls\"}]' To port forward the UI, run the following command. # Port forward the UI to https://localhost:8443/ kubectl -n numaflow-system port-forward deployment/numaflow-server 8443 :8443 This renders the following UI on https://localhost:8443/. The pipeline can be deleted by issuing the following command: kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/1-simple-pipeline.yaml Creating an advanced pipeline \u00b6 Now we will walk you through creating an advanced pipeline. In our example, this is called the even-odd pipeline, illustrated by the following diagram: There are five vertices in this example of an advanced pipeline. An HTTP source vertex which serves an HTTP endpoint to receive numbers as source data, a UDF vertex to tag the ingested numbers with the key even or odd , three Log sinks, one to print the even numbers, one to print the odd numbers, and the other one to print both the even and odd numbers. Run the following command to create the even-odd pipeline. kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml You may opt to view the list of pipelines you've created so far by running kubectl get pipeline . Otherwise, proceed to inspect the status of the pipeline, using kubectl get pods . # Wait for pods to be ready kubectl get pods NAME READY STATUS RESTARTS AGE even-odd-daemon-64d65c945d-vjs9f 1 /1 Running 0 5m3s even-odd-even-or-odd-0-pr4ze 2 /2 Running 0 30s even-odd-even-sink-0-unffo 1 /1 Running 0 22s even-odd-in-0-a7iyd 1 /1 Running 0 5m3s even-odd-number-sink-0-zmg2p 1 /1 Running 0 7s even-odd-odd-sink-0-2736r 1 /1 Running 0 15s isbsvc-default-js-0 3 /3 Running 0 10m isbsvc-default-js-1 3 /3 Running 0 10m isbsvc-default-js-2 3 /3 Running 0 10m Next, port-forward the HTTP endpoint, and make a POST request using curl . Remember to replace xxxxx with the appropriate pod names both here and in the next step. kubectl port-forward even-odd-in-0-xxxx 8444 :8443 # Post data to the HTTP endpoint curl -kq -X POST -d \"101\" https://localhost:8444/vertices/in curl -kq -X POST -d \"102\" https://localhost:8444/vertices/in curl -kq -X POST -d \"103\" https://localhost:8444/vertices/in curl -kq -X POST -d \"104\" https://localhost:8444/vertices/in Now you can watch the log for the even and odd vertices by running the commands below. # Watch the log for the even vertex kubectl logs -f even-odd-even-sink-0-xxxxx 2022 /09/07 22 :29:40 ( even-sink ) 102 2022 /09/07 22 :29:40 ( even-sink ) 104 # Watch the log for the odd vertex kubectl logs -f even-odd-odd-sink-0-xxxxx 2022 /09/07 22 :30:19 ( odd-sink ) 101 2022 /09/07 22 :30:19 ( odd-sink ) 103 View the UI for the advanced pipeline at https://localhost:8443/. The source code of the even-odd user-defined function can be found here . You also can replace the Log Sink with some other sinks like Kafka to forward the data to Kafka topics. The pipeline can be deleted by kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml A pipeline with reduce (aggregation) \u00b6 To set up an example pipeline with the Reduce UDF , see Reduce Examples . What's Next \u00b6 Try more examples in the examples directory. After exploring how Numaflow pipelines run, you can check what data Sources and Sinks Numaflow supports out of the box, or learn how to write User-defined Functions . Numaflow can also be paired with Numalogic, a collection of ML models and algorithms for real-time data analytics and AIOps including anomaly detection. Visit the Numalogic homepage for more information.","title":"Quick Start"},{"location":"quick-start/#quick-start","text":"In this page, we will guide you through the steps to: Install Numaflow. Create and run a simple pipeline. Create and run an advanced pipeline.","title":"Quick Start"},{"location":"quick-start/#before-you-begin-prerequisites","text":"To try Numaflow, you will first need to setup using one of the following options to run container images: Docker Desktop podman Then use one of the following options to create a local Kubernete Cluster: Docker Desktop Kubernetes k3d kind minikube You will also need kubectl to manage the cluster. Follow these steps to install kubectl . In case you need a refresher, all the kubectl commands used in this quick start guide can be found in the kubectl Cheat Sheet .","title":"Before you begin: prerequisites"},{"location":"quick-start/#installing-numaflow","text":"Once you have completed all the prerequisites, run the following command lines to install Numaflow and start the Inter-Step Buffer Service that handles communication between vertices. kubectl create ns numaflow-system kubectl apply -n numaflow-system -f https://raw.githubusercontent.com/numaproj/numaflow/stable/config/install.yaml kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/0-isbsvc-jetstream.yaml","title":"Installing Numaflow"},{"location":"quick-start/#creating-a-simple-pipeline","text":"As an example, we will create a simple pipeline that contains a source vertex to generate messages, a processing vertex that echos the messages, and a sink vertex that logs the messages. Run the command below to create a simple pipeline. kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/1-simple-pipeline.yaml To view a list of pipelines you've created, run: kubectl get pipeline # or \"pl\" as a short name This should create a response like the following, with AGE indicating the time elapsed since the creation of your simple pipeline. NAME PHASE MESSAGE VERTICES AGE simple-pipeline Running 3 9s To inspect the status of the pipeline, use kubectl get pods . Note that the pod names will be different from the sample response: # Wait for pods to be ready kubectl get pods NAME READY STATUS RESTARTS AGE isbsvc-default-js-0 3 /3 Running 0 19s isbsvc-default-js-1 3 /3 Running 0 19s isbsvc-default-js-2 3 /3 Running 0 19s simple-pipeline-daemon-78b798fb98-qf4t4 1 /1 Running 0 10s simple-pipeline-out-0-xc0pf 1 /1 Running 0 10s simple-pipeline-cat-0-kqrhy 2 /2 Running 0 10s simple-pipeline-in-0-rhpjm 1 /1 Running 0 11s Now you can watch the log for the output vertex. Run the command below and remember to replace xxxxx with the appropriate pod name above. kubectl logs -f simple-pipeline-out-0-xxxxx This should generate an output like the sample below: 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"VT+G+/W7Dhc=\" , \"Createdts\" :1661471977707552597 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"0TaH+/W7Dhc=\" , \"Createdts\" :1661471977707615953 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"EEGH+/W7Dhc=\" , \"Createdts\" :1661471977707618576 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"WESH+/W7Dhc=\" , \"Createdts\" :1661471977707619416 } 2022 /08/25 23 :59:38 ( out ) { \"Data\" : \"YEaH+/W7Dhc=\" , \"Createdts\" :1661471977707619936 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"qfomN/a7Dhc=\" , \"Createdts\" :1661471978707942057 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"aUcnN/a7Dhc=\" , \"Createdts\" :1661471978707961705 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"iUonN/a7Dhc=\" , \"Createdts\" :1661471978707962505 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"mkwnN/a7Dhc=\" , \"Createdts\" :1661471978707963034 } 2022 /08/25 23 :59:39 ( out ) { \"Data\" : \"jk4nN/a7Dhc=\" , \"Createdts\" :1661471978707963534 } Numaflow also comes with a built-in user interface. NOTE : Please install the metrics server if your local Kubernetes cluster does not bring it by default (e.g., Kind). You can install it by running the below command. kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml kubectl patch -n kube-system deployment metrics-server --type = json -p '[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/args/-\",\"value\":\"--kubelet-insecure-tls\"}]' To port forward the UI, run the following command. # Port forward the UI to https://localhost:8443/ kubectl -n numaflow-system port-forward deployment/numaflow-server 8443 :8443 This renders the following UI on https://localhost:8443/. The pipeline can be deleted by issuing the following command: kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/stable/examples/1-simple-pipeline.yaml","title":"Creating a simple pipeline"},{"location":"quick-start/#creating-an-advanced-pipeline","text":"Now we will walk you through creating an advanced pipeline. In our example, this is called the even-odd pipeline, illustrated by the following diagram: There are five vertices in this example of an advanced pipeline. An HTTP source vertex which serves an HTTP endpoint to receive numbers as source data, a UDF vertex to tag the ingested numbers with the key even or odd , three Log sinks, one to print the even numbers, one to print the odd numbers, and the other one to print both the even and odd numbers. Run the following command to create the even-odd pipeline. kubectl apply -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml You may opt to view the list of pipelines you've created so far by running kubectl get pipeline . Otherwise, proceed to inspect the status of the pipeline, using kubectl get pods . # Wait for pods to be ready kubectl get pods NAME READY STATUS RESTARTS AGE even-odd-daemon-64d65c945d-vjs9f 1 /1 Running 0 5m3s even-odd-even-or-odd-0-pr4ze 2 /2 Running 0 30s even-odd-even-sink-0-unffo 1 /1 Running 0 22s even-odd-in-0-a7iyd 1 /1 Running 0 5m3s even-odd-number-sink-0-zmg2p 1 /1 Running 0 7s even-odd-odd-sink-0-2736r 1 /1 Running 0 15s isbsvc-default-js-0 3 /3 Running 0 10m isbsvc-default-js-1 3 /3 Running 0 10m isbsvc-default-js-2 3 /3 Running 0 10m Next, port-forward the HTTP endpoint, and make a POST request using curl . Remember to replace xxxxx with the appropriate pod names both here and in the next step. kubectl port-forward even-odd-in-0-xxxx 8444 :8443 # Post data to the HTTP endpoint curl -kq -X POST -d \"101\" https://localhost:8444/vertices/in curl -kq -X POST -d \"102\" https://localhost:8444/vertices/in curl -kq -X POST -d \"103\" https://localhost:8444/vertices/in curl -kq -X POST -d \"104\" https://localhost:8444/vertices/in Now you can watch the log for the even and odd vertices by running the commands below. # Watch the log for the even vertex kubectl logs -f even-odd-even-sink-0-xxxxx 2022 /09/07 22 :29:40 ( even-sink ) 102 2022 /09/07 22 :29:40 ( even-sink ) 104 # Watch the log for the odd vertex kubectl logs -f even-odd-odd-sink-0-xxxxx 2022 /09/07 22 :30:19 ( odd-sink ) 101 2022 /09/07 22 :30:19 ( odd-sink ) 103 View the UI for the advanced pipeline at https://localhost:8443/. The source code of the even-odd user-defined function can be found here . You also can replace the Log Sink with some other sinks like Kafka to forward the data to Kafka topics. The pipeline can be deleted by kubectl delete -f https://raw.githubusercontent.com/numaproj/numaflow/main/examples/2-even-odd-pipeline.yaml","title":"Creating an advanced pipeline"},{"location":"quick-start/#a-pipeline-with-reduce-aggregation","text":"To set up an example pipeline with the Reduce UDF , see Reduce Examples .","title":"A pipeline with reduce (aggregation)"},{"location":"quick-start/#whats-next","text":"Try more examples in the examples directory. After exploring how Numaflow pipelines run, you can check what data Sources and Sinks Numaflow supports out of the box, or learn how to write User-defined Functions . Numaflow can also be paired with Numalogic, a collection of ML models and algorithms for real-time data analytics and AIOps including anomaly detection. Visit the Numalogic homepage for more information.","title":"What's Next"},{"location":"core-concepts/inter-step-buffer-service/","text":"Inter-Step Buffer Service \u00b6 Inter-Step Buffer Service is the service to provide Inter-Step Buffers . An Inter-Step Buffer Service is described by a Custom Resource . It is required to be existing in a namespace before Pipeline objects are created. A sample InterStepBufferService with JetStream implementation looks like below. apiVersion : numaflow.numaproj.io/v1alpha1 kind : InterStepBufferService metadata : name : default spec : jetstream : version : latest # Do NOT use \"latest\" but a specific version in your real deployment InterStepBufferService is a namespaced object. It can be used by all the Pipelines in the same namespace. By default, Pipeline objects look for an InterStepBufferService named default , so a common practice is to create an InterStepBufferService with the name default . If you give the InterStepBufferService a name other than default , then you need to give the same name in the Pipeline spec. apiVersion : numaflow.numaproj.io/v1alpha1 kind : Pipeline metadata : name : my-pipeline spec : # Optional, if not specified, defaults to \"default\" interStepBufferServiceName : different-name To query Inter-Step Buffer Service objects with kubectl : kubectl get isbsvc JetStream \u00b6 JetStream is one of the supported Inter-Step Buffer Service implementations. A keyword jetstream under spec means a JetStream cluster will be created in the namespace. Version \u00b6 Property spec.jetstream.version is required for a JetStream InterStepBufferService . Supported versions can be found from the ConfigMap numaflow-controller-config in the control plane namespace. Note The version latest in the ConfigMap should only be used for testing purpose. It's recommended that you always use a fixed version in your real workload. Replicas \u00b6 An optional property spec.jetstream.replicas (defaults to 3) can be specified, which gives the total number of nodes. Persistence \u00b6 Following example shows a JetStream InterStepBufferService with persistence. apiVersion : numaflow.numaproj.io/v1alpha1 kind : InterStepBufferService metadata : name : default spec : jetstream : version : latest # Do NOT use \"latest\" but a specific version in your real deployment persistence : storageClassName : standard # Optional, will use K8s cluster default storage class if not specified accessMode : ReadWriteOnce # Optional, defaults to ReadWriteOnce volumeSize : 10Gi # Optional, defaults to 20Gi JetStream Settings \u00b6 There are 2 places to configure JetStream settings: ConfigMap numaflow-controller-config in the control plane namespace. This is the default configuration for all the JetStream InterStepBufferService created in the Kubernetes cluster. Property spec.jetstream.settings in an InterStepBufferService object. This optional property can be used to override the default configuration defined in the ConfigMap numaflow-controller-config . A sample JetStream configuration: # https://docs.nats.io/running-a-nats-service/configuration#limits # Only \"max_payload\" is supported for configuration in this section. # Max payload size in bytes, defaults to 1 MB. It is not recommended to use values over 8MB but max_payload can be set up to 64MB. max_payload: 1048576 # # https://docs.nats.io/running-a-nats-service/configuration#jetstream # Only configure \"max_memory_store\" or \"max_file_store\" in this section, do not set \"store_dir\" as it has been hardcoded. # # e.g. 1G. -1 means no limit, up to 75% of available memory. This only take effect for streams created using memory storage. max_memory_store: -1 # e.g. 20G. -1 means no limit, Up to 1TB if available max_file_store: 1TB Buffer Configuration \u00b6 For the Inter-Step Buffers created in JetStream ISB Service, there are 2 places to configure the default properties. ConfigMap numaflow-controller-config in the control plane namespace. This is the place to configure the default properties for the streams and consumers created in all the Jet Stream ISB - Services in the Kubernetes cluster. Field spec.jetstream.bufferConfig in an InterStepBufferService object. This optional field can be used to customize the stream and consumer properties of that particular InterStepBufferService , - and the configuration will be merged into the default one from the ConfigMap numaflow-controller-config . For example, - if you only want to change maxMsgs for created streams, then you only need to give stream.maxMsgs in the field, all - the rest config will still go with the default values in the control plane ConfigMap. Both these 2 places expect a YAML format configuration like below: bufferConfig : | # The properties of the buffers (streams) to be created in this JetStream service stream: # 0: Limits, 1: Interest, 2: WorkQueue retention: 1 maxMsgs: 30000 maxAge: 168h maxBytes: -1 # 0: File, 1: Memory storage: 0 replicas: 3 duplicates: 60s # The consumer properties for the created streams consumer: ackWait: 60s maxAckPending: 20000 Note Changing the buffer configuration either in the control plane ConfigMap or in the InterStepBufferService object does NOT make any change to the buffers (streams) already existing. TLS \u00b6 TLS is optional to configure through spec.jetstream.tls: true . Enabling TLS will use a self signed CERT to encrypt the connection from Vertex Pods to JetStream service. By default TLS is not enabled. Encryption At Rest \u00b6 Encryption at rest can be enabled by setting spec.jetstream.encryption: true . Be aware this will impact the performance a bit, see the detail at official doc . Once a JetStream ISB Service is created, toggling the encryption field will cause problem for the exiting messages, so if you want to change the value, please delete and recreate the ISB Service, and you also need to restart all the Vertex Pods to pick up the new credentials. Other Configuration \u00b6 Check here for the full spec of spec.jetstream . Redis \u00b6 NOTE Today when using Redis, the pipeline will stall if Redis has any data loss, especially during failovers. Redis is supported as an Inter-Step Buffer Service implementation. A keyword native under spec.redis means several Redis nodes with a Master-Replicas topology will be created in the namespace. We also support external redis. External Redis \u00b6 If you have a managed Redis, say in AWS, etc., we can make that Redis your ISB. All you need to do is provide the external Redis endpoint name. apiVersion : numaflow.numaproj.io/v1alpha1 kind : InterStepBufferService metadata : name : default spec : redis : external : url : \" |