Packages:
Resource Types:
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
|
Field |
Description |
|---|---|
|
|
(Optional)
Metadata sets the pods’s metadata, i.e. annotations and labels |
|
|
(Optional)
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ |
|
|
(Optional)
If specified, the pod’s tolerations. |
|
|
(Optional)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. |
|
|
(Optional)
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod |
|
|
(Optional)
If specified, indicates the Redis pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
|
|
(Optional)
The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
|
|
(Optional)
The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ |
|
|
(Optional)
ServiceAccountName applied to the pod |
|
|
(Optional)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class |
|
|
(Optional)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. |
|
|
(Optional)
Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. |
|
|
(Optional)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. |
(Appears on: Sink)
|
Field |
Description |
|---|---|
|
|
(Optional)
Log sink is used to write the data to the log. |
|
|
(Optional)
Kafka sink is used to write the data to the Kafka. |
|
|
(Optional)
Blackhole sink is used to write the data to the blackhole sink, which is a sink that discards all the data written to it. |
|
|
(Optional)
UDSink sink is used to write the data to the user-defined sink. |
(Appears on: PipelineSpec, VertexSpec)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Container template for the main numa container. |
|
|
(Optional)
Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user. |
|
|
(Members of (Optional) |
|
|
(Optional) |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
|
|
(Optional)
Settings for autoscaling |
|
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
|
|
(Optional)
List of customized sidecar containers belonging to the pod. |
|
|
(Optional)
Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only. |
|
|
(Optional)
Names of the side inputs used in this vertex. |
|
|
(Optional)
Container template for the side inputs watcher container. |
(Appears on: HTTPSource, ServingSource)
|
Field |
Description |
|---|---|
|
(Optional)
A secret selector which contains bearer token To use this, the client needs to add “Authorization: Bearer ” in the header |
(Appears on: NatsAuth)
BasicAuth represents the basic authentication approach which contains a user name and a password.
|
Field |
Description |
|---|---|
|
(Optional)
Secret for auth user |
|
|
|
(Optional)
Secret for auth password |
(Appears on: AbstractSink)
Blackhole is a sink to emulate /dev/null
(Appears on: Edge)
(Appears on: InterStepBufferServiceStatus)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Appears on: VertexSpec)
CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It’s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc.
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
From vertex type. |
|
|
(Optional)
The number of partitions of the from vertex, if not provided, the default value is set to “1”. |
|
|
(Optional) |
|
|
To vertex type. |
|
|
(Optional)
The number of partitions of the to vertex, if not provided, the default value is set to “1”. |
|
|
(Optional) |
ConditionType is a valid value of Condition.Type
(Appears on: SideInput, UDF, UDSink, UDSource, UDTransformer)
Container is used to define the container properties for user-defined functions, sinks, etc.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) |
|
|
(Optional) |
|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
ContainerTemplate defines customized spec for a container
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
(Optional) |
|
|
(Optional) |
(Appears on: MonoVertexSpec, Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: CombinedEdge, PipelineSpec)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Optional)
Conditional forwarding, only allowed when “From” is a Source or UDF. |
|
|
(Optional)
OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess” |
(Appears on: Window)
FixedWindow describes a fixed window
|
Field |
Description |
|---|---|
|
|
Length is the duration of the fixed window. |
|
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
(Appears on: Edge)
|
Field |
Description |
|---|---|
|
|
Tags used to specify tags for conditional forwarding |
(Appears on: UDF)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: SASL)
GSSAPI represents a SASL GSSAPI config
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
UsernameSecret refers to the secret that contains the username |
|
|
valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH |
|
|
(Optional)
PasswordSecret refers to the secret that contains the password |
|
|
(Optional)
KeytabSecret refers to the secret that contains the keytab |
|
|
(Optional)
KerberosConfigSecret refers to the secret that contains the kerberos config |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Size of each generated message |
|
|
(Optional)
KeyCount is the number of unique keys in the payload |
|
|
(Optional)
Value is an optional uint64 value to be written in to the payload |
|
|
(Optional)
Jitter is the jitter for the message generation, used to simulate out of order messages for example if the jitter is 10s, then the message’s event time will be delayed by a random time between 0 and 10s which will result in the message being out of order by 0 to 10s |
|
|
(Optional)
ValueBlob is an optional string which is the base64 encoding of direct payload to send. This is useful for attaching a GeneratorSource to a true pipeline to test load behavior with true messages without requiring additional work to generate messages through the external source if present, the Value and MsgSize fields will be ignored. |
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: UDF)
GroupBy indicates it is a reducer UDF
|
Field |
Description |
|---|---|
|
|
Window describes the windowing strategy. |
|
|
(Optional) |
|
|
(Optional)
AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness). |
|
|
Storage is used to define the PBQ storage for a reduce vertex. |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
Whether to create a ClusterIP Service |
(Appears on: InterStepBufferServiceStatus)
(Appears on: GetDaemonDeploymentReq, GetSideInputDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)
(Appears on: Watermark)
|
Field |
Description |
|---|---|
|
|
Threshold is the duration after which a source is marked as Idle due to
lack of data. Ex: If watermark found to be idle after the Threshold
duration then the watermark is progressed by |
|
|
(Optional)
StepInterval is the duration between the subsequent increment of the
watermark as long the source remains Idle. The default value is 0s which
means that once we detect idle source, we will be incrementing the
watermark by |
|
|
IncrementBy is the duration to be added to the current watermark to progress the watermark when source is idling. |
|
Field |
Description |
||||
|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||
|
|
|||||
|
|
(Optional) |
(Appears on: InterStepBufferService)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Appears on: InterStepBufferService)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: InterStepBufferServiceSpec)
|
Field |
Description |
|---|---|
|
|
JetStream version, such as “2.7.1” |
|
|
JetStream StatefulSet size |
|
|
(Optional)
ContainerTemplate contains customized spec for NATS container |
|
|
(Optional)
ReloaderContainerTemplate contains customized spec for config reloader container |
|
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
|
|
(Optional) |
|
|
(Members of (Optional) |
|
|
(Optional)
Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream. For limits, only “max_payload” is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only “max_memory_store” and “max_file_store” are supported for configuration, do not set “store_dir” as it has been hardcoded. |
|
|
(Optional)
Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. |
|
|
(Optional)
Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). |
|
|
(Optional)
Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages. |
|
|
(Optional)
Whether enable TLS, defaults to false Enabling TLS might impact the performance |
(Appears on: BufferServiceConfig)
|
Field |
Description |
|---|---|
|
|
JetStream (NATS) URL |
|
|
|
|
|
(Optional) |
|
|
TLS enabled or not |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
|
|
Stream represents the name of the stream. |
|
|
(Optional)
TLS configuration for the nats client. |
|
|
(Optional)
Auth information |
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional) |
|
|
(Optional)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 |
|
|
(Optional)
Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 |
(Appears on: GSSAPI)
KRB5AuthType describes the kerberos auth type
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
|
|
(Optional) |
|
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
|
|
(Optional) |
|
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DeleteGracePeriodSeconds used to delete pipeline gracefully |
|
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
|
|
(Optional)
PauseGracePeriodSeconds used to pause pipeline gracefully |
(Appears on: AbstractSink)
(Appears on: TagConditions)
(Appears on: AbstractPodTemplate)
|
Field |
Description |
|---|---|
|
|
|
|
|
|
Field |
Description |
||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||||||||||||||||||||
|
|
|
||||||||||||||||||||||
|
|
(Optional) |
(Appears on: MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size from the source. |
|
|
(Optional)
Read timeout duration from the source. |
(Appears on: MonoVertexStatus)
(Appears on: MonoVertex)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
|
|
|
|
|
|
(Members of (Optional) |
|
|
(Optional)
Container template for the main numa container. |
|
|
(Optional) |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
|
|
(Optional)
Settings for autoscaling |
|
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
|
|
(Optional)
List of customized sidecar containers belonging to the pod. |
|
|
(Optional)
Template for the daemon service deployment. |
(Appears on: MonoVertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: RedisBufferService)
|
Field |
Description |
|---|---|
|
|
Redis version, such as “6.0.16” |
|
|
Redis StatefulSet size |
|
|
(Optional)
RedisContainerTemplate contains customized spec for Redis container |
|
|
(Optional)
SentinelContainerTemplate contains customized spec for Redis container |
|
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Members of (Optional) |
|
|
(Optional)
Redis configuration, if not specified, global settings in numaflow-controller-config will be used. |
(Appears on: JetStreamConfig, JetStreamSource, NatsSource)
NatsAuth defines how to authenticate the nats access
|
Field |
Description |
|---|---|
|
|
(Optional)
Basic auth which contains a username and a password |
|
(Optional)
Token auth |
|
|
(Optional)
NKey auth |
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
|
|
Subject holds the name of the subject onto which messages are published. |
|
|
Queue is used for queue subscription. |
|
|
(Optional)
TLS configuration for the nats client. |
|
|
(Optional)
Auth information |
(Appears on: PBQStorage)
NoStore means there will be no persistence storage and there will be data loss during pod restarts. Use this option only if you do not care about correctness (e.g., approx statistics pipeline like sampling rate, etc.).
(Appears on: GroupBy)
PBQStorage defines the persistence configuration for a vertex.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
(Optional) |
|
|
|
(Optional) |
(Appears on: JetStreamBufferService, NativeRedis, PBQStorage)
PersistenceStrategy defines the strategy of persistence
|
Field |
Description |
|---|---|
|
|
(Optional)
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 |
|
(Optional)
Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes |
|
|
|
Volume size, e.g. 50Gi |
|
Field |
Description |
||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||||||||||||||
|
|
|
||||||||||||||||
|
|
(Optional) |
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size for all the vertices in the pipeline, can be overridden by the vertex’s limit settings. |
|
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits. |
|
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits. |
|
|
(Optional)
Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
(Appears on: Lifecycle, PipelineStatus)
(Appears on: GetVertexPodSpecReq, Pipeline)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
|
|
|
Edges define the relationships between vertices |
|
|
(Optional)
Lifecycle define the Lifecycle properties |
|
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings |
|
|
(Optional)
Watermark enables watermark progression across the entire pipeline. |
|
|
(Optional)
Templates are used to customize additional kubernetes resources required for the Pipeline |
|
|
(Optional)
SideInputs defines the Side Inputs of a pipeline. |
(Appears on: Pipeline)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: InterStepBufferServiceSpec)
|
Field |
Description |
|---|---|
|
|
Native brings up a native Redis service |
|
|
External holds an External Redis config |
(Appears on: BufferServiceConfig, RedisBufferService)
|
Field |
Description |
|---|---|
|
|
(Optional)
Redis URL |
|
|
(Optional)
Sentinel URL, will be ignored if Redis URL is provided |
|
|
(Optional)
Only required when Sentinel is used |
|
|
(Optional)
Redis user |
|
|
(Optional)
Redis password secret selector |
|
|
(Optional)
Sentinel password secret selector |
(Appears on: NativeRedis)
|
Field |
Description |
|---|---|
|
|
(Optional)
Redis settings shared by both master and slaves, will override the global settings from controller config |
|
|
(Optional)
Special settings for Redis master node, will override the global settings from controller config |
|
|
(Optional)
Special settings for Redis replica nodes, will override the global settings from controller config |
|
|
(Optional)
Sentinel settings, will override the global settings from controller config |
(Appears on: KafkaSink, KafkaSource)
|
Field |
Description |
|---|---|
|
|
SASL mechanism to use |
|
|
(Optional)
GSSAPI contains the kerberos config |
|
|
(Optional)
SASLPlain contains the sasl plain config |
|
|
(Optional)
SASLSCRAMSHA256 contains the sasl plain config |
|
|
(Optional)
SASLSCRAMSHA512 contains the sasl plain config |
(Appears on: SASL)
|
Field |
Description |
|---|---|
|
|
UserSecret refers to the secret that contains the user |
|
|
(Optional)
PasswordSecret refers to the secret that contains the password |
|
|
(Appears on: SASL)
SASLType describes the SASL type
(Appears on: AbstractVertex, MonoVertexSpec)
Scale defines the parameters for autoscaling.
|
Field |
Description |
|---|---|
|
|
(Optional)
Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies. |
|
|
(Optional)
Minimum replicas. |
|
|
(Optional)
Maximum replicas. |
|
|
(Optional)
Lookback seconds to calculate the average pending messages and processing rate. |
|
|
(Optional)
Deprecated: Use scaleUpCooldownSeconds and scaleDownCooldownSeconds instead. Cooldown seconds after a scaling operation before another one. |
|
|
(Optional)
After scaling down the source vertex to 0, sleep how many seconds before scaling the source vertex back up to peek. |
|
|
(Optional)
TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices. |
|
|
(Optional)
TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. |
|
|
(Optional)
ReplicasPerScale defines maximum replicas can be scaled up or down at once. The is use to prevent too aggressive scaling operations |
|
|
(Optional)
ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set. |
|
|
(Optional)
ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set. |
(Appears on: Source)
ServingSource is the HTTP endpoint for Numaflow.
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
Whether to create a ClusterIP Service |
|
|
The header key from which the message id will be extracted |
|
|
Persistent store for the callbacks for serving and tracking |
(Appears on: ServingSource)
ServingStore to track and store data and metadata for tracking and serving.
|
Field |
Description |
|---|---|
|
|
URL of the persistent store to write the callbacks |
|
(Optional)
TTL for the data in the store and tracker |
(Appears on: Window)
SessionWindow describes a session window
|
Field |
Description |
|---|---|
|
|
Timeout is the duration of inactivity after which a session window closes. |
(Appears on: PipelineSpec)
SideInput defines information of a Side Input
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Optional) |
|
|
(Appears on: SideInput)
|
Field |
Description |
|---|---|
|
|
The schedule to trigger the retrievement of the side input data. It supports cron format, for example, “0 30 * * * *”. Or interval based format, such as “@hourly”, “@every 1h30m”, etc. |
|
|
(Optional) |
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Template for the side inputs manager numa container |
|
|
(Optional)
Template for the side inputs manager init container |
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
(Optional)
Fallback sink can be imagined as DLQ for primary Sink. The writes to Fallback sink will only be initiated if the ud-sink response field sets it. |
(Appears on: Window)
SlidingWindow describes a sliding window
|
Field |
Description |
|---|---|
|
|
Length is the duration of the sliding window. |
|
Slide is the slide parameter that controls the frequency at which the sliding window is created. |
|
|
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
(Appears on: AbstractVertex, MonoVertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: InterStepBufferServiceStatus, MonoVertexStatus, PipelineStatus, VertexStatus)
Status is a common structure which can be used for Status field.
|
Field |
Description |
|---|---|
|
|
(Optional)
Conditions are the latest available observations of a resource’s current state. |
(Appears on: JetStreamSource, KafkaSink, KafkaSource, NatsSource)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional)
CACertSecret refers to the secret that contains the CA cert |
|
|
(Optional)
CertSecret refers to the secret that contains the cert |
|
|
(Optional)
KeySecret refers to the secret that contains the key |
(Appears on: ForwardConditions)
|
Field |
Description |
|---|---|
|
|
(Optional)
Operator specifies the type of operation that should be used for conditional forwarding value could be “and”, “or”, “not” |
|
|
Values tag values for conditional forwarding |
(Appears on: PipelineSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
DaemonTemplate is used to customize the Daemon Deployment. |
|
|
(Optional)
JobTemplate is used to customize Jobs. |
|
|
(Optional)
SideInputsManagerTemplate is used to customize the Side Inputs Manager. |
|
|
(Optional)
VertexTemplate is used to customize the vertices of the pipeline. |
(Appears on: UDTransformer)
|
Field |
Description |
|---|---|
|
|
|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: AbstractVertex)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
(Appears on: AbstractSink)
|
Field |
Description |
|---|---|
|
|
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Appears on: Source)
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
(Appears on: VertexInstance)
|
Field |
Description |
||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
Refer to the Kubernetes API documentation for the fields of the
|
||||||||||||||
|
|
|
||||||||||||||
|
|
(Optional) |
VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.
|
Field |
Description |
|---|---|
|
|
|
|
|
|
|
|
(Appears on: AbstractVertex, CombinedEdge)
|
Field |
Description |
|---|---|
|
|
(Optional)
Read batch size from the source or buffer. It overrides the settings from pipeline limits. |
|
|
(Optional)
Read timeout duration from the source or buffer It overrides the settings from pipeline limits. |
|
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. |
|
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. |
(Appears on: VertexStatus)
(Appears on: Vertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional)
Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings. |
(Appears on: Vertex)
|
Field |
Description |
|---|---|
|
|
(Members of |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
(Appears on: Templates)
|
Field |
Description |
|---|---|
|
|
(Members of (Optional) |
|
|
(Optional)
Template for the vertex numa container |
|
|
(Optional)
Template for the vertex init container |
(Appears on: CombinedEdge)
(Appears on: PipelineSpec, VertexSpec)
|
Field |
Description |
|---|---|
|
|
(Optional)
Disabled toggles the watermark propagation, defaults to false. |
|
|
(Optional)
Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay. |
|
|
(Optional)
IdleSource defines the idle watermark properties, it could be configured in case source is idling. |
(Appears on: GroupBy)
Window describes windowing strategy
|
Field |
Description |
|---|---|
|
|
(Optional) |
|
|
(Optional) |
|
|
(Optional) |
Generated with gen-crd-api-reference-docs.