OpenModel is the Schema for the open models API
+ + +| Field | Description |
|---|---|
apiVersionstring | llmaz.io/v1alpha1 |
kindstring | OpenModel |
spec [Required]+ ModelSpec
+ |
++ No description provided. | +
status [Required]+ ModelStatus
+ |
++ No description provided. | +
Flavor defines the accelerator requirements for a model and the necessary parameters +in autoscaling. Right now, it will be used in two places:
+| Field | Description |
|---|---|
name [Required]+ FlavorName
+ |
+
+ Name represents the flavor name, which will be used in model claim. + |
+
requests+ k8s.io/api/core/v1.ResourceList
+ |
+
+ Requests defines the required accelerators to serve the model for each replica, +like <nvidia.com/gpu: 8>. For multi-hosts cases, the requests here indicates +the resource requirements for each replica, usually equals to the TP size. +Not recommended to set the cpu and memory usage here: +
|
+
nodeSelector+ map[string]string
+ |
+
+ NodeSelector represents the node candidates for Pod placements, if a node doesn't +meet the nodeSelector, it will be filtered out in the resourceFungibility scheduler plugin. +If nodeSelector is empty, it means every node is a candidate. + |
+
params+ map[string]string
+ |
+
+ Params stores other useful parameters and will be consumed by cluster-autoscaler / Karpenter +for autoscaling or be defined as model parallelism parameters like TP or PP size. +E.g. with autoscaling, when scaling up nodes with 8x Nvidia A00, the parameter can be injected +with <INSTANCE-TYPE: p4d.24xlarge> for AWS. +Preset parameters: TP, PP, INSTANCE-TYPE. + |
+
InferenceConfig represents the inference configurations for the model.
+ + +| Field | Description |
|---|---|
flavors+ []Flavor
+ |
+
+ Flavors represents the accelerator requirements to serve the model. +Flavors are fungible following the priority represented by the slice order. + |
+
ModelHub represents the model registry for model downloads.
+ + +| Field | Description |
|---|---|
name+ string
+ |
+
+ Name refers to the model registry, such as huggingface. + |
+
modelID [Required]+ string
+ |
+
+ ModelID refers to the model identifier on model hub, +such as meta-llama/Meta-Llama-3-8B. + |
+
filename [Required]+ string
+ |
+
+ Filename refers to a specified model file rather than the whole repo. +This is helpful to download a specified GGUF model rather than downloading +the whole repo which includes all kinds of quantized models. +TODO: this is only supported with Huggingface, add support for ModelScope +in the near future. +Note: once filename is set, allowPatterns and ignorePatterns should be left unset. + |
+
revision+ string
+ |
+
+ Revision refers to a Git revision id which can be a branch name, a tag, or a commit hash. + |
+
allowPatterns+ []string
+ |
+
+ AllowPatterns refers to files matched with at least one pattern will be downloaded. + |
+
ignorePatterns+ []string
+ |
+
+ IgnorePatterns refers to files matched with any of the patterns will not be downloaded. + |
+
ModelRef refers to a created Model with it's role.
+ + +| Field | Description |
|---|---|
name [Required]+ ModelName
+ |
+
+ Name represents the model name. + |
+
role+ ModelRole
+ |
+
+ Role represents the model role once more than one model is required. +Such as a draft role, which means running with SpeculativeDecoding, +and default arguments for backend will be searched in backendRuntime +with the name of speculative-decoding. + |
+
ModelSource represents the source of the model. +Only one model source will be used.
+ + +| Field | Description |
|---|---|
modelHub+ ModelHub
+ |
+
+ ModelHub represents the model registry for model downloads. + |
+
uri+ URIProtocol
+ |
+
+ URI represents a various kinds of model sources following the uri protocol, protocol://, e.g. +
|
+
ModelSpec defines the desired state of Model
+ + +| Field | Description |
|---|---|
familyName [Required]+ ModelName
+ |
+
+ FamilyName represents the model type, like llama2, which will be auto injected
+to the labels with the key of |
+
source [Required]+ ModelSource
+ |
+
+ Source represents the source of the model, there're several ways to load +the model such as loading from huggingface, OCI registry, s3, host path and so on. + |
+
inferenceConfig [Required]+ InferenceConfig
+ |
+
+ InferenceConfig represents the inference configurations for the model. + |
+
ModelStatus defines the observed state of Model
+ + +| Field | Description |
|---|---|
conditions [Required]+ []k8s.io/apimachinery/pkg/apis/meta/v1.Condition
+ |
+
+ Conditions represents the Inference condition. + |
+
URIProtocol represents the protocol of the URI.
+ + + + \ No newline at end of file diff --git a/docs/reference/inference.v1alpha1.md b/docs/reference/inference.v1alpha1.md new file mode 100644 index 0000000..9eca04a --- /dev/null +++ b/docs/reference/inference.v1alpha1.md @@ -0,0 +1,792 @@ +--- +title: llmaz inference API +content_type: tool-reference +package: inference.llmaz.io/v1alpha1 +auto_generated: true +description: Generated API reference documentation for inference.llmaz.io/v1alpha1. +--- + + +## Resource Types + + +- [Playground](#inference-llmaz-io-v1alpha1-Playground) +- [Service](#inference-llmaz-io-v1alpha1-Service) + + +## `Playground` {#inference-llmaz-io-v1alpha1-Playground} + + +**Appears in:** + + + +Playground is the Schema for the playgrounds API
+ + +| Field | Description |
|---|---|
apiVersionstring | inference.llmaz.io/v1alpha1 |
kindstring | Playground |
spec [Required]+ PlaygroundSpec
+ |
++ No description provided. | +
status [Required]+ PlaygroundStatus
+ |
++ No description provided. | +
Service is the Schema for the services API
+ + +| Field | Description |
|---|---|
apiVersionstring | inference.llmaz.io/v1alpha1 |
kindstring | Service |
spec [Required]+ ServiceSpec
+ |
++ No description provided. | +
status [Required]+ ServiceStatus
+ |
++ No description provided. | +
BackendRuntime is the Schema for the backendRuntime API
+ + +| Field | Description |
|---|---|
spec [Required]+ BackendRuntimeSpec
+ |
++ No description provided. | +
status [Required]+ BackendRuntimeStatus
+ |
++ No description provided. | +
BackendRuntimeArg is the preset arguments for easy to use. +Three preset names are provided: default, speculative-decoding, model-parallelism, +do not change the name.
+ + +| Field | Description |
|---|---|
name+ string
+ |
+
+ Name represents the identifier of the backendRuntime argument. + |
+
flags [Required]+ []string
+ |
+
+ Flags represents all the preset configurations. +Flag around with {{ .CONFIG }} is a configuration waiting for render. + |
+
| Field | Description |
|---|---|
name+ BackendName
+ |
+
+ Name represents the inference backend under the hood, e.g. vLLM. + |
+
version+ string
+ |
+
+ Version represents the backend version if you want a different one +from the default version. + |
+
args [Required]+ BackendRuntimeArg
+ |
+
+ Args represents the specified arguments of the backendRuntime, +will be append to the backendRuntime.spec.Args. + |
+
envs+ []k8s.io/api/core/v1.EnvVar
+ |
+
+ Envs represents the environments set to the container. + |
+
resources [Required]+ ResourceRequirements
+ |
+
+ Resources represents the resource requirements for backend, like cpu/mem, +accelerators like GPU should not be defined here, but at the model flavors, +or the values here will be overwritten. + |
+
BackendRuntimeSpec defines the desired state of BackendRuntime
+ + +| Field | Description |
|---|---|
commands+ []string
+ |
+
+ Commands represents the default commands for the backendRuntime. + |
+
multiHostCommands+ MultiHostCommands
+ |
+
+ MultiHostCommands represents leader and worker commands for nodes with +different roles. + |
+
image [Required]+ string
+ |
+
+ Image represents the default image registry of the backendRuntime. +It will work together with version to make up a real image. + |
+
version [Required]+ string
+ |
+
+ Version represents the default version of the backendRuntime. +It will be appended to the image as a tag. + |
+
args [Required]+ []BackendRuntimeArg
+ |
+
+ Args represents the preset arguments of the backendRuntime. +They can be appended or overwritten by the Playground backendRuntimeConfig. + |
+
envs+ []k8s.io/api/core/v1.EnvVar
+ |
+
+ Envs represents the environments set to the container. + |
+
resources [Required]+ ResourceRequirements
+ |
+
+ Resources represents the resource requirements for backendRuntime, like cpu/mem, +accelerators like GPU should not be defined here, but at the model flavors, +or the values here will be overwritten. + |
+
livenessProbe+ k8s.io/api/core/v1.Probe
+ |
+
+ Periodic probe of backend liveness. +Backend will be restarted if the probe fails. +Cannot be updated. + |
+
readinessProbe+ k8s.io/api/core/v1.Probe
+ |
+
+ Periodic probe of backend readiness. +Backend will be removed from service endpoints if the probe fails. + |
+
startupProbe+ k8s.io/api/core/v1.Probe
+ |
+
+ StartupProbe indicates that the Backend has successfully initialized. +If specified, no other probes are executed until this completes successfully. +If this probe fails, the backend will be restarted, just as if the livenessProbe failed. +This can be used to provide different probe parameters at the beginning of a backend's lifecycle, +when it might take a long time to load data or warm a cache, than during steady-state operation. + |
+
scaleTriggers+ []NamedScaleTrigger
+ |
+
+ ScaleTriggers represents a set of triggers preset to be used by Playground. +If Playground not specify the scale trigger, the 0-index trigger will be used. + |
+
BackendRuntimeStatus defines the observed state of BackendRuntime
+ + +| Field | Description |
|---|---|
conditions [Required]+ []k8s.io/apimachinery/pkg/apis/meta/v1.Condition
+ |
+
+ Conditions represents the Inference condition. + |
+
| Field | Description |
|---|---|
minReplicas+ int32
+ |
+
+ MinReplicas indicates the minimum number of inference workloads based on the traffic. +Default to 1. +MinReplicas couldn't be 0 now, will support serverless in the future. + |
+
maxReplicas+ int32
+ |
+
+ MaxReplicas indicates the maximum number of inference workloads based on the traffic. +Default to nil means there's no limit for the instance number. + |
+
scaleTriggerRef+ ScaleTriggerRef
+ |
+
+ ScaleTriggerRef refers to the configured scaleTrigger in the backendRuntime +with tuned target value. +ScaleTriggerRef and ScaleTrigger can't be set at the same time. + |
+
scaleTrigger+ ScaleTrigger
+ |
+
+ ScaleTrigger defines a set of triggers to scale the workloads. +If not defined, trigger configured in backendRuntime will be used, +otherwise, trigger defined here will overwrite the defaulted ones. +ScaleTriggerRef and ScaleTrigger can't be set at the same time. + |
+
HPATrigger represents the configuration of the HorizontalPodAutoscaler. +Inspired by kubernetes.io/pkg/apis/autoscaling/types.go#HorizontalPodAutoscalerSpec. +Note: HPA component should be installed in prior.
+ + +| Field | Description |
|---|---|
metrics+ []k8s.io/api/autoscaling/v2.MetricSpec
+ |
+
+ metrics contains the specifications for which to use to calculate the +desired replica count (the maximum replica count across all metrics will +be used). The desired replica count is calculated multiplying the +ratio between the target value and the current value by the current +number of pods. Ergo, metrics used must decrease as the pod count is +increased, and vice-versa. See the individual metric source types for +more information about how each type of metric must respond. + |
+
behavior+ k8s.io/api/autoscaling/v2.HorizontalPodAutoscalerBehavior
+ |
+
+ behavior configures the scaling behavior of the target +in both Up and Down directions (scaleUp and scaleDown fields respectively). +If not set, the default HPAScalingRules for scale up and scale down are used. + |
+
MultiHostCommands represents leader & worker commands for multiple nodes scenarios.
+ + +| Field | Description |
|---|---|
leader [Required]+ []string
+ |
++ No description provided. | +
worker [Required]+ []string
+ |
++ No description provided. | +
NamedScaleTrigger defines the rules to scale the workloads. +Only one trigger cloud work at a time. The name is used to identify +the trigger in backendRuntime.
+ + +| Field | Description |
|---|---|
name [Required]+ string
+ |
+
+ Name represents the identifier of the scale trigger, e.g. some triggers defined for +latency sensitive workloads, some are defined for throughput sensitive workloads. + |
+
hpa [Required]+ HPATrigger
+ |
+
+ HPA represents the trigger configuration of the HorizontalPodAutoscaler. + |
+
PlaygroundSpec defines the desired state of Playground
+ + +| Field | Description |
|---|---|
replicas+ int32
+ |
+
+ Replicas represents the replica number of inference workloads. + |
+
modelClaim+ ModelClaim
+ |
+
+ ModelClaim represents claiming for one model, it's a simplified use case +of modelClaims. Most of the time, modelClaim is enough. +ModelClaim and modelClaims are exclusive configured. + |
+
modelClaims+ ModelClaims
+ |
+
+ ModelClaims represents claiming for multiple models for more complicated +use cases like speculative-decoding. +ModelClaims and modelClaim are exclusive configured. + |
+
backendRuntimeConfig+ BackendRuntimeConfig
+ |
+
+ BackendRuntimeConfig represents the inference backendRuntime configuration +under the hood, e.g. vLLM, which is the default backendRuntime. + |
+
elasticConfig+ ElasticConfig
+ |
+
+ ElasticConfig defines the configuration for elastic usage, +e.g. the max/min replicas. +Note: this requires to install the HPA first or will report error. + |
+
PlaygroundStatus defines the observed state of Playground
+ + +| Field | Description |
|---|---|
conditions [Required]+ []k8s.io/apimachinery/pkg/apis/meta/v1.Condition
+ |
+
+ Conditions represents the Inference condition. + |
+
replicas [Required]+ int32
+ |
+
+ Replicas track the replicas that have been created, whether ready or not. + |
+
selector [Required]+ string
+ |
+
+ Selector points to the string form of a label selector which will be used by HPA. + |
+
TODO: Do not support DRA yet, we can support that once needed.
+ + +| Field | Description |
|---|---|
limits+ k8s.io/api/core/v1.ResourceList
+ |
+
+ Limits describes the maximum amount of compute resources allowed. +More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + |
+
requests+ k8s.io/api/core/v1.ResourceList
+ |
+
+ Requests describes the minimum amount of compute resources required. +If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, +otherwise to an implementation-defined value. Requests cannot exceed Limits. +More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + |
+
ScaleTrigger defines the rules to scale the workloads. +Only one trigger cloud work at a time, mostly used in Playground.
+ + +| Field | Description |
|---|---|
hpa [Required]+ HPATrigger
+ |
+
+ HPA represents the trigger configuration of the HorizontalPodAutoscaler. + |
+
ScaleTriggerRef refers to the configured scaleTrigger in the backendRuntime.
+ + +| Field | Description |
|---|---|
name [Required]+ string
+ |
+
+ Name represents the scale trigger name defined in the backendRuntime.scaleTriggers. + |
+
ServiceSpec defines the desired state of Service. +Service controller will maintain multi-flavor of workloads with +different accelerators for cost or performance considerations.
+ + +| Field | Description |
|---|---|
modelClaims [Required]+ ModelClaims
+ |
+
+ ModelClaims represents multiple claims for different models. + |
+
workloadTemplate [Required]+ sigs.k8s.io/lws/api/leaderworkerset/v1.LeaderWorkerSetSpec
+ |
+
+ WorkloadTemplate defines the underlying workload layout and configuration. +Note: the LWS spec might be twisted with various LWS instances to support +accelerator fungibility or other cutting-edge researches. +LWS supports both single-host and multi-host scenarios, for single host +cases, only need to care about replicas, rolloutStrategy and workerTemplate. + |
+
ServiceStatus defines the observed state of Service
+ + +| Field | Description |
|---|---|
conditions [Required]+ []k8s.io/apimachinery/pkg/apis/meta/v1.Condition
+ |
+
+ Conditions represents the Inference condition. + |
+
replicas [Required]+ int32
+ |
+
+ Replicas track the replicas that have been created, whether ready or not. + |
+
selector [Required]+ string
+ |
+
+ Selector points to the string form of a label selector, the HPA will be +able to autoscale your resource. + |
+
{{ .FieldName }}
+ {{- if not .IsOptional }} [Required]{{- end -}}
+{{ .DisplayName }}
+ {{- else -}}
+{{ .DisplayName }}
+ {{- end -}}
+ {{- end }}
+{{ .FieldName }} are embedded into this type.)
+ {{- end }}
+ {{ if .GetComment -}}
+ {{ .GetComment }}
+ {{- else -}}
+ No description provided.
+ {{- end }}
+ {{- if and (eq (.GetType.Name.Name) "ObjectMeta") -}}
+Refer to the Kubernetes API documentation for the fields of the metadata field.
+ {{- end -}}
+| Field | Description |
|---|---|
apiVersionstring | {{- .APIGroup -}} |
kindstring | {{- .Name.Name -}} |