Skip to content

kserve/modelmesh-serving

Folders and files

NameName
Last commit message
Last commit date

Latest commit

9d5e122 · Dec 3, 2023
Dec 3, 2023
Dec 2, 2023
May 31, 2023
Oct 2, 2023
Oct 6, 2023
Nov 24, 2023
Oct 6, 2023
Aug 2, 2021
Jun 2, 2023
Aug 2, 2021
Dec 3, 2023
Aug 2, 2021
Oct 6, 2023
Sep 13, 2023
Nov 3, 2022
Nov 6, 2021
Dec 3, 2023
Dec 3, 2023
Aug 4, 2021
Aug 2, 2021
Aug 15, 2023
Sep 28, 2023
Oct 18, 2021
May 5, 2023
Nov 23, 2023
Nov 23, 2023
Apr 27, 2023

Repository files navigation

Build

ModelMesh Serving

ModelMesh Serving is the Controller for managing ModelMesh, a general-purpose model serving management/routing layer.

Getting Started

To quickly get started with ModelMesh Serving, check out the Quick Start Guide.

For help, please open an issue in this repository.

Components and their Repositories

ModelMesh Serving currently comprises components spread over a number of repositories. The supported versions for the latest release are documented here.

Architecture Image

Issues across all components are tracked centrally in this repo.

Core Components

Runtime Adapters

  • modelmesh-runtime-adapter - the containers which run in each model serving pod and act as an intermediary between ModelMesh and third-party model-server containers. Its build produces a single "multi-purpose" image which can be used as an adapter to work with each of the out-of-the-box supported model servers. It also incorporates the "puller" logic which is responsible for retrieving the models from storage before handing over to the respective adapter logic to load the model (and to delete after unloading). This image is also used for a container in the load/unload path of custom ServingRuntime Pods, as a "standalone" puller.

Model Serving runtimes

ModelMesh Serving provides out-of-the-box integration with the following model servers.

ServingRuntime custom resources can be used to add support for other existing or custom-built model servers, see the docs on implementing a custom Serving Runtime

Supplementary

  • KServe V2 REST Proxy - a reverse-proxy server which translates a RESTful HTTP API into gRPC. This allows sending inference requests using the KServe V2 REST Predict Protocol to ModelMesh models which currently only support the V2 gRPC Predict Protocol.

Libraries

These are helper Java libraries used by the ModelMesh component.

  • kv-utils - Useful KV store recipes abstracted over etcd and Zookeeper
  • litelinks-core - RPC/service discovery library based on Apache Thrift, used only for communications internal to ModelMesh.

Contributing

Please read our contributing guide for details on contributing.

Building Images

# Build develop image
make build.develop

# After building the develop image,  build the runtime image
make build