Testing Multi-Node NVLink support on GB200

[klueska]

Multi-Node NVLink support with host-installed driver

Prerequisites:

• Multi-Node cluster with at least 4 GPU nodes and Multi-Node NVLink connections between all GPUs on all nodes
• R565 (or later) driver is installed on all GPU nodes
• K8s 1.32 (or later) is up and running with DRA and CDI enabled on all nodes
• IMEX packages are installed on GPU nodes with systemd service either non-existent, or disabled and masked

$ systemctl disable --now nvidia-imex.service && systemctl mask nvidia-imex.service

Setup GPU Operator and install DRA driver

Fetch the latest GPU Operator and DRA driver helm charts (from main):

helm plugin install https://github.com/aslafy-z/helm-git

(GPU_OPERATOR_VERSION=main; \
    helm repo add --force-update gpu-operator-dev-repo \
        "git+https://github.com/NVIDIA/gpu-operator@deployments/gpu-operator?ref=${GPU_OPERATOR_VERSION}")

(DRA_DRIVER_VERSION=main; \
    helm repo add --force-update nvidia-dra-driver-gpu-dev-repo \
        "git+https://github.com/NVIDIA/k8s-dra-driver-gpu@deployments/helm/nvidia-dra-driver-gpu?ref=${DRA_DRIVER_VERSION}")
        
helm repo update

Install the the GPU Operator for use with HOST managed drivers:

helm install gpu-operator gpu-operator-dev-repo/gpu-operator \
    --devel \
    --create-namespace \
    --namespace gpu-operator \
    --set operator.repository=ghcr.io/nvidia \
    --set operator.version=6171a52d \
    --set validator.repository=ghcr.io/nvidia/gpu-operator \
    --set validator.version=6171a52d \
    --set toolkit.repository=ghcr.io/nvidia \
    --set toolkit.version=5d9b27f1-ubuntu20.04 \
    --set devicePlugin.repository=ghcr.io/nvidia \
    --set devicePlugin.version=831c31e6-ubi9 \
    --set cdi.enabled=true \
    --set driver.enabled=false

Install the DRA Driver with support for HOST managed drivers:

helm install nvidia-dra-driver-gpu nvidia-dra-driver-gpu-dev-repo/nvidia-dra-driver-gpu \
    --create-namespace \
    --namespace nvidia-dra-driver-gpu \
    --set image.repository=ghcr.io/nvidia/k8s-dra-driver-gpu \
    --set image.tag=d1fad7ed-ubi9 \
    --set nvidiaDriverRoot=/ \
    --set nvidiaCtkPath=/usr/local/nvidia/toolkit/nvidia-ctk \
    --set resources.gpus.enabled=false

Validate GPU Operator and DRA driver running

Validate that all GPU Operator components are running and in a Ready state:

$ kubectl get pod -n gpu-operator
NAME                                                         READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-824ts                                  1/1     Running     0          111s
gpu-feature-discovery-b5r5m                                  1/1     Running     0          111s
gpu-feature-discovery-rzzjm                                  1/1     Running     0          111s
gpu-feature-discovery-wx2qd                                  1/1     Running     0          111s
gpu-operator-759dd5976b-m2nw4                                1/1     Running     0          16m
gpu-operator-node-feature-discovery-gc-fc549bd94-czgsv       1/1     Running     0          16m
gpu-operator-node-feature-discovery-master-b4bb855b7-rjmrm   1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-2wh9x             1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-bhvpc             1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-llhrn             1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-p68cf             1/1     Running     0          16m
nvidia-container-toolkit-daemonset-9fm8m                     1/1     Running     0          15m
nvidia-container-toolkit-daemonset-ckp7k                     1/1     Running     0          15m
nvidia-container-toolkit-daemonset-r6xb7                     1/1     Running     0          15m
nvidia-container-toolkit-daemonset-tmvcb                     1/1     Running     0          15m
nvidia-cuda-validator-5glv9                                  0/1     Completed   0          15m
nvidia-cuda-validator-94vzp                                  0/1     Completed   0          15m
nvidia-cuda-validator-h88m6                                  0/1     Completed   0          15m
nvidia-cuda-validator-xsb6g                                  0/1     Completed   0          15m
nvidia-dcgm-exporter-l5g5g                                   1/1     Running     0          15m
nvidia-dcgm-exporter-s2x4s                                   1/1     Running     0          15m
nvidia-dcgm-exporter-tlwjw                                   1/1     Running     0          15m
nvidia-dcgm-exporter-zwn9q                                   1/1     Running     0          15m
nvidia-device-plugin-daemonset-hwkt5                         1/1     Running     0          15m
nvidia-device-plugin-daemonset-nkrj8                         1/1     Running     0          15m
nvidia-device-plugin-daemonset-pxn2b                         1/1     Running     0          15m
nvidia-device-plugin-daemonset-rpz27                         1/1     Running     0          15m
nvidia-mig-manager-9bpnq                                     1/1     Running     0          15m
nvidia-mig-manager-9vzrd                                     1/1     Running     0          15m
nvidia-mig-manager-mhdjn                                     1/1     Running     0          15m
nvidia-mig-manager-v65xc                                     1/1     Running     0          15m
nvidia-operator-validator-2gmf6                              1/1     Running     0          15m
nvidia-operator-validator-6ts67                              1/1     Running     0          15m
nvidia-operator-validator-qnwml                              1/1     Running     0          15m
nvidia-operator-validator-rk6xk                              1/1     Running     0          15m

Validate that the DRA driver components are running and in a Ready state:

$ kubectl get pod -n nvidia-dra-driver-gpu
NAME                                                           READY   STATUS    RESTARTS   AGE
nvidia-dra-driver-k8s-dra-driver-controller-67cb99d84b-5q7kj   1/1     Running   0          7m26s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-7kdg9          1/1     Running   0          7m27s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-bd6gn          1/1     Running   0          7m27s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-bzm6p          1/1     Running   0          7m26s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-xjm4p          1/1     Running   0          7m27s

Confirm that all GPU nodes are labeled with clique ids:

$ (echo -e "NODE\tLABEL\tCLIQUE"; kubectl get nodes -o json | \
    jq -r '.items[] | [.metadata.name, "nvidia.com/gpu.clique", .metadata.labels["nvidia.com/gpu.clique"]] | @tsv') | \
    column -t
NODE                  LABEL                  CLIQUE
sc-starwars-mab1-b00  nvidia.com/gpu.clique  1fbed3a8-bd74-4c83-afcb-cfb75ebc9304.1
sc-starwars-mab2-b00  nvidia.com/gpu.clique  1fbed3a8-bd74-4c83-afcb-cfb75ebc9304.1
sc-starwars-mab3-b00  nvidia.com/gpu.clique  1fbed3a8-bd74-4c83-afcb-cfb75ebc9304.1
sc-starwars-mab4-b00  nvidia.com/gpu.clique  1fbed3a8-bd74-4c83-afcb-cfb75ebc9304.1

Simple IMEX channel injection with IMEX daemon running

Run a simple test to validate IMEX daemons are started and IMEX channels are injected:

cat <<EOF > imex-channel-injection.yaml
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: imex-channel-injection
spec:
  numNodes: 1
  channel:
    resourceClaimTemplate:
      name: imex-channel-0
---
apiVersion: v1
kind: Pod
metadata:
  name: imex-channel-injection
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.clique
            operator: Exists
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["ls -la /dev/nvidia-caps-imex-channels; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: imex-channel-0
  resourceClaims:
  - name: imex-channel-0
    resourceClaimTemplateName: imex-channel-0
EOF

$ kubectl apply -f imex-channel-injection.yaml
computedomain.resource.nvidia.com/imex-channel-injection created
pod/imex-channel-injection created

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
imex-channel-injection   1/1     Running   0          3s

$ kubectl get pods -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain
NAME                                 READY   STATUS    RESTARTS   AGE
imex-channel-injection-6k9sx-ffgpf   1/1     Running   0          3s

$ kubectl logs imex-channel-injection
total 0
drwxr-xr-x 2 root root     60 Feb 19 10:43 .
drwxr-xr-x 6 root root    380 Feb 19 10:43 ..
crw-rw-rw- 1 root root 507, 0 Feb 19 10:43 channel0

$ kubectl logs -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain --tail=-1
IMEX Log initializing at: 2/19/2025 10:43:30.155
[Feb 19 2025 10:43:30] [INFO] [tid 35] IMEX version 570.00 is running with the following configuration options

[Feb 19 2025 10:43:30] [INFO] [tid 35] Logging level = 4

[Feb 19 2025 10:43:30] [INFO] [tid 35] Logging file name/path = /var/log/nvidia-imex.log

[Feb 19 2025 10:43:30] [INFO] [tid 35] Append to log file = 0

[Feb 19 2025 10:43:30] [INFO] [tid 35] Max Log file size = 1024 (MBs)

[Feb 19 2025 10:43:30] [INFO] [tid 35] Use Syslog file = 0

[Feb 19 2025 10:43:30] [INFO] [tid 35] IMEX Library communication bind interface =

[Feb 19 2025 10:43:30] [INFO] [tid 35] IMEX library communication bind port = 50000

[Feb 19 2025 10:43:30] [INFO] [tid 35] Identified this node as ID 0, using bind IP of '10.136.206.42', and network interface of enP2s2f1
[Feb 19 2025 10:43:30] [INFO] [tid 35] nvidia-imex persistence file /var/run/nvidia-imex/persist.dat does not exist.  Assuming no previous importers.
[Feb 19 2025 10:43:30] [INFO] [tid 35] NvGpu Library version matched with GPU Driver version
[Feb 19 2025 10:43:30] [INFO] [tid 59] Started processing of incoming messages.
[Feb 19 2025 10:43:30] [INFO] [tid 60] Started processing of incoming messages.
[Feb 19 2025 10:43:30] [INFO] [tid 61] Started processing of incoming messages.
[Feb 19 2025 10:43:30] [INFO] [tid 62] Started processing of incoming messages.
[Feb 19 2025 10:43:30] [INFO] [tid 35] Creating gRPC channels to all peers (nPeers = 1).
[Feb 19 2025 10:43:30] [INFO] [tid 35] IMEX_WAIT_FOR_QUORUM != FULL, continuing initialization without waiting for connections to all nodes.
[Feb 19 2025 10:43:30] [INFO] [tid 35] GPU event successfully subscribed

  $ kubectl delete -f imex-channel-injection.yaml
computedomain.resource.nvidia.com "imex-channel-injection" deleted
pod "imex-channel-injection" deleted

Multi-Node MPI test

Install the latest version of the MPI Operator:

kubectl create -f https://github.com/kubeflow/mpi-operator/releases/download/v0.6.0/mpi-operator.yaml

Run a multi-node nvbandwidth test requiring IMEX channels with MPI:

cat <<EOF > nvbandwidth-test-job.yaml
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: nvbandwidth-test-compute-domain
spec:
  numNodes: 4
  channel:
    resourceClaimTemplate:
      name: nvbandwidth-test-compute-domain-channel

---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nvbandwidth-test
spec:
  slotsPerWorker: 2
  launcherCreationPolicy: WaitForWorkersReady
  runPolicy:
    cleanPodPolicy: Running
  sshAuthMountPath: /home/mpiuser/.ssh
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        metadata:
          labels:
            nvbandwidth-test-replica: mpi-launcher
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
          containers:
          - image: ghcr.io/nvidia/k8s-samples:nvbandwidth-v0.7-8d103163
            name: mpi-launcher
            securityContext:
              runAsUser: 1000
            command:
            - mpirun
            args:
            - --bind-to
            - core
            - --map-by
            - ppr:2:node
            - -np
            - "8"
            - --report-bindings
            - -q
            - nvbandwidth
            - -t
            - multinode_device_to_device_memcpy_read_ce
    Worker:
      replicas: 4
      template:
        metadata:
          labels:
            nvbandwidth-test-replica: mpi-worker
        spec:
          affinity:
            podAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: nvbandwidth-test-replica
                    operator: In
                    values:
                    - mpi-worker
                topologyKey: nvidia.com/gpu.clique
          containers:
          - image: ghcr.io/nvidia/k8s-samples:nvbandwidth-v0.7-8d103163
            name: mpi-worker
            securityContext:
              runAsUser: 1000
            env:
            command:
            - /usr/sbin/sshd
            args:
            - -De
            - -f
            - /home/mpiuser/.sshd_config
            resources:
              limits:
                nvidia.com/gpu: 2
              claims:
              - name: compute-domain-channel
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: nvbandwidth-test-compute-domain-channel
EOF

$ kubectl apply -f nvbandwidth-test-job.yaml
computedomain.resource.nvidia.com/nvbandwidth-test-compute-domain created
mpijob.kubeflow.org/nvbandwidth-test created

$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
nvbandwidth-test-launcher-sqlwc   1/1     Running   0          8s
nvbandwidth-test-worker-0         1/1     Running   0          15s
nvbandwidth-test-worker-1         1/1     Running   0          15s
nvbandwidth-test-worker-2         1/1     Running   0          15s
nvbandwidth-test-worker-3         1/1     Running   0          14s

$ kubectl get pods -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain
NAME                                          READY   STATUS    RESTARTS   AGE
nvbandwidth-test-compute-domain-j8c7j-bkckf   1/1     Running   0          5s
nvbandwidth-test-compute-domain-j8c7j-d54ts   1/1     Running   0          5s
nvbandwidth-test-compute-domain-j8c7j-hw2sc   1/1     Running   0          5s
nvbandwidth-test-compute-domain-j8c7j-rhgx8   1/1     Running   0          5s

$ kubectl logs --tail=-1 -l job-name=nvbandwidth-test-launcher
Warning: Permanently added '[nvbandwidth-test-worker-0.nvbandwidth-test.default.svc]:2222' (ECDSA) to the list of known hosts.
Warning: Permanently added '[nvbandwidth-test-worker-2.nvbandwidth-test.default.svc]:2222' (ECDSA) to the list of known hosts.
Warning: Permanently added '[nvbandwidth-test-worker-1.nvbandwidth-test.default.svc]:2222' (ECDSA) to the list of known hosts.
Warning: Permanently added '[nvbandwidth-test-worker-3.nvbandwidth-test.default.svc]:2222' (ECDSA) to the list of known hosts.
[nvbandwidth-test-worker-2:00011] MCW rank 4 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-2:00011] MCW rank 5 bound to socket 0[core 1[hwt 0]]: [./B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-1:00011] MCW rank 2 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-1:00011] MCW rank 3 bound to socket 0[core 1[hwt 0]]: [./B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-0:00011] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-0:00011] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-3:00011] MCW rank 6 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-3:00011] MCW rank 7 bound to socket 0[core 1[hwt 0]]: [./B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
nvbandwidth Version: v0.7
Built from Git version: v0.7

MPI version: Open MPI v4.1.4, package: Debian OpenMPI, ident: 4.1.4, repo rev: v4.1.4, May 26, 2022
CUDA Runtime Version: 12080
CUDA Driver Version: 12080
Driver Version: 570.00

Process 0 (nvbandwidth-test-worker-0): device 0: NVIDIA GH200 96GB HBM3 (00000009:01:00)
Process 1 (nvbandwidth-test-worker-0): device 1: NVIDIA GH200 96GB HBM3 (00000019:01:00)
Process 2 (nvbandwidth-test-worker-1): device 0: NVIDIA GH200 96GB HBM3 (00000009:01:00)
Process 3 (nvbandwidth-test-worker-1): device 1: NVIDIA GH200 96GB HBM3 (00000019:01:00)
Process 4 (nvbandwidth-test-worker-2): device 0: NVIDIA GH200 96GB HBM3 (00000009:01:00)
Process 5 (nvbandwidth-test-worker-2): device 1: NVIDIA GH200 96GB HBM3 (00000019:01:00)
Process 6 (nvbandwidth-test-worker-3): device 0: NVIDIA GH200 96GB HBM3 (00000009:01:00)
Process 7 (nvbandwidth-test-worker-3): device 1: NVIDIA GH200 96GB HBM3 (00000019:01:00)

Running multinode_device_to_device_memcpy_read_ce.
memcpy CE GPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0       N/A    391.89    391.86    392.06    391.98    391.84    392.04    391.86
 1    391.89       N/A    391.66    392.09    391.91    391.86    391.89    391.81
 2    391.86    391.82       N/A    392.09    391.97    391.84    391.93    391.93
 3    391.95    391.73    391.91       N/A    392.02    391.89    391.98    391.93
 4    391.86    391.81    391.91    391.81       N/A    391.84    392.04    391.89
 5    391.97    391.61    391.98    391.81    391.81       N/A    391.91    391.81
 6    391.95    391.73    391.84    391.82    391.66    391.75       N/A    391.88
 7    392.02    391.75    391.93    391.81    391.79    391.75    391.73       N/A

SUM multinode_device_to_device_memcpy_read_ce 21944.96

NOTE: The reported results may not reflect the full capabilities of the platform.
Performance can vary with software drivers, hardware clocks, and system topology.

$ kubectl delete -f nvbandwidth-test-job.yaml
computedomain.resource.nvidia.com "nvbandwidth-test-compute-domain" deleted
mpijob.kubeflow.org "nvbandwidth-test" deleted

[liayan]

@klueska , how do you create the channel device btw, created by driver or imex ds here?

I only saw a validation logic in the container yaml file.

/dev/nvidia-caps-imex-channels/channel0

is imex-channel-0 the resource name or the real char device name?

[klueska]

It is created by the DRA driver at the point that it injects it into the container:
https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/cmd/compute-domain-kubelet-plugin/device_state.go#L348

[klueska]

imex-channel-0 is the name of the abstract device advertised by the DRA driver in its resource slice. Once allocated to a pod, the logic linked above is triggered to create it and eventually inject it into the pod requesting it.