Volcano on Volcano

User Guide

Tue, 03 Feb 2026 00:00:00 +0000

This section contains the User Guide for Volcano.

Scheduling Gates Queue Admission User Guide

Mon, 04 May 2026 00:00:00 +0000

Overview

This page describes how to enable and use the SchedulingGatesQueueAdmission feature to prevent cluster autoscalers (such as Cluster Autoscaler or Karpenter) from triggering unnecessary scale-ups when pods are waiting for Volcano queue capacity.

Problem

Volcano marks pods as Unschedulable for any allocation failure, whether it’s due to insufficient cluster resources (where autoscaling is appropriate) or queue capacity limits (where autoscaling is not needed). Cluster autoscalers cannot distinguish between these scenarios, causing unnecessary node scale-ups.

The problem is described in detail in the design document.

Solution

This feature uses Kubernetes schedulingGates to hold pods until the queue has capacity. While gated, pods are invisible to autoscalers. The gate is removed only after the queue capacity check passes and if the pod then cannot be scheduled due to missing nodes, it is marked as Unschedulable, allowing autoscalers to respond correctly.

Prerequisites

Volcano v1.15+ with the SchedulingGatesQueueAdmission feature gate enabled.
The capacity plugin configured in the scheduler (the feature is implemented in the capacity plugin and will soon be integrated in proportion as well).

1. Enable the Feature Gate

The feature is Alpha and disabled by default. Enable it on both the scheduler and webhook-manager.

Using Helm

helm install volcano volcano/volcano --namespace volcano-system --create-namespace \
  --set custom.scheduler_feature_gates="SchedulingGatesQueueAdmission=true" \
  --set custom.admission_feature_gates="SchedulingGatesQueueAdmission=true"

Using kubectl apply

Add the following flag to both the volcano-scheduler and volcano-admission deployments:

--feature-gates=SchedulingGatesQueueAdmission=true

Optionally, configure the number of async gate removal workers (default: 5):

--gate-removal-worker-num=10

These workers asynchronously process gate removals — each worker picks up a pod whose queue capacity check has passed and removes its scheduling gate, allowing the pod to proceed to scheduling. Increasing this number can help throughput when many pods are being ungated concurrently.

2. Configure the Capacity Plugin

Ensure the capacity plugin is enabled in your scheduler configuration. The reserved resource tracking that prevents race conditions between gate removal and pod allocation is implemented in this plugin.

Example scheduler configuration:

actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
- plugins:
  - name: predicates
  - name: capacity
  - name: nodeorder

3. Opt-in Pods

The feature is opt-in per pod, and one can start using it by adding the following annotation to pods that should use gate-controlled queue admission:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  annotations:
    # Opt-in annotation
    scheduling.volcano.sh/queue-allocation-gate: "true"
spec:
  schedulerName: volcano
  containers:
  - name: worker
    image: nginx
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"

When this pod is created:

The Volcano webhook injects a scheduling.volcano.sh/queue-allocation-gate scheduling gate.
The pod stays gated (invisible to autoscalers) until the queue has capacity.
Once capacity is available, the scheduler removes the gate.
If the pod can be placed on a node, it gets scheduled normally.
If no node matches (e.g., needs a specific node type), it gets marked Unschedulable, correctly triggering the autoscaler.

4. Verify the Feature is Working

After creating an opted-in pod, verify the gate was injected by the mutation webhook:

kubectl get pod my-pod -o jsonpath='{.spec.schedulingGates}'

Expected output (while waiting for queue capacity):

[{"name":"scheduling.volcano.sh/queue-allocation-gate"}]

Once the queue has capacity and the scheduler removes the gate, the field will be empty:

kubectl get pod my-pod -o jsonpath='{.spec.schedulingGates}'
# empty output

Interaction with other Scheduling Gates

If a pod has additional scheduling gates from other controllers (e.g., example.com/my-gate), Volcano will not remove its gate until the pod has only the Volcano gate remaining. This ensures Volcano does not interfere with other gate controllers and avoids reserving queue capacity for pods that are still blocked by external dependencies.

Limitations

Once a pod’s gate is removed, it reserves queue capacity until it is scheduled or deleted. If the pod remains unschedulable (e.g., waiting for the autoscaler to add nodes), it continues to hold queue capacity, potentially blocking other pods. Additionally, the feature currently does not implement a timeout for reserved capacity. Operators should be aware that pods that have been ungated but remain unschedulable can hold queue capacity indefinitely.
The feature is only implemented in the capacity plugin. Users relying on the proportion plugin for queue resource management will still face false autoscaler scale-ups, as the scheduling gates mechanism is not yet integrated with proportion. Tracking issue: #5271.

Ascend vNPU User Guide

Tue, 03 Feb 2026 00:00:00 +0000

Introduction

Volcano supports two vNPU modes for sharing Ascend devices:

1. MindCluster mode

Description:

The initial version of MindCluster—the official Ascend cluster scheduling add-on—required custom modifications and recompilation of Volcano. Furthermore, it was limited to Volcano release1.7 and release1.9, which complicated its use and restricted access to newer Volcano features.

To address this, we have integrated its core scheduling logic for Ascend vNPU into Volcano’s native device-share plugin, which is designed specifically for scheduling and sharing heterogeneous resources like GPUs and NPUs. This integration provides seamless access to vNPU capabilities through the procedure below, while maintaining full compatibility with the latest Volcano features.

Use case:

vNPU cluster for Ascend 310 series
with support for more chip types to come

2. HAMi mode

Description:

This mode is developed by a third-party community ‘HAMi’, which is the developer of volcano-vgpu feature. It supports vNPU feature for both Ascend 310 and Ascend 910. It also supports managing heterogeneous Ascend cluster(Cluster with multiple Ascend types, i.e. 910A,910B2,910B3,310P)

Use case:

NPU and vNPU cluster for Ascend 910 series
NPU and vNPU cluster for Ascend 310 series
Heterogeneous Ascend cluster

Installation

To enable vNPU scheduling, the following components must be set up based on the selected mode:

Prerequisites:

Kubernetes >= 1.16
Volcano >= 1.14
ascend-docker-runtime (for HAMi Mode)

Install Volcano:

Follow instructions in Volcano Installer Guide

Follow instructions in Volcano Installer Guide

Install ascend-device-plugin and third-party components

In this step, you need to select different ascend-device-plugin based on the vNPU mode you selected. MindCluster mode requires additional components from Ascend to be installed.

MindCluster Mode

Install Third-Party Components

Follow the official Ascend documentation to install the following components:

NodeD
Ascend Device Plugin
Ascend Docker Runtime
ClusterD
Ascend Operator

Note: Skip the installation of ascend-volcano mentioned in the document above, as we have already installed the native Volcano from the Volcano community in the Prerequisites part.

Configuration Adjustment for Ascend Device Plugin:

When installing ascend-device-plugin, you must set the presetVirtualDevice parameter to "false" in the device-plugin-310P-volcano-v{version}.yaml file to enable dynamic virtualization of 310P:


...
args:
  [
    "device-plugin",
    "-useAscendDocker=true",
    "-volcanoType=true",
    "-presetVirtualDevice=false",
    "-logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log",
    "-logLevel=0",
  ]
...

For detailed information, please consult the official Ascend MindCluster documentation.

Scheduler Config Update

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: predicates
      - name: deviceshare
        arguments:
          deviceshare.AscendMindClusterVNPUEnable: true   # enable ascend vnpu
    configurations:
    ...
    - name: init-params
      arguments: {"grace-over-time":"900","presetVirtualDevice":"false"}  # to enable dynamic virtualization, presetVirtualDevice need to be set false

HAMi mode

Label the Node with `ascend=on`

kubectl label node {ascend-node} ascend=on

Deploy `hami-scheduler-device` ConfigMap

kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/refs/heads/main/ascend-device-configmap.yaml

Deploy ascend-device-plugin

kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/refs/heads/main/ascend-device-plugin.yaml

For more information, refer to the ascend-device-plugin documentation.

Scheduler Config Update

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: predicates
      - name: deviceshare
        arguments:
          deviceshare.AscendHAMiVNPUEnable: true   # enable ascend vnpu
          deviceshare.SchedulePolicy: binpack  # scheduling policy. binpack / spread
          deviceshare.KnownGeometriesCMNamespace: kube-system
          deviceshare.KnownGeometriesCMName: hami-scheduler-device

Note: You may notice that, ‘volcano-vgpu’ has its own GeometriesCMName and GeometriesCMNamespace, which means if you want to use both vNPU and vGPU in a same volcano cluster, you need to merge the configMap from both sides and set it here.

Usage

Usage is different depending on the mode you selected

MindCluster mode

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: mindx-dls
  namespace: vnpu
  labels:
    ring-controller.atlas: ascend-310P
spec:
  minAvailable: 1
  schedulerName: volcano
  policies:
    - event: PodEvicted
      action: RestartJob
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 3
  queue: default
  tasks:
    - name: "default-test"
      replicas: 1
      template:
        metadata:
          labels:
            app: infers
            ring-controller.atlas: ascend-310P
            vnpu-dvpp: "null"
            vnpu-level: low
        spec:
          schedulerName: volcano
          containers:
            - name: resnet50infer
              image: swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.1.RC1-300I-Duo-py311-openeuler24.03-lts
              imagePullPolicy: IfNotPresent
              securityContext:
                privileged: false
              command: ["/bin/bash", "-c", "tail -f /dev/null"]
              resources:
                requests:
                  huawei.com/npu-core: 8
                limits:
                  huawei.com/npu-core: 8
          nodeSelector:
            host-arch: huawei-arm

The supported Ascend chips and their ResourceNames are shown in the following table:

ChipName	JobLabel and TaskLabel	ResourceName
310P3	ring-controller.atlas: ascend-310P	huawei.com/npu-core

Description of Labels in the Virtualization Task YAML

Key	Value	Description
vnpu-level	low	Low configuration (default). Selects the lowest-configuration “virtualized instance template.”
	high	Performance-first. When cluster resources are sufficient, the scheduler will choose the highest-configured virtualized instance template possible. When most physical NPUs in the cluster are already in use and only a few AI Cores remain on each device, the scheduler will allocate templates that match the remaining AI Core count rather than forcing high-profile templates. For details, refer to the table below.
vnpu-dvpp	yes	The Pod uses DVPP.
	no	The Pod does not use DVPP.
	null	Default value. DVPP usage is not considered.
ring-controller.atlas	ascend-310P	Indicates that the task uses products from the Atlas inference series.

Effect of DVPP and Level Configurations

Product Model	Requested AI Core Count	vnpu-dvpp	vnpu-level	Downgrade	Selected Template
Atlas Inference Series (8 AI Cores)	1	`null`	Any value	–	`vir01`
	2	`null`	`low` / other values	–	`vir02_1c`
	2	`null`	`high`	No	`vir02`
	2	`null`	`high`	Yes	`vir02_1c`
	4	`yes`	`low` / other values	–	`vir04_4c_dvpp`
	4	`no`	`low` / other values	–	`vir04_3c_ndvpp`
	4	`null`	`low` / other values	–	`vir04_3c`
	4	`yes`	`high`	–	`vir04_4c_dvpp`
	4	`no`	`high`	–	`vir04_3c_ndvpp`
	4	`null`	`high`	No	`vir04`
	4	`null`	`high`	Yes	`vir04_3c`
	8 or multiples of 8	Any value	Any value	–	–

Notice

For chip virtualization (non-full card usage), the value of vnpu-dvpp must strictly match the corresponding value listed in the above table. Any other values will cause the task to fail to be dispatched.

For detailed information, please consult the official Ascend MindCluster documentation.

HAMi mode

apiVersion: v1
kind: Pod
metadata:
  name: ascend-pod
spec:
  schedulerName: volcano
  containers:
    - name: ubuntu-container
      image: swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-pytorch:24.0.RC1-A2-1.11.0-ubuntu20.04
      command: ["sleep"]
      args: ["100000"]
      resources:
        limits:
          huawei.com/Ascend310P: "1"
          huawei.com/Ascend310P-memory: "4096"

The supported Ascend chips and their ResourceNames are shown in the following table:

ChipName	ResourceName	ResourceMemoryName
910A	huawei.com/Ascend910A	huawei.com/Ascend910A-memory
910B2	huawei.com/Ascend910B2	huawei.com/Ascend910B2-memory
910B3	huawei.com/Ascend910B3	huawei.com/Ascend910B3-memory
910B4	huawei.com/Ascend910B4	huawei.com/Ascend910B4-memory
910B4-1	huawei.com/Ascend910B4-1	huawei.com/Ascend910B4-1-memory
310P3	huawei.com/Ascend310P	huawei.com/Ascend310P-memory

Hami vNPU scene memory allocation restrictions

When a Pod requests a single vNPU device, the memory can be configured to any value, and the memory request of the job will automatically align with the closest sharding strategy
- Example: Single card memory is 65536, virtualization templates are ¹⁄₄ (16384), ¹⁄₂ (32768)
- Pod requests 1 vNPU device and requests 1024 memory, so the actual allocated memory is 16384
- Pod requests 1 vNPU device with a requested memory of 20480, resulting in an actual allocated memory of 32768
When a Pod requests multiple vNPU devices, the memory resource request can be left unspecified or filled with the maximum value, The memory allocated to Pod is the actual memory of the entire card

Capacity Plugin User Guide

Tue, 03 Feb 2026 00:00:00 +0000

Introduction

Capacity plugin is a replacement of proportion plugin, but instead of dividing the queue’s deserved resources by weight, it realizes elastic queue capacity management i.e., queue’s resource borrowing and lending mechanism by specifying the amount of deserved resources for each dimension resource of the queue.

A queue can use the idle resources of other queues, and when other queues submit jobs, they can reclaim the resources that have been lent, and the amount of reclaimed resources is the amount of queue’s deserved resources. For more detail, please see Capacity scheduling design

Environment setup

Install volcano

Refer to Install Guide to install volcano.

After installed, update the scheduler configuration:

kubectl edit cm -n volcano-system volcano-scheduler-configmap

Please make sure

reclaim action is enabled.
capacity plugin is enabled and proportion plugin is removed.

Note: capacity and proportion plugin are in conflict, the two plugins cannot be used together.

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill, reclaim" # add reclaim action.
    tiers:
    - plugins:
      - name: priority
      - name: gang
        enablePreemptable: false
      - name: conformance
    - plugins:
      - name: drf
        enablePreemptable: false
      - name: predicates
      - name: capacity # add this field and remove proportion plugin.
      - name: nodeorder
      - name: binpack

Config queue’s deserved resources

Assume there are two nodes and two queues named queue1 and queue2 in your kubernetes cluster, and each node has 4 CPU and 16Gi memory, then there will be total 8 CPU and 32Gi memory in your cluster.

allocatable:
  cpu: "4"
  memory: 16Gi
  pods: "110"

config queue1’s deserved field with 2 cpu and 8Gi memory.

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: queue1
spec:
  reclaimable: true
  deserved: # set the deserved field.
    cpu: 2
    memory: 8Gi

config queue2’s deserved field with 6 cpu and 24Gi memory.

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: queue2
spec:
  reclaimable: true
  deserved: # set the deserved field.
    cpu: 6
    memory: 24Gi

Submit pods to each queue

First, submit a deployment named demo-1 to queue1 with replicas=8 and each pod requests 1 cpu and 4Gi memory, because queue2 is idle, so queue1 can use the whole clusters’ resources, and you can see that 8 pods are in Running state.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-1
spec:
  selector:
    matchLabels:
      app: demo-1
  replicas: 8
  template:
    metadata:
      labels:
        app: demo-1
      annotations:
        scheduling.volcano.sh/queue-name: "queue1" # set the queue
    spec:
      schedulerName: volcano
      containers:
      - name: nginx
        image: nginx:1.14.2
        resources:
          requests:
            cpu: 1
            memory: 4Gi
        ports:
        - containerPort: 80

Expected result:

$ kubectl get po                                                                                             
NAME                      READY   STATUS    RESTARTS   AGE
demo-1-7bc649f544-2wjg7   1/1     Running   0          5s
demo-1-7bc649f544-cvsmr   1/1     Running   0          5s
demo-1-7bc649f544-j5lzp   1/1     Running   0          5s
demo-1-7bc649f544-jvlbx   1/1     Running   0          5s
demo-1-7bc649f544-mzgg2   1/1     Running   0          5s
demo-1-7bc649f544-ntrs2   1/1     Running   0          5s
demo-1-7bc649f544-nv424   1/1     Running   0          5s
demo-1-7bc649f544-zd6d9   1/1     Running   0          5s

Then submit a deployment named demo-2 to queue2 with replicas=8 and each pod requests 1 cpu and 4Gi memory.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-2
spec:
  selector:
    matchLabels:
      app: demo-2
  replicas: 8
  template:
    metadata:
      labels:
        app: demo-2
      annotations:
        scheduling.volcano.sh/queue-name: "queue2" # set the queue
    spec:
      schedulerName: volcano
      containers:
      - name: nginx
        image: nginx:1.14.2
        resources:
          requests:
            cpu: 1
            memory: 4Gi
        ports:
        - containerPort: 80

Because queue1 occupied queue2’s resources, so queue2 will reclaim its deserved resources with 6 cpu and 24Gi memory. And each pod of demo-2 request 1 cpu and 4Gi memory, so there will be 6 Pods in Running state of demo-2, and demo-1’s pods will be evicted.

Finally, you can see that there are 2 Running pods in demo-1(belongs to queue1), and 6 Running pods in demo-2(belongs to queue2), which meets queue’s deserved resources respectively.

$ kubectl get po                                                                                             
NAME                      READY   STATUS    RESTARTS   AGE
demo-1-7bc649f544-4vvdv   0/1     Pending   0          37s
demo-1-7bc649f544-c6mds   0/1     Pending   0          37s
demo-1-7bc649f544-j5lzp   1/1     Running   0          14m
demo-1-7bc649f544-mzgg2   1/1     Running   0          14m
demo-1-7bc649f544-pqdgk   0/1     Pending   0          37s
demo-1-7bc649f544-tx6wp   0/1     Pending   0          37s
demo-1-7bc649f544-wmshq   0/1     Pending   0          37s
demo-1-7bc649f544-wrhrr   0/1     Pending   0          37s
demo-2-6dfb86c49b-2jvgm   0/1     Pending   0          37s
demo-2-6dfb86c49b-dnjzv   1/1     Running   0          37s
demo-2-6dfb86c49b-fzvmp   1/1     Running   0          37s
demo-2-6dfb86c49b-jlf69   1/1     Running   0          37s
demo-2-6dfb86c49b-k62f7   1/1     Running   0          37s
demo-2-6dfb86c49b-k9b9v   1/1     Running   0          37s
demo-2-6dfb86c49b-rpzvg   0/1     Pending   0          37s
demo-2-6dfb86c49b-zch7w   1/1     Running   0          37s

Cooldown Protection Plugin User Guide

Tue, 03 Feb 2026 00:00:00 +0000

Background

When we need to enable elastic training or serving, preemptible job’s pods can be preempted or back to running repeatedly, if no cooldown protection set, these pods can be preempted again after they just started for a short time, this may cause service stability dropped. So we add “cdp” plugin to ensure preemptible job’s pods can run for at least some time set by user.

Environment setup

Install volcano

Refer to Install Guide to install volcano.

Update scheduler configmap

After installed, update the scheduler configuration:

kubectl edit configmap -n volcano-system volcano-scheduler-configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, preempt, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
      - name: cdp
    - plugins:
      - name: drf
      - name: predicates
      - name: task-topology
        arguments:
          task-topology.weight: 10
      - name: proportion
      - name: nodeorder
      - name: binpack

Running Jobs

Take a simple volcano job as sample.

original job yaml is as below, which has “ps” and “worker” task

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: test-job
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "worker"
      template:
        metadata:
          name: worker
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure
    - replicas: 2
      name: "ps"
      template:
        metadata:
          name: ps
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure

Edit yaml of vcjob

add annotations in volcano job in format below.
1. volcano.sh/preemptable annotation indicates that job or task is preemptable
2. volcano.sh/cooldown-time annotation indicates cooldown time for the entire job or dedicated task. Value for the annotation indicates cooldown time, valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”.
```
    volcano.sh/preemptable: "true"
    volcano.sh/cooldown-time: "600s"
```

Example 1

Add annotation to entire job, then “ps” and “worker” task can be preempted and all have cooldown time support.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: test-job
  annotations:
    volcano.sh/preemptable: "true"
    volcano.sh/cooldown-time: "600s"
spec:
  ... # below keep the same

Example 2

Add annotation to dedicated task, as shown below, only “worker” can be preempted and have cooldown time support.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: test-job
spec:
  minAvailable: 3
  schedulerName: volcano
  priorityClassName: high-priority
  plugins:
    ssh: []
    env: []
    svc: []
  maxRetry: 5
  queue: default
  volumes:
    - mountPath: "/myinput"
    - mountPath: "/myoutput"
      volumeClaimName: "testvolumeclaimname"
      volumeClaim:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "my-storage-class"
        resources:
          requests:
            storage: 1Gi
  tasks:
    - replicas: 6
      name: "worker"
      template:
        metadata:
          name: worker
          annotations:     # add annotation in tasks
            volcano.sh/preemptable: "true"
            volcano.sh/cooldown-time: "600s"
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure
    - replicas: 2
      name: "ps"
      template:
        metadata:
          name: ps
        spec:
          containers:
            - image: nginx
              imagePullPolicy: IfNotPresent
              name: nginx
              resources:
                requests:
                  cpu: "1"
          restartPolicy: OnFailure

Extender User Guide

Tue, 03 Feb 2026 00:00:00 +0000

Install volcano

1. Install from source

Refer to Install Guide to install volcano.

2. Deploy extender

Deploy extender into kubernetes cluster. Extender needs to expose domain name or IP address and verbs that can be provided.

3. Update Volcano configuration

kubectl edit cm -n volcano-system volcano-scheduler-configmap

Users can view the meaning of the parameters through the documentation

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "reclaim, allocate, backfill, preempt"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: predicates
      - name: extender
        arguments:
          extender.urlPrefix: http://127.0.0.1:8713
          extender.httpTimeout: 100ms
          extender.onSessionOpenVerb: onSessionOpen
          extender.onSessionCloseVerb: onSessionClose
          extender.predicateVerb: predicate
          extender.prioritizeVerb: prioritize
          extender.preemptableVerb: preemptable
          extender.reclaimableVerb: reclaimable
          extender.queueOverusedVerb: queueOverused
          extender.jobEnqueueableVerb: jobEnqueueable
          extender.ignorable: true

Verify Extender is working

The user can see in the log something like : ‘Initialize extender plugin with configuration : {your configuration}’

GPU Number User guide

Tue, 03 Feb 2026 00:00:00 +0000

Important Note

Note GPU Number is deprecated in volcano v1.9, recommended to use the Volcano VGPU feature, which is provided by HAMI project, click here

Environment setup

Install volcano

1. Install from source

Refer to Install Guide to install volcano.

After installed, update the scheduler configuration:

kubectl edit cm -n volcano-system volcano-scheduler-configmap

For volcano v1.8.2+(v1.8.2 excluded), use the following configMap

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: deviceshare
        arguments:
          deviceshare.GPUNumberEnable: true # enable gpu number
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack

For volcano v1.8.2-(v1.8.2 included), use the following configMap

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: predicates
        arguments:
          predicate.GPUNumberEnable: true # enable gpu number
      - name: proportion
      - name: nodeorder
      - name: binpack

2. Install from release package.

Same as above, after installed, update the scheduler configuration in volcano-scheduler-configmap configmap.

Install Volcano device plugin

Please refer to volcano device plugin

Remember to config volcano device plugin to support gpu-number, users need to config volcano device plugin –gpu-strategy=number. For more information volcano device plugin configuration

Verify environment is ready

Check the node status, it is ok volcano.sh/gpu-number is included in the allocatable resources.

$ kubectl get node {node name} -oyaml
...
Capacity:
  attachable-volumes-gce-pd:  127
  cpu:                        2
  ephemeral-storage:          98868448Ki
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     7632596Ki
  pods:                       110
  volcano.sh/gpu-memory:      0
  volcano.sh/gpu-number:      1
Allocatable:
  attachable-volumes-gce-pd:  127
  cpu:                        1930m
  ephemeral-storage:          47093746742
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     5752532Ki
  pods:                       110
  volcano.sh/gpu-memory:      0
  volcano.sh/gpu-number:      1

Running Jobs With Multiple GPU Cards

Jobs can have multiple exclusive NVIDIA GPUs cards via defining container level resource requirements volcano.sh/gpu-number:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod1
spec:
  containers:
    - name: cuda-container
      image: nvidia/cuda:9.0-devel
      command: ["sleep"]
      args: ["100000"]
      resources:
        limits:
          volcano.sh/gpu-number: 1 # requesting 1 gpu cards
EOF

If the above pods claim multiple gpu cards, you can see each of them has exclusive gpu cards:

$ kubectl exec -ti  gpu-pod1 env
...
NVIDIA_VISIBLE_DEVICES=0
VOLCANO_GPU_ALLOCATED=1
...

Understanding How Multiple GPU Cards Requirement Works

The main architecture is similar as the previous, but the gpu-index results of each pod will be a list of gpu cards index.

create a pod with volcano.sh/gpu-number resource request,

volcano scheduler predicates and allocate gpu cards to the pod. Add the below annotation

annotations:
volcano.sh/gpu-index: "0"
volcano.sh/predicate-time: "159376446655083530"

kubelet watches the pod bound to itself, and calls allocate API to set env before running the container.
```
env:
NVIDIA_VISIBLE_DEVICES: "0" # GPU card index
VOLCANO_GPU_ALLOCATED: "1" # GPU number allocated
```

GPU Sharing User guide

Tue, 03 Feb 2026 00:00:00 +0000

Important Note

Note GPU sharing is deprecated in volcano v1.9, recommended to use the Volcano VGPU feature, which is provided by HAMI project, click here

Environment setup

Install volcano

1. Install from source

Refer to Install Guide to install volcano.

After installed, update the scheduler configuration:

kubectl edit cm -n volcano-system volcano-scheduler-configmap

For volcano v1.8.2+(v1.8.2 excluded), use the following configMap

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: deviceshare
        arguments:
          deviceshare.GPUSharingEnable: true # enable gpu sharing
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack

For volcano v1.8.2-(v1.8.2 included), use the following configMap

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: predicates
        arguments:
          predicate.GPUSharingEnable: true # enable gpu sharing
      - name: proportion
      - name: nodeorder
      - name: binpack

2. Install from release package

Same as above, after installed, update the scheduler configuration in volcano-scheduler-configmap configmap.

Install Volcano device plugin

Please refer to volcano device plugin

By default volcano device plugin supports shared GPUs, users do not need to config volcano device plugin. Default setting is the same as setting –gpu-strategy=share. For more information volcano device plugin configuration

Verify environment is ready

Check the node status, it is ok if volcano.sh/gpu-memory and volcano.sh/gpu-number are included in the allocatable resources.

$ kubectl get node {node name} -oyaml
...
status:
  addresses:
  - address: 172.17.0.3
    type: InternalIP
  - address: volcano-control-plane
    type: Hostname
  allocatable:
    cpu: "4"
    ephemeral-storage: 123722704Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8174332Ki
    pods: "110"
    volcano.sh/gpu-memory: "89424"
    volcano.sh/gpu-number: "8"    # GPU resource
  capacity:
    cpu: "4"
    ephemeral-storage: 123722704Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8174332Ki
    pods: "110"
    volcano.sh/gpu-memory: "89424"
    volcano.sh/gpu-number: "8"   # GPU resource

NVIDIA GPUs can now be shared via container level resource requirements using the resource name volcano.sh/gpu-memory:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod1
spec:
  schedulerName: volcano
  containers:
    - name: cuda-container
      image: nvidia/cuda:9.0-devel
      command: ["sleep"]
      args: ["100000"]
      resources:
        limits:
          volcano.sh/gpu-memory: 1024 # requesting 1024MB GPU memory
EOF

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod2
spec:
  schedulerName: volcano
  containers:
    - name: cuda-container
      image: nvidia/cuda:9.0-devel
      command: ["sleep"]
      args: ["100000"]
      resources:
        limits:
          volcano.sh/gpu-memory: 1024 # requesting 1024MB GPU memory
EOF

If only the above pods are claiming gpu resource in a cluster, you can see the pods sharing one gpu card:

$ kubectl exec -ti  gpu-pod1 env
...
VOLCANO_GPU_MEMORY_TOTAL=11178
VOLCANO_GPU_ALLOCATED=1024
NVIDIA_VISIBLE_DEVICES=0
...

$ kubectl exec -ti  gpu-pod1 env
...
VOLCANO_GPU_MEMORY_TOTAL=11178
VOLCANO_GPU_ALLOCATED=1024
NVIDIA_VISIBLE_DEVICES=0
...

The GPU sharing workflow is depicted as below:

create a pod with volcano.sh/gpu-memory resource request,
volcano scheduler predicates and allocate gpu resource for the pod. Adding the below annotation
```
annotations:
volcano.sh/gpu-index: "0"
volcano.sh/predicate-time: "1593764466550835304"
```

kubelet watches the pod bound to itself, and call allocate API to set env before running the container.

env:
NVIDIA_VISIBLE_DEVICES: "0" # GPU card index
VOLCANO_GPU_ALLOCATED: "1024" # GPU allocated
VOLCANO_GPU_MEMORY_TOTAL: "11178" # GPU memory of the card

How to configure priorityclass for job

Tue, 03 Feb 2026 00:00:00 +0000

Background

When a user creates a job, if there is no PriorityClassName specified in the template of the task, the task will use the PriorityClassName specified by the job. The user can specify PriorityClassName in each task’s template to override the configuration of job, so that priority can be configured separately for each task.

Key Points

If the task does not specify PriorityClassName but the job does, the task will use the job’s PriorityClass, and the PreemptionPolicy and priority value will also be the same as the job.
When user needs to allow task in job to preempt other tasks if the resources are insufficient, a separate PriorityClassName must be set in the task’s template, and it is important to note that if there are multiple tasks that need to be set to different priorities, then user need to set the PriorityClassName for all of them, otherwise the task that does not specify PriorityClassName will use the PriorityClass of the job it belongs to.

Example

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: job-priority
value: 1
preemptionPolicy: Never
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: task-priority
value: 10
---
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: job
spec:
  schedulerName: volcano
  priorityClassName: job-priority
  minAvailable: 1
  tasks:
    - replicas: 1
      name: preempt-task
      template:
        spec:
          priorityClassName: task-priority # The priorityclass is specified individually for this task
          containers:
            - image: alpine
              command: ["/bin/sh", "-c", "sleep 1000"]
              name: preempt
              resources:
                requests:
                  cpu: 1
    - replicas: 1
      name: non-preempt-task # This task does not specify PriorityClassName, so it will use the "job-priority" priorityclass specified by job
      template:
        spec:
          containers:
            - image: alpine
              command: ["/bin/sh", "-c", "sleep 1000"]
              name: non-preempt
              resources:
                requests:
                  cpu: 1

How to Configure Scheduler

Tue, 03 Feb 2026 00:00:00 +0000

Requirements

Before reading the guidance, please make sure you are aware of basic concepts such as action plugin session tier volcano job podgroup queue and so on. If they are still strange to you, please refer to Volcano Docs for more details.
Before reading the guidance, please make sure you have general understanding of Volcano scheduling workflow.

Background

In order to adjust the scheduling process and algorithms to different scenarios, Volcano allows users to configure actions and plugins for volcano scheduler. The scheduling pipeline consists of a series of actions. The plugins implement the algorithms, which will be called in actions as registered session functions. As what you can see in the configmap volcano-scheduler-configmap, plugins are divided into 2 tiers by default. It may confuse some users. Besides, it is necessary to provide a guidance about how to configure volcano scheduler.

Key Points

All the configurations are configured in the configmap volcano-scheduler-configmap, which is under the namespace volcano-system.
The configuration is made up of 2 parts: actions and tiers.
actions defines the scheduling pipeline. They will be executed in order in each session.
tiers divides the plugins into several categories. All the functions defined in the plugins will be registered when a session is open and called when actions are executed.
In some scenarios, users may configure different plugins which registers the same functions. It will depend on the business requirement to decide how to combine these functions. That’s why tier is required.

Actions

Action implements the main logic of scheduling.
Volcano allow users to make self-defined actions.
Volcano provides 7 built-in actions until April 2022. The details are as follows.

ID	Name	Required	Description
1	enqueue	Y	Judge whether the idle resource in the cluster can satisfy the basic demand of a workload. If yes, set the podgroup of the workload to be `inqueue`, otherwise keep the podgroup `pending`. Notice that the default value for parameter `overcommit-factor` is `1.2`.
2	allocate	Y	Try to allocate resource to workloads whose corresponding podgroup status is `inqueue`.
3	backfill	N	Try to allocate resource to workloads whose pods are `BestEffort`.
4	preempt	N	Recognise workloads with high priority. Try to evict pods with low priority and allocate the resource to them.
5	reclaim	N	Pick out queues whose resources have been borrowed by other queues and reclaim them back.
6	elect	N	Select a workload satisfying some conditions. It is designed to work with resource reservation for target workload. Will deprecated at future releases.
7	reserve	N	Select a series of nodes and reserve resource. It is designed to work with resource reservation for target workload. Will deprecated at future releases.

Tiers and Plugins

Plugin provides implementation details about scheduling algorithms by registering a series of functions. These functions will be called during actions are executed.
In general, a plugin mainly consists of 3 functions: Name OnSessionOpen OnSessionClose. Name provides the name of the plugin. OnSessionOpen executes some operations when a session starts and register some functions about scheduling details. OnSessionClose clean up some resource when a session finishes.
Some plugins provide arguments for users to match their custom scenarios.
Different plugins may register same functions with different logic. Please make sure they can work together when configuring plugins.
Volcano provides 15 built-in plugins until April 2022. The details are as follows.

ID	Name	Arguments	Registered Functions	Description
1	binpack	* binpack.weight * binpack.cpu * binpack.memory * binpack.resources	* nodeOrderFn	Try to bind pods to nodes with high resource usage to reduce fragmentation.
2	conformance	/	* preemptableFn * reclaimableFn	Skip critical pods and not evict them.
3	drf	/	* preemptableFn * queueOrderFn * reclaimFn * jobOrderFn * namespaceOrderFn	Provide fair resource shares for all queues.
4	extender	* extender.urlPrefix * extender.httpTimeout * extender.onSessionOpenVerb * extender.onSessionCloseVerb * extender.predicateVerb * extender.prioritizeVerb * extender.preemptableVerb * extender.reclaimableVerb * extender.queueOverusedVerb * extender.jobEnqueueableVerb * extender.ignorable	* predicateFn * batchNodeOrderFn * preemptableFn * reclaimableFn * jobEnqueueableFn * overusedFn	Add outer http server to execute custom actions.
5	gang	/	* jobValidFn * reclaimableFn * preemptableFn * jobOrderFn * JobReadyFn * jobPipelineFn * jobStarvingFn	Consider the minimal resource requirement or member number for a workload when allocate resource to it.
6	nodeorder	* nodeaffinity.weight * podaffinity.weight * leastrequested.weight * balancedresource.weight * mostrequested.weight * tainttoleration.weight * imagelocality.weight	* nodeOrderFn * batchNodeOrderFn	Sort all nodes in custom way.
7	numaaware	* weight	* predicateFn * batchNodeOrderFn	Consider CPU Numa as a key factor when binding a pod to a node.
8	overcommit	* overcommit-factor	* jobEnqueueableFn * jobEnqueuedFn	Set the available resource as the given times of the whole resource of the cluster.
9	predicate	* predicate.GPUSharingEnable * predicate.CacheEnable * predicate.ProportionalEnable * predicate.resources * predicate.resources.nvidia.com/gpu.cpu * predicate.resources.nvidia.com/gpu.memory	* predicateFn	Add custom functions about how to filter nodes for pods.
10	priority	/	* taskOrderFn * jobOrderFn * preemptableFn * jobStarvingFn	Defines priority for workloads.
11	proportion	/	* queueOrderFn * reclaimableFn * overusedFn * allocatableFn * jobEnqueueableFn	Divide the whole resources of the cluster to all queues as proportion according to queues’ configurations
12	reservation	/	* targetJobFn * reservedNodesFn	Sort nodes as resource usage and lock parts for target workload as reservation.
13	sla	* sla-waiting-time	* jobOrderFn * jobEnqueueableFn * JobPipelinedFn	Sort workloads according to the SLA settings.
14	task-topology	/	* taskOrderFn * nodeOrderFn	Bind pods with different roles to nodes according to the given policy.
15	tdm	* tdm.revocable-zone.rz1 * tdm.revocable-zone.rz2 * tdm.evict.period	* predicateFn * nodeOrderFn * preemptableFn * victimTasksFn * jobOrderFn * jobPipelinedFn * jobStarvingFn	Enable part of nodes to be in the charge of K8s and other clusters in different period.

Examples

# default configuration for scheduler
actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
  - name: conformance
- plugins:
  - name: overcommit
  - name: drf
  - name: predicates
  - name: proportion
  - name: nodeorder
  - name: binpack

Note:

According to the default configuration, the scheduling process works as follows at a session. The scheduler will run the following pipeline regularly. The period is 1s by default.
```
graph LR
1(Start) --> 2(OpenSession) --> 3(enqueue) --> 4(allocate) --> 5(backfill) --> 6(CloseSession) --> 7(End)
```
All the functions in the configured plugins will be registered when executing OpenSession and called when executing the configured actions. For example, jobEnqueueable function, which is registered in overcommit plugin and called at enqueue action, aims to judge whether the idle resource of the cluster can satisfy the minimal demand of a workload.
Both overcommit and proportion plugin have registered function jobEnqueueableFn, which will be called in the function JobEnqueueable. Besides, overcommit and proportion are in the same tier. According to the implementation of JobEnqueueable, it will get through all the plugins in different tiers in order. If any jobEnqueueableFn returns a value belows 0, it stops executing the jobEnqueueableFn registered in the following plugins and returns false. Namely, if the jobEnqueueableFn registered in overcommit returns a value belows 0, jobEnqueueableFn, which is called in enqueue action, will return false and never call the jobEnqueueableFn registered in the proportion plugin.

FAQ

How can I decide which plugins should be grouped into a tier? How many tiers should I set for my business? > In most scenarios, users should not concern about how to divide plugins to different tiers. It’s OK to configure all plugins within a single tier. Only on condition that it is related with eviction will you need to think of how to organize the plugins in different tiers. For example, when enable reclaim action, the scheduler will try to collect a set of victims. In order to reduce the influence to users’ business, it is reasonable to pick out victims as less as possible. Then you can configure the plugins evicting the least pods in the first tier. And configure other plugins with eviction at the second tier. If the first tier can pick out victims, it will not call the functions registered in the plugins, which is configured at the second tier.

How to Enable Dynamic Resource Allocation (DRA) in Volcano Scheduler

Tue, 03 Feb 2026 00:00:00 +0000

This document describes the steps required to enable Dynamic Resource Allocation (DRA) support in the Volcano scheduler.

Prerequisites

Before proceeding with the configuration steps, ensure your cluster meets the following prerequisites:

Configure Cluster Nodes (Containerd)

For nodes running containerd as the container runtime, you must enable the Container Device Interface (CDI) feature. This is crucial for containerd to properly interact with DRA drivers and inject dynamic resources into Pods.

Modify the containerd configuration file on each node (typically /etc/containerd/config.toml) to ensure the following setting is present:

# Enable CDI as described in
# https://tags.cncf.io/container-device-interface#containerd-configuration
[plugins."io.containerd.grpc.v1.cri"]
  enable_cdi = true
  cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

After modifying the configuration, restart the containerd service on each node for the changes to take effect. For example: sudo systemctl restart containerd

If you are using other container runtimes, please refer to: how-to-configure-cdi

1. Configure Kube-apiserver

DRA-related APIs are k8s built-in resources instead of CRD resources, and these resources are not registered by default in v1.32, so you need to set the startup parameters of kube-apiserver to manually register DRA-related APIs, add or ensure the following flag is present in your kube-apiserver manifest or configuration:

--runtime-config=resource.k8s.io/v1beta1=true

2. Install Volcano With DRA feature gates enabled

When installing Volcano, you need to enable the DRA related feature gates, e.g., DynamicResourceAllocation must be enabled when you need to use DRA, you can also choose to enable the DRAAdminAccess feature gate to manage devices as your need.

When you are using helm to install Volcano, you can use following command to install Volcano with DRA feature gates enabled:

helm install volcano volcano/volcano --namespace volcano-system --create-namespace \
  --set custom.scheduler_feature_gates="DynamicResourceAllocation=true" \
  # Add other necessary Helm values for your installation

When you directly use kubectl apply -f to install Volcano, you need to add or ensure the following flag is present in your volcano-scheduler manifest:

--feature-gates=DynamicResourceAllocation=true

3. Configure Volcano Scheduler Plugins

After installing Volcano, you need to configure the Volcano scheduler’s plugin configuration to enable the DRA plugin within the predicates plugin arguments.

Locate your Volcano scheduler configuration (A ConfigMap contains the configuration). Find the predicates plugin configuration and add or modify its arguments to enable DRA plugin.

An example snippet of the scheduler configuration (within the volcano-scheduler.conf key of the ConfigMap) might look like this:

actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
- plugins:
  - name: drf
  - name: predicates
    arguments:
      predicate.DynamicResourceAllocationEnable: true
  - name: proportion
  - name: nodeorder
  - name: binpack

4. Deploy a DRA Driver

To utilize Dynamic Resource Allocation, you need to deploy a DRA driver in your cluster. The driver is responsible for managing the lifecycle of dynamic resources. For example, you can refer to the kubernetes-sigs/dra-example-driver to deploy a example DRA driver for testing.

For some DRA Drivers which have already been used in actual production, you can refer to: - NVIDIA/k8s-dra-driver-gpu - intel/intel-resource-drivers-for-kubernetes

How to Tune Volcano Performance in Large-Scale Scenarios

Tue, 03 Feb 2026 00:00:00 +0000

This article originates from the “Volcano Large-Scale Performance Testing and Tuning” project of the Open Source Promotion Plan (OSPP), organized by the Chinese Academy of Sciences. All experimental data and analysis have been published in the author’s blog series (@Freshwlnd).

1. Introduction

1.1 Background and Objectives

Volcano is a mainstream cloud-native batch processing system widely used in AI, big data, and HPC scenarios. This project aims to systematically reproduce and identify performance bottlenecks in Volcano under a load of tens of thousands of pods through large-scale performance testing, ultimately producing a practical tuning guide and proposing future architectural optimization directions.

1.2 Summary of Overall Conclusions

Through systematic experiments, we have reached the following core conclusions:

Scenario-Dependent Performance: Volcano’s performance is highly dependent on the scheduling scenario. When Gang Scheduling is enabled, its performance far exceeds that of other schedulers, highlighting its core advantages in batch processing and AI scenarios. Under the general scenario where Gang is not enabled, due to the additional overhead it brings to the Job management function designed for complex scenarios, there is still some room for optimization.
Webhook as the Main Bottleneck: In large-scale, resource-constrained clusters, the default Webhook configuration (10s timeout) is the direct cause of large-scale pod creation failures. Additionally, the overhead introduced by the Webhook validation chain itself is a significant performance factor.
Optimal Value for Controller Worker Threads: The --worker-threads parameter is not a case of “the higher, the better.” An improper configuration can degrade performance by saturating computing resources or exacerbating API-Server contention, exhibiting a “V-shaped” performance curve.
API-Server Contention between CREATE/SCHEDULE: In high-concurrency scenarios, concurrent write operations to the K8s API-Server/ETCD by the Controller (CREATE) and Scheduler (SCHEDULE) lead to queuing/retries and a macroscopic phenomenon of “stair-step stalls” where they appear to execute alternately. Meanwhile, blindly increasing the CREATE rate may actually prolong the overall scheduling time.

2. Test and Monitoring Environment Description

The core of this testing was to extend the open-source kube-scheduling-perf framework to simulate large-scale pod scheduling scenarios in a local Kind cluster. We have conducted a detailed source code analysis of this testing framework and written a blog series titled “Cloud-Native Batch Scheduling in Practice: Volcano Monitoring and Performance Testing,” available on the author’s blog (@Freshwlnd).

Testing Framework: The wzshiming/kube-scheduling-perf project uses a Makefile for automated orchestration, enabling one-click cluster setup, execution of multi-scenario benchmarks, monitoring data collection, and results archiving.
Data Collection: The framework uses audit-exporter to accurately capture timestamps of Pod Creation and Pod Schedule events from the kube-apiserver audit logs (audit.log).
Core Metrics:
- X-axis: Test run time (seconds).
- Y-axis: Cumulative number of Pods that have completed CREATE or SCHEDULE.
- Curve Slope: Represents instantaneous throughput (Pods per second), a key performance metric.
- Illustrative Example:

3. Performance Phenomena

3.1 Core Performance Phenomena

In summary, we observed the following performance characteristics during our tests.

When Gang Scheduling was not enabled, we observed unique performance characteristics in Volcano:
1. Throughput and Stair-Step Stalls: The CREATED and SCHEDULED curves exhibit a distinct stair-step pattern, with alternating periods of “surges” and “stalls.”
2. Large-Scale Pod Creation Failures: In some cases, when the number of pods within a single job is very large (e.g., 20 Jobs × 500 Pods), fewer than 2,000 pods were successfully created under the default configuration.
Enabling Gang Scheduling is Volcano’s core use case. In this scenario:
1. Superior Performance: Volcano significantly outperforms other schedulers, thanks to its efficient PodGroup management and Gang scheduling algorithm.
2. Necessary Overhead: Although all schedulers experience longer total runtimes with Gang Scheduling due to more complex computations, Volcano’s architecture maintains the highest efficiency. This validates the necessity and sophistication of its job management features. It is essential to provide adequate resources for Volcano in production environments to leverage its full capabilities.

3.2 Analysis of Specific Performance Phenomena

With a total of 10k pods, we adjusted the number of jobs and the number of pods per job, both with and without Gang Scheduling enabled. We focused primarily on the non-Gang Scheduling scenario.

Test Environment Description

Test Environment Configuration: - Hardware: Intel Xeon Gold 6230 @ 2.10GHz, 8-core CPU, 15GB RAM, 79GB Storage - Software: Docker 27.5.1, Kubernetes 1.32.2 (Kind cluster), kubectl v1.33.2 - Test Scale: A single test result occupied 1.3GB-2.7GB of storage, totaling 15GB of test data. - Comparative Verification: Comparative tests were conducted in a high-configuration environment (24-core CPU, 96GB RAM) to verify the impact of hardware resources on the results.

Test Methodology: - Extended tests were based on the open-source kube-scheduling-perf framework. - audit-exporter was used to accurately capture Pod Creation and Pod Schedule events from kube-apiserver audit logs. - Validation analysis was performed by comparing results with the video from the KubeCon Europe 2025 tech talk “A Comparative Analysis of Kueue, Volcano, and YuniKorn - Wei Huang, Apple & Shiming Zhang, DaoCloud”.

Test Results Analysis

Without Gang Scheduling, we observed Volcano’s unique performance characteristics across four typical benchmark combinations (all with a total of 10,000 pods):

Benchmark 1: 10,000 Jobs × 1 Pod/Job

Observation: The CREATE and SCHEDULE curves almost completely overlapped, and the overall slope (throughput) was lower compared to scenarios with fewer Jobs, mainly due to the overhead of complex job management features. The results were the same in both low and high-resource environments.
Conclusion: In this scenario, the processing overhead of the Job objects themselves is substantial, making the pod creation (CREATE) phase the absolute performance bottleneck.

Benchmark 2: 500 Jobs × 20 Pods/Job

Observation: The SCHEDULED curve significantly lagged behind the CREATED curve, and both curves showed the clearest stair-step stall pattern. Volcano exhibited periodic surges in both CREATED and SCHEDULE phases, indicating intermittent stalls in the controller. The results were the same in both low and high-resource environments.
Conclusion: The scheduling (SCHEDULE) phase became the bottleneck, and the resource contention between the Controller and Scheduler was most prominent.

Benchmark 3 & 4: 20 Jobs × 500 Pods/Job & 1 Job × 10,000 Pods/Job

Observation: The CREATED curve rose rapidly at the beginning and then flattened, while the SCHEDULED curve grew slowly and linearly, with a huge gap between them. The stair-step stall pattern in the CREATED curve disappeared. In our resource-limited local test environment (8-core, 16GB), a large number of pod creation failures were observed.
Conclusion: When the total number of jobs is small and the number of pods per job is extremely large, the Controller can create pods very quickly, making the scheduling (SCHEDULE) phase the absolute performance bottleneck.

Key Finding: In all four cases, Volcano’s total scheduling time was roughly proportional to the number of Jobs, indicating that its performance bottleneck is primarily caused by Job processing.

4. Performance Bottleneck Hypotheses and Verification

4.1 Hypothesis 1: Webhook is a Performance Bottleneck

4.1.1 Verification (Timeout Issue)

By analyzing audit.log, we found that a large number of pod creation failures were due to an insufficient Webhook timeoutSeconds (10s). The evidence includes:

Pod Creation Failure Rate: 98.7% of pod creation requests failed due to timeouts.
Log Evidence: 4.9GB of audit logs recorded a massive number of timeout errors.
Timeout Configuration: The webhook timeout was set to 10 seconds.

Specifically, we analyzed the 4.9GB audit log with the following commands:

# Count the number of errors
grep -c "context deadline exceeded" kube-apiserver-audit.volcano.log
# Output: 520120

# Count the number of Webhook calls
grep -c "validatepod.volcano.sh" kube-apiserver-audit.volcano.log
# Output: 515531

# Count the number of successful/failed pod creations
grep -c "ResponseComplete.*pods.*create.*Success" kube-apiserver-audit.volcano.log
# Output: 712
grep -c "ResponseComplete.*pods.*create.*Failure" kube-apiserver-audit.volcano.log
# Output: 520518

Analyzing the error messages in the logs revealed a consistent pattern:

{
  "status": "Failure",
  "message": "Internal error occurred: failed calling webhook \"validatepod.volcano.sh\": failed to call webhook: Post \"https://volcano-admission-service.volcano-system.svc:443/pods/validate?timeout=10s\": context deadline exceeded",
  "reason": "InternalError",
  "code": 500
}

The statistical summary is as follows:

Metric	Count	Percentage
Total Pod Creation Requests	526,767	100%
Successful Creations	712	0.13%
Failed Creations	520,518	98.7%
Webhook Timeout Errors	520,120	98.7%

Experimental Design: To verify this hypothesis, we increased the webhook timeout from 10 seconds to 30 seconds:

# Batch modify the timeout in all webhook configuration files
sed -i 's/timeoutSeconds: 10/timeoutSeconds: 30/g' schedulers/volcano/admission-service-*.yaml

Verification Result: After increasing the timeout to 30s, the number of pod creations returned to normal. In benchmark 3 and 4, the number of created pods recovered from “less than 1000” to the normal state of “10,000”, and the pod creation success rate increased from 1.3% to nearly 100%.

Before fix:

After fix:

4.1.2 Verification (Inherent Overhead)

Even after resolving the timeout issue, the complex validation process of the Webhook (TLS, network transport) itself introduces overhead. Specifically, Volcano’s Webhook system includes multiple components, and each pod creation request must pass through:

Mutating Webhook: Modifies Pod configuration (e.g., adding fields like maxRetry, minAvailable).
Validating Webhook: Validates the legality of the Pod configuration.
Admission Service: The service that handles Webhook requests.
TLS Certificate Validation: Ensures the security of Webhook calls.

Common Webhook configurations include:

Type	Configuration Name	Description
Mutating	`volcano-admission-service-jobs-mutate`	Modifies Job configuration, adds fields like `maxRetry`.
Mutating	`volcano-admission-service-podgroups-mutate`	Modifies PodGroup configuration.
Mutating	`volcano-admission-service-pods-mutate`	Modifies Pod configuration.
Mutating	`volcano-admission-service-queues-mutate`	Modifies Queue configuration.
Validating	`volcano-admission-service-jobs-validate`	Validates Job configuration legality.
Validating	`volcano-admission-service-pods-validate`	Validates Pod configuration legality.
Validating	`volcano-admission-service-queues-validate`	Validates Queue configuration legality.

Considering the development trends of modern Kubernetes versions, some Webhook functionalities could potentially be replaced by:

Strengthening Controller Validation: Moving necessary validation logic (e.g., null pointer checks) into the Volcano Controller Manager to support disabling webhooks in simpler scenarios.
Using CRD Validation Rules: Using the Common Expression Language (CEL) for K8s admission control to validate CRD values (using the K8s v1.29 [stable] x-kubernetes-validations extension), thereby replacing parts of the Validating Webhook with Kubernetes’ native CRD validation capabilities.
Pre-configured Templates: Reducing runtime modification needs by pre-configuring Job templates.

Verification Result: Comparative experiments showed that disabling the Webhook led to a significant performance improvement. Pod creation/scheduling speed increased noticeably, and overall scheduler performance improved. While the Webhook system provides important validation and modification functions, its processing overhead becomes a bottleneck in large-scale pod creation scenarios. Taking the 10000 Jobs x 1 Pod scenario as an example, the overall scheduling time was reduced from about 250 seconds to about 180 seconds, a throughput increase of nearly 30%, which is a significant improvement.

4.2 Hypothesis 2: Controller Worker Threads Need Tuning

After analyzing the Controller source code, we found that the --worker-threads parameter (default 50) determines the number of Jobs that can be processed concurrently. Further experiments revealed that this value is not a case of “the more, the better”:

Too Few Threads: Fails to fully utilize computing resources, with I/O waits becoming the bottleneck.
Too Many Threads: Exacerbates CPU and API-Server contention, leading to a performance decrease.
Verification: By conducting multiple experiments with different values for the --worker-threads parameter of volcano-controller-manager, we observed a “V-shaped” performance curve. Too few threads underutilize the CPU; too many threads increase resource contention.

4.2.1 Source Code Analysis of Controller Creation Logic

Origin of Worker Goroutines

// cmd/controller-manager/app/server.go:134-139
func startControllers(config *rest.Config, opt *options.ServerOption) func(ctx context.Context) {
    ...
    controllerOpt.WorkerNum = opt.WorkerThreads
    ...
}

The --worker-threads parameter defaults to 50 (if not specified) and determines the number of goroutines that the JobController uses to concurrently process Job requests.

⚠️ Note: These worker threads determine the parallelism for Jobs, not Pods.

Hashing and Queues

// pkg/controllers/job/job_controller.go:318-333
func (cc *jobcontroller) belongsToThisRoutine(key string, count uint32) bool {
    val := cc.genHash(key)
    return val % cc.workers == count
}

func (cc *jobcontroller) getWorkerQueue(key string) workqueue.TypedRateLimitingInterface[any] {
	val := cc.genHash(key)
	queue := cc.queueList[val%cc.workers]
	return queue
}

// genHash source code
func (cc *jobcontroller) genHash(key string) uint32 {
    hashVal := fnv.New32() // FNV-1a non-cryptographic hash
    hashVal.Write([]byte(key))
    return hashVal.Sum32()
}

FNV (Fowler–Noll–Vo) is a fast, low-collision non-cryptographic hash function that serves as a consistent hashing mechanism in Volcano:

Serializing Single-Job Operations: It ensures that the same JobKey is always routed to the same worker, preventing race conditions (like version conflicts or duplicate pod creation) that could arise from multiple goroutines modifying the same Job state concurrently.
Load Balancing: It distributes different Jobs evenly across the workers queues, improving parallelism.

❓ What if “same Job → same goroutine” is not guaranteed? * Multiple goroutines could enter the state machine for the same Job simultaneously, causing Status conflicts (retries due to ResourceVersion mismatch, optimistic lock failures). * Duplicate creation of Pods / PodGroups, leading to resource leaks and Gang Scheduling failures. * Without this mechanism, a global lock or fine-grained optimistic retries would be necessary, which would be less efficient.

PodGroup Creation

// pkg/controllers/job/job_controller_actions.go:190-214
func (cc *jobcontroller) createOrUpdatePodGroup(job *batch.Job) error {
    ...
    pg := &scheduling.PodGroup{ ... }
    vcClient.SchedulingV1beta1().PodGroups(...).Create(..., pg, ...)
    ...
}

A PodGroup is created with a single API call, and each Job has only one PodGroup. The creation itself is relatively simple (it’s just a logical unit) and is unlikely to be a major bottleneck: * A PodGroup is essentially a CRD object (with a Spec & Metadata of only a few dozen bytes, as defined in pkg/controllers/job/job_controller_actions.go). The creation process is just a single write operation from kube-apiserver to etcd. * It does not involve scheduling decisions, node communication, or resource calculations. It returns immediately upon success without subsequent long-running processes.

However, it’s worth noting that if MinMember is set too high or Queue resources are insufficient, the scheduler may still block Job startup in a later stage because the PodGroup conditions are not met. This is a scheduling issue, not a CREATE issue.

Pod Creation

// pkg/controllers/job/job_controller_actions.go
func (cc *jobcontroller) syncJob(jobInfo *apis.JobInfo, updateStatus state.UpdateStatusFn) error {
    ...
    waitCreationGroup := sync.WaitGroup{}
    ...
	var podToCreateEachTask []*v1.Pod
	for _, ts := range job.Spec.Tasks {
        for i := 0; i < int(ts.Replicas); i++ {           // Collect pods to be created
            ...
            newPod := createJobPod(job, tc, ts.TopologyPolicy, i, jobForwarding)
            ...
            podToCreateEachTask = append(podToCreateEachTask, newPod)
            waitCreationGroup.Add(1)
            ...
        }
        podToCreate[ts.Name] = podToCreateEachTask
    }
    ...
	for taskName, podToCreateEachTask := range podToCreate {
        ...
        go func(taskName string, podToCreateEachTask []*v1.Pod) {
            ...
            for _, pod := range podToCreateEachTask {
                go func(pod *v1.Pod) {
                    defer waitCreationGroup.Done()
                    kubeClient.CoreV1().Pods(...).Create(...)
                }(pod)
            }
            ...
        }(taskName, podToCreateEachTask)
    }
    ...
    waitCreationGroup.Wait()  // ⬅ Blocking: does not return until the entire batch is complete
    ...
}

Observation: All pods within a Job must be created in the same batch. The Wait() call blocks the worker goroutine from servicing other Jobs, becoming a performance bottleneck.

Unlike PodGroups, Pods are native K8s objects with many complex fields. They require more default field population and take longer to write to etcd, making them a more likely bottleneck.

4.2.2 Experimental Design for Worker Thread Optimization

Experiment Setup: - Worker Threads: 1, 5, 10, 25, 50, 100, 150, 200, 400, 600 - Job and Pod Combinations: 10000×1, 5000×2, 2000×5, 1000×10, 500×20, 200×50, 100×100, 50×200, 20×500, 1×10000 - Metrics: Time from test start to the completion of the last CREATE and SCHEDULE event.

4.2.3 Analysis of Experimental Results

The experiments led to two key conclusions:

Impact of Increasing Job Count with the Same Number of Worker Threads: - With a fixed number of worker threads, as the number of Jobs increased, the CREATE time generally increased. - This confirms that a higher number of Jobs leads to a larger computational load on the Controller. Furthermore, the CREATE requests are fragmented into smaller pieces, exacerbating contention with SCHEDULE requests.

Impact of Increasing Worker Threads with the Same Job Count: - With a fixed number of Jobs, as the number of Controller worker threads increased, the CREATE time showed a distinct “V-shaped” curve: it first decreased, then increased.

Analysis of the V-Shaped Performance Curve:

Too Few Threads (Left Side of the V): Fails to fully utilize computing resources, with I/O waits becoming the bottleneck.
- The bottleneck is in API-Server communication and remote etcd I/O.
- Computing resources are underutilized. A small number of threads quickly finish their computations, send CREATE requests, and then block.
- Increasing the number of threads here can better utilize computing resources and eliminate bubble time spent waiting on communication/I/O.
Too Many Threads (Right Side of the V): Exacerbates CPU and API-Server contention, leading to a performance decrease.
- The bottleneck becomes the computing resources themselves, plus increased cross-job Pod queuing and blocking.
- A large number of threads have already saturated all computing resources. Pods from different jobs now compete for creation in a fair-queuing manner.
- The last few pods in a job block the entire job, which makes the surge-and-stall phenomenon more severe.

4.3 Hypothesis 3: Contention Between Controller and Scheduler

We observed that the stair-step stalls for CREATE (performed by the Controller) and SCHEDULE (performed by the Scheduler) never occurred simultaneously. Further investigation revealed that the root cause is contention from two independent components concurrently writing to a shared resource, the K8s API-Server/ETCD. Short‑term queuing/retries manifest macroscopically as alternating execution.

4.3.1 Analysis of Independence at the Source Code Level

A careful analysis of the Volcano source code shows that the CREATE functionality belongs to the Controller component, while the SCHEDULE functionality belongs to the Scheduler component. There are no direct calls, dependencies, or other relationships between them at the code level:

// pkg/controllers/job/job_controller_actions.go
func (cc *jobcontroller) syncJob(jobInfo *apis.JobInfo, updateStatus state.UpdateStatusFn) error {
    // Controller is responsible for Pod creation
    for _, pod := range podToCreateEachTask {
        go func(pod *v1.Pod) {
            defer waitCreationGroup.Done()
            newPod, err := cc.kubeClient.CoreV1().Pods(pod.Namespace).Create(context.TODO(), pod, metav1.CreateOptions{})
            // ...
        }(pod)
    }
    waitCreationGroup.Wait()
}

// pkg/scheduler/actions/allocate/allocate.go
func (alloc *Action) Execute(ssn *framework.Session) {
    // Scheduler is responsible for Pod scheduling
    for _, task := range tasks {
        // Scheduling decision and binding
        ssn.Bind(task, node)
    }
}

4.3.2 Analysis of Contention at the API-Server Level

The real contention occurs at the K8s API-Server level. After completing their respective operations, both components must submit their results (creating a Pod / updating a Pod’s status) to the K8s API-Server, which then handles writing to etcd. When both components operate on the same Pod object simultaneously, optimistic concurrency control triggers short‑term retries/queuing. Under scale, this ultimately manifests as an alternating execution pattern where one works while the other waits.

Request Flow Analysis:

// Controller's Pod creation request flow
JobController.syncJob() 
  → kubeClient.CoreV1().Pods().Create() 
    → kube-apiserver 
      → etcd write

// Scheduler's Pod scheduling request flow
VolcanoScheduler.allocate() 
  → ssn.Bind() 
    → kubeClient.CoreV1().Pods().Update() 
      → kube-apiserver 
        → etcd write

4.3.3 In-depth Analysis of Queuing and Retry Mechanisms

Impact of Job Count on Request Patterns:

Request Fragmentation: When the number of jobs is high, a single batch of 10,000 Pod CREATE requests gets fragmented into many smaller batches.
Contention and Interleaving: Time gaps appear between these small batches of CREATE requests. SCHEDULE (Pod Update) requests sent by the Scheduler “fill in” these gaps.

Amplifying Effect of Goroutine Blocking and Request Retries:

The Controller processes jobs one at a time per worker. A worker goroutine waits for all pods within one job to be created before moving on to the next. During this process, if some pods in a job are blocked, it has an amplifying effect, blocking the entire goroutine. Furthermore, resource updates for a series of pods belonging to the same job will trigger updates to the same PodGroup or even the Job object, increasing the likelihood of transient retries and short stalls.

// pkg/controllers/job/job_controller_actions.go
func (cc *jobcontroller) syncJob(jobInfo *apis.JobInfo, updateStatus state.UpdateStatusFn) error {
    // ...
    waitCreationGroup := sync.WaitGroup{}
    // Collect all pods to be created
    for _, ts := range job.Spec.Tasks {
        for i := 0; i < int(ts.Replicas); i++ {
            // ...
            waitCreationGroup.Add(1)
        }
    }
    
    // Create pods concurrently
    for taskName, podToCreateEachTask := range podToCreate {
        go func(taskName string, podToCreateEachTask []*v1.Pod) {
            for _, pod := range podToCreateEachTask {
                go func(pod *v1.Pod) {
                    defer waitCreationGroup.Done()
                    // API-Server call
                    newPod, err := cc.kubeClient.CoreV1().Pods(pod.Namespace).Create(context.TODO(), pod, metav1.CreateOptions{})
                }(pod)
            }
        }(taskName, podToCreateEachTask)
    }
    
    // ⚠️ Key blocking point: waits for all pods to be created
    waitCreationGroup.Wait()
}

4.3.4 Experimental Evidence

Direct evidence of resource contention was found during testing:

Under multi-job workloads, the “stair-step” patterns in the cumulative pod count curves for Volcano’s Pod creation (CREATE) and scheduling (SCHEDULE) did not occur at the same time. In the per-second throughput curves, the two alternated between performance peaks and troughs but were never in a trough simultaneously.

4.4 Other Hypotheses: Ruled-Out Factors

We also hypothesized and tested other factors. Experiments showed that the enqueue scheduling phase and Volcano version differences were not the core factors causing the performance issues.

4.4.1 Verification of Enqueue Functionality

Experimental Analysis: enqueue is an important stage in the Volcano scheduler’s workflow, primarily responsible for:

Job Enqueue Management: Adding submitted Jobs to the scheduling queue.
Priority Sorting: Sorting Jobs based on factors like priority and submission time.
Resource Pre-check: Preliminary check to see if cluster resources meet Job requirements.
Queue Capacity Control: Managing queue capacity limits and admission control.

The complete scheduling flow of the Volcano scheduler is as follows:

graph LR
    A[Job Submission] --> B[enqueue]
    B --> C[allocate]
    C --> D[backfill]
    D --> E[reclaim]
    E --> F[preempt]
    
    B1[enqueue stage] --> B2[Queue Management]
    B1 --> B3[Priority Sorting]
    B1 --> B4[Resource Pre-check]
    
    C1[allocate stage] --> C2[Resource Allocation]
    C1 --> C3[Node Selection]
    C1 --> C4[Pod Binding]

enqueue could potentially impact scheduling performance by restricting pod creation based on a preliminary resource check: it checks for sufficient resources before pod creation and may limit the creation rate of new pods if resources are scarce. This could cause the CREATED event curve to stagnate and become a bottleneck in some scenarios.

Experimental Design: We disabled the enqueue functionality to test this hypothesis.

# Modify scheduler configuration to remove the enqueue action
actions: "allocate, backfill, reclaim"  # Original: actions: "enqueue, allocate, backfill"

Experimental Result: After disabling enqueue, the test results were largely consistent with previous local tests. The periodic surges in CREATED events still occurred, and there was no significant improvement in scheduling performance. The enqueue action is mainly responsible for task queuing and priority sorting, and its direct impact on pod creation and scheduling is limited.

4.4.2 Verification of Version Differences

Experimental Design: We noticed that the latest Volcano version was newer than our test version and suspected this might cause performance differences. We tested this by upgrading Volcano from v1.11.0 to v1.12.0-alpha.0:

# Batch replace version numbers
sed -i 's/v1\.11\.0/v1.12.0-alpha.0/g' schedulers/volcano/*/deployment.yaml
sed -i 's/v1\.11\.0/v1.12.0-alpha.0/g' schedulers/volcano/*/job.yaml

Experimental Result: After upgrading to v1.12.0-alpha.0, the test results were consistent with previous local tests. The abnormal phenomena in CREATED events persisted, and there was no significant improvement in scheduling performance. While version upgrades may bring some improvements, the core performance bottlenecks remained.

5. Tuning Guide and Future Outlook

5.1 Short-Term Optimization Solutions

Adjust Webhook Timeout: In the deployment for volcano-admission-service, increase the webhook’s timeoutSeconds from the default of 10 to 30.
Increase Webhook Resources: Allocate more CPU and Memory resources to the volcano-admission Pod (e.g., refer to “Huawei Cloud - Volcano Scheduler - Recommended Resources” and use limits: cpu: "2500m", memory: "4Gi" for clusters with 1000 nodes or more).

The resource quota for the volcano-admission component depends on the cluster scale, as shown in Table 1. The resource quotas for the volcano-controller and volcano-scheduler components are related to the cluster’s node and Pod scale, with the following recommendations: - For clusters with fewer than 100 nodes, the default configuration can be used: CPU request of 500m and limit of 2000m; Memory request of 500Mi and limit of 2000Mi. - For clusters with more than 100 nodes, for every additional 100 nodes (or 10,000 pods), it is recommended to increase the CPU request by 500m and memory request by 1000Mi. The CPU limit should be 1500m higher than the request, and the memory limit should be 1000Mi higher than the request. Note: Recommended calculation formulas for requests: - CPU Request: Calculate the value of “target node count * target Pod scale” and find the closest value in Table 2 via interpolation, rounding up to the nearest specification for request and limit values. For example, in a scenario with 2000 nodes and 20k pods, “target node count * target Pod scale” is 4000w. The closest higher spec is 700/7w (“cluster node count * Pod scale” equals 4900w), so the recommended CPU request is 4000m and the limit is 5500m. - Memory Request: It is recommended to allocate 2.4G of memory per 1000 nodes and 1G of memory per 10k pods. These values should be added together. That is: Memory Request = (Target Node Count / 1000) * 2.4G + (Target Pod Scale / 10k) * 1G. For example, in a 2000-node, 20k-pod scenario, the memory request would be 2 * 2.4G + 2 * 1G = 6.8G.
Configure Controller Worker Threads Reasonably: Adjust the --worker-threads startup parameter for volcano-controller-manager based on cluster size and Job shape. For scenarios with a large number of Jobs (>500), it is recommended to increase this value (e.g., to 100-200).

5.2 Long-Term Optimization Solutions

Replace Webhook: Consider moving some of the Webhook’s validation logic down into the Controller or using K8s CRD Validation Rules (CEL) to reduce RPC overhead.
Optimize Controller and Scheduler Interaction: Design a dynamic rate-matching mechanism to balance the CREATE and SCHEDULE rates. This would achieve global optimization rather than blindly improving the performance of a single stage.

HyperNode Auto Discovery User Guide

Tue, 03 Feb 2026 00:00:00 +0000

Introduction

This document describes how to use the HyperNode network topology auto-discovery feature in Volcano. This feature automatically discovers the network topology within the cluster and creates and maintains HyperNode custom resources (CRs) based on the discovered information. The Volcano scheduler leverages these HyperNode CRs for scheduling decisions, eliminating the need for users to manually maintain HyperNode information.

Prerequisites

Please Install Volcano with version >= v1.12.0 first.

Configuration

The HyperNode network topology discovery feature is configured via a ConfigMap. The ConfigMap contains the configuration for the discovery sources, such as UFM, RoCE, and label, you can modify the configuration according to your own cluster environments. Please note that you should replace with your Volcano namespace if Volcano is not installed in the default namespace.

Secret Configuration (Required First Step)

Before configuring the UFM discovery, you must first create a Kubernetes Secret to store your UFM credentials:

kubectl create secret generic ufm-credentials \
  --from-literal=username='your-ufm-username' \
  --from-literal=password='your-ufm-password' \
  -n volcano-system

Note: Replace your-ufm-username and your-ufm-password with your actual UFM credentials, and adjust the namespace if needed.

Example ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: volcano-controller-configmap
  namespace: volcano-system # Replace with your Volcano namespace if Volcano is not installed in the default namespace.
data:
  volcano-controller.conf: |
    networkTopologyDiscovery:
      - source: ufm
        enabled: true
        interval: 10m
        credentials:
          secretRef:
            name: ufm-credentials # Replace with the secret name that stores the UFM credentials.
            namespace: volcano-system #Replace with the secret namespace that stores the UFM credentials.
        config:
          endpoint: https://ufm-server:8080
          insecureSkipVerify: true
      - source: roce
        enabled: false
        interval: 15m
        config:
          endpoint: https://roce-server:9090
      - source: label
        enabled: true
        config:
          networkTopologyTypes:
            topologyA2:
              - nodeLabel: "volcano.sh/tor" # The label that indicates which tor a node belongs to. If the values corresponding to this label on different nodes are the same, it means these nodes belong to the same tor. 
              - nodeLabel: "kubernetes.io/hostname" # A standard label automatically added to each node in a Kubernetes cluster, used to identify the hostname of the node.
            topologyA3:
              - nodeLabel: "volcano.sh/hypercluster" # The label that indicates which hypercluster a node belongs to. If the values corresponding to this label on different nodes are the same, it means these nodes belong to the same hypercluster.
              - nodeLabel: "volcano.sh/hypernode" # The label that indicates which hypernode a node belongs to. If the values corresponding to this label on different nodes are the same, it means these nodes belong to the same hypernode.
              - nodeLabel: "kubernetes.io/hostname" # A standard label automatically added to each node in a Kubernetes cluster, used to identify the hostname of the node.

Configuration Options

source: The discovery source. Supported values are ufm, roce, and label.
enabled: Whether the discovery source is enabled.
interval: The interval between discovery operations. If not specified, the default value is 1 hour.
config: The configuration for the discovery source. The configuration options vary depending on the discovery source.
credentials: The credentials configuration for accessing the discovery source.
- secretRef: Reference to a Kubernetes Secret containing credentials.
  - name: The name of the Secret.
  - namespace: The namespace of the Secret.

UFM Configuration Options

endpoint: The UFM API endpoint.
insecureSkipVerify: Whether to skip TLS certificate verification. This should only be used in development environments.

RoCE Configuration Options(Currently not supported)

endpoint: The RoCE API endpoint.
token: The RoCE API token.

Label Configuration Options

networkTopologyTypes: The structure that supports different types of network topologies, including those for GPU, NPU, etc. Below is an example of the NPU cluster network topology.
- topologyA2: The network topology type of A2(Ascend 910B) cluster.
  - nodeLabel: For the labels on a node, when there are multiple labels, hypernodes are constructed from bottom to top. The bottommost label is kubernetes.io/hostname, which is a standard built-in label key in Kubernetes, and the label above it is volcano.sh/tor, indicates which tor a node belongs to.
- topologyA3: The network topology type of A3(Ascend 910C) cluster.
  - nodeLabel: For the labels on a node, when there are multiple labels, hypernodes are constructed from bottom to top. The bottommost label is kubernetes.io/hostname, which is a standard built-in label key in Kubernetes, and the label above it is volcano.sh/hypernode and volcano.sh/hypercluster, volcano.sh/hypernode indicates which hypernode a node belongs to, volcano.sh/hypercluster indicates which hypercluster a node belongs to.

    tier2                     s4                                 s5                         
                      /               \                   /              \                 
    tier1           s0                s1                 s2              s3              
                 /      \          /      \           /      \        /      \         
              node0    node1    node2    node3      node4   node5   node6   node7

       The labels of each node in the cluster:
        node0:   kubernetes.io/hostname=192.168.1.10 # Node Ip
                 volcano.sh/hypernode=s0             # HyperNode Name
                 volcano.sh/hypercluster=s4          # HyperCluster Name
        node1:   kubernetes.io/hostname=192.168.1.11           
                 volcano.sh/hypernode=s0             
                 volcano.sh/hypercluster=s4          
        node2:   kubernetes.io/hostname=192.168.1.12 
                 volcano.sh/hypernode=s1             
                 volcano.sh/hypercluster=s4          
        node3:   kubernetes.io/hostname=192.168.1.13 
                 volcano.sh/hypernode=s1             
                 volcano.sh/hypercluster=s4          
        node4:   kubernetes.io/hostname=192.168.1.14 
                 volcano.sh/hypernode=s2
                 volcano.sh/hypercluster=s5 
        node5:   kubernetes.io/hostname=192.168.1.15 
                 volcano.sh/hypernode=s2
                 volcano.sh/hypercluster=s5 
        node6:   kubernetes.io/hostname=192.168.1.16 
                 volcano.sh/hypernode=s3
                 volcano.sh/hypercluster=s5
        node7:   kubernetes.io/hostname=192.168.1.17 
                 volcano.sh/hypernode=s3
                 volcano.sh/hypercluster=s5

Verification

Check the Volcano controller logs to ensure that the discovery sources are started successfully.

kubectl logs -n volcano-system -l app=volcano-controllers -c volcano-controllers | grep "Successfully started all network topology discoverers"

Check the created HyperNode resources.

kubectl get hypernodes -l volcano.sh/network-topology-source=<source>

Replace <source> with the discovery source you configured, such as ufm or label.

Troubleshooting

If the discovery sources are not started successfully, check the Volcano controller logs for errors.
If the HyperNode resources are not created, check the discovery source configuration and ensure that the discovery source is able to connect to the network topology data source.

Best Practices

Volcano uses Kubernetes-standard Secrets to store sensitive credential information (username/password or token). For more stringent key encryption requirements, users should consider additional mechanisms like Encrypting Secret Data at Rest.
The credential Secrets can be placed in a specified namespace for better isolation.
For UFM discoverer, the controller only needs read access to the specific Secret containing credentials.
For label discoverer, the controller needs to pre-label the nodes with the tags corresponding to the hypernodes.
When deploying in production environments, proper RBAC policies should be configured to limit access to Secrets.
TLS certificate verification should be enabled in production environments to prevent MITM attacks.
Monitor the Volcano controller logs for errors.
Set a reasonable discovery interval to avoid overloading the network topology data source.

MPI Plugin User Guide

Tue, 03 Feb 2026 00:00:00 +0000

Introduction

MPI plugin is designed to optimize the user experience when running MPI jobs, it not only allows users to write less yaml, but also ensures the normal operation of MPI jobs.

How the MPI Plugin Works

The MPI plugin will do three things:

Open ports used by MPI for all containers of the job
Force open ssh and svc plugins
add MPI_HOST environment variable for master pod, this environment variable includes the worker’s domain name, It is used by the --host parameter of mpiexec

Parameters of the MPI Plugin

Key Points

If master or worker is configured, please ensure that the tasks corresponding to their values exist, and the roles of these tasks correspond to the meaning of the parameters
If port is configured, make the port value of sshd the same as the value of the parameter.
If the gang plugin is enabled, then make sure that the value of minAvailable is equal to the number of replicas of the worker.

Arguments

ID	Name	Type	Default Value	Required	Description	Example
1	master	string	master	No	Name of MPI master	–master=mpimaster
2	worker	string	worker	No	Name of MPI worker	–worker=mpiworker
3	port	string	22	No	The port to open for the container	–port=5000

Examples

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: lm-mpi-job
spec:
  minAvailable: 1
  schedulerName: volcano
  plugins:
    mpi: ["--master=mpimaster","--worker=mpiworker","--port=22"]  ## MPI plugin register
  tasks:
    - replicas: 1
      name: mpimaster
      policies:
        - event: TaskCompleted
          action: CompleteJob
      template:
        spec:
          containers:
            - command:
                - /bin/sh
                - -c
                - |
                  mkdir -p /var/run/sshd; /usr/sbin/sshd;
                  mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world;
              image: volcanosh/example-mpi:0.0.3
              name: mpimaster
              workingDir: /home
          restartPolicy: OnFailure
    - replicas: 2
      name: mpiworker
      template:
        spec:
          containers:
            - command:
                - /bin/sh
                - -c
                - |
                  mkdir -p /var/run/sshd; /usr/sbin/sshd -D;
              image: volcanosh/example-mpi:0.0.3
              name: mpiworker
              workingDir: /home
          restartPolicy: OnFailure

Network Topology Aware Scheduling User Guide

Tue, 03 Feb 2026 00:00:00 +0000

1 Background

In the context of AI large model training, Model Parallelism divides the model across multiple nodes, requiring frequent and substantial data exchange between these nodes during training. At this point, the network transmission performance between nodes often becomes the bottleneck of training, significantly affecting training efficiency. Data centers have diverse network types (such as IB, RoCE, NVSwitch, etc.), and the network topology is complex, typically involving multiple layers of switches. The fewer switches between two nodes, the lower the communication latency and the higher the throughput. Therefore, users want to schedule workloads to the best performance domain with the highest throughput and lowest latency, minimizing cross-switch communication to accelerate data exchange and improve training efficiency.

To address this, Volcano proposed the Network Topology Aware Scheduling strategy, which uses a unified network topology API and intelligent scheduling policies to solve the network communication performance issues in large-scale data center AI training tasks.

2 Features

2.1 Unified Network Topology API: Accurately Expressing Network Topology

To shield the differences in data center network types, Volcano defines a new CRD HyperNode to represent the network topology, providing a standardized API interface. Compared to the traditional method of using node labels to represent network topology, HyperNode has the following advantages:

Unified Semantics: HyperNode provides a standardized way to describe network topology, avoiding the semantic inconsistency issues of the label method.
Hierarchical Structure: HyperNode supports a tree-like hierarchical structure, allowing for more precise representation of the actual network topology.
Easy Management: Cluster administrators can manually create HyperNodes or use network topology auto-discovery tools to maintain HyperNodes.

A HyperNode represents a network topology performance domain, typically mapped to a switch or tor. Multiple HyperNodes are connected hierarchically to form a tree structure. For example, the following diagram shows a network topology composed of multiple HyperNodes:

In this structure, the communication efficiency between nodes depends on the HyperNode hierarchy span between them. For example:

node0 and node1 belong to s0, achieving the highest communication efficiency.
node1 and node2 need to cross two layers of HyperNodes (s0→s4→s1), resulting in lower communication efficiency.
node0 and node4 need to cross three layers of HyperNodes (s0→s4→s6), resulting in the worst communication efficiency.

2.2 HyperNode Auto Discovery: Simplify Network Topology Management

To further reduce the management burden of network topology information, Volcano provides the HyperNode auto-discovery feature. This feature automatically discovers network topology structures within clusters and creates, updates, or deletes corresponding HyperNode Custom Resources (CRs) based on the discovery results.

The auto-discovery feature offers the following key benefits:

Automated Management: Automatically discovers and maintains HyperNode information from various data sources (such as UFM, RoCE, or node labels), eliminating the need for manual maintenance.
Real-time Updates: Periodically synchronizes network topology changes to ensure HyperNode information remains current with the actual network state.
Extensible Architecture: Supports pluggable Discoverer components, allowing users to develop custom discovery logic for their specific network management tools.

Through this automated discovery mechanism, users can focus on job scheduling configuration without worrying about the complexities of HyperNode creation and maintenance, significantly simplifying the deployment and management of network topology-aware scheduling.

2.3 Network Topology Constraints: Improve Network Communication Efficiency

In Volcano Jobs, the NetworkTopology field can be configured to describe the network topology constraints during job deployment. The specific constraint configuration is as follows:

mode: Supports hard and soft modes.
- hard: Hard constraint, tasks within the job must be deployed within the same HyperNode.
- soft: Soft constraint, tasks are deployed within the same HyperNode as much as possible.
highestTierAllowed: Used with hard mode, indicating the highest tier of HyperNode allowed for job deployment. This field is not required when mode is soft.

For example, the following configuration means the job can only be deployed within HyperNodes of tier 2 or lower, such as s4 and s5, and their child nodes s0, s1, s2, s3. Otherwise, the job will remain in the Pending state:

spec:
  networkTopology:
    mode: hard
    highestTierAllowed: 2

By configuring this scheduling constraint, users can precisely control the network topology constraints of the job, ensuring that the job runs in the best performance domain that meets the conditions, thereby significantly improving training efficiency.

2.4 SubGroup Affinity Policy: Fine-grained Control of Scheduling Constraints for Distributed Jobs

In large model training scenarios, the entire training task requires enormous resources, which typically cannot be deployed within a single network performance domain. It’s necessary to split the training task into pipeline parallelism (PP) or data parallelism (DP), allowing parallel tasks to be deployed across network performance domains.

To address this, Volcano provides the SubGroup Affinity Policy. In a Volcano Job, the partitionPolicy field can be configured to partition Pods within a Job, and network topology constraints can be configured for each partition.

During scheduling, each partition will follow its own network topology constraints, thereby meeting the network communication performance requirements of each parallel task partition.

In addition, each partition must follow the Gang scheduling constraints. That is, a partition is allowed to be scheduled only when all pods within it meet the scheduling conditions.

For example, the following configuration indicates that the entire job is only allowed to be scheduled to HyperNodes at Tier 2 or below. And within the job, the 8 Pods are divided into 2 partitions, and each partition can only be scheduled to HyperNodes at Tier 1.

spec:
  networkTopology:
    mode: hard
    highestTierAllowed: 2
  tasks:
    - name: "task1"
      replica: 8
      partitionPolicy:
        totalPartitions: 2
        partitionSize: 4
        networkTopology:
          mode: hard
          highestTierAllowed: 1
      template:
      # pod template

2.5 HyperNode-Level Bin Packing: Improve Resource Utilization for Network Topology Performance Domain

When scheduling workloads with network topology constraints, the scheduler will prioritize scheduling them to HyperNodes with higher current resource utilization to improve resource utilization in the network performance domain.

As shown in the diagram below, suppose there are two Tier 1 HyperNodes in the cluster, where HyperNode0 already has some resources occupied by other tasks. A user issues a Volcano Job, dividing it into two partitions, each configured with highestTierAllowed=1.

Without HyperNode-level bin packing, the scheduling result might be that partition 0 is scheduled to HyperNode0, and partition 1 is scheduled to HyperNode1.
With HyperNode-Level bin packing, both partition 0 and 1 will be prioritized for scheduling to HyperNode0. In this case, HyperNode1 can be used as a fully idle HyperNode for other tasks.

Furthermore, for workloads without network topology constraints, the scheduler will prioritize scheduling them to nodes with higher hypernode-level resource utilization (where each tier of hypernodes will be considered), in order to reduce the hypernode-level resource fragmentation.

For example, as shown in the figure below, there is a cluster consisting of 8 nodes and 7 hypernodes. Currently, the resources of node0, node2 and node4 are occupied by existing tasks, while those of the other nodes are idle. Then, at this point, a user submits a volcano job with two independent pods to this cluster.

Without HyperNode-level bin packing, the scheduler may assign these two pods to node1 and node6, or node3 and node7, making the hypernode-level resource fragmentation more severe.
With HyperNode-level bin packing, the scheduler will prefer to assign them to node1 and node3, leaving node5, node6 and node7 together for other larger network-topology-constrained workloads.

3 User Guide

3.1 Installing Volcano

Refer to Install Guide to install Volcano.

After installed, update the scheduler configuration:

kubectl edit cm -n volcano-system volcano-scheduler-configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack
      - name: network-topology-aware # Add it to enable network-topology-aware plugin
        arguments:
          weight: 10
          hypernode.binpack.cpu: 5                                     # HyperNode-level bin packing weight for CPU
          hypernode.binpack.memory: 1                                  # HyperNode-Level bin packing weight for memory
          hypernode.binpack.resources: nvidia.com/gpu, example.com/foo # Custom resource names to be considered by the bin packing strategy
          hypernode.binpack.resources.nvidia.com/gpu: 2                # HyperNode-Level bin packing weight for "nvidia.com/gpu" resources
          hypernode.binpack.resources.example.com/foo: 3               # HyperNode-Level bin packing weight for "example.com/foo" resources
          hypernode.binpack.normal-pod.enable: true                    # Whether or not to enable HyperNode-level bin packing for normal pods
          hypernode.binpack.normal-pod.fading: 0.8                     # Parameter to control the weights of hypernodes of different tiers, i.e., the weights of hypernodes of tier `i` are math.Pow(fading, i-1)

3.2 Building Network Topology

3.2.1 Build via HyperNode Auto-Discovery (Recommended)

Refers to How to Use HyperNode Auto Discovery

3.2.2 Build Manually

You can build Network Topology by creating HyperNode CRs manually. Refer to the definition of CRD hypernodes.topology.volcano.sh for more details.

3.3 Deploy Workloads Using Network Topology Aware Scheduling

Based on the following network topology, this chapter will demonstrate how workloads can be deployed using Network Topology Aware Scheduling.

3.3.1 Deploying Using Volcano Job With Network Topology Constraints Configured

Create a Volcano job where each Pod requests the full CPU resources of one node. Example follows:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: network-topology-job
 namespace: default
spec:
 schedulerName: volcano
 minAvailable: 4
 networkTopology:
   mode: hard
   highestTierAllowed: 1
 tasks:
   - name: t0
     replicas: 8
     template:
       spec:
         containers:
           - name: c0
             image: nginx:latest
             resources:
               requests:
                 cpu: "4"
               limits:
                 cpu: "4"

The scheduling result is as follows:

$ kubectl get pod -owide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE
network-topology-job-t0-0   1/1     Running   0          5s    192.168.0.10   node4
network-topology-job-t0-1   1/1     Running   0          5s    192.168.0.11   node5
network-topology-job-t0-2   1/1     Running   0          5s    192.168.0.12   node6
network-topology-job-t0-3   1/1     Running   0          5s    192.168.0.13   node7
network-topology-job-t0-4   0/1     Pending   0          5s    <none>         <none>
network-topology-job-t0-5   0/1     Pending   0          5s    <none>         <none>
network-topology-job-t0-6   0/1     Pending   0          5s    <none>         <none>
network-topology-job-t0-7   0/1     Pending   0          5s    <none>         <none>

In this example, Pods within the Job cannot be scheduled across Tier 1 HyperNodes. The Job is scheduled to HyperNode1, which only has 4 nodes (Node4~Node7), so only 4 Pods are running.

3.3.2 Deploying Using Volcano Job With Network Topology Constraints And SubGroup Affinity Policy Configured

Create a Volcano job where each Pod requests the full CPU resources of one node; and configure a SubGroup Affinity Policy to divide the 8 Pods within the Job into 2 partitions. Example follows:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: network-topology-job
 namespace: default
spec:
 schedulerName: volcano
 minAvailable: 4
 networkTopology:
   mode: hard
   highestTierAllowed: 2
 tasks:
   - name: t0
     replicas: 8
     partitionPolicy:
       totalPartitions: 2
       partitionSize: 4
       networkTopology:
         mode: hard
         highestTierAllowed: 1
     template:
       spec:
         containers:
           - name: c0
             image: nginx:latest
             resources:
               requests:
                 cpu: "4"
               limits:
                 cpu: "4"

The scheduling result is as follows:

$ kubectl get pod -owide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE
network-topology-job-t0-0   1/1     Running   0          5s    192.168.0.10   node2
network-topology-job-t0-1   1/1     Running   0          5s    192.168.0.11   node3
network-topology-job-t0-2   1/1     Running   0          5s    192.168.0.12   node1
network-topology-job-t0-3   1/1     Running   0          5s    192.168.0.13   node0
network-topology-job-t0-4   1/1     Running   0          5s    192.168.0.14   node4
network-topology-job-t0-5   1/1     Running   0          5s    192.168.0.15   node5
network-topology-job-t0-6   1/1     Running   0          5s    192.168.0.16   node6
network-topology-job-t0-7   1/1     Running   0          5s    192.168.0.17   node7

In this example, the entire Job is scheduled to HyperNode2 (Node0~Node7). The first partition (Pod0~Pod3) is scheduled to HyperNode0 (Node0~Node3), the second partitions (Pod4~Pod7) is scheduled to HyperNode1 (Node4~Node7), and each partition satisfies its own network topology constraints.

3.3.3 Deploying Using Volcano Job Without Network Topology Constraints

Create a Volcano job with two independent pods, where each Pod requests the full CPU resources of one node. Example follows:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: network-topology-job
 namespace: default
spec:
 schedulerName: volcano
 minAvailable: 2
 tasks:
   - name: t0
     replicas: 2
     template:
       spec:
         containers:
           - name: c0
             image: nginx:latest
             resources:
               requests:
                 cpu: "4"
               limits:
                 cpu: "4"

The scheduling result is as follows:
```
$ kubectl get pod -owide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE
network-topology-job-t0-0   1/1     Running   0          5s    192.168.0.10   node0
network-topology-job-t0-1   1/1     Running   0          5s    192.168.0.11   node1
```
Since the resources are all idle, the first pod, i.e., network-topology-job-t0-0, will be scheduled to any node. It is node0 in this example. Then, the second pod, i.e., network-topology-job-t0-1, must be scheduled to node1, node2 or node3. It is node1 in this example. This is because the resource utilization of HyperNode0 is currently higher than HyperNode1.

Volcano on Volcano

User Guide

Scheduling Gates Queue Admission User Guide

Overview

Problem

Solution

Prerequisites

1. Enable the Feature Gate

Using Helm

Using kubectl apply

2. Configure the Capacity Plugin

3. Opt-in Pods

4. Verify the Feature is Working

Interaction with other Scheduling Gates

Limitations

Related

Ascend vNPU User Guide

Introduction

1. MindCluster mode

2. HAMi mode

Installation

Install Volcano:

Install ascend-device-plugin and third-party components

MindCluster Mode

Install Third-Party Components

Scheduler Config Update

HAMi mode

Label the Node with ascend=on

Deploy hami-scheduler-device ConfigMap

Deploy ascend-device-plugin

Scheduler Config Update

Usage

MindCluster mode

HAMi mode

Hami vNPU scene memory allocation restrictions

Capacity Plugin User Guide

Introduction

Environment setup

Install volcano

Config queue’s deserved resources

Submit pods to each queue

Cooldown Protection Plugin User Guide

Background

Environment setup

Install volcano

Update scheduler configmap

Running Jobs

Edit yaml of vcjob

Extender User Guide

Install volcano

1. Install from source

2. Deploy extender

3. Update Volcano configuration

Verify Extender is working

GPU Number User guide

Important Note

Environment setup

Install volcano

1. Install from source

2. Install from release package.

Install Volcano device plugin

Verify environment is ready

Running Jobs With Multiple GPU Cards

Understanding How Multiple GPU Cards Requirement Works

GPU Sharing User guide

Important Note

Environment setup

Install volcano

1. Install from source

2. Install from release package

Install Volcano device plugin

Verify environment is ready

Running GPU Sharing Jobs

Understanding how GPU sharing works

How to configure priorityclass for job

Background

Key Points

Example

How to Configure Scheduler

Requirements

Label the Node with `ascend=on`

Deploy `hami-scheduler-device` ConfigMap