https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/ Recent content in Introduction on Volcano Hugo -- gohugo.io en-us ©2025 Volcano Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/actions/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/actions/ Enqueue Overview The Enqueue action filters qualified jobs into the queue to be scheduled. When the minimum number of resource requests under a Job cannot be met, even if the scheduling action is performed for a pod under a Job, pod will not be able to schedule because the “Gang” constraint is not reached. A state refresh from “Pending” to “Inqueue” can only happen if the minimum resource size of the job is met. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/architecture/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/architecture/ Overall Architecture Application scenarios of Volcano Volcano is designed for high-performance workloads running on Kubernetes. It follows the design and mechanisms of Kubernetes. Volcano architecture Volcano consists of scheduler / controllermanager / admission / vcctl: Scheduler Volcano Scheduler schedules jobs to the most suitable node based on actions and plug-ins. Volcano supplements Kubernetes to support multiple scheduling algorithms for jobs. ControllerManager (CM) Volcano CMs manage the lifecycle of Custom Resource Definitions (CRDs). https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/cli/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/cli/ Introduction A Command Line Interface (CLI) is provided for you to manage resources. Configuration You can obtain the latest executable file by cloning the code from GitHub and running the following command in the root directory of the project: # make vcctl Copy the executable file to $PATH. You then can execute it anywhere. Command Line List Listing all jobs vcctl job list # vcctl job list Name Creation Phase JobType Replicas Min Pending Running Succeeded Failed Unknown RetryCount job-1 2020-09-01 Running Batch 1 1 0 1 0 0 0 0 Deleting a specific job vcctl job delete –name job-name [–namespace job-namespace] https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/membership/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/membership/ Volcano Community Membership Note: We regularly update this document with the latest information and incorporate feedback from the Volcano community. This document gives a brief overview of the Volcano community roles and the requirements and responsibilities associated with them. Role Requirements Responsibilities Privileges Member Sponsored by 2 approvers, active in the community, has contributed to Volcano Gets new contributors on board Member of the Volcano GitHub organization Approver Sponsored by 2 maintainers, has extensive experience and knowledge of a particular domain, has actively contributed to code and review Reviews and approves contributions from community members Has write access to specific packages in relevant repositories Maintainer Sponsored by 2 owners, has shown good technical judgment in feature design/development and PR review Release planning and feature development/maintenance Has top-level write access to relevant repositories; name entry in the Maintainers file of the repositories Owner Sponsored by 3 owners, has proven ability to lead the entire Volcano project Drives the overall technical roadmap of the project and sets priorities of activities in release planning Has admin access to the Volcano GitHub organization Note: All Volcano community members must follow the Volcano Code of Conduct. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/flink_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/flink_on_volcano/ Flink introduction Apache Flink is an open-source streaming framework developed by the Apache Software Foundation. At its core, Apache Flink is a distributed streaming data streaming engine written in Java and Scala. Flink executes any stream data program in data parallelism and pipelining. Flink’s pipelined runtime system can execute both batch and stream programs. In addition, the Flink runtime itself supports the execution of iterative algorithms. The premise condition Make sure the deployed Kubernetes, Kubectl, Volcano are installed correctly. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/installation/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/installation/ This document guides how to install the v1.9.0 version of version. Prerequisites Kubernetes 1.12+ with CRD support. Installation Modes Install with YAML files Install from code Install with Helm Install with YAML files Install Volcano on an existing Kubernetes cluster. This way is both available for x86_64 and arm64 architecture. kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/release-1.9/installer/volcano-development.yaml Install from code If you don’t have a Kubernetes cluster, try one-click install from code base. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/kubeflow_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/kubeflow_on_volcano/ Kubeflow introduction Kubernetes has become the de facto standard for cloud native application choreography and management, and more and more applications are migrating to Kubernetes. The field of artificial intelligence and machine learning naturally contains a large number of computation-intensive tasks. Developers are very willing to build an AI platform based on Kubernetes and make full use of the resource management, application scheduling, operation and maintenance monitoring capabilities provided by Kubernetes. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/mindspore_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/mindspore_on_volcano/ MindSpore introduction MindSpore is a new generation of deep learning framework launched by Huawei. It is derived from the best practices of the whole industry, which best matches the computing power of the rise processor, supports the flexible deployment of the whole scene of terminal, edge and cloud, and creates a brand new AI programming paradigm and lowers the threshold of AI development. MindSpore on Volcano Create a new mindSpore-cpu. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/mpi_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/mpi_on_volcano/ HPC introduction High Performance Computing (HPC) refers to the use of aggregated Computing power to handle data-intensive Computing tasks that cannot be performed by standard workstations. HPC = PBS + Maui + OpenMPI[1] PBS：Resource manager, which is responsible for managing resources for all nodes in the cluster Maui：Third-party task scheduler, support resource reservation, support various complex priority policies, support preemption, etc. OpenMPI：The upper communication environment, taking into account the functions of communication library, compilation and distributed start task. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/schduler_introduction/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/schduler_introduction/ Introduction Volcano scheduler is the component responsible for pod scheduling. It consists of a series of actions and plugins. Actions define the action that should be executed in every step. Plugins provide the action algorithm details in different scenarios. Volcano scheduler is highly scalable. You can specify and implement actions and plugins based on your requirements. Workflow Volcano scheduler workflow Volcano scheduler works as follows: Watches for and caches the jobs submitted by the client. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/pp_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/pp_on_volcano/ paddlepaddle introduction PaddlePaddle is a deep learning framework open source by Baidu in September 2016, aiming to provide a safe, efficient, flexible, easy to use and extensible deep learning platform. In October 2018, the Paddle Fluid 1.0 version was released by the Paddle Fluid team, which comprehensively upgraded core capabilities such as neural network description, large-scale distributed training, and high-performance inference engine. For example, in the latest Paddle Fluid 1. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/plugins/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/plugins/ Gang gang plugin Overview The Gang scheduling strategy is one of the core scheduling algorithms of the Volcano-Scheduler. It meets the scheduling requirements of “All or nothing” in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of Pod. The Gang scheduler algorithm is to observe whether the scheduled number of Pods under Job meets the minimum number of runs. When the minimum number of https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/podgroup/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/podgroup/ Introduction PodGroup is a group of pods with strong association and is mainly used in batch scheduling, for example, ps and worker tasks in TensorFlow. PodGroup is of a Custom Resource Definition (CRD) type. Example apiVersion: scheduling.volcano.sh/v1beta1 kind: PodGroup metadata: creationTimestamp: "2020-08-11T12:28:55Z" generation: 5 name: test namespace: default ownerReferences: - apiVersion: batch.volcano.sh/v1alpha1 blockOwnerDeletion: true controller: true kind: Job name: test uid: 028ecfe8-0ff9-477d-836c-ac5676491a38 resourceVersion: "109074" selfLink: /apis/scheduling.volcano.sh/v1beta1/namespaces/default/podgroups/job-1 uid: eb2508f5-3349-439c-b94d-4ac23afd71ff spec: minMember: 1 minResources: cpu: "3" memory: "2048Mi" priorityClassName: high-priority queue: default status: conditions: - lastTransitionTime: "2020-08-11T12:28:57Z" message: '1/0 tasks in gang unschedulable: pod group is not ready, 1 minAvailable. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/queue/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/queue/ Introduction Queue is a collection of PodGroups, which adopts FIFO. It is also used as the basis for resource division. Example apiVersion: scheduling.volcano.sh/v1beta1 kind: Queue metadata: creationTimestamp: "2020-08-10T11:54:36Z" generation: 1 name: default resourceVersion: "559" selfLink: /apis/scheduling.volcano.sh/v1beta1/queues/default uid: 14082e4c-bef6-4248-a414-1e06d8352bf0 spec: reclaimable: true weight: 1 capability: cpu: "4" memory: "4096Mi" status: state: Open Key Fields weight weight indicates the relative weight of a queue in cluster resource division. The resource allocated to the queue equals (weight/total-weight) x total-resource. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/referrals/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/referrals/ Background AI development process Deep learning generally involves four stages: data acquisition and processing, model training and evolution, model deployment, and model evaluation [1]. At the beginning of an enterprise’s involvement in deep learning, due to the lack of relevant experience and resources for deep learning, the team members usually choose to complete the above steps manually step by step in order to quickly meet the immediate needs of the project development period. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/spark_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/spark_on_volcano/ Spark introduction Spark is a fast and versatile big data clustering computing system. It provides high-level APIs for Scala, Java, Python, and R, as well as an optimization engine that supports a generic computational graph for data analysis. It also supports a rich set of advanced tools, including Spark SQL for SQL and Dataframes, MLLib for machine learning, GraphX for graphics processing, and Spark Streaming for Streaming. Spark on Volcano Currently, there are two ways to support the integration of Spark on Kubernetes and volcano. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/tf_on_volcano/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/tf_on_volcano/ TensorFlow introduction TensorFlow is a symbolic mathematical system based on data flow programming, which is widely used in programming and realization of various machine learning algorithms. Its predecessor is DistBelief, a neural network algorithm library of Google. TensorFlow on Volcano PS-worker model: Parameter Server performs model-related services, Work Server trains related services, inference calculation, gradient calculation, etc[1]. ps-worker TensorFlow on Kubernetes has many problems: Resource isolation. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/tutorials/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/tutorials/ Here is a simple example of how to use Volcano with CRD resources. Step: 1 Create a queue named “test”. # kubectl create -f queue.yaml // queue.yaml apiVersion: scheduling.volcano.sh/v1beta1 kind: Queue metadata: name: test spec: weight: 1 reclaimable: false capability: cpu: 2 Step: 2 Create a VolcanoJob named “job-1”. # kubectl create -f vcjob.yaml // vcjob.yaml apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: job-1 spec: minAvailable: 1 schedulerName: volcano queue: test policies: - event: PodEvicted action: RestartJob tasks: - replicas: 1 name: nginx policies: - event: TaskCompleted action: CompleteJob template: spec: containers: - command: - sleep - 10m image: nginx:latest name: nginx resources: requests: cpu: 1 limits: cpu: 1 restartPolicy: Never Step: 3 Check the status of the custom job. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/contribution/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/contribution/ Welcome Welcome to Volcano! Before You Start Code of Conduct Community discussions Community Expectations Getting Started Your First Contribution Find Something to Work On Find a Good Topic Work on an Issue File an Issue Contribution Workflow Open a Pull Request Code Review Commit Message Format Testing Before You Start Code of Conduct All Volcano contributors must read and observe the Code of Conduct. https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/vcjob/ Tue, 21 May 2024 00:00:00 +0000 https://deploy-preview-499--volcano-sh.netlify.app/en/docs/v1-9-0/vcjob/ Introduction VolcanoJob, referred to as vcjob, is a CRD object for Volcano. Different from a Kubernetes job, it provides more advanced features such as specified scheduler, minimum number of members, task definition, lifecycle management, specific queue, and specific priority. VolcanoJob is ideal for high performance computing scenarios such as machine learning, big data applications, and scientific computing. Example apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: test-job spec: minAvailable: 3 schedulerName: volcano priorityClassName: high-priority policies: - event: PodEvicted action: RestartJob plugins: ssh: [] env: [] svc: [] maxRetry: 5 queue: default volumes: - mountPath: "/myinput" - mountPath: "/myoutput" volumeClaimName: "testvolumeclaimname" volumeClaim: accessModes: [ "ReadWriteOnce" ] storageClassName: "my-storage-class" resources: requests: storage: 1Gi tasks: - replicas: 6 name: "default-nginx" template: metadata: name: web spec: containers: - image: nginx imagePullPolicy: IfNotPresent name: nginx resources: requests: cpu: "1" restartPolicy: OnFailure Key Fields schedulerName schedulerName indicates the scheduler that will schedule the job.