Mistral models with vLLM user guide on GDC air-gapped

This document provides step-by-step instructions for deploying open weight models from Mistral AI on Google Distributed Cloud (GDC) air-gapped environments. It covers using vLLM for high-throughput serving, using the GDC platform's capabilities, including Kubernetes, Harbor, and GPU resources. Select the model and the GPU option in the Model section; every other step is the same for all models.

Supported models

Model	Hugging Face repository	Parameters	License	GPU options
Mistral 7B Instruct	`mistralai/Mistral-7B-Instruct-v0.2`	7.2B (dense)	Apache 2.0	1 x any GPU
Mistral NeMo 12B Instruct	`mistralai/Mistral-Nemo-Instruct-2407`	12B (dense)	Apache 2.0	1 x any GPU
Mistral Small 3.2 24B Instruct	`mistralai/Mistral-Small-3.2-24B-Instruct-2506`	24B (dense, with vision encoder)	Apache 2.0	1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB or B300 288 GB
Ministral 8B Instruct	`mistralai/Ministral-8B-Instruct-2410`	8B (dense)	Mistral AI Research License (research and non-commercial use only)	1 x any GPU
Mixtral 8x7B Instruct	`mistralai/Mixtral-8x7B-Instruct-v0.1`	46.7B total / 12.9B active (mixture of experts, 8x7B)	Apache 2.0	2 x NVIDIA A100 80 GB 2 x NVIDIA H100 94 GB 1 x NVIDIA A100 80 GB or H100 94 GB with `--quantization=fp8` 1 x NVIDIA H200 141 GB or B300 288 GB
Magistral Small 1.2 24B	`mistralai/Magistral-Small-2509`	24B (dense, reasoning, with vision encoder)	Apache 2.0	1 x NVIDIA A100 80 GB 1 x NVIDIA H100 94 GB 1 x NVIDIA H200 141 GB or B300 288 GB

Architecture

The solution involves deploying a containerized serving backend (vLLM) as a Deployment within a standard cluster. Model weights are stored on Persistent Volumes, populated from files downloaded locally and uploaded using a loader pod. A Kubernetes Service of type LoadBalancer exposes the backend API, and NetworkPolicy / ProjectNetworkPolicy objects secure access to it.

Before you begin

Complete the Open Weight Models with vLLM reference implementation on the workstation, and make sure you have access to the ${HOME}/gdcag-solutions directory.

Identity and Access Management

Ensure that the necessary IAM accounts, roles, and permissions are properly configured.

GDC User roles on organization:

Optional: Standard Cluster Admin (View clusters from the console)

GDC User roles on project:

Standard Cluster Admin (standard-cluster-admin)

GDC User role on the standard cluster:

StandardClusterRoleBinding to the StandardClusterRole cluster-admin (created by a Project IAM Admin, see the Identity and Access Management section of the reference implementation)

Workstation

Run all commands in this guide from a workstation with the required tools and connectivity to the environment and internet, unless a step states otherwise.

The environment configuration is layered:

${HOME}/gdcag-solutions/env.sh loads the platform and registry settings.
${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh adds the vLLM settings.
This user guide adds env.d/model.sh (the model) and env.d/accelerator.sh (the GPU option and runtime settings), which you write in the Model section; and env.d/model-common.sh (the values derived from them).

Create the user guide directory structure:

mkdir -p ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d

Create the common model environment configuration file. It derives every path and resource name from GDCS_HF_MODEL, so it is the same for all models:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model-common.sh && echo "Successfully created." || echo "Failed to create!"
# Model (derived from env.d/model.sh)
export GDCS_HF_DOWNLOAD_PATH="${GDCS_USER_GUIDE_HOME}/hf-models"
export GDCS_HF_MODEL_LOWERCASE="${GDCS_HF_MODEL,,}"
export GDCS_HF_MODEL_DOWNLOAD_DIR="${GDCS_HF_DOWNLOAD_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
export GDCS_HF_PV_MOUNT_PATH="/mnt/hf-model"

GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_LOWERCASE//\//-}"
export GDCS_HF_MODEL_KUBERNETES="${GDCS_HF_MODEL_KUBERNETES//./-}"

# Workload
export GDCS_WORKLOAD_NAMESPACE="vllm-mistral"
export GDCS_WORKLOAD_NAME="${GDCS_HF_MODEL_KUBERNETES}"
export GDCS_MODEL_LOADER_POD_NAME="loader-${GDCS_HF_MODEL_KUBERNETES}"
EOF

Create the user guide environment loader file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.sh && echo "Successfully created." || echo "Failed to create!"
source "${HOME}/gdcag-solutions/open-weight-models/vllm/env.sh"

export GDCS_USER_GUIDE_HOME="${HOME}/gdcag-solutions/open-weight-models/vllm/mistral"
echo "GDCS_USER_GUIDE_HOME=${GDCS_USER_GUIDE_HOME}"

# Sourced in dependency order: the model selection, the accelerator and runtime settings, then the values derived from them
source "${GDCS_USER_GUIDE_HOME}/env.d/model.sh"
source "${GDCS_USER_GUIDE_HOME}/env.d/accelerator.sh"
source "${GDCS_USER_GUIDE_HOME}/env.d/model-common.sh"
echo "GDCS_HF_MODEL=${GDCS_HF_MODEL} (${GDCS_WORKLOAD_ACCELERATOR_COUNT} x ${GDCS_WORKLOAD_ACCELERATOR_TYPE:-unset})"
EOF

Model

Select the model and the GPU option that matches the GPU nodes of your standard cluster. GDC air-gapped offers the following GPUs (see Cluster node machine types); the accelerator resource type is the name under which the nodes expose the GPU to Kubernetes.

GPU	Memory	Machine types	Accelerator resource type
NVIDIA A100	40 GB	`a2-highgpu-1g-gdc`	`nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB`
NVIDIA A100	80 GB	`a2-ultragpu-1g-gdc`, `a2-ultragpu-2g-gdc`	`nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE`
NVIDIA H100	94 GB	`a3-highgpu-1g-gdc` (Preview), `a3-highgpu-2g-gdc` (Preview), `a3-highgpu-4g-gdc`	`nvidia.com/gpu-pod-NVIDIA_H100_NVL`
NVIDIA H200	141 GB	`a3-ultragpu-8g-gdc`	`nvidia.com/gpu-pod-NVIDIA_H200`
NVIDIA B300	288 GB	`a4-ultragpu-8g-gdc`	`nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC`

Each tab lists the model facts, the command that writes the model environment file env.d/model.sh, and nested tabs that write the accelerator environment file env.d/accelerator.sh for your chosen GPU option with pre-filled resource types and vLLM settings. To switch models or GPU options later, write different environment files, source the environment again, and repeat the Implementation section; the workloads of several models can run side by side in the same namespace because every resource name is derived from the model name.

Create the model and accelerator environment files for the model and GPU option you want to deploy.

Mistral 7B Instruct

Attribute	Value
Hugging Face repository	`mistralai/Mistral-7B-Instruct-v0.2`
Parameters	7.2B (dense)
License	Apache 2.0
Weights used by vLLM	Hugging Face format (`model-*.safetensors`), about 14.5 GB
Context length	32k tokens; this guide serves `--max-model-len 32768`

Create the model environment file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mistral 7B Instruct
export GDCS_HF_MODEL="mistralai/Mistral-7B-Instruct-v0.2"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="consolidated* *.bin"
EOF

Create the accelerator environment file for your GPU option:

NVIDIA A100 40 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 40 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="60"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Mistral NeMo 12B Instruct

Attribute	Value
Hugging Face repository	`mistralai/Mistral-Nemo-Instruct-2407`
Parameters	12B (dense)
License	Apache 2.0
Weights used by vLLM	Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 24.5 GB
Context length	128k tokens; this guide serves `--max-model-len 32768`

Create the model environment file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mistral NeMo 12B Instruct
export GDCS_HF_MODEL="mistralai/Mistral-Nemo-Instruct-2407"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Create the accelerator environment file for your GPU option:

NVIDIA A100 40 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 40 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="180"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Mistral Small 3.2 24B Instruct

Attribute	Value
Hugging Face repository	`mistralai/Mistral-Small-3.2-24B-Instruct-2506`
Parameters	24B (dense, with vision encoder)
License	Apache 2.0
Weights used by vLLM	Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 48 GB
Context length	128k tokens; this guide serves `--max-model-len 32768`

Create the model environment file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mistral Small 3.2 24B Instruct
export GDCS_HF_MODEL="mistralai/Mistral-Small-3.2-24B-Instruct-2506"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Create the accelerator environment file for your GPU option:

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Ministral 8B Instruct

Attribute	Value
Hugging Face repository	`mistralai/Ministral-8B-Instruct-2410`
Parameters	8B (dense)
License	Mistral AI Research License (research and non-commercial use only)
Weights used by vLLM	Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 16 GB
Context length	128k tokens; this guide serves `--max-model-len 32768`

Create the model environment file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Ministral 8B Instruct
export GDCS_HF_MODEL="mistralai/Ministral-8B-Instruct-2410"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Create the accelerator environment file for your GPU option:

NVIDIA A100 40 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 40 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_PCIE_40GB"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="120"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Mixtral 8x7B Instruct

Attribute	Value
Hugging Face repository	`mistralai/Mixtral-8x7B-Instruct-v0.1`
Parameters	46.7B total / 12.9B active (mixture of experts, 8x7B)
License	Apache 2.0
Weights used by vLLM	Hugging Face format (`model-.safetensors`), about 93 GB (the repository also carries `consolidated..pt` PyTorch files, excluded)
Context length	32k tokens; this guide serves `--max-model-len 32768`

Create the model environment file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Mixtral 8x7B Instruct
export GDCS_HF_MODEL="mistralai/Mixtral-8x7B-Instruct-v0.1"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="consolidated* *.bin"
EOF

Create the accelerator environment file for your GPU option:

NVIDIA A100 80 GB (2 GPUs)

Requires a node with two A100 80 GB GPUs (a2-ultragpu-2g-gdc); vLLM runs with --tensor-parallel-size 2.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB (2 GPUs)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="2"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB (2 GPUs)

Requires a node with at least two H100 GPUs (a3-highgpu-2g-gdc or a3-highgpu-4g-gdc); vLLM runs with --tensor-parallel-size 2.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB (2 GPUs)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="2"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA A100 80 GB (FP8)

--quantization=fp8 quantizes the weights to FP8 while loading (about 47 GB instead of 93 GB), with slightly lower output quality.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB (FP8)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS="--quantization=fp8"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H100 94 GB (FP8)

--quantization=fp8 quantizes the weights to FP8 while loading (about 47 GB instead of 93 GB), with slightly lower output quality.

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB (FP8)
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS="--quantization=fp8"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="32768"
export GDCS_VLLM_EXTRA_ARGS=""
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="480"
export GDCS_VLLM_TEST_MAX_TOKENS="128"
EOF

Magistral Small 1.2 24B

Attribute	Value
Hugging Face repository	`mistralai/Magistral-Small-2509`
Parameters	24B (dense, reasoning, with vision encoder)
License	Apache 2.0
Weights used by vLLM	Mistral format (`consolidated.safetensors`, `params.json`, `tekken.json`), about 48 GB
Context length	128k tokens (40k recommended by Mistral); this guide serves `--max-model-len 40960`

Create the model environment file:

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/model.sh && echo "Successfully created." || echo "Failed to create!"
# Model: Magistral Small 1.2 24B
export GDCS_HF_MODEL="mistralai/Magistral-Small-2509"
export GDCS_HF_MODEL_REVISION="main"  # Hugging Face branch, tag, or commit to download
export GDCS_HF_DOWNLOAD_EXCLUDE="model-*.safetensors *.bin"
EOF

Create the accelerator environment file for your GPU option:

NVIDIA A100 80 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA A100 80 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_A100_80GB_PCIE"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

NVIDIA H100 94 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H100 94 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H100_NVL"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

NVIDIA H200 141 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA H200 141 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_H200"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

NVIDIA B300 288 GB

cat << 'EOF' > ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.d/accelerator.sh && echo "Successfully created." || echo "Failed to create!"
# Accelerator: NVIDIA B300 288 GB
export GDCS_WORKLOAD_ACCELERATOR_TYPE="nvidia.com/gpu-pod-NVIDIA_B300_SXM6_AC"
export GDCS_WORKLOAD_ACCELERATOR_COUNT="1"

# vLLM
export GDCS_VLLM_MAX_MODEL_LEN="40960"
export GDCS_VLLM_EXTRA_ARGS="--reasoning-parser=mistral"
export GDCS_VLLM_STARTUP_FAILURE_THRESHOLD="300"
export GDCS_VLLM_TEST_MAX_TOKENS="1024"
EOF

Source the environment file:

source ${HOME}/gdcag-solutions/open-weight-models/vllm/mistral/env.sh

The output is similar to the following:

GDCS_ROOT_HOME=HOME_DIRECTORY_PATH/gdcag-solutions
GDCS_IMPLEMENTATION_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm
GDCS_USER_GUIDE_HOME=HOME_DIRECTORY_PATH/gdcag-solutions/open-weight-models/vllm/mistral
GDCS_HF_MODEL=HUGGING_FACE_REPOSITORY (ACCELERATOR_COUNT x ACCELERATOR_RESOURCE_TYPE)

Cluster

Configure gdcloud:

gdcloud config set core/account "default-user"
gdcloud config set core/organization_console_url "${GDC_ZONAL_CONSOLE_URL}"
gdcloud config set core/project "${GDC_PROJECT}"
gdcloud config set core/zone "${GDC_ZONE}"

Authenticate to the GDC environment:
```
gdcloud auth login
```

Retrieve cluster credentials:

gdcloud clusters get-credentials "${GDC_STANDARD_CLUSTER_NAME}" \
--project="${GDC_PROJECT}" \
--standard \
--zone="${GDC_ZONE}"

Verify connectivity to the cluster:

kubectl get nodes -L node.cluster.private.gdc.goog/machine-class

Verify that the accelerator resource type of the accelerator environment file is available in the cluster. The command lists the accelerator resource types that the nodes expose (the name depends on the installed GPU model; it is empty on clusters without GPU nodes) and the configured value:
```
kubectl get nodes --output=json | jq -r '[.items[].status.allocatable | keys[] | select(startswith("nvidia.com/"))] | unique[]'
echo "GDCS_WORKLOAD_ACCELERATOR_TYPE=${GDCS_WORKLOAD_ACCELERATOR_TYPE}"
```
Note: If the configured value isn't in the list, set GDCS_WORKLOAD_ACCELERATOR_TYPE in env.d/accelerator.sh to one of the listed resource types and source the environment file again.

Implementation

Namespace

Create the namespace:

kubectl create namespace "${GDCS_WORKLOAD_NAMESPACE}" \
--dry-run=client \
--output=yaml | kubectl apply -f -

Add the imagePullSecret:

kubectl create secret docker-registry "${GDCS_HARBOR_K8S_PULL_SECRET}" \
--dry-run=client \
--from-file=.dockerconfigjson=${GDCS_HARBOR_K8S_DOCKER_CONFIG}/config.json \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" \
--output=yaml | kubectl apply -f -

Model weights

Download the model weights (only the files vLLM needs for the selected model, GDCS_HF_DOWNLOAD_EXCLUDE skips the rest) and remove the .cache directory:

EXCLUDE_ARGS=()
if [[ -n "${GDCS_HF_DOWNLOAD_EXCLUDE:-}" ]]; then
  read -r -a EXCLUDE_PATTERNS <<< "${GDCS_HF_DOWNLOAD_EXCLUDE}"
  EXCLUDE_ARGS=(--exclude "${EXCLUDE_PATTERNS[@]}")
fi
hf download "${GDCS_HF_MODEL}" --revision "${GDCS_HF_MODEL_REVISION}" --local-dir "${GDCS_HF_MODEL_DOWNLOAD_DIR}" "${EXCLUDE_ARGS[@]}" && \
rm -rf "${GDCS_HF_MODEL_DOWNLOAD_DIR}/.cache"

Calculate the required PersistentVolumeClaim size from the downloaded model weights with 25% overhead (rounded up to the next whole GiB):

MODEL_KIB=$(du -sk "${GDCS_HF_MODEL_DOWNLOAD_DIR}" | awk '{print $1}')
export GDCS_MODEL_PVC_SIZE="$(( (MODEL_KIB * 125 / 100 + 1048575) / 1048576 ))Gi"
echo "GDCS_MODEL_PVC_SIZE=${GDCS_MODEL_PVC_SIZE}"

Create the manifest for the ReadWriteOnce PersistentVolumeClaim for the model weights:

cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ${GDCS_WORKLOAD_NAME}-rwo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: ${GDCS_MODEL_PVC_SIZE}
  storageClassName: standard-rwo
  volumeMode: Filesystem
EOF

Apply the manifest for the PersistentVolumeClaim for the model weights:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-rwo-pvc.yaml" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Wait for the PersistentVolumeClaim to bind and print the values that the Infrastructure Operator needs to adjust the I/O quality of service (QoS) policy:

kubectl wait \
--for=jsonpath='{.status.phase}'=Bound \
"pvc/${GDCS_WORKLOAD_NAME}-rwo" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" \
--timeout=120s
BACKING_PVC=$(kubectl get pvc "${GDCS_WORKLOAD_NAME}-rwo" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" \
--output=jsonpath='{.spec.volumeName}')
echo "BACKING_PVC=${BACKING_PVC} ORG=${GDC_ORG}"

If performance-rwo isn't available on your cluster, ask your Infrastructure Operator to apply the high-performance QoS policy (system-performance-<ORG>-admin, 30 IOPS per GiB with a 6,148 IOPS minimum floor) to the volume that backs the PersistentVolumeClaim. Provide the BACKING_PVC and ORG values that the previous step printed. The Infrastructure Operator applies the policy as described in FILE-R0097; it takes effect on the live volume, so you can continue with the next steps.

Note: If high-performance block storage is enabled on your cluster, you can set storageClassName: performance-rwo in the PersistentVolumeClaim manifest instead of requesting the QoS policy. If kubectl get storageclass performance-rwo returns nothing, the Infrastructure Operator has not enabled high-performance block storage.

Create the manifest for the model loader Pod:

cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml"
apiVersion: v1
kind: Pod
metadata:
  name: ${GDCS_MODEL_LOADER_POD_NAME}
spec:
  containers:
    - name: uploader
      image: ${GDCS_HARBOR_PROJECT_URI}/library/busybox:${GDCS_BUSYBOX_IMAGE_TAG}
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: model-data
          mountPath: ${GDCS_HF_PV_MOUNT_PATH}
  imagePullSecrets:
    - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
  volumes:
    - name: model-data
      persistentVolumeClaim:
        claimName: ${GDCS_WORKLOAD_NAME}-rwo
EOF

Apply the manifest for the model loader Pod:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/${GDCS_MODEL_LOADER_POD_NAME}-pod.yaml" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Wait for the model loader Pod to be Running:

watch --color --interval 5 --no-title \
"kubectl get pod/${GDCS_MODEL_LOADER_POD_NAME} \
--namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"

Upload the model weights to the PersistentVolume (the script skips files that already exist with the same size, so it can be re-run):

export LOCAL_MODEL_DIR="${GDCS_HF_MODEL_DOWNLOAD_DIR}"
export REMOTE_MODEL_DIR="${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
export REMOTE_MOUNT_DIR="${GDCS_HF_PV_MOUNT_PATH}"

${GDCS_IMPLEMENTATION_HOME}/model_weight_upload.sh

Delete the model loader Pod:

kubectl delete pod/${GDCS_MODEL_LOADER_POD_NAME} \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Inference server

Create the manifest for the vLLM Deployment. The ConfigMap carries the model-specific vLLM settings from the environment, so the manifest is the same for every model and GPU option:

cat <<EOF > "${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
  name: ${GDCS_WORKLOAD_NAME}
data:
  MAX_MODEL_LEN: "${GDCS_VLLM_MAX_MODEL_LEN}"
  MODEL: "${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}"
  PORT: "8000"
  TENSOR_PARALLEL_SIZE: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
  VLLM_EXTRA_ARGS: "${GDCS_VLLM_EXTRA_ARGS}"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${GDCS_WORKLOAD_NAME}
  labels:
    app: ${GDCS_WORKLOAD_NAME}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${GDCS_WORKLOAD_NAME}
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ${GDCS_WORKLOAD_NAME}
    spec:
      containers:
        - command:
            - /bin/sh
            - -c
          args:
            - exec vllm serve "\$MODEL" --port="\$PORT" --tensor-parallel-size="\$TENSOR_PARALLEL_SIZE" --max-model-len="\$MAX_MODEL_LEN" \$VLLM_EXTRA_ARGS
          envFrom:
            - configMapRef:
                name: ${GDCS_WORKLOAD_NAME}
          image: ${GDCS_HARBOR_PROJECT_URI}/vllm/vllm-openai:${GDCS_VLLM_IMAGE_TAG}-fips
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 20
            timeoutSeconds: 5
          name: vllm
          ports:
            - containerPort: 8000
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits:
              ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
            requests:
              ${GDCS_WORKLOAD_ACCELERATOR_TYPE}: "${GDCS_WORKLOAD_ACCELERATOR_COUNT}"
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: ${GDCS_VLLM_STARTUP_FAILURE_THRESHOLD}
            periodSeconds: 10
            timeoutSeconds: 5
          volumeMounts:
            - name: model-weights
              mountPath: ${GDCS_HF_PV_MOUNT_PATH}
            - name: shm
              mountPath: /dev/shm
      securityContext:
        seLinuxOptions:
          type: spc_t
      imagePullSecrets:
        - name: ${GDCS_HARBOR_K8S_PULL_SECRET}
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: ${GDCS_WORKLOAD_NAME}-rwo
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: "8Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: ${GDCS_WORKLOAD_NAME}
spec:
  ports:
    - name: vllm-http
      port: 80
      protocol: TCP
      targetPort: 8000
  selector:
    app: ${GDCS_WORKLOAD_NAME}
  type: LoadBalancer
EOF

Apply the manifest for the vLLM Deployment:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/${GDCS_WORKLOAD_NAME}-deployment.yaml" \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Create the manifest for the vLLM NetworkPolicy:

cat << EOF > "${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml" && echo "Successfully created." || echo "Failed to create!"
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-vllm-ingress
  namespace: ${GDCS_WORKLOAD_NAMESPACE}
spec:
  ingress:
  - from:
    - ipBlock:
        cidr: 0.0.0.0/0
    - namespaceSelector: {}
    ports:
    - port: 8000
      protocol: TCP
  podSelector: {}
  policyTypes:
  - Ingress
EOF

Apply the manifest for the vLLM NetworkPolicy:

kubectl apply \
--filename="${GDCS_USER_GUIDE_HOME}/vllm-networkpolicy.yaml"

Wait for the Pod to be Running:

watch --color --interval 5 --no-title \
"kubectl get pods --selector=app=${GDCS_WORKLOAD_NAME} \
--namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e 'Running'"

Wait for the Deployment to be Ready:

watch --color --interval 5 --no-title \
"kubectl get deployment/${GDCS_WORKLOAD_NAME} \
--namespace=${GDCS_WORKLOAD_NAMESPACE} | GREP_COLORS='mt=01;92' grep -E --color=always -e '^' -e '1/1     1            1'"

Optional: Stream the container logs to monitor startup progress:

kubectl logs deployment/${GDCS_WORKLOAD_NAME} \
--follow \
--namespace="${GDCS_WORKLOAD_NAMESPACE}"

Validation

Retrieve the LoadBalancer IP address for vLLM:

export VLLM_LB_IP=$(kubectl get service/${GDCS_WORKLOAD_NAME} --namespace="${GDCS_WORKLOAD_NAMESPACE}" --output=jsonpath='{.status.loadBalancer.ingress[*].ip}')
echo "VLLM_LB_IP=${VLLM_LB_IP}"

Send a test request to vLLM. The max_tokens value is GDCS_VLLM_TEST_MAX_TOKENS from env.d/accelerator.sh; reasoning models need a larger budget because the reasoning phase consumes output tokens:

curl http://${VLLM_LB_IP}/v1/models

curl http://${VLLM_LB_IP}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "'${GDCS_HF_PV_MOUNT_PATH}/${GDCS_HF_MODEL_LOWERCASE}'",
  "messages": [
    {"role": "user", "content": "Explain Google Distributed Cloud air-gapped briefly."}
  ],
  "max_tokens": '${GDCS_VLLM_TEST_MAX_TOKENS}'
}'

If the LoadBalancer IP address isn't reachable from the workstation, for example because no ProjectNetworkPolicy admits it yet, send the requests through port forwarding:

kubectl port-forward service/${GDCS_WORKLOAD_NAME} \
--namespace="${GDCS_WORKLOAD_NAMESPACE}" 8000:80 &
PF_PID=$!

sleep 1

curl http://127.0.0.1:8000/v1/models

kill ${PF_PID}

Mistral models with vLLM user guide on GDC air-gapped Stay organized with collections Save and categorize content based on your preferences.

Supported models

Architecture

Before you begin

Identity and Access Management

Workstation

Model

Mistral 7B Instruct

NVIDIA A100 40 GB

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Mistral NeMo 12B Instruct

NVIDIA A100 40 GB

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Mistral Small 3.2 24B Instruct

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Ministral 8B Instruct

NVIDIA A100 40 GB

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Mixtral 8x7B Instruct

NVIDIA A100 80 GB (2 GPUs)

NVIDIA H100 94 GB (2 GPUs)

NVIDIA A100 80 GB (FP8)

NVIDIA H100 94 GB (FP8)

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Magistral Small 1.2 24B

NVIDIA A100 80 GB

NVIDIA H100 94 GB

NVIDIA H200 141 GB

NVIDIA B300 288 GB

Cluster

Implementation

Namespace

Model weights

Inference server

Validation

Mistral models with vLLM user guide on GDC air-gapped