vGPU快速起步¶
备注
本文是综合了以下实践的 精炼提纯 版本:
安装NVIDIA license服务器¶
NVIDIA License Server 安装是一个独立步骤
vgpu_unlock¶
NVIDIA GRID vGPU驱动需要以 动态内核模块支持(DKMS) 模块方式安装:
./NVIDIA-Linux-x86_64-510.85.03-vgpu-kvm.run --dkms
安装NVIDIA Virtual GPU Manager¶
使用
mdevctl
命令扫描GPU支持的 NVIDIA Virtual GPU (vGPU) 组合设备规格:
mdevctl types
我的 Nvidia Tesla P10 GPU运算卡 支持的 NVIDIA Virtual GPU (vGPU) 组合:
0000:82:00.0
nvidia-156
Available instances: 12
Device API: vfio-pci
Name: GRID P40-2B
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-215
Available instances: 12
Device API: vfio-pci
Name: GRID P40-2B4
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-241
Available instances: 24
Device API: vfio-pci
Name: GRID P40-1B4
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-283
Available instances: 6
Device API: vfio-pci
Name: GRID P40-4C
Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6
nvidia-284
Available instances: 4
Device API: vfio-pci
Name: GRID P40-6C
Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4
nvidia-285
Available instances: 3
Device API: vfio-pci
Name: GRID P40-8C
Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3
nvidia-286
Available instances: 2
Device API: vfio-pci
Name: GRID P40-12C
Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2
nvidia-287
Available instances: 1
Device API: vfio-pci
Name: GRID P40-24C
Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1
nvidia-46
Available instances: 24
Device API: vfio-pci
Name: GRID P40-1Q
Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-47
Available instances: 12
Device API: vfio-pci
Name: GRID P40-2Q
Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
nvidia-48
Available instances: 8
Device API: vfio-pci
Name: GRID P40-3Q
Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
nvidia-49
Available instances: 6
Device API: vfio-pci
Name: GRID P40-4Q
Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
nvidia-50
Available instances: 4
Device API: vfio-pci
Name: GRID P40-6Q
Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
nvidia-51
Available instances: 3
Device API: vfio-pci
Name: GRID P40-8Q
Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
nvidia-52
Available instances: 2
Device API: vfio-pci
Name: GRID P40-12Q
Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
nvidia-53
Available instances: 1
Device API: vfio-pci
Name: GRID P40-24Q
Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
nvidia-54
Available instances: 24
Device API: vfio-pci
Name: GRID P40-1A
Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
nvidia-55
Available instances: 12
Device API: vfio-pci
Name: GRID P40-2A
Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
nvidia-56
Available instances: 8
Device API: vfio-pci
Name: GRID P40-3A
Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
nvidia-57
Available instances: 6
Device API: vfio-pci
Name: GRID P40-4A
Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
nvidia-58
Available instances: 4
Device API: vfio-pci
Name: GRID P40-6A
Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
nvidia-59
Available instances: 3
Device API: vfio-pci
Name: GRID P40-8A
Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
nvidia-60
Available instances: 2
Device API: vfio-pci
Name: GRID P40-12A
Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
nvidia-61
Available instances: 1
Device API: vfio-pci
Name: GRID P40-24A
Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
nvidia-62
Available instances: 24
Device API: vfio-pci
Name: GRID P40-1B
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
NVIDIA Virtual GPU (vGPU) 设备类型(
mdev_supported_types
每种规格末尾有一个 A/B/C/Q 标识类型 )
类型 |
建议用途 |
---|---|
A |
虚拟应用程序(Virtual Applications / vApps) |
B |
虚拟桌面(Virtual Desktops / vPC) |
C |
AI/机器学习/训练( vCS or vWS ) |
Q |
虚拟工作站 (Virtual Workstations / vWS) |
我规划构建2块
P40-12C
用于 Machine Learning Atlas 的 GPU Kubernetes ,执行以下脚本构建mdev
设备:
# GPU芯片(按实际配置) => 0000:82:00.0
gpu_bdf="0000:"`lspci | grep GP102GL | awk '{print $1}'`
# vGPU profile从 mdevctl types 中查询
vgpu_type="nvidia-286"
# 生成2个uuid用于vgpu设备标识
id_list=/tmp/vgpu_uuid
uuid -n 2 > $id_list
# 创建vGPU
for id in `cat $id_list`;do
sudo mdevctl start -u $id -p $gpu_bdf -t $vgpu_type
sudo mdevctl define -a -u $id
cat << EOF
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
<source>
<address uuid='$id'/>
</source>
</hostdev>
EOF
done
将执行脚本的输出信息:
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
<source>
<address uuid='3eb9d560-0b31-11ee-91a9-bb28039c61eb'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
<source>
<address uuid='3eb9d718-0b31-11ee-91aa-2b17f51ee12d'/>
</source>
</hostdev>
(对应2个vGPU)分别添加到 2个 虚拟机 y-k8s-n-1
和 y-k8s-n-2
中,然后启动虚拟机
虚拟机启动后,在 Host 物理主机上检查
nvidia-smi vgpu
可以看到如下输出信息,表明两块 NVIDIA Virtual GPU (vGPU) 已经正确插入到2个虚拟机中:
Thu Jun 15 23:33:21 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.85.03 Driver Version: 510.85.03 |
|---------------------------------+------------------------------+------------+
| GPU Name | Bus-Id | GPU-Util |
| vGPU ID Name | VM ID VM Name | vGPU-Util |
|=================================+==============================+============|
| 0 NVIDIA Graphics Device | 00000000:82:00.0 | 0% |
| 3251634189 GRID P40-12C | 10a1... y-k8s-n-1 | 0% |
| 3251634195 GRID P40-12C | b102... y-k8s-n-2 | 0% |
+---------------------------------+------------------------------+------------+
恢复 mdev
设备¶
当操作系统升级(内核)之后,重启物理服务器,上述 mdev
设备丢失,此时启动 y-k8s-n-1
报错:
error: Failed to start domain 'y-k8s-n-1'
error: device not found: mediated device '3eb9d560-0b31-11ee-91a9-bb28039c61eb' not found
则需要恢复原先的 mdev
设备(依然使用原先的uuid),也就是将上述 vgpu_create
脚本后半段抠出来直接运行(uuid沿用分配给虚拟机的设备uuid):
# GPU芯片(按实际配置) => 0000:82:00.0
gpu_bdf="0000:"`lspci | grep GP102GL | awk '{print $1}'`
# vGPU profile从 mdevctl types 中查询
vgpu_type="nvidia-286"
cat << EOF > mdev_id
3eb9d560-0b31-11ee-91a9-bb28039c61eb
3eb9d718-0b31-11ee-91aa-2b17f51ee12d
EOF
for id in `cat mdev_id`;do
sudo mdevctl start -u $id -p $gpu_bdf -t $vgpu_type
sudo mdevctl define -a -u $id
done
不过,这里会提示:
Device 3eb9d560-0b31-11ee-91a9-bb28039c61eb on 0000:82:00.0 already defined, try modify?
Device 3eb9d718-0b31-11ee-91aa-2b17f51ee12d on 0000:82:00.0 already defined, try modify?
也就是说,只要执行 mdevctl start
就可以了
安装NVIDIA Virtual GPU Guest Driver¶
虚拟机内部安装 GCC, Linux Kernel Headers 以及
dkms
:
sudo apt install gcc linux-headers-$(uname -r) dkms
登陆虚拟机,安装Guest虚拟机的GRID驱动:
sudo dpkg -i nvidia-linux-grid-510_510.85.02_amd64.deb
然后重启虚拟机
在Ubuntu虚拟机中编辑
/etc/nvidia/gridd.conf
配置:
# Description: Set License Server Address
# Data type: string
# Format: "<address>"
ServerAddress=192.168.6.248
# Description: Set License Server port number
# Data type: integer
# Format: <port>, default is 7070
ServerPort=7070
# Description: Set Feature to be enabled
# Data type: integer
# Possible values:
# 0 => for unlicensed state
# 1 => for NVIDIA vGPU (Optional, autodetected as per vGPU type)
# 2 => for NVIDIA RTX Virtual Workstation
# 4 => for NVIDIA Virtual Compute Server
# All other values reserved
FeatureType=4
启动
nvidia-gridd
:
systemctl start nvidia-gridd
观察 Lince Server 服务器的 License Feature Usage
可以看到Licence计数已经减少了1个,也就是被vGPU客户端使用了