vGPU快速起步¶

备注

本文是综合了以下实践的 精炼提纯 版本:

安装NVIDIA license服务器 ¶

NVIDIA License Server 安装是一个独立步骤

vgpu_unlock ¶

NVIDIA GRID vGPU驱动需要以动态内核模块支持(DKMS) 模块方式安装:

使用DKMS方式安装NVIDIA vGPU驱动¶

./NVIDIA-Linux-x86_64-510.85.03-vgpu-kvm.run --dkms

安装NVIDIA Virtual GPU Manager ¶

使用 mdevctl 命令扫描GPU支持的 NVIDIA Virtual GPU (vGPU) 组合设备规格:

使用 mdevctl types 命令扫描 mdev_supported_types 目录获得 NVIDIA Virtual GPU (vGPU) 设备配置¶

mdevctl types

我的 Nvidia Tesla P10 GPU运算卡支持的 NVIDIA Virtual GPU (vGPU) 组合:

使用 mdevctl types 命令扫描 mdev_supported_types 目录获得 NVIDIA Virtual GPU (vGPU) 设备配置¶

0000:82:00.0
  nvidia-156
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-215
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-241
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-283
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4C
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6
  nvidia-284
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6C
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4
  nvidia-285
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8C
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3
  nvidia-286
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12C
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2
  nvidia-287
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24C
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1
  nvidia-46
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-47
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
  nvidia-48
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3Q
    Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
  nvidia-49
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
  nvidia-50
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6Q
    Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
  nvidia-51
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
  nvidia-52
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12Q
    Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
  nvidia-53
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24Q
    Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
  nvidia-54
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
  nvidia-55
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
  nvidia-56
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3A
    Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
  nvidia-57
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
  nvidia-58
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6A
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
  nvidia-59
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
  nvidia-60
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12A
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
  nvidia-61
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24A
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
  nvidia-62
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24

NVIDIA Virtual GPU (vGPU) 设备类型( mdev_supported_types 每种规格末尾有一个 A/B/C/Q 标识类型 )

vGPU类型和用途关系¶
类型	建议用途
A	虚拟应用程序(Virtual Applications / vApps)
B	虚拟桌面(Virtual Desktops / vPC)
C	AI/机器学习/训练( vCS or vWS )
Q	虚拟工作站 (Virtual Workstations / vWS)

我规划构建2块 P40-12C 用于 Machine Learning Atlas 的 GPU Kubernetes ，执行以下脚本构建 mdev 设备:

vgpu_create 脚本创建2个 P40-12C NVIDIA Virtual GPU (vGPU)¶

# GPU芯片(按实际配置) => 0000:82:00.0
gpu_bdf="0000:"`lspci | grep GP102GL | awk '{print $1}'`

# vGPU profile从 mdevctl types 中查询
vgpu_type="nvidia-286"

# 生成2个uuid用于vgpu设备标识
id_list=/tmp/vgpu_uuid
uuid -n 2 > $id_list

# 创建vGPU
for id in `cat $id_list`;do
    sudo mdevctl start -u $id -p $gpu_bdf -t $vgpu_type
    sudo mdevctl define -a -u $id

cat << EOF
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
    <source>
        <address uuid='$id'/>
    </source>
</hostdev>
EOF

done

将执行脚本的输出信息:

执行 vgpu_create 脚本创建2个 P40-12C NVIDIA Virtual GPU (vGPU) 输出信息第 一 个vgpu设备¶

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
    <source>
        <address uuid='3eb9d560-0b31-11ee-91a9-bb28039c61eb'/>
    </source>
</hostdev>

执行 vgpu_create 脚本创建2个 P40-12C NVIDIA Virtual GPU (vGPU) 输出信息第 二 个vgpu设备¶

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
    <source>
        <address uuid='3eb9d718-0b31-11ee-91aa-2b17f51ee12d'/>
    </source>
</hostdev>

(对应2个vGPU)分别添加到 2个虚拟机 y-k8s-n-1 和 y-k8s-n-2 中，然后启动虚拟机

虚拟机启动后，在 Host 物理主机上检查 nvidia-smi vgpu 可以看到如下输出信息，表明两块 NVIDIA Virtual GPU (vGPU) 已经正确插入到2个虚拟机中:

执行 nvidia-smi vgpu 可以看到两块vGPU已经分配给两个虚拟机¶

Thu Jun 15 23:33:21 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.85.03              Driver Version: 510.85.03                 |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  NVIDIA Graphics Device     | 00000000:82:00.0             |   0%       |
|      3251634189  GRID P40-12C   | 10a1...  y-k8s-n-1           |      0%    |
|      3251634195  GRID P40-12C   | b102...  y-k8s-n-2           |      0%    |
+---------------------------------+------------------------------+------------+

恢复 `mdev` 设备¶

当操作系统升级(内核)之后，重启物理服务器，上述 mdev 设备丢失，此时启动 y-k8s-n-1 报错:

error: Failed to start domain 'y-k8s-n-1'
error: device not found: mediated device '3eb9d560-0b31-11ee-91a9-bb28039c61eb' not found

则需要恢复原先的 mdev 设备(依然使用原先的uuid)，也就是将上述 vgpu_create 脚本后半段抠出来直接运行(uuid沿用分配给虚拟机的设备uuid):

恢复原先创建过的 mdev 设备¶

# GPU芯片(按实际配置) => 0000:82:00.0
gpu_bdf="0000:"`lspci | grep GP102GL | awk '{print $1}'`

# vGPU profile从 mdevctl types 中查询
vgpu_type="nvidia-286"

cat << EOF > mdev_id
3eb9d560-0b31-11ee-91a9-bb28039c61eb
3eb9d718-0b31-11ee-91aa-2b17f51ee12d
EOF

for id in `cat mdev_id`;do
    sudo mdevctl start -u $id -p $gpu_bdf -t $vgpu_type
    sudo mdevctl define -a -u $id
done

不过，这里会提示:

Device 3eb9d560-0b31-11ee-91a9-bb28039c61eb on 0000:82:00.0 already defined, try modify?
Device 3eb9d718-0b31-11ee-91aa-2b17f51ee12d on 0000:82:00.0 already defined, try modify?

也就是说，只要执行 mdevctl start 就可以了

安装NVIDIA Virtual GPU Guest Driver ¶

虚拟机内部安装 GCC, Linux Kernel Headers 以及 dkms :

在Ubuntu Guest虚拟机中需要安装GCC，Linux Kernel Headers和dkms¶

sudo apt install gcc linux-headers-$(uname -r) dkms

登陆虚拟机，安装Guest虚拟机的GRID驱动:

在Ubuntu Guest虚拟机中安装 nvidia-linux-grid Guest驱动¶

sudo dpkg -i nvidia-linux-grid-510_510.85.02_amd64.deb

然后重启虚拟机
在Ubuntu虚拟机中编辑 /etc/nvidia/gridd.conf 配置:

配置虚拟机 /etc/nvidia/gridd.conf 连接License服务器¶

# Description: Set License Server Address
# Data type: string
# Format:  "<address>"
ServerAddress=192.168.6.248

# Description: Set License Server port number
# Data type: integer
# Format:  <port>, default is 7070
ServerPort=7070

# Description: Set Feature to be enabled
# Data type: integer
# Possible values:
#    0 => for unlicensed state
#    1 => for NVIDIA vGPU (Optional, autodetected as per vGPU type)
#    2 => for NVIDIA RTX Virtual Workstation
#    4 => for NVIDIA Virtual Compute Server
# All other values reserved
FeatureType=4

启动 nvidia-gridd :

配置Lince Server的IP和端口以及请求License，然后启动 nvidia-gridd¶

systemctl start nvidia-gridd

观察 Lince Server 服务器的 License Feature Usage 可以看到Licence计数已经减少了1个，也就是被vGPU客户端使用了

备注

一切就绪，现在可以开始:

GPU Kubernetes 部署
虚拟机内部尝试一下 PyTorch快速起步
部署 Looking Glass 玩微软飞行模拟

vGPU快速起步¶

安装NVIDIA license服务器¶

vgpu_unlock¶

安装NVIDIA Virtual GPU Manager¶

恢复 mdev 设备¶

安装NVIDIA Virtual GPU Guest Driver¶

安装NVIDIA license服务器 ¶

vgpu_unlock ¶

安装NVIDIA Virtual GPU Manager ¶

恢复 `mdev` 设备¶

安装NVIDIA Virtual GPU Guest Driver ¶