vGPU快速起步

安装NVIDIA license服务器

NVIDIA License Server 安装是一个独立步骤

vgpu_unlock

使用DKMS方式安装NVIDIA vGPU驱动
./NVIDIA-Linux-x86_64-510.85.03-vgpu-kvm.run --dkms

安装NVIDIA Virtual GPU Manager

使用 mdevctl types 命令扫描 mdev_supported_types 目录获得 NVIDIA Virtual GPU (vGPU) 设备配置
mdevctl types

我的 Nvidia Tesla P10 GPU运算卡 支持的 NVIDIA Virtual GPU (vGPU) 组合:

使用 mdevctl types 命令扫描 mdev_supported_types 目录获得 NVIDIA Virtual GPU (vGPU) 设备配置
0000:82:00.0
  nvidia-156
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-215
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-241
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-283
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4C
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6
  nvidia-284
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6C
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4
  nvidia-285
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8C
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3
  nvidia-286
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12C
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2
  nvidia-287
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24C
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1
  nvidia-46
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-47
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
  nvidia-48
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3Q
    Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
  nvidia-49
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
  nvidia-50
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6Q
    Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
  nvidia-51
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
  nvidia-52
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12Q
    Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
  nvidia-53
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24Q
    Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
  nvidia-54
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
  nvidia-55
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
  nvidia-56
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3A
    Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
  nvidia-57
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
  nvidia-58
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6A
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
  nvidia-59
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
  nvidia-60
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12A
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
  nvidia-61
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24A
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
  nvidia-62
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
vGPU类型和用途关系

类型

建议用途

A

虚拟应用程序(Virtual Applications / vApps)

B

虚拟桌面(Virtual Desktops / vPC)

C

AI/机器学习/训练( vCS or vWS )

Q

虚拟工作站 (Virtual Workstations / vWS)

vgpu_create 脚本创建2个 P40-12C NVIDIA Virtual GPU (vGPU)
# GPU芯片(按实际配置) => 0000:82:00.0
gpu_bdf="0000:"`lspci | grep GP102GL | awk '{print $1}'`

# vGPU profile从 mdevctl types 中查询
vgpu_type="nvidia-286"

# 生成2个uuid用于vgpu设备标识
id_list=/tmp/vgpu_uuid
uuid -n 2 > $id_list

# 创建vGPU
for id in `cat $id_list`;do
    sudo mdevctl start -u $id -p $gpu_bdf -t $vgpu_type
    sudo mdevctl define -a -u $id

cat << EOF
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
    <source>
        <address uuid='$id'/>
    </source>
</hostdev>
EOF

done

将执行脚本的输出信息:

执行 vgpu_create 脚本创建2个 P40-12C NVIDIA Virtual GPU (vGPU) 输出信息第 个vgpu设备
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
    <source>
        <address uuid='3eb9d560-0b31-11ee-91a9-bb28039c61eb'/>
    </source>
</hostdev>
执行 vgpu_create 脚本创建2个 P40-12C NVIDIA Virtual GPU (vGPU) 输出信息第 个vgpu设备
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
    <source>
        <address uuid='3eb9d718-0b31-11ee-91aa-2b17f51ee12d'/>
    </source>
</hostdev>

(对应2个vGPU)分别添加到 2个 虚拟机 y-k8s-n-1y-k8s-n-2 中,然后启动虚拟机

  • 虚拟机启动后,在 Host 物理主机上检查 nvidia-smi vgpu 可以看到如下输出信息,表明两块 NVIDIA Virtual GPU (vGPU) 已经正确插入到2个虚拟机中:

执行 nvidia-smi vgpu 可以看到两块vGPU已经分配给两个虚拟机
Thu Jun 15 23:33:21 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.85.03              Driver Version: 510.85.03                 |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  NVIDIA Graphics Device     | 00000000:82:00.0             |   0%       |
|      3251634189  GRID P40-12C   | 10a1...  y-k8s-n-1           |      0%    |
|      3251634195  GRID P40-12C   | b102...  y-k8s-n-2           |      0%    |
+---------------------------------+------------------------------+------------+

恢复 mdev 设备

当操作系统升级(内核)之后,重启物理服务器,上述 mdev 设备丢失,此时启动 y-k8s-n-1 报错:

error: Failed to start domain 'y-k8s-n-1'
error: device not found: mediated device '3eb9d560-0b31-11ee-91a9-bb28039c61eb' not found

则需要恢复原先的 mdev 设备(依然使用原先的uuid),也就是将上述 vgpu_create 脚本后半段抠出来直接运行(uuid沿用分配给虚拟机的设备uuid):

恢复原先创建过的 mdev 设备
# GPU芯片(按实际配置) => 0000:82:00.0
gpu_bdf="0000:"`lspci | grep GP102GL | awk '{print $1}'`

# vGPU profile从 mdevctl types 中查询
vgpu_type="nvidia-286"

cat << EOF > mdev_id
3eb9d560-0b31-11ee-91a9-bb28039c61eb
3eb9d718-0b31-11ee-91aa-2b17f51ee12d
EOF

for id in `cat mdev_id`;do
    sudo mdevctl start -u $id -p $gpu_bdf -t $vgpu_type
    sudo mdevctl define -a -u $id
done

不过,这里会提示:

Device 3eb9d560-0b31-11ee-91a9-bb28039c61eb on 0000:82:00.0 already defined, try modify?
Device 3eb9d718-0b31-11ee-91aa-2b17f51ee12d on 0000:82:00.0 already defined, try modify?

也就是说,只要执行 mdevctl start 就可以了

安装NVIDIA Virtual GPU Guest Driver

  • 虚拟机内部安装 GCC, Linux Kernel Headers 以及 dkms :

在Ubuntu Guest虚拟机中需要安装GCC,Linux Kernel Headers和dkms
sudo apt install gcc linux-headers-$(uname -r) dkms
  • 登陆虚拟机,安装Guest虚拟机的GRID驱动:

在Ubuntu Guest虚拟机中安装 nvidia-linux-grid Guest驱动
sudo dpkg -i nvidia-linux-grid-510_510.85.02_amd64.deb
  • 然后重启虚拟机

  • 在Ubuntu虚拟机中编辑 /etc/nvidia/gridd.conf 配置:

配置虚拟机 /etc/nvidia/gridd.conf 连接License服务器
# Description: Set License Server Address
# Data type: string
# Format:  "<address>"
ServerAddress=192.168.6.248

# Description: Set License Server port number
# Data type: integer
# Format:  <port>, default is 7070
ServerPort=7070

# Description: Set Feature to be enabled
# Data type: integer
# Possible values:
#    0 => for unlicensed state
#    1 => for NVIDIA vGPU (Optional, autodetected as per vGPU type)
#    2 => for NVIDIA RTX Virtual Workstation
#    4 => for NVIDIA Virtual Compute Server
# All other values reserved
FeatureType=4
  • 启动 nvidia-gridd :

配置Lince Server的IP和端口以及请求License,然后启动 nvidia-gridd
systemctl start nvidia-gridd

观察 Lince Server 服务器的 License Feature Usage 可以看到Licence计数已经减少了1个,也就是被vGPU客户端使用了

备注

一切就绪,现在可以开始: