PLX 8747硬件PCIe bifurcation

在 PCIe bifurcation 探索时，我曾经想采用硬件来实现PCIe bifucation，市场上主流的硬件PCIe bifurcation是PLX公司芯片，例如 PEX 8747

检查验证

• 先安装一个NVMe存储，启动服务器

• 启动服务器后，检查设备识别，使用 lspci 命令:

使用 lspci 检查输出

lspci | grep -i -e pex -e nvme

输出显示:

使用 lspci 检查输出

82:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:09.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:11.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
85:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO

既然没有识别正常，那么我们再安装一块nvme看看 PCIe bifurcation 效果

• 再次重启服务器，使用相同的 lspci 命令检查，可以看到输出识别出了2个nvme设备:

使用 lspci 检查输出

82:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:09.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
83:11.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
85:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
86:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO

• 检查 fdisk -l 磁盘输出:

使用 fdisk -l 检查输出

Disk /dev/nvme1n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: SAMSUNG MZVL21T0HCLR-00B00
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D3860CB9-4291-4532-B16D-2FCCD292EFBA

Device             Start        End   Sectors   Size Type
/dev/nvme1n1p1      2048  976562175 976560128 465.7G Linux filesystem
/dev/nvme1n1p2 976562176 1562499071 585936896 279.4G Linux LVM


Disk /dev/nvme0n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: SAMSUNG MZVL21T0HCLR-00B00
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BF78F6A8-7654-4646-83B7-8331F77921E1

Device             Start        End   Sectors   Size Type
/dev/nvme0n1p1      2048  976562175 976560128 465.7G Linux filesystem
/dev/nvme0n1p2 976562176 1562499071 585936896 279.4G Linux LVM

验证成功，能够正确识别多NVMe设备

读写验证

备注

由于我的NVMe磁盘已经输出给 Ceph Atlas 虚拟机，难以直接测试读写，暂时忽略