Gentoo 虚拟化

备注

我在Gentoo上的虚拟化实践还结合了 Gentoo上运行ZFS(xcloud) ，目标是构建在 ZFS 存储上的虚拟机，以获得灵活的存储管理和性能提升。

硬件

大多数现代计算机架构都包括对硬件级别虚拟化的支持:

AMD 的 AMD-V (svm) / Intel 的 Vt-x (vmx):

检查硬件虚拟化支持

grep --color -E "vmx|svm" /proc/cpuinfo

虚拟化扩展必须受处理器支持并在系统固件（通常是主板的固件菜单）中启用，以便可由 Guest 操作系统访问

软件

Hypervisors: QEMU
虚拟化管理工具包: Libvirt虚拟机管理器
GUI: virt-manager

Kernel

在内核激活IOMMU的配置如下

内核激活IOMMU支持

Device Drivers --->
  [*] IOMMU Hardware Support --->
            Generic IOMMU Pagetable Support ----
      [*]   AMD IOMMU support
      <*>     AMD IOMMU Version 2 driver
      [*]   Support for Intel IOMMU using DMA Remapping Devices
      [*]     Support for Shared Virtual Memory with Intel IOMMU
      [*]     Enable Intel DMA Remapping Devices by default
      [*]   Support for Interrupt Remapping

备注

我的内核实践实际上是在 BLFS QEMU 中完成，也就是构建 LFS(Linux from scratch) 自定义内核，所以实际配置略有不同，见在QEMU中运行GPU passthrough的Debian :

LFS(Linux from scratch) 内核激活IOMMU支持

Device Drivers --->
  [*] IOMMU Hardware Support --->
            Generic IOMMU Pagetable Support ----
      [ ]   AMD IOMMU support
      [*]   Support for Intel IOMMU using DMA Remapping Devices
      [*]     Support for Shared Virtual Memory with Intel IOMMU
      [*]     Enable Intel DMA Remapping Devices by default
      [*]     Enable Intel IOMMU scalable mode by default
      [*]     Intel IOMMU performance events
      < >   IOMMU Userspace API
      [*]   Support for Interrupt Remapping
      < >   Virtio IOMMU driver

注意，如果内核配置了 CONFIG_TRIM_UNUSED_KSYMS `` ( **Trim unused exported kernel symbols** )，一种内核安全裁剪功能，常用于嵌入式系统，那么需要多某些符号表设置白名单。否则会出现 ``Failed to add group <n> to KVM VFIO device: Invalid argument 错误消息。详见 kernel 4.7.0 breaks pci passthrough [SOLVED] 和 KVM/VFIO passthrough not working when TRIM_UNUSED_KSYMS is enabled ( 我没有激活这个功能 ):

内核激活了 Trim unused exported kernel symbols 一定要设置符号白名单

[*] Enable loadable module support --->
    [*]   Trim unused exported kernel symbols
    (/path/to/whitelist) Whitelist of symbols to keep in ksymtab

/path/to/whitelist内容如下:

vfio_group_get_external_user
vfio_external_group_match_file
vfio_group_put_external_user
vfio_group_set_kvm
vfio_external_check_extension
vfio_external_user_iommu_id
mdev_get_iommu_device
mdev_bus_type

内核激活VFIO支持:

内核激活VFIO支持

Device Drivers --->
  <M> VFIO Non-Privileged userpsace driver framework --->
      [*]   VFIO No-IOMMU support ----
      <M>   VFIO support for PCI devices
      [*]     VFIO PCI support for VGA devices
      < >   Mediated device driver framework

备注

我的内核实践在 BLFS QEMU 中完成在QEMU中运行GPU passthrough的Debian 最新内核配置有所不同:

LFS(Linux from scratch) 内核激活VFIO支持

Device Drivers --->
  <M> VFIO Non-Privileged userpsace driver framework --->
    -*-   Support for the VFIO group /dev/vfio/$group_id
    -*-     Support for the VFIO container /dev/vfio/vfio
    [*]     VFIO No-IOMMU support
    [ ]   Export VFIO internals in DebugFS (NEW)
    VFIO support for PCI devices  ---> 
      <M>   Generic VFIO support for any PCI devices
      [*]     Generic VFIO PCI support for VGA devices
      [ ]   Generic VFIO PCI extensions for Intel graphics (GVT-d)
      < > VFIO support for VIRTIO NET PCI devices (NEW)

USE flags

QEMU 

/etc/portage/package.use/qemu

# qemu: enable the usbredir and spice USE flags on the qemu package for virt-manager
app-emulation/qemu aio alsa bpf bzip2 curl usbredir spice

Libvirt虚拟机管理器 

备注

If policykit USE flag is not enabled for libvirt package, the libvirt group will not be created when app-emulation/libvirt is emerged. If this is the case, another group, such as wheel must be used for unix_sock_group.

livvirt 有很多USE参数，以下参数是我参考 gentoo linux wiki: libvirt 设置

/etc/portage/package.use/libvirt

# Gentoo官方文档推荐参数
# app-emulation/libvirt pcap virt-network numa fuse macvtap vepa qemu

# 我的初步参数增加了ZFS支持
# libvirt: policykit USE flag , libvirt group will created
app-emulation/libvirt pcap virt-network numa fuse macvtap vepa qemu policykit zfs

# 更为全面的支持参数
# app-emulation/libvirt pcap fuse macvtap vepa glusterfs iscsi iscsi-direct libssh libvirtd lvm nfs numa parted pcap policykit qemu rbd sasl udev virt-network wireshark-plugins zfs

`virt-manager`

/etc/portage/package.use/virtual

net-dns/dnsmasq script
net-libs/gnutls tools pkcs11

# virt-manager
app-emulation/virt-manager gui
net-misc/spice-gtk usbredir
sys-apps/dbus X

# virt-viewer
net-misc/spice-gtk gtk3

# qemu: enable the usbredir and spice USE flags on the qemu package for virt-manager
app-emulation/qemu usbredir spice

# libvirt: policykit USE flag , libvirt group will created
app-emulation/libvirt policykit

建议 virt-manager 启用 gui 方便维护，部分参数 view-viewer / libvirt 重合

`virt-viewer`

对于使用 spice 图形界面协议，需要安装 virt-viewer 软件包并且开启 spice USE flag:

/etc/portage/package.use/virt-viewer

# required by app-emulation/virt-viewer-11.0::gentoo[spice]
# required by virt-viewer (argument)
>=net-misc/spice-gtk-0.42-r4 gtk3

安装

emerge --ask app-emulation/libvirt app-emulation/qemu app-emulation/virt-manager app-emulation/virt-viewer

安装了netcat 出现 virt-manager 但由于包被阻止而失败:

virt-manager 但由于包被阻止而失败

[blocks B      ] net-analyzer/netcat ("net-analyzer/netcat" is soft blocking net-analyzer/openbsd-netcat-1.219_p1)

emerge virt-manager failed with block package (Solved) : 删除 netcat

删除 netcat

emerge -acv net-analyzer/netcat

启动

用户添加到libvirt组:

用户添加到libvirt组

usermod -a -G libvirt huatai

从 libvirtd 配置文件中取消注释以下行:

/etc/libvirt/libvirtd.conf

auth_unix_ro = "none"
auth_unix_rw = "none"
unix_sock_group = "libvirt"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"

Virt-manager 使用 libvirt 作为管理虚拟机的后端: 需要启动 libvirt 守护进程:

添加libvirtd服务

rc-service libvirtd restart
rc-update add libvirtd

此时使用普通用户huatai执行 virsh list 可以观察运行的虚拟机

ZFS存储

如前所述，我在部署Gentoo平台的虚拟化之前，先构建 Gentoo上运行ZFS(xcloud) ，在具备了ZFS基础之后，现在来构建 libvirt ZFS存储池 :

Gentoo上运行ZFS(xcloud) 完成后的当前存储现状

当前存储状态

# zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zpool-data   356G   184K   356G        -         -     0%     0%  1.00x    ONLINE  -

# zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
zpool-data   184K   345G    24K  /var/lib/docker

备注

这里作为测试环境，我采用了混用 libvirt ZFS存储池和 Docker ZFS 存储驱动

构建 libvirt ZFS存储池 :

定义名为 images_zfs 的ZFS类型存储池

virsh pool-define-as --name images_zfs --source-name zpool-data --type zfs

启动这个定义好的ZFS存储池且设置为自动启动:

启动ZFS存储池

virsh pool-autostart images_zfs
virsh pool-start images_zfs

如果一切正常，则检查存储池状态:

检查ZFS存储池状态

virsh pool-list

输出如下:

检查ZFS存储池状态显示已经激活了ZFS存储池

 Name         State    Autostart
----------------------------------
 images_zfs   active   yes

使用ZFS存储创建虚拟机

结合创建KVM虚拟机创建一个Fedora Sway spin 40发行版安装虚拟机:

先创建ZFS卷，然后再使用这个卷创建Fedora虚拟机

# virsh vol-create-as images_zfs fedora-sway-40 10G

virt-install \
     --network bridge:virbr0 \
     --name fedora-sway-40 \
     --ram=4096 \
     --vcpus=2 \
     --os-variant=fedora40 \
     --boot uefi --cpu host-passthrough \
     --disk vol=images_zfs/fedora-sway-40,sparse=false,format=raw,bus=virtio,cache=none,io=native \
     --graphics spice \
     --cdrom=/var/lib/libvirt/images/Fedora-Xfce-Live-x86_64-40-1.14.iso
     #--cdrom=/var/lib/libvirt/images/Fedora-Sway-Live-x86_64-40-1.14.iso

不过，上述仅仅是一个案例，我的真正目标是玩 LFS(Linux from scratch) ，Let's go...

Gentoo 虚拟化

硬件

软件

Kernel

USE flags

QEMU

Libvirt虚拟机管理器

virt-manager

virt-viewer

安装

启动