排查ARM虚拟机磁盘异常

我在 arch linux ARM KVM虚拟化 解决了虚拟机运行问题之后,终于可以进行 Fedora 37 Server安装 。但是,看似顺利的步骤,在安装过程中,却出现磁盘格式化报错(连续尝试了2个虚拟机):

../../_images/arm_vm_disk_fail.png

排查

在虚拟机的终端界面,按下 ctrl-b 2 可以进入 shell 界面(按 3 则是log)

  • 检查磁盘:

    fdisk -l
    

可以看到分区已经如安装过程配置:

Device      Start      End  Sectors  Size Type
/dev/vda1    2048   526335   524288  256M EFI System
/dev/vda2  526336 25163775 24637440 11.7G Linux filesystem


Disk /dev/vdb: 55 GiB, 59055800320 bytes, 115343360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

这说明分区完成

  • 文件系统写入报错,我尝试 mkfs.xfs /dev/sda2 出现如下报错:

    mkfs.xfs: pwrite failed: Input/output error
    libxfs_bwrite: write failed on (unknown) bno 0x177ef00/0x100, err=5
    mkfs.xfs: Releasing dirty buffer to free list!
    found dirty buffer (bulk) on free list!
    mkfs.xfs: pwrite failed: Input/output error
    libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=5
    mkfs.xfs: Releasing dirty buffer to free list!
    found dirty buffer (bulk) on free list!
    mkfs.xfs: pwrite failed: Input/output error
    libxfs_bwrite: write failed on xfs_sb bno 0x0/0x1, err=5
    mkfs.xfs: Releasing dirty buffer to free list!
    found dirty buffer (bulk) on free list!
    mkfs.xfs: pwrite failed: Input/output error
    libxfs_bwrite: write failed on (unknown) bno 0xbbf830/0x2, err=5
    mkfs.xfs: Releasing dirty buffer to free list!
    found dirty buffer (bulk) on free list!
    mkfs.xfs: read failed: Input/output error
    mkfs.xfs: data size check failed
    mkfs.xfs: filesystem failed to initialize
    
  • 检查虚拟机XML配置:

虚拟机a-b-data-2的配置XML
<domain type='kvm' id='1'>
  <name>a-b-data-2</name>
  <uuid>666e6083-bd59-433c-89ec-6614a5006750</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://fedoraproject.org/fedora/37"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='aarch64' machine='virt-7.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-armvirt/aarch64/QEMU_CODE.fd</loader>
    <nvram template='/usr/share/edk2-armvirt/aarch64/QEMU_VARS.fd'>/var/lib/libvirt/qemu/nvram/a-b-data-2_VARS.fd</nvram>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <gic version='3'/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/vg-libvirt/a-b-data-2' index='3'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/vg-libvirt/a-b-data-2-store' index='2'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/Fedora-Server-dvd-aarch64-37-1.7.iso' index='1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x11'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x12'/>
      <alias name='pci.11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x13'/>
      <alias name='pci.12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x14'/>
      <alias name='pci.13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x15'/>
      <alias name='pci.14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:7d:43:23'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/6'/>
      <target type='system-serial' port='0'>
        <model name='pl011'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/6'>
      <source path='/dev/pts/6'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-a-b-data-2/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <audio id='1' type='none'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+968:+968</label>
    <imagelabel>+968:+968</imagelabel>
  </seclabel>
</domain>

libvirt LVM卷管理存储池 看不出异常,但是考虑到之前在 virt-builder快速构建VM工具 验证过使用官方下载的 Fedora Server ARM64虚拟机镜像是可以正常运行的。所以,尝试先 虚拟机磁盘镜像转换成LVM卷管理 :

使用dd命令将raw格式虚拟磁盘复制到LVM卷
dd if=Fedora-Server-37-1.7.aarch64.raw of=/dev/vg-libvirt/a-b-data-2 bs=4M
  • 执行 virsh dumpxml a-b-data-2 > a-b-data-2.xml 备份虚拟机配置

  • 启动虚拟机 virsh start a-b-data-2

  • 通过 virsh console a-b-data-2 观察控制台输出,发现进入了 UEFI shell

难道是 libvirt LVM卷管理存储池 存在问题

  • a-b-data-2 虚拟机的配置中磁盘 /dev/vg-libvirt/a-b-data-2 ,导入到能够正常运行的虚拟机 fedora37 :

将怀疑存在问题的LVM卷 /dev/vg-libvirt/a-b-data-2 添加到可以正常运行的虚拟机 fedora37
virsh attach-disk fedora37 --source /dev/vg-libvirt/a-b-data-1 --target vdb --persistent --driver qemu --subdriver raw

然后启动 fedora37

virsh start fedora37

在虚拟机中,我发现完全可以对 /dev/vg-libvirt/a-b-data-2 对应的 /dev/vdb 进行分区和格式化,复制文件也没有问题。

这说明我在 arch linux ARM KVM虚拟化 创建虚拟机针对磁盘挂载参数(参数在 x86 虚拟机环境运行没有问题)在ARM64平台存在不兼容的问题:

对比我在 arch linux ARM KVM虚拟化 创建虚拟机 virt-install 生成的虚拟机配置磁盘部分,可以看出差异:

  • 存在异常的磁盘配置:

使用virt-install指定磁盘参数后生成的虚拟机磁盘配置,存在无法写入异常
<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
  <source dev='/dev/vg-libvirt/a-b-data-2'/>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
  • 正常的磁盘配置

使用virsh attach-disk命令插入虚拟磁盘,自动生成的虚拟机磁盘配置,可以正常读写
<disk type='block' device='disk'>
  <driver name='qemu' type='raw'/>
  <source dev='/dev/vg-libvirt/a-b-data-2'/>
  <target dev='vdb' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</disk>

可以看出差异部分是磁盘驱动的参数 cache='none' io='native' discard='unmap' ,这些参数是我在 arch linux ARM KVM虚拟化 创建虚拟机命令 virt-install 传递的命令:

virt-install安装指定虚拟磁盘参数 cache=none,io=native 会导致ARM64虚拟机无法写入磁盘
#ARM64架构虚拟磁盘可能不能使用 cache=none,io=native
#但是如果不加参数,则默认就是 cache=none,io=native 导致无法写入磁盘
virt-install \
  --network bridge:virbr0 \
  --name a-b-data-1 \
  --ram=6144 \
  --vcpus=2 \
  --arch aarch64 \
  --boot uefi \
  --disk path=/dev/vg-libvirt/a-b-data-1,format=raw,bus=virtio,cache=none,io=native \
  --disk path=/dev/vg-libvirt/a-b-data-1-store,format=raw,bus=virtio,cache=none,io=native \
  --cdrom=/var/lib/libvirt/images/Fedora-Server-dvd-aarch64-37-1.7.iso

带入的参数 cache=none,io=native

看起来在 ARM64 架构下使用 libvirt LVM卷管理存储池 ,支持 io=native 可能存在异常( 一些资料显示 io=native 可以获得姣好性能 )。并且, 如果不指定任何参数 ,则默认还是 cache=none,io=native ,依然无法写磁盘。

我经过对比尝试另一个常用的优化参数 io=threads 可以解决这个异常:

virt-install安装指定虚拟磁盘参数 cache=writeback,io=threads 可以解决ARM64虚拟机磁盘写入问题
#ARM64架构虚拟磁盘可能不能使用 cache=none,io=native
#但是如果不加参数,则默认就是 cache=none,io=native 导致无法写入磁盘
#所以改为另一种常用模式 io=threads

host="a-b-data-1"

virt-install \
  --network bridge:virbr0 \
  --name ${host} \
  --ram=6144 \
  --vcpus=2 \
  --arch aarch64 \
  --boot uefi \
  --disk path=/dev/vg-libvirt/${host},format=raw,bus=virtio,cache=writeback,io=threads \
  --disk path=/dev/vg-libvirt/${host}-store,format=raw,bus=virtio,cache=writeback,io=threads \
  --cdrom=/var/lib/libvirt/images/Fedora-Server-dvd-aarch64-37-1.7.iso

详情可以参考 arch linux: PCI passthrough via OVMF > Virtio disk > Considerations