排查ARM虚拟机磁盘异常¶
我在 arch linux ARM KVM虚拟化 解决了虚拟机运行问题之后，终于可以进行 Fedora 37 Server安装 。但是，看似顺利的步骤，在安装过程中，却出现磁盘格式化报错(连续尝试了2个虚拟机):
排查¶
在虚拟机的终端界面，按下 ctrl-b 2 可以进入 shell 界面(按 3 则是log)
检查磁盘:
fdisk -l
可以看到分区已经如安装过程配置:
Device Start End Sectors Size Type
/dev/vda1 2048 526335 524288 256M EFI System
/dev/vda2 526336 25163775 24637440 11.7G Linux filesystem
Disk /dev/vdb: 55 GiB, 59055800320 bytes, 115343360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
这说明分区完成
文件系统写入报错，我尝试
mkfs.xfs /dev/sda2出现如下报错:mkfs.xfs: pwrite failed: Input/output error libxfs_bwrite: write failed on (unknown) bno 0x177ef00/0x100, err=5 mkfs.xfs: Releasing dirty buffer to free list! found dirty buffer (bulk) on free list! mkfs.xfs: pwrite failed: Input/output error libxfs_bwrite: write failed on (unknown) bno 0x0/0x100, err=5 mkfs.xfs: Releasing dirty buffer to free list! found dirty buffer (bulk) on free list! mkfs.xfs: pwrite failed: Input/output error libxfs_bwrite: write failed on xfs_sb bno 0x0/0x1, err=5 mkfs.xfs: Releasing dirty buffer to free list! found dirty buffer (bulk) on free list! mkfs.xfs: pwrite failed: Input/output error libxfs_bwrite: write failed on (unknown) bno 0xbbf830/0x2, err=5 mkfs.xfs: Releasing dirty buffer to free list! found dirty buffer (bulk) on free list! mkfs.xfs: read failed: Input/output error mkfs.xfs: data size check failed mkfs.xfs: filesystem failed to initialize
检查虚拟机XML配置:
虚拟机a-b-data-2的配置XML¶
<domain type='kvm' id='1'>
<name>a-b-data-2</name>
<uuid>666e6083-bd59-433c-89ec-6614a5006750</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://fedoraproject.org/fedora/37"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>6291456</memory>
<currentMemory unit='KiB'>6291456</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='aarch64' machine='virt-7.0'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2-armvirt/aarch64/QEMU_CODE.fd</loader>
<nvram template='/usr/share/edk2-armvirt/aarch64/QEMU_VARS.fd'>/var/lib/libvirt/qemu/nvram/a-b-data-2_VARS.fd</nvram>
<boot dev='cdrom'/>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<gic version='3'/>
</features>
<cpu mode='host-passthrough' check='none'/>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>destroy</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-aarch64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/vg-libvirt/a-b-data-2' index='3'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/vg-libvirt/a-b-data-2-store' index='2'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/Fedora-Server-dvd-aarch64-37-1.7.iso' index='1'/>
<backingStore/>
<target dev='sda' bus='scsi'/>
<readonly/>
<alias name='scsi0-0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<alias name='scsi0'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x8'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x9'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0xa'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0xb'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0xc'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0xd'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0xe'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0xf'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x10'/>
<alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x11'/>
<alias name='pci.10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x12'/>
<alias name='pci.11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x13'/>
<alias name='pci.12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x14'/>
<alias name='pci.13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x15'/>
<alias name='pci.14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:7d:43:23'/>
<source bridge='virbr0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/6'/>
<target type='system-serial' port='0'>
<model name='pl011'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/6'>
<source path='/dev/pts/6'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-a-b-data-2/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<audio id='1' type='none'/>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<alias name='rng0'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+968:+968</label>
<imagelabel>+968:+968</imagelabel>
</seclabel>
</domain>
从 libvirt LVM卷管理存储池 看不出异常，但是考虑到之前在 virt-builder快速构建VM工具 验证过使用官方下载的 Fedora Server ARM64虚拟机镜像是可以正常运行的。所以，尝试先 虚拟机磁盘镜像转换成LVM卷管理 :
使用dd命令将raw格式虚拟磁盘复制到LVM卷¶
dd if=Fedora-Server-37-1.7.aarch64.raw of=/dev/vg-libvirt/a-b-data-2 bs=4M
执行
virsh dumpxml a-b-data-2 > a-b-data-2.xml备份虚拟机配置启动虚拟机
virsh start a-b-data-2通过
virsh console a-b-data-2观察控制台输出，发现进入了 UEFI shell
难道是 libvirt LVM卷管理存储池 存在问题
将
a-b-data-2虚拟机的配置中磁盘/dev/vg-libvirt/a-b-data-2，导入到能够正常运行的虚拟机fedora37:
将怀疑存在问题的LVM卷 /dev/vg-libvirt/a-b-data-2 添加到可以正常运行的虚拟机 fedora37¶
virsh attach-disk fedora37 --source /dev/vg-libvirt/a-b-data-1 --target vdb --persistent --driver qemu --subdriver raw
然后启动 fedora37
virsh start fedora37
在虚拟机中，我发现完全可以对 /dev/vg-libvirt/a-b-data-2 对应的 /dev/vdb 进行分区和格式化，复制文件也没有问题。
这说明我在 arch linux ARM KVM虚拟化 创建虚拟机针对磁盘挂载参数(参数在 x86 虚拟机环境运行没有问题)在ARM64平台存在不兼容的问题:
对比我在 arch linux ARM KVM虚拟化 创建虚拟机 virt-install 生成的虚拟机配置磁盘部分，可以看出差异:
存在异常的磁盘配置:
使用virt-install指定磁盘参数后生成的虚拟机磁盘配置，存在无法写入异常¶
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<source dev='/dev/vg-libvirt/a-b-data-2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
正常的磁盘配置
使用virsh attach-disk命令插入虚拟磁盘，自动生成的虚拟机磁盘配置，可以正常读写¶
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/vg-libvirt/a-b-data-2'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</disk>
可以看出差异部分是磁盘驱动的参数 cache='none' io='native' discard='unmap' ，这些参数是我在 arch linux ARM KVM虚拟化 创建虚拟机命令 virt-install 传递的命令:
virt-install安装指定虚拟磁盘参数 cache=none,io=native 会导致ARM64虚拟机无法写入磁盘¶
#ARM64架构虚拟磁盘可能不能使用 cache=none,io=native
#但是如果不加参数，则默认就是 cache=none,io=native 导致无法写入磁盘
virt-install \
--network bridge:virbr0 \
--name a-b-data-1 \
--ram=6144 \
--vcpus=2 \
--arch aarch64 \
--boot uefi \
--disk path=/dev/vg-libvirt/a-b-data-1,format=raw,bus=virtio,cache=none,io=native \
--disk path=/dev/vg-libvirt/a-b-data-1-store,format=raw,bus=virtio,cache=none,io=native \
--cdrom=/var/lib/libvirt/images/Fedora-Server-dvd-aarch64-37-1.7.iso
带入的参数 cache=none,io=native
看起来在 ARM64 架构下使用 libvirt LVM卷管理存储池 ，支持 io=native 可能存在异常( 一些资料显示 io=native 可以获得姣好性能 )。并且， 如果不指定任何参数 ，则默认还是 cache=none,io=native ，依然无法写磁盘。
我经过对比尝试另一个常用的优化参数 io=threads 可以解决这个异常:
virt-install安装指定虚拟磁盘参数 cache=writeback,io=threads 可以解决ARM64虚拟机磁盘写入问题¶
#ARM64架构虚拟磁盘可能不能使用 cache=none,io=native
#但是如果不加参数，则默认就是 cache=none,io=native 导致无法写入磁盘
#所以改为另一种常用模式 io=threads
host="a-b-data-1"
virt-install \
--network bridge:virbr0 \
--name ${host} \
--ram=6144 \
--vcpus=2 \
--arch aarch64 \
--boot uefi \
--disk path=/dev/vg-libvirt/${host},format=raw,bus=virtio,cache=writeback,io=threads \
--disk path=/dev/vg-libvirt/${host}-store,format=raw,bus=virtio,cache=writeback,io=threads \
--cdrom=/var/lib/libvirt/images/Fedora-Server-dvd-aarch64-37-1.7.iso
详情可以参考 arch linux: PCI passthrough via OVMF > Virtio disk > Considerations