On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote: On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote: On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host. I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64. The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ? It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
These patches have not been pushed to upstream?
The above attachment is my dts dumped from qemu, no socket/core/thread
My qemu version is 4.2.0 from upstream
-smp 60,sockets=2,cores=30,threads=1 -bios A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
/usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM. No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.