On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com> wrote:

On 8/20/2020 6:23 PM, Haibin Zhang wrote:

On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com> wrote:



On 8/20/2020 3:44 PM, Haibin Zhang wrote:

Hi, Ying Fang,
I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung.
Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/openEuler-20.03-LTS.aarch64.qcow2.xz
Qemu command:
qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \
 -cpu host -m 122880 -object iothread,id=iothread1 \
 -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \
 -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \

It works after getting rid of these two memory-backend-ram objects

I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.

 -smp 60,sockets=2,cores=30,threads=1  -bios /usr/share/AAVMF/AAVMF_CODE.fd \
 -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
 -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
 -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \
 -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
 -net none -serial telnet::9001,server,nowait -monitor stdio
Guest startup logs:
[    3.769390] smp: Brought up 2 nodes, 60 CPUs
[    4.125100] SMP: Total of 60 processors activated.
[    4.126246] CPU features: detected: Privileged Access Never
[    4.127625] CPU features: detected: LSE atomic instructions
[    4.128969] CPU features: detected: User Access Override
[    4.130261] CPU features: detected: Data cache clean to Point of Persistence
[    4.131949] CPU features: detected: RAS Extension Support
[    4.133253] CPU features: detected: ARM64 MPAM Extension Support
[    4.134704] CPU features: detected: CRC32 instructions
[    4.184286] CPU: All CPU(s) started at EL1
[    4.185891] alternatives: patching kernel code
[   64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0)
[   64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0
[   64.315251] swapper/0       R  running task        0     1      0 0x0000002a
[   64.316930] Call trace:
[   64.317533]  dump_backtrace+0x0/0x198
[   64.318432]  show_stack+0x24/0x30
[   64.319248]  sched_show_task+0x12c/0x160
[   64.320206]  rcu_check_callbacks+0x780/0x7a8
[   64.321253]  update_process_times+0x34/0x60
[   64.322296]  tick_periodic+0x48/0x100
[   64.323193]  tick_handle_periodic+0x38/0xd8
[   64.324224]  arch_timer_handler_virt+0x38/0x58
[   64.325320]  handle_percpu_devid_irq+0x90/0x248
[   64.326419]  generic_handle_irq+0x34/0x50
[   64.327383]  __handle_domain_irq+0x68/0xc0
[   64.328367]  gic_handle_irq+0x6c/0x170
[   64.329275]  el1_irq+0xb8/0x140
[   64.330040]  __bitmap_weight+0x48/0xa0
[   64.330942]  build_sched_domains+0xd58/0xf50
[   64.331962]  sched_init_domains+0xd4/0x108
[   64.332950]  sched_init_smp+0x3c/0x8c
[   64.333842]  kernel_init_freeable+0x17c/0x34c
[   64.334891]  kernel_init+0x18/0x118
[   64.335735]  ret_from_fork+0x10/0x18

You mean there is something wrong with the NUMA feature, right ?

please try using the xml below and see if you can luanch your VM.

No numa specified in below xml file.

<domain type='kvm' id='17'>
 <name>openeuler-test</name>
 <memory unit='GiB'>32</memory>
 <currentMemory unit='GiB'>32</currentMemory>
 <memtune>
   <hard_limit unit='GiB'>32</hard_limit>
 </memtune>
 <vcpu placement='static' cpuset='7-10'>60</vcpu>
 <iothreads>5</iothreads>
 <cputune>
   <emulatorpin cpuset='7-10'/>
 </cputune>
 <resource>
   <partition>/machine</partition>
 </resource>
 <os>
   <type arch='aarch64' machine='virt-4.1'>hvm</type>
   <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader>
   <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram>
   <boot dev='hd'/>
   <boot dev='cdrom'/>
 </os>
 <features>
   <gic version='3'/>
<acpi/>
 </features>
 <cpu mode='host-passthrough' check='none'>
   <topology sockets='2' cores='30' threads='1'/>
 </cpu>
 <clock offset='utc'>
 </clock>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>restart</on_crash>
 <devices>
   <emulator>/usr/bin/qemu-kvm</emulator>
   <disk type='file' device='disk'>
     <driver name='qemu' type='qcow2' cache='none' io='native'/>
     <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/>
     <backingStore/>
     <target dev='sda' bus='scsi'/>
     <alias name='scsi0-0-0-0'/>
     <address type='drive' controller='0' bus='0' target='0' unit='0'/>
   </disk>
   <controller type='usb' index='0' model='ehci'>
     <alias name='usb'/>
     <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
   </controller>
   <controller type='scsi' index='0' model='virtio-scsi'>
     <alias name='scsi0'/>
     <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
   </controller>
   <controller type='pci' index='0' model='pcie-root'>
     <alias name='pcie.0'/>
   </controller>
   <controller type='pci' index='1' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='1' port='0x8'/>
     <alias name='pci.1'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
   </controller>
   <controller type='pci' index='2' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='2' port='0x9'/>
     <alias name='pci.2'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
   </controller>
   <controller type='pci' index='3' model='pcie-to-pci-bridge'>
     <model name='pcie-pci-bridge'/>
     <alias name='pci.3'/>
     <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
   </controller>
   <controller type='pci' index='4' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='4' port='0xa'/>
     <alias name='pci.4'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
   </controller>
   <controller type='pci' index='5' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='5' port='0xb'/>
     <alias name='pci.5'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
   </controller>
   <controller type='pci' index='6' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='6' port='0xc'/>
     <alias name='pci.6'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
   </controller>
   <controller type='pci' index='7' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='7' port='0xd'/>
     <alias name='pci.7'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
   </controller>
   <controller type='pci' index='8' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='8' port='0xe'/>
     <alias name='pci.8'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
   </controller>
   <controller type='pci' index='9' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='9' port='0xf'/>
     <alias name='pci.9'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
   </controller>
   <controller type='pci' index='10' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='10' port='0x10'/>
     <alias name='pci.10'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
   </controller>
   <controller type='pci' index='11' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='11' port='0x11'/>
     <alias name='pci.11'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
   </controller>
   <controller type='pci' index='12' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='12' port='0x12'/>
     <alias name='pci.12'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
   </controller>
   <controller type='pci' index='13' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='13' port='0x13'/>
     <alias name='pci.13'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
   </controller>
   <controller type='pci' index='14' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='14' port='0x14'/>
     <alias name='pci.14'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
   </controller>
   <controller type='pci' index='15' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='15' port='0x15'/>
     <alias name='pci.15'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
   </controller>
   <controller type='pci' index='16' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='16' port='0x16'/>
     <alias name='pci.16'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
   </controller>
   <controller type='pci' index='17' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='17' port='0x17'/>
     <alias name='pci.17'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
   </controller>
   <controller type='pci' index='18' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='18' port='0x18'/>
     <alias name='pci.18'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
   </controller>
   <controller type='pci' index='19' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='19' port='0x19'/>
     <alias name='pci.19'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
   </controller>
   <controller type='pci' index='20' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='20' port='0x1a'/>
     <alias name='pci.20'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
   </controller>
   <controller type='pci' index='21' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='21' port='0x1b'/>
     <alias name='pci.21'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
   </controller>
   <controller type='pci' index='22' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='22' port='0x1c'/>
     <alias name='pci.22'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
   </controller>
   <controller type='pci' index='23' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='23' port='0x1d'/>
     <alias name='pci.23'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
   </controller>
   <controller type='pci' index='24' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='24' port='0x1e'/>
     <alias name='pci.24'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/>
   </controller>
   <controller type='pci' index='25' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='25' port='0x1f'/>
     <alias name='pci.25'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
   </controller>
   <controller type='pci' index='26' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='26' port='0x20'/>
     <alias name='pci.26'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
   </controller>
   <controller type='pci' index='27' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='27' port='0x21'/>
     <alias name='pci.27'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
   </controller>
   <controller type='pci' index='28' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='28' port='0x22'/>
     <alias name='pci.28'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
   </controller>
   <controller type='pci' index='29' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='29' port='0x23'/>
     <alias name='pci.29'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/>
   </controller>
   <controller type='pci' index='30' model='pcie-root-port'>
     <model name='pcie-root-port'/>
     <target chassis='30' port='0x24'/>
     <alias name='pci.30'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/>
   </controller>
   <controller type='virtio-serial' index='0'>
     <alias name='virtio-serial0'/>
     <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
   </controller>
   <serial type='pty'>
     <source path='/dev/pts/2'/>
     <target port='0'/>
     <alias name='serial0'/>
   </serial>
   <console type='pty' tty='/dev/pts/2'>
     <source path='/dev/pts/2'/>
     <target type='serial' port='0'/>
     <alias name='serial0'/>
   </console>
   <channel type='unix'>
     <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/>
     <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/>
     <alias name='channel0'/>
     <address type='virtio-serial' controller='0' bus='0' port='1'/>
   </channel>
   <channel type='unix'>
     <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/>
     <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
     <alias name='channel1'/>
     <address type='virtio-serial' controller='0' bus='0' port='2'/>
   </channel>
   <channel type='unix'>
     <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/>
     <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/>
     <alias name='channel2'/>
     <address type='virtio-serial' controller='0' bus='0' port='3'/>
   </channel>
   <channel type='unix'>
     <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/>
     <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/>
     <alias name='channel3'/>
     <address type='virtio-serial' controller='0' bus='0' port='4'/>
   </channel>
   <input type='tablet' bus='usb'>
     <alias name='input0'/>
     <address type='usb' bus='0' port='1'/>
   </input>
   <input type='keyboard' bus='usb'>
     <alias name='input1'/>
     <address type='usb' bus='0' port='2'/>
   </input>
   <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'>
     <listen type='address' address='0.0.0.0'/>
   </graphics>
   <video>
     <model type='virtio' heads='1' primary='yes'/>
     <alias name='video0'/>
     <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/>
   </video>
   <memballoon model='virtio'>
     <alias name='balloon0'/>
     <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/>
   </memballoon>
 </devices>
 <seclabel type='dynamic' model='dac' relabel='yes'>
   <label>+0:+0</label>
   <imagelabel>+0:+0</imagelabel>
 </seclabel>
</domain>

I will test your cmdline way later to see if it can be reproduced.

Thanks.
Ying.

.