[Virt] Re: openEuler guest with multi-numa cannot startup

20 Aug 2020


      On 8/20/2020 3:44 PM, Haibin Zhang wrote:
...
Hi, Ying Fang,
I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler 
guest, but it hung.
Qcow2 is downloaded from 
https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o...
Qemu command:
qemu-system-aarch64 -name guest=12345,debug-threads=on -machine 
virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \
  -cpu host -m 122880 -object iothread,id=iothread1 \
  -object 
memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred 
-numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \
  -object 
memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred 
-numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \
  -smp 60,sockets=2,cores=30,threads=1  -bios 
/usr/share/AAVMF/AAVMF_CODE.fd \
  -device 
pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 
\
  -device 
pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
  -drive 
file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none 
\
  -device 
virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
\
  -net none -serial telnet::9001,server,nowait -monitor stdio
Guest startup logs:
[    3.769390] smp: Brought up 2 nodes, 60 CPUs
[    4.125100] SMP: Total of 60 processors activated.
[    4.126246] CPU features: detected: Privileged Access Never
[    4.127625] CPU features: detected: LSE atomic instructions
[    4.128969] CPU features: detected: User Access Override
[    4.130261] CPU features: detected: Data cache clean to Point of 
Persistence
[    4.131949] CPU features: detected: RAS Extension Support
[    4.133253] CPU features: detected: ARM64 MPAM Extension Support
[    4.134704] CPU features: detected: CRC32 instructions
[    4.184286] CPU: All CPU(s) started at EL1
[    4.185891] alternatives: patching kernel code
[   64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0)
[   64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 
(4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0
[   64.315251] swapper/0       R  running task        0     1      0 
0x0000002a
[   64.316930] Call trace:
[   64.317533]  dump_backtrace+0x0/0x198
[   64.318432]  show_stack+0x24/0x30
[   64.319248]  sched_show_task+0x12c/0x160
[   64.320206]  rcu_check_callbacks+0x780/0x7a8
[   64.321253]  update_process_times+0x34/0x60
[   64.322296]  tick_periodic+0x48/0x100
[   64.323193]  tick_handle_periodic+0x38/0xd8
[   64.324224]  arch_timer_handler_virt+0x38/0x58
[   64.325320]  handle_percpu_devid_irq+0x90/0x248
[   64.326419]  generic_handle_irq+0x34/0x50
[   64.327383]  __handle_domain_irq+0x68/0xc0
[   64.328367]  gic_handle_irq+0x6c/0x170
[   64.329275]  el1_irq+0xb8/0x140
[   64.330040]  __bitmap_weight+0x48/0xa0
[   64.330942]  build_sched_domains+0xd58/0xf50
[   64.331962]  sched_init_domains+0xd4/0x108
[   64.332950]  sched_init_smp+0x3c/0x8c
[   64.333842]  kernel_init_freeable+0x17c/0x34c
[   64.334891]  kernel_init+0x18/0x118
[   64.335735]  ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
<domain type='kvm' id='17'>
   <name>openeuler-test</name>
   <memory unit='GiB'>32</memory>
   <currentMemory unit='GiB'>32</currentMemory>
   <memtune>
     <hard_limit unit='GiB'>32</hard_limit>
   </memtune>
   <vcpu placement='static' cpuset='7-10'>60</vcpu>
   <iothreads>5</iothreads>
   <cputune>
     <emulatorpin cpuset='7-10'/>
   </cputune>
   <resource>
     <partition>/machine</partition>
   </resource>
   <os>
     <type arch='aarch64' machine='virt-4.1'>hvm</type>
     <loader readonly='yes' 
type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader>
     <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram>
     <boot dev='hd'/>
     <boot dev='cdrom'/>
   </os>
   <features>
     <gic version='3'/>
    <acpi/>
   </features>
   <cpu mode='host-passthrough' check='none'>
     <topology sockets='2' cores='30' threads='1'/>
   </cpu>
   <clock offset='utc'>
   </clock>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
   <on_crash>restart</on_crash>
   <devices>
     <emulator>/usr/bin/qemu-kvm</emulator>
     <disk type='file' device='disk'>
       <driver name='qemu' type='qcow2' cache='none' io='native'/>
       <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/>
       <backingStore/>
       <target dev='sda' bus='scsi'/>
       <alias name='scsi0-0-0-0'/>
       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
     </disk>
     <controller type='usb' index='0' model='ehci'>
       <alias name='usb'/>
       <address type='pci' domain='0x0000' bus='0x03' slot='0x01' 
function='0x0'/>
     </controller>
     <controller type='scsi' index='0' model='virtio-scsi'>
       <alias name='scsi0'/>
       <address type='pci' domain='0x0000' bus='0x04' slot='0x00' 
function='0x0'/>
     </controller>
     <controller type='pci' index='0' model='pcie-root'>
       <alias name='pcie.0'/>
     </controller>
     <controller type='pci' index='1' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='1' port='0x8'/>
       <alias name='pci.1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x0' multifunction='on'/>
     </controller>
     <controller type='pci' index='2' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='2' port='0x9'/>
       <alias name='pci.2'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
     </controller>
     <controller type='pci' index='3' model='pcie-to-pci-bridge'>
       <model name='pcie-pci-bridge'/>
       <alias name='pci.3'/>
       <address type='pci' domain='0x0000' bus='0x01' slot='0x00' 
function='0x0'/>
     </controller>
     <controller type='pci' index='4' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='4' port='0xa'/>
       <alias name='pci.4'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
     </controller>
     <controller type='pci' index='5' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='5' port='0xb'/>
       <alias name='pci.5'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x3'/>
     </controller>
     <controller type='pci' index='6' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='6' port='0xc'/>
       <alias name='pci.6'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x4'/>
     </controller>
     <controller type='pci' index='7' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='7' port='0xd'/>
       <alias name='pci.7'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x5'/>
     </controller>
     <controller type='pci' index='8' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='8' port='0xe'/>
       <alias name='pci.8'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x6'/>
     </controller>
     <controller type='pci' index='9' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='9' port='0xf'/>
       <alias name='pci.9'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x7'/>
     </controller>
     <controller type='pci' index='10' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='10' port='0x10'/>
       <alias name='pci.10'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0' multifunction='on'/>
     </controller>
     <controller type='pci' index='11' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='11' port='0x11'/>
       <alias name='pci.11'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x1'/>
     </controller>
     <controller type='pci' index='12' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='12' port='0x12'/>
       <alias name='pci.12'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x2'/>
     </controller>
     <controller type='pci' index='13' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='13' port='0x13'/>
       <alias name='pci.13'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x3'/>
     </controller>
     <controller type='pci' index='14' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='14' port='0x14'/>
       <alias name='pci.14'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x4'/>
     </controller>
     <controller type='pci' index='15' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='15' port='0x15'/>
       <alias name='pci.15'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x5'/>
     </controller>
     <controller type='pci' index='16' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='16' port='0x16'/>
       <alias name='pci.16'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x6'/>
     </controller>
     <controller type='pci' index='17' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='17' port='0x17'/>
       <alias name='pci.17'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x7'/>
     </controller>
     <controller type='pci' index='18' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='18' port='0x18'/>
       <alias name='pci.18'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0' multifunction='on'/>
     </controller>
     <controller type='pci' index='19' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='19' port='0x19'/>
       <alias name='pci.19'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x1'/>
     </controller>
     <controller type='pci' index='20' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='20' port='0x1a'/>
       <alias name='pci.20'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x2'/>
     </controller>
     <controller type='pci' index='21' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='21' port='0x1b'/>
       <alias name='pci.21'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x3'/>
     </controller>
     <controller type='pci' index='22' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='22' port='0x1c'/>
       <alias name='pci.22'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x4'/>
     </controller>
     <controller type='pci' index='23' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='23' port='0x1d'/>
       <alias name='pci.23'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x5'/>
     </controller>
     <controller type='pci' index='24' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='24' port='0x1e'/>
       <alias name='pci.24'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x6'/>
     </controller>
     <controller type='pci' index='25' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='25' port='0x1f'/>
       <alias name='pci.25'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x7'/>
     </controller>
     <controller type='pci' index='26' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='26' port='0x20'/>
       <alias name='pci.26'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x0' multifunction='on'/>
     </controller>
     <controller type='pci' index='27' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='27' port='0x21'/>
       <alias name='pci.27'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x1'/>
     </controller>
     <controller type='pci' index='28' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='28' port='0x22'/>
       <alias name='pci.28'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x2'/>
     </controller>
     <controller type='pci' index='29' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='29' port='0x23'/>
       <alias name='pci.29'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x3'/>
     </controller>
     <controller type='pci' index='30' model='pcie-root-port'>
       <model name='pcie-root-port'/>
       <target chassis='30' port='0x24'/>
       <alias name='pci.30'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x4'/>
     </controller>
     <controller type='virtio-serial' index='0'>
       <alias name='virtio-serial0'/>
       <address type='pci' domain='0x0000' bus='0x03' slot='0x02' 
function='0x0'/>
     </controller>
     <serial type='pty'>
       <source path='/dev/pts/2'/>
       <target port='0'/>
       <alias name='serial0'/>
     </serial>
     <console type='pty' tty='/dev/pts/2'>
       <source path='/dev/pts/2'/>
       <target type='serial' port='0'/>
       <alias name='serial0'/>
     </console>
     <channel type='unix'>
       <source mode='bind' 
path='/var/run/libvirt/qemu/fangying-test.extend'/>
       <target type='virtio' name='org.qemu.guest_agent.1' 
state='connected'/>
       <alias name='channel0'/>
       <address type='virtio-serial' controller='0' bus='0' port='1'/>
     </channel>
     <channel type='unix'>
       <source mode='bind' 
path='/var/run/libvirt/qemu/fangying-test.agent'/>
       <target type='virtio' name='org.qemu.guest_agent.0' 
state='connected'/>
       <alias name='channel1'/>
       <address type='virtio-serial' controller='0' bus='0' port='2'/>
     </channel>
     <channel type='unix'>
       <source mode='bind' 
path='/var/run/libvirt/qemu/fangying-test.hostd'/>
       <target type='virtio' name='org.qemu.guest_agent.2' 
state='connected'/>
       <alias name='channel2'/>
       <address type='virtio-serial' controller='0' bus='0' port='3'/>
     </channel>
     <channel type='unix'>
       <source mode='bind' 
path='/var/run/libvirt/qemu/fangying-test.upgraded'/>
       <target type='virtio' name='org.qemu.guest_agent.3' 
state='connected'/>
       <alias name='channel3'/>
       <address type='virtio-serial' controller='0' bus='0' port='4'/>
     </channel>
     <input type='tablet' bus='usb'>
       <alias name='input0'/>
       <address type='usb' bus='0' port='1'/>
     </input>
     <input type='keyboard' bus='usb'>
       <alias name='input1'/>
       <address type='usb' bus='0' port='2'/>
     </input>
     <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'>
       <listen type='address' address='0.0.0.0'/>
     </graphics>
     <video>
       <model type='virtio' heads='1' primary='yes'/>
       <alias name='video0'/>
       <address type='pci' domain='0x0000' bus='0x03' slot='0x04' 
function='0x0'/>
     </video>
     <memballoon model='virtio'>
       <alias name='balloon0'/>
       <address type='pci' domain='0x0000' bus='0x03' slot='0x03' 
function='0x0'/>
     </memballoon>
   </devices>
   <seclabel type='dynamic' model='dac' relabel='yes'>
     <label>+0:+0</label>
     <imagelabel>+0:+0</imagelabel>
   </seclabel>
</domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks.
Ying.

2025

2024

2023

2022

2021

2020

[Virt] Re: openEuler guest with multi-numa cannot startup