On Aug 21, 2020, at 1:51 PM, Haibin Zhang <haibincheung@hotmail.com> wrote:
On Aug 21, 2020, at 12:39 PM, Ying Fang <fangying1@huawei.com> wrote:
On 8/21/2020 12:05 PM, Haibin Zhang wrote:
On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.com <mailto:fangying1@huawei.com>> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote:
It is partially supported, qemu has support for CPU topologyOn Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.com <mailto:fangying1@huawei.com><mailto:fangying1@huawei.com>> wrote:The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
On 8/21/2020 10:21 AM, Haibin Zhang wrote:
Yes, this may be the clue and I'm quite intrested in this issue.On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com <mailto:fangying1@huawei.com><mailto:fangying1@huawei.com><mailto:fangying1@huawei.com>> wrote:I have specified policy=preferred not strict.
On 8/20/2020 6:23 PM, Haibin Zhang wrote:
I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com <mailto:fangying1@huawei.com> <mailto:fangying1@huawei.com> <mailto:fangying1@huawei.com>> wrote:It works after getting rid of these two memory-backend-ram objects
On 8/20/2020 3:44 PM, Haibin Zhang wrote:
Hi, Ying Fang,
I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung.
Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/openEuler-20.03-LTS.aarch64.qcow2.xz
Qemu command:
qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \
-cpu host -m 122880 -object iothread,id=iothread1 \
-object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \
-object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \
I get some clues below:
https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@hisilicon.com/
It says numa confguration is conflict with cpu affinity specified by MPIDR
Since currently KVM does not support userspace mpdir configuration.
I am now doing some work to enable MPIDR emulation in KVM on aarch64.
configuration. The topology infomation can be reported to GuestOS
via ACPI or fdt. However KVM does not support MPIDR emulation now,
which means guest can see the topology but MPIDR is different.
I'm now working on it to enable MPIDR emulation in KVM so that
the ACPI info can be coordinate with MPIDR.
Please refer to:
https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1-drjones@redhat.com/
These patches have not been pushed to upstream?
The above attachment is my dts dumped from qemu, no socket/core/thread
My qemu version is 4.2.0 from upstream
No, it has not been accepted by upstream qemu, but CPU topology need it.
And besides KVM must support MPIDR emulation which I am now working on.
MRS <Xt>, MPIDR_EL1
In guest, how can I let above instruction trap to kvm?
If arm archtecture support banked MPIDR_EL1 just like ICC and ICV, that would be best.
A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week.-smp 60,sockets=2,cores=30,threads=1 -bios
Please keep in touch with me.
No numa specified in below xml file./usr/share/AAVMF/AAVMF_CODE.fd \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \
-device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-net none -serial telnet::9001,server,nowait -monitor stdio
Guest startup logs:
[ 3.769390] smp: Brought up 2 nodes, 60 CPUs
[ 4.125100] SMP: Total of 60 processors activated.
[ 4.126246] CPU features: detected: Privileged Access Never
[ 4.127625] CPU features: detected: LSE atomic instructions
[ 4.128969] CPU features: detected: User Access Override
[ 4.130261] CPU features: detected: Data cache clean to Point of Persistence
[ 4.131949] CPU features: detected: RAS Extension Support
[ 4.133253] CPU features: detected: ARM64 MPAM Extension Support
[ 4.134704] CPU features: detected: CRC32 instructions
[ 4.184286] CPU: All CPU(s) started at EL1
[ 4.185891] alternatives: patching kernel code
[ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0)
[ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0
[ 64.315251] swapper/0 R running task 0 1 0 0x0000002a
[ 64.316930] Call trace:
[ 64.317533] dump_backtrace+0x0/0x198
[ 64.318432] show_stack+0x24/0x30
[ 64.319248] sched_show_task+0x12c/0x160
[ 64.320206] rcu_check_callbacks+0x780/0x7a8
[ 64.321253] update_process_times+0x34/0x60
[ 64.322296] tick_periodic+0x48/0x100
[ 64.323193] tick_handle_periodic+0x38/0xd8
[ 64.324224] arch_timer_handler_virt+0x38/0x58
[ 64.325320] handle_percpu_devid_irq+0x90/0x248
[ 64.326419] generic_handle_irq+0x34/0x50
[ 64.327383] __handle_domain_irq+0x68/0xc0
[ 64.328367] gic_handle_irq+0x6c/0x170
[ 64.329275] el1_irq+0xb8/0x140
[ 64.330040] __bitmap_weight+0x48/0xa0
[ 64.330942] build_sched_domains+0xd58/0xf50
[ 64.331962] sched_init_domains+0xd4/0x108
[ 64.332950] sched_init_smp+0x3c/0x8c
[ 64.333842] kernel_init_freeable+0x17c/0x34c
[ 64.334891] kernel_init+0x18/0x118
[ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
.
<domain type='kvm' id='17'>
<name>openeuler-test</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<memtune>
<hard_limit unit='GiB'>32</hard_limit>
</memtune>
<vcpu placement='static' cpuset='7-10'>60</vcpu>
<iothreads>5</iothreads>
<cputune>
<emulatorpin cpuset='7-10'/>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='aarch64' machine='virt-4.1'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader>
<nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram>
<boot dev='hd'/>
<boot dev='cdrom'/>
</os>
<features>
<gic version='3'/>
<acpi/>
</features>
<cpu mode='host-passthrough' check='none'>
<topology sockets='2' cores='30' threads='1'/>
</cpu>
<clock offset='utc'>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='native'/>
<source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/>
<backingStore/>
<target dev='sda' bus='scsi'/>
<alias name='scsi0-0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ehci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<alias name='scsi0'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x8'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x9'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-to-pci-bridge'>
<model name='pcie-pci-bridge'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0xa'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0xb'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0xc'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0xd'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0xe'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0xf'/>
<alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x10'/>
<alias name='pci.10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x11'/>
<alias name='pci.11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x12'/>
<alias name='pci.12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x13'/>
<alias name='pci.13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x14'/>
<alias name='pci.14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='15' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='15' port='0x15'/>
<alias name='pci.15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='16' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='16' port='0x16'/>
<alias name='pci.16'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='17' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='17' port='0x17'/>
<alias name='pci.17'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='18' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='18' port='0x18'/>
<alias name='pci.18'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='19' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='19' port='0x19'/>
<alias name='pci.19'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='20' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='20' port='0x1a'/>
<alias name='pci.20'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='21' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='21' port='0x1b'/>
<alias name='pci.21'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='22' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='22' port='0x1c'/>
<alias name='pci.22'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='23' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='23' port='0x1d'/>
<alias name='pci.23'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
</controller>
<controller type='pci' index='24' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='24' port='0x1e'/>
<alias name='pci.24'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/>
</controller>
<controller type='pci' index='25' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='25' port='0x1f'/>
<alias name='pci.25'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
</controller>
<controller type='pci' index='26' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='26' port='0x20'/>
<alias name='pci.26'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='27' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='27' port='0x21'/>
<alias name='pci.27'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
</controller>
<controller type='pci' index='28' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='28' port='0x22'/>
<alias name='pci.28'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
</controller>
<controller type='pci' index='29' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='29' port='0x23'/>
<alias name='pci.29'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/>
</controller>
<controller type='pci' index='30' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='30' port='0x24'/>
<alias name='pci.30'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/>
</controller>
<serial type='pty'>
<source path='/dev/pts/2'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/>
<target type='virtio' name='org.qemu.guest_agent.1' state='connected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/>
<target type='virtio' name='org.qemu.guest_agent.2' state='connected'/>
<alias name='channel2'/>
<address type='virtio-serial' controller='0' bus='0' port='3'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/>
<target type='virtio' name='org.qemu.guest_agent.3' state='connected'/>
<alias name='channel3'/>
<address type='virtio-serial' controller='0' bus='0' port='4'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='keyboard' bus='usb'>
<alias name='input1'/>
<address type='usb' bus='0' port='2'/>
</input>
<graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='virtio' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks.
Ying.