On 7/2/2020 3:18 PM, Haibin Zhang wrote:
Hi, Ying Fang.
Hi Haibin, Thanks for report this issue for us. We will verify and response to you later.
I use qemu create a VM (96 cores and 360g ram) on my kunpeng 920-4826 server, then do unixbench tests. The performance of pipe-based context switching decreases severely, which shows below:
Guest# ./context1 60 COUNT|4779535|1|lps COUNT|4779536|1|lps
Host# ./context1 60 COUNT|13990745|1|lps COUNT|13990746|1|lps
Could you provide more details about your OS and virtualization software version information for us?
As you know, pipe-based context switching measures the number of times two processes can exchange an increasing integer through a pipe in a duration (like 60 seconds). Times of guest is only 34% as that of host. Attachment is the context1.c from unixbench. It looks simple, just spawn a child process with which it carries on a bi-directional pipe conversation.
Yes, we had knowledge about this context test now. Thanks.
Guest# taskset -c 20 ./context1 60 COUNT|18059443|1|lps COUNT|18059443|1|lps
Host# taskset -c 20 ./context1 60 COUNT|17881974|1|lps COUNT|17881975|1|lps
Use taskset to set cpu affinity, which makes results better.
Can you help me fix this issue? Maybe WFE/WFI traps lead the performance cost? Is there PLE(Pause Loop Exit) feature on Kunpeng?
We had some performance tuning techniques in the openEuler OS. And Xie XiangYou is the expert on this area, maybe he could help you with it.
Moreover, you can drop us an issue on gitee via: https://gitee.com/src-openeuler/qemu
On Jul 2, 2020, at 3:36 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 7/2/2020 3:18 PM, Haibin Zhang wrote: Hi, Ying Fang. Hi Haibin, Thanks for report this issue for us. We will verify and response to you later.
I use qemu create a VM (96 cores and 360g ram) on my kunpeng 920-4826 server, then do unixbench tests. The performance of pipe-based context switching decreases severely, which shows below: Guest# ./context1 60 COUNT|4779535|1|lps COUNT|4779536|1|lps Host# ./context1 60 COUNT|13990745|1|lps COUNT|13990746|1|lps Could you provide more details about your OS and virtualization software version information for us?
Host/guest OS : CentOS-8.1.1911-aarch64 Qemu : https://gitee.com/src-openeuler/qemu.git
As you know, pipe-based context switching measures the number of times two processes can exchange an increasing integer through a pipe in a duration (like 60 seconds). Times of guest is only 34% as that of host. Attachment is the context1.c from unixbench. It looks simple, just spawn a child process with which it carries on a bi-directional pipe conversation.
Yes, we had knowledge about this context test now. Thanks. Guest# taskset -c 20 ./context1 60 COUNT|18059443|1|lps COUNT|18059443|1|lps Host# taskset -c 20 ./context1 60 COUNT|17881974|1|lps COUNT|17881975|1|lps Use taskset to set cpu affinity, which makes results better. Can you help me fix this issue? Maybe WFE/WFI traps lead the performance cost? Is there PLE(Pause Loop Exit) feature on Kunpeng?
We had some performance tuning techniques in the openEuler OS. And Xie XiangYou is the expert on this area, maybe he could help you with it.
Ok, install openEuler OS and test again
Moreover, you can drop us an issue on gitee via: https://gitee.com/src-openeuler/qemu
On Jul 2, 2020, at 4:07 PM, Haibin Zhang <haibincheung@hotmail.commailto:haibincheung@hotmail.com> wrote:
On Jul 2, 2020, at 3:36 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 7/2/2020 3:18 PM, Haibin Zhang wrote: Hi, Ying Fang. Hi Haibin, Thanks for report this issue for us. We will verify and response to you later.
I use qemu create a VM (96 cores and 360g ram) on my kunpeng 920-4826 server, then do unixbench tests. The performance of pipe-based context switching decreases severely, which shows below: Guest# ./context1 60 COUNT|4779535|1|lps COUNT|4779536|1|lps Host# ./context1 60 COUNT|13990745|1|lps COUNT|13990746|1|lps Could you provide more details about your OS and virtualization software version information for us?
Host/guest OS : CentOS-8.1.1911-aarch64 Qemu : https://gitee.com/src-openeuler/qemu.git
As you know, pipe-based context switching measures the number of times two processes can exchange an increasing integer through a pipe in a duration (like 60 seconds). Times of guest is only 34% as that of host. Attachment is the context1.c from unixbench. It looks simple, just spawn a child process with which it carries on a bi-directional pipe conversation.
Yes, we had knowledge about this context test now. Thanks. Guest# taskset -c 20 ./context1 60 COUNT|18059443|1|lps COUNT|18059443|1|lps Host# taskset -c 20 ./context1 60 COUNT|17881974|1|lps COUNT|17881975|1|lps Use taskset to set cpu affinity, which makes results better. Can you help me fix this issue? Maybe WFE/WFI traps lead the performance cost? Is there PLE(Pause Loop Exit) feature on Kunpeng?
We had some performance tuning techniques in the openEuler OS. And Xie XiangYou is the expert on this area, maybe he could help you with it.
Ok, install openEuler OS and test again
Host/Guest: openEuler release 20.03 (LTS)
Unixbench results: System Benchmarks Partial Index BASELINE RESULT INDEX Pipe-based Context Switching 4000.0 206994.4 517.5 -> Host Pipe-based Context Switching 4000.0 108939.5 272.3 -> Guest
Moreover, you can drop us an issue on gitee via: https://gitee.com/src-openeuler/qemu
On 7/6/2020 5:16 PM, Haibin Zhang wrote:
On Jul 2, 2020, at 4:07 PM, Haibin Zhang <haibincheung@hotmail.com mailto:haibincheung@hotmail.com> wrote:
On Jul 2, 2020, at 3:36 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 7/2/2020 3:18 PM, Haibin Zhang wrote:
Hi, Ying Fang.
Hi Haibin, Thanks for report this issue for us. We will verify and response to you later.
I use qemu create a VM (96 cores and 360g ram) on my kunpeng 920-4826 server, then do unixbench tests. The performance of pipe-based context switching decreases severely, which shows below: Guest# ./context1 60 COUNT|4779535|1|lps COUNT|4779536|1|lps Host# ./context1 60 COUNT|13990745|1|lps COUNT|13990746|1|lps
Could you provide more details about your OS and virtualization software version information for us?
Host/guest OS : CentOS-8.1.1911-aarch64 Qemu : https://gitee.com/src-openeuler/qemu.git
As you know, pipe-based context switching measures the number of times two processes can exchange an increasing integer through a pipe in a duration (like 60 seconds). Times of guest is only 34% as that of host. Attachment is the context1.c from unixbench. It looks simple, just spawn a child process with which it carries on a bi-directional pipe conversation.
Yes, we had knowledge about this context test now. Thanks.
Guest# taskset -c 20 ./context1 60 COUNT|18059443|1|lps COUNT|18059443|1|lps Host# taskset -c 20 ./context1 60 COUNT|17881974|1|lps COUNT|17881975|1|lps Use taskset to set cpu affinity, which makes results better. Can you help me fix this issue? Maybe WFE/WFI traps lead the performance cost? Is there PLE(Pause Loop Exit) feature on Kunpeng?
We had some performance tuning techniques in the openEuler OS. And Xie XiangYou is the expert on this area, maybe he could help you with it.
Ok, install openEuler OS and test again
Host/Guest: openEuler release 20.03 (LTS)
Unixbench results: System Benchmarks Partial Index BASELINE RESULT INDEX Pipe-based Context Switching 4000.0 206994.4 517.5 -> Host Pipe-based Context Switching 4000.0 108939.5 272.3 -> Guest
Hi, Haibin:
We did a test for UnixBench context1 on KunPeng 930 Server:
HostOS 128U256G: openeulerversion=openEuler-20.03-LTS compiletime=2020-03-28-18-00-01 gccversion=7.3.0-20190804.h30.oe1 kernelversion=4.19.90-2003.4.0.0036.oe1 openjdkversion=1.8.0.242.b08-1.h5.oe1
GuestOS 16U32G: openeulerversion=openEuler-20.03-LTS compiletime=2020-03-24-06-52-56 gccversion=7.3.0-20190804.h30.oe1 kernelversion=4.19.90-2003.4.0.0036.oe1 openjdkversion=1.8.0.242.b08-1.h5.oe1
HostOS: COUNT|14025377|1|lps COUNT|14025378|1|lps GuestOS: COUNT|13498345|1|lps COUNT|13498345|1|lps
Moreover, you can drop us an issue on gitee via: https://gitee.com/src-openeuler/qemu
On Jul 7, 2020, at 4:41 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 7/6/2020 5:16 PM, Haibin Zhang wrote: On Jul 2, 2020, at 4:07 PM, Haibin Zhang <haibincheung@hotmail.commailto:haibincheung@hotmail.com mailto:haibincheung@hotmail.com> wrote:
On Jul 2, 2020, at 3:36 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 7/2/2020 3:18 PM, Haibin Zhang wrote: Hi, Ying Fang. Hi Haibin, Thanks for report this issue for us. We will verify and response to you later.
I use qemu create a VM (96 cores and 360g ram) on my kunpeng 920-4826 server, then do unixbench tests. The performance of pipe-based context switching decreases severely, which shows below: Guest# ./context1 60 COUNT|4779535|1|lps COUNT|4779536|1|lps Host# ./context1 60 COUNT|13990745|1|lps COUNT|13990746|1|lps Could you provide more details about your OS and virtualization software version information for us?
Host/guest OS : CentOS-8.1.1911-aarch64 Qemu : https://gitee.com/src-openeuler/qemu.git
As you know, pipe-based context switching measures the number of times two processes can exchange an increasing integer through a pipe in a duration (like 60 seconds). Times of guest is only 34% as that of host. Attachment is the context1.c from unixbench. It looks simple, just spawn a child process with which it carries on a bi-directional pipe conversation.
Yes, we had knowledge about this context test now. Thanks. Guest# taskset -c 20 ./context1 60 COUNT|18059443|1|lps COUNT|18059443|1|lps Host# taskset -c 20 ./context1 60 COUNT|17881974|1|lps COUNT|17881975|1|lps Use taskset to set cpu affinity, which makes results better. Can you help me fix this issue? Maybe WFE/WFI traps lead the performance cost? Is there PLE(Pause Loop Exit) feature on Kunpeng?
We had some performance tuning techniques in the openEuler OS. And Xie XiangYou is the expert on this area, maybe he could help you with it.
Ok, install openEuler OS and test again Host/Guest: openEuler release 20.03 (LTS) Unixbench results: System Benchmarks Partial Index BASELINE RESULT INDEX Pipe-based Context Switching 4000.0 206994.4 517.5 -> Host Pipe-based Context Switching 4000.0 108939.5 272.3 -> Guest
Hi, Haibin:
We did a test for UnixBench context1 on KunPeng 930 Server:
HostOS 128U256G: openeulerversion=openEuler-20.03-LTS compiletime=2020-03-28-18-00-01 gccversion=7.3.0-20190804.h30.oe1 kernelversion=4.19.90-2003.4.0.0036.oe1 openjdkversion=1.8.0.242.b08-1.h5.oe1
GuestOS 16U32G: openeulerversion=openEuler-20.03-LTS compiletime=2020-03-24-06-52-56 gccversion=7.3.0-20190804.h30.oe1 kernelversion=4.19.90-2003.4.0.0036.oe1 openjdkversion=1.8.0.242.b08-1.h5.oe1
HostOS: COUNT|14025377|1|lps COUNT|14025378|1|lps GuestOS: COUNT|13498345|1|lps COUNT|13498345|1|lps
Are there some hardware enhancements on Kunpeng 930? On Kunpeng 920, results show as follow.
HostOS: COUNT|12008901|1|lps COUNT|12008901|1|lps GuestOS: COUNT|7288718|1|lps COUNT|7288718|1|lps
I can make sure that WFI TRAP causes the performance lost. Stats from /sys/kernel/debug/kvm/ shows that guest WFI_exits more frequently during the test. Context1 processes wait for block pipe and call WFI instruction to yield cpu core.
Moreover, you can drop us an issue on gitee via: https://gitee.com/src-openeuler/qemu
On 7/8/2020 11:49 AM, Haibin Zhang wrote:
On Jul 7, 2020, at 4:41 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 7/6/2020 5:16 PM, Haibin Zhang wrote:
On Jul 2, 2020, at 4:07 PM, Haibin Zhang <haibincheung@hotmail.com mailto:haibincheung@hotmail.commailto:haibincheung@hotmail.com> wrote:
On Jul 2, 2020, at 3:36 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 7/2/2020 3:18 PM, Haibin Zhang wrote:
Hi, Ying Fang.
Hi Haibin, Thanks for report this issue for us. We will verify and response to you later.
I use qemu create a VM (96 cores and 360g ram) on my kunpeng 920-4826 server, then do unixbench tests. The performance of pipe-based context switching decreases severely, which shows below: Guest# ./context1 60 COUNT|4779535|1|lps COUNT|4779536|1|lps Host# ./context1 60 COUNT|13990745|1|lps COUNT|13990746|1|lps
Could you provide more details about your OS and virtualization software version information for us?
Host/guest OS : CentOS-8.1.1911-aarch64 Qemu : https://gitee.com/src-openeuler/qemu.git
As you know, pipe-based context switching measures the number of times two processes can exchange an increasing integer through a pipe in a duration (like 60 seconds). Times of guest is only 34% as that of host. Attachment is the context1.c from unixbench. It looks simple, just spawn a child process with which it carries on a bi-directional pipe conversation.
Yes, we had knowledge about this context test now. Thanks.
Guest# taskset -c 20 ./context1 60 COUNT|18059443|1|lps COUNT|18059443|1|lps Host# taskset -c 20 ./context1 60 COUNT|17881974|1|lps COUNT|17881975|1|lps Use taskset to set cpu affinity, which makes results better. Can you help me fix this issue? Maybe WFE/WFI traps lead the performance cost? Is there PLE(Pause Loop Exit) feature on Kunpeng?
We had some performance tuning techniques in the openEuler OS. And Xie XiangYou is the expert on this area, maybe he could help you with it.
Ok, install openEuler OS and test again
Host/Guest: openEuler release 20.03 (LTS) Unixbench results: System Benchmarks Partial Index BASELINE RESULT INDEX Pipe-based Context Switching 4000.0 206994.4 517.5 -> Host Pipe-based Context Switching 4000.0 108939.5 272.3 -> Guest
Hi, Haibin:
We did a test for UnixBench context1 on KunPeng 930 Server:
HostOS 128U256G: openeulerversion=openEuler-20.03-LTS compiletime=2020-03-28-18-00-01 gccversion=7.3.0-20190804.h30.oe1 kernelversion=4.19.90-2003.4.0.0036.oe1 openjdkversion=1.8.0.242.b08-1.h5.oe1
GuestOS 16U32G: openeulerversion=openEuler-20.03-LTS compiletime=2020-03-24-06-52-56 gccversion=7.3.0-20190804.h30.oe1 kernelversion=4.19.90-2003.4.0.0036.oe1 openjdkversion=1.8.0.242.b08-1.h5.oe1
HostOS: COUNT|14025377|1|lps COUNT|14025378|1|lps GuestOS: COUNT|13498345|1|lps COUNT|13498345|1|lps
Are there some hardware enhancements on Kunpeng 930?
Sorry, it's spelling mistake, I mean Hisilicon 1630 thus the KunPeng 920 server.
On Kunpeng 920, results show as follow.
HostOS: COUNT|12008901|1|lps COUNT|12008901|1|lps GuestOS: COUNT|7288718|1|lps COUNT|7288718|1|lps
I can make sure that WFI TRAP causes the performance lost. Stats from /sys/kernel/debug/kvm/ shows that guest WFI_exits more frequently during the test. Context1 processes wait for block pipe and call WFI instruction to yield cpu core.
This performance lost seems to be related with CentOS and KunPeng 920 Server(with 96 cores), and We have noticed this. Our colleague kezhiming has also noticed this, we are investigating on this.
Moreover, you can drop us an issue on gitee via: https://gitee.com/src-openeuler/qemu
Hi, Ying Fang,
I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o...
Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ -smp 60,sockets=2,cores=30,threads=1 -bios /usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio
Guest startup logs:
[ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
On 8/20/2020 3:44 PM, Haibin Zhang wrote:
Hi, Ying Fang,
I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o...
Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ -smp 60,sockets=2,cores=30,threads=1 -bios /usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio
Guest startup logs:
[ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
On Aug 20, 2020, at 5:08 PM, Ying Fang fangying1@huawei.com wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote:
Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \
It works after getting rid of these two memory-backend-ram objects
-smp 60,sockets=2,cores=30,threads=1 -bios /usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
On 8/20/2020 6:23 PM, Haibin Zhang wrote:
On Aug 20, 2020, at 5:08 PM, Ying Fang fangying1@huawei.com wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote:
Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \
It works after getting rid of these two memory-backend-ram objects
I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.
-smp 60,sockets=2,cores=30,threads=1 -bios /usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.
I have specified policy=preferred not strict.
I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his...
It says numa confguration is conflict with cpu affinity specified by MPIDR
-smp 60,sockets=2,cores=30,threads=1 -bios /usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM. No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On 8/21/2020 10:21 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote:
On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote:
Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \
It works after getting rid of these two memory-backend-ram objects
I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.
I have specified policy=preferred not strict.
I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his...
It says numa confguration is conflict with cpu affinity specified by MPIDR
Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64.
-smp 60,sockets=2,cores=30,threads=1 -bios
A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
/usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote: On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host. I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64.
The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
-smp 60,sockets=2,cores=30,threads=1 -bios A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
/usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM. No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On 8/21/2020 11:23 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote:
On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: > Hi, Ying Fang, > I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a > openEuler guest, but it hung. > Qcow2 is downloaded from > https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... > Qemu command: > qemu-system-aarch64 -name guest=12345,debug-threads=on -machine > virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ > -cpu host -m 122880 -object iothread,id=iothread1 \ > -object > memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred > -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ > -object > memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred > -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \
It works after getting rid of these two memory-backend-ram objects
I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.
I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR
Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64.
The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
> -smp 60,sockets=2,cores=30,threads=1 -bios
A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
> /usr/share/AAVMF/AAVMF_CODE.fd \ > -device > pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 > \ > -device > pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ > -drive > file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none > \ > -device > virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > \ > -net none -serial telnet::9001,server,nowait -monitor stdio > Guest startup logs: > [ 3.769390] smp: Brought up 2 nodes, 60 CPUs > [ 4.125100] SMP: Total of 60 processors activated. > [ 4.126246] CPU features: detected: Privileged Access Never > [ 4.127625] CPU features: detected: LSE atomic instructions > [ 4.128969] CPU features: detected: User Access Override > [ 4.130261] CPU features: detected: Data cache clean to Point > of Persistence > [ 4.131949] CPU features: detected: RAS Extension Support > [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support > [ 4.134704] CPU features: detected: CRC32 instructions > [ 4.184286] CPU: All CPU(s) started at EL1 > [ 4.185891] alternatives: patching kernel code > [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: > [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) > [ 64.312260] rcu: All QSes seen, last rcu_sched kthread > activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, > root ->qsmask 0x0 > [ 64.315251] swapper/0 R running task 0 1 > 0 0x0000002a > [ 64.316930] Call trace: > [ 64.317533] dump_backtrace+0x0/0x198 > [ 64.318432] show_stack+0x24/0x30 > [ 64.319248] sched_show_task+0x12c/0x160 > [ 64.320206] rcu_check_callbacks+0x780/0x7a8 > [ 64.321253] update_process_times+0x34/0x60 > [ 64.322296] tick_periodic+0x48/0x100 > [ 64.323193] tick_handle_periodic+0x38/0xd8 > [ 64.324224] arch_timer_handler_virt+0x38/0x58 > [ 64.325320] handle_percpu_devid_irq+0x90/0x248 > [ 64.326419] generic_handle_irq+0x34/0x50 > [ 64.327383] __handle_domain_irq+0x68/0xc0 > [ 64.328367] gic_handle_irq+0x6c/0x170 > [ 64.329275] el1_irq+0xb8/0x140 > [ 64.330040] __bitmap_weight+0x48/0xa0 > [ 64.330942] build_sched_domains+0xd58/0xf50 > [ 64.331962] sched_init_domains+0xd4/0x108 > [ 64.332950] sched_init_smp+0x3c/0x8c > [ 64.333842] kernel_init_freeable+0x17c/0x34c > [ 64.334891] kernel_init+0x18/0x118 > [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM.
No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote: On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote: On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host. I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64. The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ? It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
These patches have not been pushed to upstream?
The above attachment is my dts dumped from qemu, no socket/core/thread
My qemu version is 4.2.0 from upstream
-smp 60,sockets=2,cores=30,threads=1 -bios A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
/usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM. No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On 8/21/2020 12:05 PM, Haibin Zhang wrote:
On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: >> On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com >> mailto:fangying1@huawei.com mailto:fangying1@huawei.com >> mailto:fangying1@huawei.com> wrote: >> >> >> >> On 8/20/2020 3:44 PM, Haibin Zhang wrote: >>> Hi, Ying Fang, >>> I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a >>> openEuler guest, but it hung. >>> Qcow2 is downloaded from >>> https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... >>> Qemu command: >>> qemu-system-aarch64 -name guest=12345,debug-threads=on -machine >>> virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ >>> -cpu host -m 122880 -object iothread,id=iothread1 \ >>> -object >>> memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred >>> -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ >>> -object >>> memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred >>> -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ > It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.
I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR
Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64.
The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
These patches have not been pushed to upstream?
The above attachment is my dts dumped from qemu, no socket/core/thread
My qemu version is 4.2.0 from upstream
No, it has not been accepted by upstream qemu, but CPU topology need it. And besides KVM must support MPIDR emulation which I am now working on.
>>> -smp 60,sockets=2,cores=30,threads=1 -bios
A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
>>> /usr/share/AAVMF/AAVMF_CODE.fd \ >>> -device >>> pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 >>> \ >>> -device >>> pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 >>> \ >>> -drive >>> file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none >>> \ >>> -device >>> virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>> \ >>> -net none -serial telnet::9001,server,nowait -monitor stdio >>> Guest startup logs: >>> [ 3.769390] smp: Brought up 2 nodes, 60 CPUs >>> [ 4.125100] SMP: Total of 60 processors activated. >>> [ 4.126246] CPU features: detected: Privileged Access Never >>> [ 4.127625] CPU features: detected: LSE atomic instructions >>> [ 4.128969] CPU features: detected: User Access Override >>> [ 4.130261] CPU features: detected: Data cache clean to >>> Point of Persistence >>> [ 4.131949] CPU features: detected: RAS Extension Support >>> [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support >>> [ 4.134704] CPU features: detected: CRC32 instructions >>> [ 4.184286] CPU: All CPU(s) started at EL1 >>> [ 4.185891] alternatives: patching kernel code >>> [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: >>> [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) >>> [ 64.312260] rcu: All QSes seen, last rcu_sched kthread >>> activity 15002 (4294908290-4294893288), >>> jiffies_till_next_fqs=1, root ->qsmask 0x0 >>> [ 64.315251] swapper/0 R running task 0 1 >>> 0 0x0000002a >>> [ 64.316930] Call trace: >>> [ 64.317533] dump_backtrace+0x0/0x198 >>> [ 64.318432] show_stack+0x24/0x30 >>> [ 64.319248] sched_show_task+0x12c/0x160 >>> [ 64.320206] rcu_check_callbacks+0x780/0x7a8 >>> [ 64.321253] update_process_times+0x34/0x60 >>> [ 64.322296] tick_periodic+0x48/0x100 >>> [ 64.323193] tick_handle_periodic+0x38/0xd8 >>> [ 64.324224] arch_timer_handler_virt+0x38/0x58 >>> [ 64.325320] handle_percpu_devid_irq+0x90/0x248 >>> [ 64.326419] generic_handle_irq+0x34/0x50 >>> [ 64.327383] __handle_domain_irq+0x68/0xc0 >>> [ 64.328367] gic_handle_irq+0x6c/0x170 >>> [ 64.329275] el1_irq+0xb8/0x140 >>> [ 64.330040] __bitmap_weight+0x48/0xa0 >>> [ 64.330942] build_sched_domains+0xd58/0xf50 >>> [ 64.331962] sched_init_domains+0xd4/0x108 >>> [ 64.332950] sched_init_smp+0x3c/0x8c >>> [ 64.333842] kernel_init_freeable+0x17c/0x34c >>> [ 64.334891] kernel_init+0x18/0x118 >>> [ 64.335735] ret_from_fork+0x10/0x18 >> >> You mean there is something wrong with the NUMA feature, right ? >> >> please try using the xml below and see if you can luanch your VM. > No numa specified in below xml file. >> >> <domain type='kvm' id='17'> >> <name>openeuler-test</name> >> <memory unit='GiB'>32</memory> >> <currentMemory unit='GiB'>32</currentMemory> >> <memtune> >> <hard_limit unit='GiB'>32</hard_limit> >> </memtune> >> <vcpu placement='static' cpuset='7-10'>60</vcpu> >> <iothreads>5</iothreads> >> <cputune> >> <emulatorpin cpuset='7-10'/> >> </cputune> >> <resource> >> <partition>/machine</partition> >> </resource> >> <os> >> <type arch='aarch64' machine='virt-4.1'>hvm</type> >> <loader readonly='yes' >> type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> >> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> >> <boot dev='hd'/> >> <boot dev='cdrom'/> >> </os> >> <features> >> <gic version='3'/> >> <acpi/> >> </features> >> <cpu mode='host-passthrough' check='none'> >> <topology sockets='2' cores='30' threads='1'/> >> </cpu> >> <clock offset='utc'> >> </clock> >> <on_poweroff>destroy</on_poweroff> >> <on_reboot>restart</on_reboot> >> <on_crash>restart</on_crash> >> <devices> >> <emulator>/usr/bin/qemu-kvm</emulator> >> <disk type='file' device='disk'> >> <driver name='qemu' type='qcow2' cache='none' io='native'/> >> <source >> file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> >> <backingStore/> >> <target dev='sda' bus='scsi'/> >> <alias name='scsi0-0-0-0'/> >> <address type='drive' controller='0' bus='0' target='0' >> unit='0'/> >> </disk> >> <controller type='usb' index='0' model='ehci'> >> <alias name='usb'/> >> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' >> function='0x0'/> >> </controller> >> <controller type='scsi' index='0' model='virtio-scsi'> >> <alias name='scsi0'/> >> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' >> function='0x0'/> >> </controller> >> <controller type='pci' index='0' model='pcie-root'> >> <alias name='pcie.0'/> >> </controller> >> <controller type='pci' index='1' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='1' port='0x8'/> >> <alias name='pci.1'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x0' multifunction='on'/> >> </controller> >> <controller type='pci' index='2' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='2' port='0x9'/> >> <alias name='pci.2'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x1'/> >> </controller> >> <controller type='pci' index='3' model='pcie-to-pci-bridge'> >> <model name='pcie-pci-bridge'/> >> <alias name='pci.3'/> >> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' >> function='0x0'/> >> </controller> >> <controller type='pci' index='4' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='4' port='0xa'/> >> <alias name='pci.4'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x2'/> >> </controller> >> <controller type='pci' index='5' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='5' port='0xb'/> >> <alias name='pci.5'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x3'/> >> </controller> >> <controller type='pci' index='6' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='6' port='0xc'/> >> <alias name='pci.6'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x4'/> >> </controller> >> <controller type='pci' index='7' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='7' port='0xd'/> >> <alias name='pci.7'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x5'/> >> </controller> >> <controller type='pci' index='8' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='8' port='0xe'/> >> <alias name='pci.8'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x6'/> >> </controller> >> <controller type='pci' index='9' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='9' port='0xf'/> >> <alias name='pci.9'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x7'/> >> </controller> >> <controller type='pci' index='10' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='10' port='0x10'/> >> <alias name='pci.10'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x0' multifunction='on'/> >> </controller> >> <controller type='pci' index='11' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='11' port='0x11'/> >> <alias name='pci.11'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x1'/> >> </controller> >> <controller type='pci' index='12' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='12' port='0x12'/> >> <alias name='pci.12'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x2'/> >> </controller> >> <controller type='pci' index='13' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='13' port='0x13'/> >> <alias name='pci.13'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x3'/> >> </controller> >> <controller type='pci' index='14' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='14' port='0x14'/> >> <alias name='pci.14'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x4'/> >> </controller> >> <controller type='pci' index='15' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='15' port='0x15'/> >> <alias name='pci.15'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x5'/> >> </controller> >> <controller type='pci' index='16' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='16' port='0x16'/> >> <alias name='pci.16'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x6'/> >> </controller> >> <controller type='pci' index='17' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='17' port='0x17'/> >> <alias name='pci.17'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x7'/> >> </controller> >> <controller type='pci' index='18' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='18' port='0x18'/> >> <alias name='pci.18'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x0' multifunction='on'/> >> </controller> >> <controller type='pci' index='19' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='19' port='0x19'/> >> <alias name='pci.19'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x1'/> >> </controller> >> <controller type='pci' index='20' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='20' port='0x1a'/> >> <alias name='pci.20'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x2'/> >> </controller> >> <controller type='pci' index='21' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='21' port='0x1b'/> >> <alias name='pci.21'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x3'/> >> </controller> >> <controller type='pci' index='22' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='22' port='0x1c'/> >> <alias name='pci.22'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x4'/> >> </controller> >> <controller type='pci' index='23' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='23' port='0x1d'/> >> <alias name='pci.23'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x5'/> >> </controller> >> <controller type='pci' index='24' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='24' port='0x1e'/> >> <alias name='pci.24'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x6'/> >> </controller> >> <controller type='pci' index='25' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='25' port='0x1f'/> >> <alias name='pci.25'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x7'/> >> </controller> >> <controller type='pci' index='26' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='26' port='0x20'/> >> <alias name='pci.26'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >> function='0x0' multifunction='on'/> >> </controller> >> <controller type='pci' index='27' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='27' port='0x21'/> >> <alias name='pci.27'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >> function='0x1'/> >> </controller> >> <controller type='pci' index='28' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='28' port='0x22'/> >> <alias name='pci.28'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >> function='0x2'/> >> </controller> >> <controller type='pci' index='29' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='29' port='0x23'/> >> <alias name='pci.29'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >> function='0x3'/> >> </controller> >> <controller type='pci' index='30' model='pcie-root-port'> >> <model name='pcie-root-port'/> >> <target chassis='30' port='0x24'/> >> <alias name='pci.30'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >> function='0x4'/> >> </controller> >> <controller type='virtio-serial' index='0'> >> <alias name='virtio-serial0'/> >> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' >> function='0x0'/> >> </controller> >> <serial type='pty'> >> <source path='/dev/pts/2'/> >> <target port='0'/> >> <alias name='serial0'/> >> </serial> >> <console type='pty' tty='/dev/pts/2'> >> <source path='/dev/pts/2'/> >> <target type='serial' port='0'/> >> <alias name='serial0'/> >> </console> >> <channel type='unix'> >> <source mode='bind' >> path='/var/run/libvirt/qemu/fangying-test.extend'/> >> <target type='virtio' name='org.qemu.guest_agent.1' >> state='connected'/> >> <alias name='channel0'/> >> <address type='virtio-serial' controller='0' bus='0' port='1'/> >> </channel> >> <channel type='unix'> >> <source mode='bind' >> path='/var/run/libvirt/qemu/fangying-test.agent'/> >> <target type='virtio' name='org.qemu.guest_agent.0' >> state='connected'/> >> <alias name='channel1'/> >> <address type='virtio-serial' controller='0' bus='0' port='2'/> >> </channel> >> <channel type='unix'> >> <source mode='bind' >> path='/var/run/libvirt/qemu/fangying-test.hostd'/> >> <target type='virtio' name='org.qemu.guest_agent.2' >> state='connected'/> >> <alias name='channel2'/> >> <address type='virtio-serial' controller='0' bus='0' port='3'/> >> </channel> >> <channel type='unix'> >> <source mode='bind' >> path='/var/run/libvirt/qemu/fangying-test.upgraded'/> >> <target type='virtio' name='org.qemu.guest_agent.3' >> state='connected'/> >> <alias name='channel3'/> >> <address type='virtio-serial' controller='0' bus='0' port='4'/> >> </channel> >> <input type='tablet' bus='usb'> >> <alias name='input0'/> >> <address type='usb' bus='0' port='1'/> >> </input> >> <input type='keyboard' bus='usb'> >> <alias name='input1'/> >> <address type='usb' bus='0' port='2'/> >> </input> >> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> >> <listen type='address' address='0.0.0.0'/> >> </graphics> >> <video> >> <model type='virtio' heads='1' primary='yes'/> >> <alias name='video0'/> >> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' >> function='0x0'/> >> </video> >> <memballoon model='virtio'> >> <alias name='balloon0'/> >> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' >> function='0x0'/> >> </memballoon> >> </devices> >> <seclabel type='dynamic' model='dac' relabel='yes'> >> <label>+0:+0</label> >> <imagelabel>+0:+0</imagelabel> >> </seclabel> >> </domain> >> >> I will test your cmdline way later to see if it can be reproduced. >> >> Thanks. >> Ying. >> > .
On Aug 21, 2020, at 12:39 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 12:05 PM, Haibin Zhang wrote: On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote: On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote: On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host. I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64. The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ? It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1... These patches have not been pushed to upstream? The above attachment is my dts dumped from qemu, no socket/core/thread My qemu version is 4.2.0 from upstream
No, it has not been accepted by upstream qemu, but CPU topology need it. And besides KVM must support MPIDR emulation which I am now working on.
MRS <Xt>, MPIDR_EL1
In guest, how can I let above instruction trap to kvm?
If arm archtecture support banked MPIDR_EL1 just like ICC and ICV, that would be best.
-smp 60,sockets=2,cores=30,threads=1 -bios A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
/usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM. No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On Aug 21, 2020, at 1:51 PM, Haibin Zhang <haibincheung@hotmail.commailto:haibincheung@hotmail.com> wrote:
On Aug 21, 2020, at 12:39 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 12:05 PM, Haibin Zhang wrote: On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote: On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote: On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote: On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.commailto:fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/20/2020 3:44 PM, Haibin Zhang wrote: Hi, Ying Fang, I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a openEuler guest, but it hung. Qcow2 is downloaded from https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... Qemu command: qemu-system-aarch64 -name guest=12345,debug-threads=on -machine virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ -cpu host -m 122880 -object iothread,id=iothread1 \ -object memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ -object memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host. I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... It says numa confguration is conflict with cpu affinity specified by MPIDR Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64. The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ? It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
I backport above patches to my qemu 4.2.0, but it doesn’t work.
Can you show me the patches on openEuler qemu?
These patches have not been pushed to upstream? The above attachment is my dts dumped from qemu, no socket/core/thread My qemu version is 4.2.0 from upstream
No, it has not been accepted by upstream qemu, but CPU topology need it. And besides KVM must support MPIDR emulation which I am now working on.
MRS <Xt>, MPIDR_EL1
In guest, how can I let above instruction trap to kvm?
If arm archtecture support banked MPIDR_EL1 just like ICC and ICV, that would be best.
-smp 60,sockets=2,cores=30,threads=1 -bios A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
/usr/share/AAVMF/AAVMF_CODE.fd \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -drive file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none \ -device virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -net none -serial telnet::9001,server,nowait -monitor stdio Guest startup logs: [ 3.769390] smp: Brought up 2 nodes, 60 CPUs [ 4.125100] SMP: Total of 60 processors activated. [ 4.126246] CPU features: detected: Privileged Access Never [ 4.127625] CPU features: detected: LSE atomic instructions [ 4.128969] CPU features: detected: User Access Override [ 4.130261] CPU features: detected: Data cache clean to Point of Persistence [ 4.131949] CPU features: detected: RAS Extension Support [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support [ 4.134704] CPU features: detected: CRC32 instructions [ 4.184286] CPU: All CPU(s) started at EL1 [ 4.185891] alternatives: patching kernel code [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) [ 64.312260] rcu: All QSes seen, last rcu_sched kthread activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, root ->qsmask 0x0 [ 64.315251] swapper/0 R running task 0 1 0 0x0000002a [ 64.316930] Call trace: [ 64.317533] dump_backtrace+0x0/0x198 [ 64.318432] show_stack+0x24/0x30 [ 64.319248] sched_show_task+0x12c/0x160 [ 64.320206] rcu_check_callbacks+0x780/0x7a8 [ 64.321253] update_process_times+0x34/0x60 [ 64.322296] tick_periodic+0x48/0x100 [ 64.323193] tick_handle_periodic+0x38/0xd8 [ 64.324224] arch_timer_handler_virt+0x38/0x58 [ 64.325320] handle_percpu_devid_irq+0x90/0x248 [ 64.326419] generic_handle_irq+0x34/0x50 [ 64.327383] __handle_domain_irq+0x68/0xc0 [ 64.328367] gic_handle_irq+0x6c/0x170 [ 64.329275] el1_irq+0xb8/0x140 [ 64.330040] __bitmap_weight+0x48/0xa0 [ 64.330942] build_sched_domains+0xd58/0xf50 [ 64.331962] sched_init_domains+0xd4/0x108 [ 64.332950] sched_init_smp+0x3c/0x8c [ 64.333842] kernel_init_freeable+0x17c/0x34c [ 64.334891] kernel_init+0x18/0x118 [ 64.335735] ret_from_fork+0x10/0x18
You mean there is something wrong with the NUMA feature, right ?
please try using the xml below and see if you can luanch your VM. No numa specified in below xml file.
<domain type='kvm' id='17'> <name>openeuler-test</name> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <memtune> <hard_limit unit='GiB'>32</hard_limit> </memtune> <vcpu placement='static' cpuset='7-10'>60</vcpu> <iothreads>5</iothreads> <cputune> <emulatorpin cpuset='7-10'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='aarch64' machine='virt-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <gic version='3'/> <acpi/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='30' threads='1'/> </cpu> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ehci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x11'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='12' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='12' port='0x12'/> <alias name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='13' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='13' port='0x13'/> <alias name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='14' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='14' port='0x14'/> <alias name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='15' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='15' port='0x15'/> <alias name='pci.15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='16' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='16' port='0x16'/> <alias name='pci.16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='17' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='17' port='0x17'/> <alias name='pci.17'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='pci' index='18' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='18' port='0x18'/> <alias name='pci.18'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='19' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='19' port='0x19'/> <alias name='pci.19'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller> <controller type='pci' index='20' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='20' port='0x1a'/> <alias name='pci.20'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x1b'/> <alias name='pci.21'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0x1c'/> <alias name='pci.22'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/> </controller> <controller type='pci' index='23' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='23' port='0x1d'/> <alias name='pci.23'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/> </controller> <controller type='pci' index='24' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='24' port='0x1e'/> <alias name='pci.24'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/> </controller> <controller type='pci' index='25' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='25' port='0x1f'/> <alias name='pci.25'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/> </controller> <controller type='pci' index='26' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='26' port='0x20'/> <alias name='pci.26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='27' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='27' port='0x21'/> <alias name='pci.27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='pci' index='28' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='28' port='0x22'/> <alias name='pci.28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='29' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='29' port='0x23'/> <alias name='pci.29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/> </controller> <controller type='pci' index='30' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='30' port='0x24'/> <alias name='pci.30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x02' function='0x0'/> </controller> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.extend'/> <target type='virtio' name='org.qemu.guest_agent.1' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.hostd'/> <target type='virtio' name='org.qemu.guest_agent.2' state='connected'/> <alias name='channel2'/> <address type='virtio-serial' controller='0' bus='0' port='3'/> </channel> <channel type='unix'> <source mode='bind' path='/var/run/libvirt/qemu/fangying-test.upgraded'/> <target type='virtio' name='org.qemu.guest_agent.3' state='connected'/> <alias name='channel3'/> <address type='virtio-serial' controller='0' bus='0' port='4'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='keyboard' bus='usb'> <alias name='input1'/> <address type='usb' bus='0' port='2'/> </input> <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='virtio' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x04' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain>
I will test your cmdline way later to see if it can be reproduced.
Thanks. Ying.
.
On 8/21/2020 5:25 PM, Haibin Zhang wrote:
On Aug 21, 2020, at 1:51 PM, Haibin Zhang <haibincheung@hotmail.com mailto:haibincheung@hotmail.com> wrote:
On Aug 21, 2020, at 12:39 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 12:05 PM, Haibin Zhang wrote:
On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote:
> On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.com > mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> > wrote: > > > > On 8/21/2020 10:21 AM, Haibin Zhang wrote: >>> On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com >>> mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> >>> wrote: >>> >>> >>> >>> On 8/20/2020 6:23 PM, Haibin Zhang wrote: >>>>> On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com >>>>> mailto:fangying1@huawei.commailto:fangying1@huawei.com >>>>> mailto:fangying1@huawei.com mailto:fangying1@huawei.com> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On 8/20/2020 3:44 PM, Haibin Zhang wrote: >>>>>> Hi, Ying Fang, >>>>>> I use qemu(https://gitee.com/src-openeuler/qemu.git) launch >>>>>> a openEuler guest, but it hung. >>>>>> Qcow2 is downloaded >>>>>> fromhttps://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... >>>>>> Qemu command: >>>>>> qemu-system-aarch64 -name guest=12345,debug-threads=on >>>>>> -machine >>>>>> virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ >>>>>> -cpu host -m 122880 -object iothread,id=iothread1 \ >>>>>> -object >>>>>> memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred >>>>>> -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ >>>>>> -object >>>>>> memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred >>>>>> -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ >>>> It works after getting rid of these two memory-backend-ram objects >>> I am wondering whether there is enough memory on each NUMA node >>> you specified in the command line on your host. You can use >>> numactl -H to check numa info of your host. >> I have specified policy=preferred not strict. >> I get some clues below: >> https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... >> It says numa confguration is conflict with cpu affinity >> specified by MPIDR > Yes, this may be the clue and I'm quite intrested in this issue. > > Since currently KVM does not support userspace mpdir configuration. > I am now doing some work to enable MPIDR emulation in KVM on aarch64. The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
I backport above patches to my qemu 4.2.0, but it doesn’t work.
Can you show me the patches on openEuler qemu?
It has been introduced in patch [1] in openEuler qemu.
[1] https://gitee.com/src-openeuler/qemu/blob/master/hw-arm-virt-Introduce-cpu-t...
These patches have not been pushed to upstream? The above attachment is my dts dumped from qemu, no socket/core/thread My qemu version is 4.2.0 from upstream
No, it has not been accepted by upstream qemu, but CPU topology need it. And besides KVM must support MPIDR emulation which I am now working on.
MRS <Xt>, MPIDR_EL1
In guest, how can I let above instruction trap to kvm?
If arm archtecture support banked MPIDR_EL1 just like ICC and ICV, that would be best.
> >>>>>> -smp 60,sockets=2,cores=30,threads=1 -bios > A 2(scokets) * 30(cores) * 1(threads) topology may confusing the > guest kernel. I'll test this situation later this week. > Please keep in touch with me. > >>>>>> /usr/share/AAVMF/AAVMF_CODE.fd \ >>>>>> -device >>>>>> pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 >>>>>> \ >>>>>> -device >>>>>> pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 >>>>>> \ >>>>>> -drive >>>>>> file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none >>>>>> \ >>>>>> -device >>>>>> virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>> \ >>>>>> -net none -serial telnet::9001,server,nowait -monitor stdio >>>>>> Guest startup logs: >>>>>> [ 3.769390] smp: Brought up 2 nodes, 60 CPUs >>>>>> [ 4.125100] SMP: Total of 60 processors activated. >>>>>> [ 4.126246] CPU features: detected: Privileged Access Never >>>>>> [ 4.127625] CPU features: detected: LSE atomic instructions >>>>>> [ 4.128969] CPU features: detected: User Access Override >>>>>> [ 4.130261] CPU features: detected: Data cache clean to >>>>>> Point of Persistence >>>>>> [ 4.131949] CPU features: detected: RAS Extension Support >>>>>> [ 4.133253] CPU features: detected: ARM64 MPAM Extension >>>>>> Support >>>>>> [ 4.134704] CPU features: detected: CRC32 instructions >>>>>> [ 4.184286] CPU: All CPU(s) started at EL1 >>>>>> [ 4.185891] alternatives: patching kernel code >>>>>> [ 64.308834] rcu: INFO: rcu_sched detected stalls on >>>>>> CPUs/tasks: >>>>>> [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, >>>>>> q=0) >>>>>> [ 64.312260] rcu: All QSes seen, last rcu_sched kthread >>>>>> activity 15002 (4294908290-4294893288), >>>>>> jiffies_till_next_fqs=1, root ->qsmask 0x0 >>>>>> [ 64.315251] swapper/0 R running task 0 >>>>>> 1 0 0x0000002a >>>>>> [ 64.316930] Call trace: >>>>>> [ 64.317533] dump_backtrace+0x0/0x198 >>>>>> [ 64.318432] show_stack+0x24/0x30 >>>>>> [ 64.319248] sched_show_task+0x12c/0x160 >>>>>> [ 64.320206] rcu_check_callbacks+0x780/0x7a8 >>>>>> [ 64.321253] update_process_times+0x34/0x60 >>>>>> [ 64.322296] tick_periodic+0x48/0x100 >>>>>> [ 64.323193] tick_handle_periodic+0x38/0xd8 >>>>>> [ 64.324224] arch_timer_handler_virt+0x38/0x58 >>>>>> [ 64.325320] handle_percpu_devid_irq+0x90/0x248 >>>>>> [ 64.326419] generic_handle_irq+0x34/0x50 >>>>>> [ 64.327383] __handle_domain_irq+0x68/0xc0 >>>>>> [ 64.328367] gic_handle_irq+0x6c/0x170 >>>>>> [ 64.329275] el1_irq+0xb8/0x140 >>>>>> [ 64.330040] __bitmap_weight+0x48/0xa0 >>>>>> [ 64.330942] build_sched_domains+0xd58/0xf50 >>>>>> [ 64.331962] sched_init_domains+0xd4/0x108 >>>>>> [ 64.332950] sched_init_smp+0x3c/0x8c >>>>>> [ 64.333842] kernel_init_freeable+0x17c/0x34c >>>>>> [ 64.334891] kernel_init+0x18/0x118 >>>>>> [ 64.335735] ret_from_fork+0x10/0x18 >>>>> >>>>> You mean there is something wrong with the NUMA feature, right ? >>>>> >>>>> please try using the xml below and see if you can luanch your VM. >>>> No numa specified in below xml file. >>>>> >>>>> <domain type='kvm' id='17'> >>>>> <name>openeuler-test</name> >>>>> <memory unit='GiB'>32</memory> >>>>> <currentMemory unit='GiB'>32</currentMemory> >>>>> <memtune> >>>>> <hard_limit unit='GiB'>32</hard_limit> >>>>> </memtune> >>>>> <vcpu placement='static' cpuset='7-10'>60</vcpu> >>>>> <iothreads>5</iothreads> >>>>> <cputune> >>>>> <emulatorpin cpuset='7-10'/> >>>>> </cputune> >>>>> <resource> >>>>> <partition>/machine</partition> >>>>> </resource> >>>>> <os> >>>>> <type arch='aarch64' machine='virt-4.1'>hvm</type> >>>>> <loader readonly='yes' >>>>> type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> >>>>> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> >>>>> <boot dev='hd'/> >>>>> <boot dev='cdrom'/> >>>>> </os> >>>>> <features> >>>>> <gic version='3'/> >>>>> <acpi/> >>>>> </features> >>>>> <cpu mode='host-passthrough' check='none'> >>>>> <topology sockets='2' cores='30' threads='1'/> >>>>> </cpu> >>>>> <clock offset='utc'> >>>>> </clock> >>>>> <on_poweroff>destroy</on_poweroff> >>>>> <on_reboot>restart</on_reboot> >>>>> <on_crash>restart</on_crash> >>>>> <devices> >>>>> <emulator>/usr/bin/qemu-kvm</emulator> >>>>> <disk type='file' device='disk'> >>>>> <driver name='qemu' type='qcow2' cache='none' io='native'/> >>>>> <source >>>>> file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> >>>>> <backingStore/> >>>>> <target dev='sda' bus='scsi'/> >>>>> <alias name='scsi0-0-0-0'/> >>>>> <address type='drive' controller='0' bus='0' target='0' >>>>> unit='0'/> >>>>> </disk> >>>>> <controller type='usb' index='0' model='ehci'> >>>>> <alias name='usb'/> >>>>> <address type='pci' domain='0x0000' bus='0x03' >>>>> slot='0x01' function='0x0'/> >>>>> </controller> >>>>> <controller type='scsi' index='0' model='virtio-scsi'> >>>>> <alias name='scsi0'/> >>>>> <address type='pci' domain='0x0000' bus='0x04' >>>>> slot='0x00' function='0x0'/> >>>>> </controller> >>>>> <controller type='pci' index='0' model='pcie-root'> >>>>> <alias name='pcie.0'/> >>>>> </controller> >>>>> <controller type='pci' index='1' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='1' port='0x8'/> >>>>> <alias name='pci.1'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x0' multifunction='on'/> >>>>> </controller> >>>>> <controller type='pci' index='2' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='2' port='0x9'/> >>>>> <alias name='pci.2'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x1'/> >>>>> </controller> >>>>> <controller type='pci' index='3' model='pcie-to-pci-bridge'> >>>>> <model name='pcie-pci-bridge'/> >>>>> <alias name='pci.3'/> >>>>> <address type='pci' domain='0x0000' bus='0x01' >>>>> slot='0x00' function='0x0'/> >>>>> </controller> >>>>> <controller type='pci' index='4' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='4' port='0xa'/> >>>>> <alias name='pci.4'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x2'/> >>>>> </controller> >>>>> <controller type='pci' index='5' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='5' port='0xb'/> >>>>> <alias name='pci.5'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x3'/> >>>>> </controller> >>>>> <controller type='pci' index='6' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='6' port='0xc'/> >>>>> <alias name='pci.6'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x4'/> >>>>> </controller> >>>>> <controller type='pci' index='7' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='7' port='0xd'/> >>>>> <alias name='pci.7'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x5'/> >>>>> </controller> >>>>> <controller type='pci' index='8' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='8' port='0xe'/> >>>>> <alias name='pci.8'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x6'/> >>>>> </controller> >>>>> <controller type='pci' index='9' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='9' port='0xf'/> >>>>> <alias name='pci.9'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x01' function='0x7'/> >>>>> </controller> >>>>> <controller type='pci' index='10' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='10' port='0x10'/> >>>>> <alias name='pci.10'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x0' multifunction='on'/> >>>>> </controller> >>>>> <controller type='pci' index='11' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='11' port='0x11'/> >>>>> <alias name='pci.11'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x1'/> >>>>> </controller> >>>>> <controller type='pci' index='12' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='12' port='0x12'/> >>>>> <alias name='pci.12'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x2'/> >>>>> </controller> >>>>> <controller type='pci' index='13' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='13' port='0x13'/> >>>>> <alias name='pci.13'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x3'/> >>>>> </controller> >>>>> <controller type='pci' index='14' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='14' port='0x14'/> >>>>> <alias name='pci.14'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x4'/> >>>>> </controller> >>>>> <controller type='pci' index='15' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='15' port='0x15'/> >>>>> <alias name='pci.15'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x5'/> >>>>> </controller> >>>>> <controller type='pci' index='16' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='16' port='0x16'/> >>>>> <alias name='pci.16'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x6'/> >>>>> </controller> >>>>> <controller type='pci' index='17' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='17' port='0x17'/> >>>>> <alias name='pci.17'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x02' function='0x7'/> >>>>> </controller> >>>>> <controller type='pci' index='18' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='18' port='0x18'/> >>>>> <alias name='pci.18'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x0' multifunction='on'/> >>>>> </controller> >>>>> <controller type='pci' index='19' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='19' port='0x19'/> >>>>> <alias name='pci.19'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x1'/> >>>>> </controller> >>>>> <controller type='pci' index='20' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='20' port='0x1a'/> >>>>> <alias name='pci.20'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x2'/> >>>>> </controller> >>>>> <controller type='pci' index='21' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='21' port='0x1b'/> >>>>> <alias name='pci.21'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x3'/> >>>>> </controller> >>>>> <controller type='pci' index='22' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='22' port='0x1c'/> >>>>> <alias name='pci.22'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x4'/> >>>>> </controller> >>>>> <controller type='pci' index='23' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='23' port='0x1d'/> >>>>> <alias name='pci.23'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x5'/> >>>>> </controller> >>>>> <controller type='pci' index='24' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='24' port='0x1e'/> >>>>> <alias name='pci.24'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x6'/> >>>>> </controller> >>>>> <controller type='pci' index='25' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='25' port='0x1f'/> >>>>> <alias name='pci.25'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x03' function='0x7'/> >>>>> </controller> >>>>> <controller type='pci' index='26' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='26' port='0x20'/> >>>>> <alias name='pci.26'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x04' function='0x0' multifunction='on'/> >>>>> </controller> >>>>> <controller type='pci' index='27' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='27' port='0x21'/> >>>>> <alias name='pci.27'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x04' function='0x1'/> >>>>> </controller> >>>>> <controller type='pci' index='28' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='28' port='0x22'/> >>>>> <alias name='pci.28'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x04' function='0x2'/> >>>>> </controller> >>>>> <controller type='pci' index='29' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='29' port='0x23'/> >>>>> <alias name='pci.29'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x04' function='0x3'/> >>>>> </controller> >>>>> <controller type='pci' index='30' model='pcie-root-port'> >>>>> <model name='pcie-root-port'/> >>>>> <target chassis='30' port='0x24'/> >>>>> <alias name='pci.30'/> >>>>> <address type='pci' domain='0x0000' bus='0x00' >>>>> slot='0x04' function='0x4'/> >>>>> </controller> >>>>> <controller type='virtio-serial' index='0'> >>>>> <alias name='virtio-serial0'/> >>>>> <address type='pci' domain='0x0000' bus='0x03' >>>>> slot='0x02' function='0x0'/> >>>>> </controller> >>>>> <serial type='pty'> >>>>> <source path='/dev/pts/2'/> >>>>> <target port='0'/> >>>>> <alias name='serial0'/> >>>>> </serial> >>>>> <console type='pty' tty='/dev/pts/2'> >>>>> <source path='/dev/pts/2'/> >>>>> <target type='serial' port='0'/> >>>>> <alias name='serial0'/> >>>>> </console> >>>>> <channel type='unix'> >>>>> <source mode='bind' >>>>> path='/var/run/libvirt/qemu/fangying-test.extend'/> >>>>> <target type='virtio' name='org.qemu.guest_agent.1' >>>>> state='connected'/> >>>>> <alias name='channel0'/> >>>>> <address type='virtio-serial' controller='0' bus='0' >>>>> port='1'/> >>>>> </channel> >>>>> <channel type='unix'> >>>>> <source mode='bind' >>>>> path='/var/run/libvirt/qemu/fangying-test.agent'/> >>>>> <target type='virtio' name='org.qemu.guest_agent.0' >>>>> state='connected'/> >>>>> <alias name='channel1'/> >>>>> <address type='virtio-serial' controller='0' bus='0' >>>>> port='2'/> >>>>> </channel> >>>>> <channel type='unix'> >>>>> <source mode='bind' >>>>> path='/var/run/libvirt/qemu/fangying-test.hostd'/> >>>>> <target type='virtio' name='org.qemu.guest_agent.2' >>>>> state='connected'/> >>>>> <alias name='channel2'/> >>>>> <address type='virtio-serial' controller='0' bus='0' >>>>> port='3'/> >>>>> </channel> >>>>> <channel type='unix'> >>>>> <source mode='bind' >>>>> path='/var/run/libvirt/qemu/fangying-test.upgraded'/> >>>>> <target type='virtio' name='org.qemu.guest_agent.3' >>>>> state='connected'/> >>>>> <alias name='channel3'/> >>>>> <address type='virtio-serial' controller='0' bus='0' >>>>> port='4'/> >>>>> </channel> >>>>> <input type='tablet' bus='usb'> >>>>> <alias name='input0'/> >>>>> <address type='usb' bus='0' port='1'/> >>>>> </input> >>>>> <input type='keyboard' bus='usb'> >>>>> <alias name='input1'/> >>>>> <address type='usb' bus='0' port='2'/> >>>>> </input> >>>>> <graphics type='vnc' port='5903' autoport='yes' >>>>> listen='0.0.0.0'> >>>>> <listen type='address' address='0.0.0.0'/> >>>>> </graphics> >>>>> <video> >>>>> <model type='virtio' heads='1' primary='yes'/> >>>>> <alias name='video0'/> >>>>> <address type='pci' domain='0x0000' bus='0x03' >>>>> slot='0x04' function='0x0'/> >>>>> </video> >>>>> <memballoon model='virtio'> >>>>> <alias name='balloon0'/> >>>>> <address type='pci' domain='0x0000' bus='0x03' >>>>> slot='0x03' function='0x0'/> >>>>> </memballoon> >>>>> </devices> >>>>> <seclabel type='dynamic' model='dac' relabel='yes'> >>>>> <label>+0:+0</label> >>>>> <imagelabel>+0:+0</imagelabel> >>>>> </seclabel> >>>>> </domain> >>>>> >>>>> I will test your cmdline way later to see if it can be >>>>> reproduced. >>>>> >>>>> Thanks. >>>>> Ying. >>>>> >>>> .
On 8/21/2020 1:51 PM, Haibin Zhang wrote:
On Aug 21, 2020, at 12:39 PM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 12:05 PM, Haibin Zhang wrote:
On Aug 21, 2020, at 11:31 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote: >> On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com >> mailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.commailto:fangying1@huawei.com> >> wrote: >> >> >> >> On 8/20/2020 6:23 PM, Haibin Zhang wrote: >>>> On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com >>>> mailto:fangying1@huawei.commailto:fangying1@huawei.com >>>> mailto:fangying1@huawei.com mailto:fangying1@huawei.com> >>>> wrote: >>>> >>>> >>>> >>>> On 8/20/2020 3:44 PM, Haibin Zhang wrote: >>>>> Hi, Ying Fang, >>>>> I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a >>>>> openEuler guest, but it hung. >>>>> Qcow2 is downloaded >>>>> fromhttps://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... >>>>> Qemu command: >>>>> qemu-system-aarch64 -name guest=12345,debug-threads=on >>>>> -machine >>>>> virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ >>>>> -cpu host -m 122880 -object iothread,id=iothread1 \ >>>>> -object >>>>> memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred >>>>> -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ >>>>> -object >>>>> memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred >>>>> -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ >>> It works after getting rid of these two memory-backend-ram objects >> I am wondering whether there is enough memory on each NUMA node >> you specified in the command line on your host. You can use >> numactl -H to check numa info of your host. > I have specified policy=preferred not strict. > I get some clues below: > https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his... > It says numa confguration is conflict with cpu affinity specified > by MPIDR Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64.
The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
These patches have not been pushed to upstream? The above attachment is my dts dumped from qemu, no socket/core/thread My qemu version is 4.2.0 from upstream
No, it has not been accepted by upstream qemu, but CPU topology need it. And besides KVM must support MPIDR emulation which I am now working on.
MRS <Xt>, MPIDR_EL1
In guest, how can I let above instruction trap to kvm?
If arm archtecture support banked MPIDR_EL1 just like ICC and ICV, that would be best.
On aarch64 platform, we have VPIDR_EL2 and VMPIDR_EL2 to help emulate MPIDR in guest. Please have a look at [1].
[1] https://developer.arm.com/architectures/learn-the-architecture/armv8-a-virtu...
>>>>> -smp 60,sockets=2,cores=30,threads=1 -bios A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
>>>>> /usr/share/AAVMF/AAVMF_CODE.fd \ >>>>> -device >>>>> pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 >>>>> \ >>>>> -device >>>>> pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 >>>>> \ >>>>> -drive >>>>> file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none >>>>> \ >>>>> -device >>>>> virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>> \ >>>>> -net none -serial telnet::9001,server,nowait -monitor stdio >>>>> Guest startup logs: >>>>> [ 3.769390] smp: Brought up 2 nodes, 60 CPUs >>>>> [ 4.125100] SMP: Total of 60 processors activated. >>>>> [ 4.126246] CPU features: detected: Privileged Access Never >>>>> [ 4.127625] CPU features: detected: LSE atomic instructions >>>>> [ 4.128969] CPU features: detected: User Access Override >>>>> [ 4.130261] CPU features: detected: Data cache clean to >>>>> Point of Persistence >>>>> [ 4.131949] CPU features: detected: RAS Extension Support >>>>> [ 4.133253] CPU features: detected: ARM64 MPAM Extension >>>>> Support >>>>> [ 4.134704] CPU features: detected: CRC32 instructions >>>>> [ 4.184286] CPU: All CPU(s) started at EL1 >>>>> [ 4.185891] alternatives: patching kernel code >>>>> [ 64.308834] rcu: INFO: rcu_sched detected stalls on >>>>> CPUs/tasks: >>>>> [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) >>>>> [ 64.312260] rcu: All QSes seen, last rcu_sched kthread >>>>> activity 15002 (4294908290-4294893288), >>>>> jiffies_till_next_fqs=1, root ->qsmask 0x0 >>>>> [ 64.315251] swapper/0 R running task 0 1 >>>>> 0 0x0000002a >>>>> [ 64.316930] Call trace: >>>>> [ 64.317533] dump_backtrace+0x0/0x198 >>>>> [ 64.318432] show_stack+0x24/0x30 >>>>> [ 64.319248] sched_show_task+0x12c/0x160 >>>>> [ 64.320206] rcu_check_callbacks+0x780/0x7a8 >>>>> [ 64.321253] update_process_times+0x34/0x60 >>>>> [ 64.322296] tick_periodic+0x48/0x100 >>>>> [ 64.323193] tick_handle_periodic+0x38/0xd8 >>>>> [ 64.324224] arch_timer_handler_virt+0x38/0x58 >>>>> [ 64.325320] handle_percpu_devid_irq+0x90/0x248 >>>>> [ 64.326419] generic_handle_irq+0x34/0x50 >>>>> [ 64.327383] __handle_domain_irq+0x68/0xc0 >>>>> [ 64.328367] gic_handle_irq+0x6c/0x170 >>>>> [ 64.329275] el1_irq+0xb8/0x140 >>>>> [ 64.330040] __bitmap_weight+0x48/0xa0 >>>>> [ 64.330942] build_sched_domains+0xd58/0xf50 >>>>> [ 64.331962] sched_init_domains+0xd4/0x108 >>>>> [ 64.332950] sched_init_smp+0x3c/0x8c >>>>> [ 64.333842] kernel_init_freeable+0x17c/0x34c >>>>> [ 64.334891] kernel_init+0x18/0x118 >>>>> [ 64.335735] ret_from_fork+0x10/0x18 >>>> >>>> You mean there is something wrong with the NUMA feature, right ? >>>> >>>> please try using the xml below and see if you can luanch your VM. >>> No numa specified in below xml file. >>>> >>>> <domain type='kvm' id='17'> >>>> <name>openeuler-test</name> >>>> <memory unit='GiB'>32</memory> >>>> <currentMemory unit='GiB'>32</currentMemory> >>>> <memtune> >>>> <hard_limit unit='GiB'>32</hard_limit> >>>> </memtune> >>>> <vcpu placement='static' cpuset='7-10'>60</vcpu> >>>> <iothreads>5</iothreads> >>>> <cputune> >>>> <emulatorpin cpuset='7-10'/> >>>> </cputune> >>>> <resource> >>>> <partition>/machine</partition> >>>> </resource> >>>> <os> >>>> <type arch='aarch64' machine='virt-4.1'>hvm</type> >>>> <loader readonly='yes' >>>> type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> >>>> <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> >>>> <boot dev='hd'/> >>>> <boot dev='cdrom'/> >>>> </os> >>>> <features> >>>> <gic version='3'/> >>>> <acpi/> >>>> </features> >>>> <cpu mode='host-passthrough' check='none'> >>>> <topology sockets='2' cores='30' threads='1'/> >>>> </cpu> >>>> <clock offset='utc'> >>>> </clock> >>>> <on_poweroff>destroy</on_poweroff> >>>> <on_reboot>restart</on_reboot> >>>> <on_crash>restart</on_crash> >>>> <devices> >>>> <emulator>/usr/bin/qemu-kvm</emulator> >>>> <disk type='file' device='disk'> >>>> <driver name='qemu' type='qcow2' cache='none' io='native'/> >>>> <source >>>> file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> >>>> <backingStore/> >>>> <target dev='sda' bus='scsi'/> >>>> <alias name='scsi0-0-0-0'/> >>>> <address type='drive' controller='0' bus='0' target='0' >>>> unit='0'/> >>>> </disk> >>>> <controller type='usb' index='0' model='ehci'> >>>> <alias name='usb'/> >>>> <address type='pci' domain='0x0000' bus='0x03' >>>> slot='0x01' function='0x0'/> >>>> </controller> >>>> <controller type='scsi' index='0' model='virtio-scsi'> >>>> <alias name='scsi0'/> >>>> <address type='pci' domain='0x0000' bus='0x04' >>>> slot='0x00' function='0x0'/> >>>> </controller> >>>> <controller type='pci' index='0' model='pcie-root'> >>>> <alias name='pcie.0'/> >>>> </controller> >>>> <controller type='pci' index='1' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='1' port='0x8'/> >>>> <alias name='pci.1'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x0' multifunction='on'/> >>>> </controller> >>>> <controller type='pci' index='2' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='2' port='0x9'/> >>>> <alias name='pci.2'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x1'/> >>>> </controller> >>>> <controller type='pci' index='3' model='pcie-to-pci-bridge'> >>>> <model name='pcie-pci-bridge'/> >>>> <alias name='pci.3'/> >>>> <address type='pci' domain='0x0000' bus='0x01' >>>> slot='0x00' function='0x0'/> >>>> </controller> >>>> <controller type='pci' index='4' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='4' port='0xa'/> >>>> <alias name='pci.4'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x2'/> >>>> </controller> >>>> <controller type='pci' index='5' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='5' port='0xb'/> >>>> <alias name='pci.5'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x3'/> >>>> </controller> >>>> <controller type='pci' index='6' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='6' port='0xc'/> >>>> <alias name='pci.6'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x4'/> >>>> </controller> >>>> <controller type='pci' index='7' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='7' port='0xd'/> >>>> <alias name='pci.7'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x5'/> >>>> </controller> >>>> <controller type='pci' index='8' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='8' port='0xe'/> >>>> <alias name='pci.8'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x6'/> >>>> </controller> >>>> <controller type='pci' index='9' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='9' port='0xf'/> >>>> <alias name='pci.9'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x01' function='0x7'/> >>>> </controller> >>>> <controller type='pci' index='10' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='10' port='0x10'/> >>>> <alias name='pci.10'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x0' multifunction='on'/> >>>> </controller> >>>> <controller type='pci' index='11' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='11' port='0x11'/> >>>> <alias name='pci.11'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x1'/> >>>> </controller> >>>> <controller type='pci' index='12' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='12' port='0x12'/> >>>> <alias name='pci.12'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x2'/> >>>> </controller> >>>> <controller type='pci' index='13' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='13' port='0x13'/> >>>> <alias name='pci.13'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x3'/> >>>> </controller> >>>> <controller type='pci' index='14' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='14' port='0x14'/> >>>> <alias name='pci.14'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x4'/> >>>> </controller> >>>> <controller type='pci' index='15' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='15' port='0x15'/> >>>> <alias name='pci.15'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x5'/> >>>> </controller> >>>> <controller type='pci' index='16' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='16' port='0x16'/> >>>> <alias name='pci.16'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x6'/> >>>> </controller> >>>> <controller type='pci' index='17' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='17' port='0x17'/> >>>> <alias name='pci.17'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x02' function='0x7'/> >>>> </controller> >>>> <controller type='pci' index='18' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='18' port='0x18'/> >>>> <alias name='pci.18'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x0' multifunction='on'/> >>>> </controller> >>>> <controller type='pci' index='19' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='19' port='0x19'/> >>>> <alias name='pci.19'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x1'/> >>>> </controller> >>>> <controller type='pci' index='20' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='20' port='0x1a'/> >>>> <alias name='pci.20'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x2'/> >>>> </controller> >>>> <controller type='pci' index='21' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='21' port='0x1b'/> >>>> <alias name='pci.21'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x3'/> >>>> </controller> >>>> <controller type='pci' index='22' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='22' port='0x1c'/> >>>> <alias name='pci.22'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x4'/> >>>> </controller> >>>> <controller type='pci' index='23' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='23' port='0x1d'/> >>>> <alias name='pci.23'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x5'/> >>>> </controller> >>>> <controller type='pci' index='24' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='24' port='0x1e'/> >>>> <alias name='pci.24'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x6'/> >>>> </controller> >>>> <controller type='pci' index='25' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='25' port='0x1f'/> >>>> <alias name='pci.25'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x03' function='0x7'/> >>>> </controller> >>>> <controller type='pci' index='26' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='26' port='0x20'/> >>>> <alias name='pci.26'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x04' function='0x0' multifunction='on'/> >>>> </controller> >>>> <controller type='pci' index='27' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='27' port='0x21'/> >>>> <alias name='pci.27'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x04' function='0x1'/> >>>> </controller> >>>> <controller type='pci' index='28' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='28' port='0x22'/> >>>> <alias name='pci.28'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x04' function='0x2'/> >>>> </controller> >>>> <controller type='pci' index='29' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='29' port='0x23'/> >>>> <alias name='pci.29'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x04' function='0x3'/> >>>> </controller> >>>> <controller type='pci' index='30' model='pcie-root-port'> >>>> <model name='pcie-root-port'/> >>>> <target chassis='30' port='0x24'/> >>>> <alias name='pci.30'/> >>>> <address type='pci' domain='0x0000' bus='0x00' >>>> slot='0x04' function='0x4'/> >>>> </controller> >>>> <controller type='virtio-serial' index='0'> >>>> <alias name='virtio-serial0'/> >>>> <address type='pci' domain='0x0000' bus='0x03' >>>> slot='0x02' function='0x0'/> >>>> </controller> >>>> <serial type='pty'> >>>> <source path='/dev/pts/2'/> >>>> <target port='0'/> >>>> <alias name='serial0'/> >>>> </serial> >>>> <console type='pty' tty='/dev/pts/2'> >>>> <source path='/dev/pts/2'/> >>>> <target type='serial' port='0'/> >>>> <alias name='serial0'/> >>>> </console> >>>> <channel type='unix'> >>>> <source mode='bind' >>>> path='/var/run/libvirt/qemu/fangying-test.extend'/> >>>> <target type='virtio' name='org.qemu.guest_agent.1' >>>> state='connected'/> >>>> <alias name='channel0'/> >>>> <address type='virtio-serial' controller='0' bus='0' >>>> port='1'/> >>>> </channel> >>>> <channel type='unix'> >>>> <source mode='bind' >>>> path='/var/run/libvirt/qemu/fangying-test.agent'/> >>>> <target type='virtio' name='org.qemu.guest_agent.0' >>>> state='connected'/> >>>> <alias name='channel1'/> >>>> <address type='virtio-serial' controller='0' bus='0' >>>> port='2'/> >>>> </channel> >>>> <channel type='unix'> >>>> <source mode='bind' >>>> path='/var/run/libvirt/qemu/fangying-test.hostd'/> >>>> <target type='virtio' name='org.qemu.guest_agent.2' >>>> state='connected'/> >>>> <alias name='channel2'/> >>>> <address type='virtio-serial' controller='0' bus='0' >>>> port='3'/> >>>> </channel> >>>> <channel type='unix'> >>>> <source mode='bind' >>>> path='/var/run/libvirt/qemu/fangying-test.upgraded'/> >>>> <target type='virtio' name='org.qemu.guest_agent.3' >>>> state='connected'/> >>>> <alias name='channel3'/> >>>> <address type='virtio-serial' controller='0' bus='0' >>>> port='4'/> >>>> </channel> >>>> <input type='tablet' bus='usb'> >>>> <alias name='input0'/> >>>> <address type='usb' bus='0' port='1'/> >>>> </input> >>>> <input type='keyboard' bus='usb'> >>>> <alias name='input1'/> >>>> <address type='usb' bus='0' port='2'/> >>>> </input> >>>> <graphics type='vnc' port='5903' autoport='yes' >>>> listen='0.0.0.0'> >>>> <listen type='address' address='0.0.0.0'/> >>>> </graphics> >>>> <video> >>>> <model type='virtio' heads='1' primary='yes'/> >>>> <alias name='video0'/> >>>> <address type='pci' domain='0x0000' bus='0x03' >>>> slot='0x04' function='0x0'/> >>>> </video> >>>> <memballoon model='virtio'> >>>> <alias name='balloon0'/> >>>> <address type='pci' domain='0x0000' bus='0x03' >>>> slot='0x03' function='0x0'/> >>>> </memballoon> >>>> </devices> >>>> <seclabel type='dynamic' model='dac' relabel='yes'> >>>> <label>+0:+0</label> >>>> <imagelabel>+0:+0</imagelabel> >>>> </seclabel> >>>> </domain> >>>> >>>> I will test your cmdline way later to see if it can be reproduced. >>>> >>>> Thanks. >>>> Ying. >>>> >>> .
On 8/21/2020 11:31 AM, Ying Fang via Virt wrote:
On 8/21/2020 11:23 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 10:49 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.com> wrote:
On 8/21/2020 10:21 AM, Haibin Zhang wrote:
On Aug 21, 2020, at 9:09 AM, Ying Fang <fangying1@huawei.com mailto:fangying1@huawei.commailto:fangying1@huawei.com> wrote:
On 8/20/2020 6:23 PM, Haibin Zhang wrote:
> On Aug 20, 2020, at 5:08 PM, Ying Fang <fangying1@huawei.com > mailto:fangying1@huawei.com mailto:fangying1@huawei.com> wrote: > > > > On 8/20/2020 3:44 PM, Haibin Zhang wrote: >> Hi, Ying Fang, >> I use qemu(https://gitee.com/src-openeuler/qemu.git) launch a >> openEuler guest, but it hung. >> Qcow2 is downloaded from >> https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/o... >> >> Qemu command: >> qemu-system-aarch64 -name guest=12345,debug-threads=on -machine >> virt,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 \ >> -cpu host -m 122880 -object iothread,id=iothread1 \ >> -object >> memory-backend-ram,id=ram-node0,size=64424509440,host-nodes=0,policy=preferred >> -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 \ >> -object >> memory-backend-ram,id=ram-node1,size=64424509440,host-nodes=1,policy=preferred >> -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 \ It works after getting rid of these two memory-backend-ram objects
I am wondering whether there is enough memory on each NUMA node you specified in the command line on your host. You can use numactl -H to check numa info of your host.
I have specified policy=preferred not strict. I get some clues below: https://lore.kernel.org/lkml/1577088979-8545-1-git-send-email-prime.zeng@his...
It says numa confguration is conflict with cpu affinity specified by MPIDR
Yes, this may be the clue and I'm quite intrested in this issue.
Since currently KVM does not support userspace mpdir configuration. I am now doing some work to enable MPIDR emulation in KVM on aarch64.
The “-smp x,sockets=n,cores=m” of qemu doesn’t work in aarch64 ?
It is partially supported, qemu has support for CPU topology configuration. The topology infomation can be reported to GuestOS via ACPI or fdt. However KVM does not support MPIDR emulation now, which means guest can see the topology but MPIDR is different. I'm now working on it to enable MPIDR emulation in KVM so that the ACPI info can be coordinate with MPIDR.
Please refer to: https://patchwork.ozlabs.org/project/qemu-devel/cover/20180704124923.32483-1...
The patch series related with this issue has been posted to upstream: https://patchwork.kernel.org/cover/11781339/
You can pick up and test it.
>> -smp 60,sockets=2,cores=30,threads=1 -bios
A 2(scokets) * 30(cores) * 1(threads) topology may confusing the guest kernel. I'll test this situation later this week. Please keep in touch with me.
>> /usr/share/AAVMF/AAVMF_CODE.fd \ >> -device >> pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 >> \ >> -device >> pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 >> \ >> -drive >> file=/data/euler/euler.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none >> \ >> -device >> virtio-blk-pci,scsi=off,bus=pci.1,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >> \ >> -net none -serial telnet::9001,server,nowait -monitor stdio >> Guest startup logs: >> [ 3.769390] smp: Brought up 2 nodes, 60 CPUs >> [ 4.125100] SMP: Total of 60 processors activated. >> [ 4.126246] CPU features: detected: Privileged Access Never >> [ 4.127625] CPU features: detected: LSE atomic instructions >> [ 4.128969] CPU features: detected: User Access Override >> [ 4.130261] CPU features: detected: Data cache clean to Point >> of Persistence >> [ 4.131949] CPU features: detected: RAS Extension Support >> [ 4.133253] CPU features: detected: ARM64 MPAM Extension Support >> [ 4.134704] CPU features: detected: CRC32 instructions >> [ 4.184286] CPU: All CPU(s) started at EL1 >> [ 4.185891] alternatives: patching kernel code >> [ 64.308834] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: >> [ 64.310827] rcu: (detected by 0, t=15002 jiffies, g=-719, q=0) >> [ 64.312260] rcu: All QSes seen, last rcu_sched kthread >> activity 15002 (4294908290-4294893288), jiffies_till_next_fqs=1, >> root ->qsmask 0x0 >> [ 64.315251] swapper/0 R running task 0 1 >> 0 0x0000002a >> [ 64.316930] Call trace: >> [ 64.317533] dump_backtrace+0x0/0x198 >> [ 64.318432] show_stack+0x24/0x30 >> [ 64.319248] sched_show_task+0x12c/0x160 >> [ 64.320206] rcu_check_callbacks+0x780/0x7a8 >> [ 64.321253] update_process_times+0x34/0x60 >> [ 64.322296] tick_periodic+0x48/0x100 >> [ 64.323193] tick_handle_periodic+0x38/0xd8 >> [ 64.324224] arch_timer_handler_virt+0x38/0x58 >> [ 64.325320] handle_percpu_devid_irq+0x90/0x248 >> [ 64.326419] generic_handle_irq+0x34/0x50 >> [ 64.327383] __handle_domain_irq+0x68/0xc0 >> [ 64.328367] gic_handle_irq+0x6c/0x170 >> [ 64.329275] el1_irq+0xb8/0x140 >> [ 64.330040] __bitmap_weight+0x48/0xa0 >> [ 64.330942] build_sched_domains+0xd58/0xf50 >> [ 64.331962] sched_init_domains+0xd4/0x108 >> [ 64.332950] sched_init_smp+0x3c/0x8c >> [ 64.333842] kernel_init_freeable+0x17c/0x34c >> [ 64.334891] kernel_init+0x18/0x118 >> [ 64.335735] ret_from_fork+0x10/0x18 > > You mean there is something wrong with the NUMA feature, right ? > > please try using the xml below and see if you can luanch your VM. No numa specified in below xml file. > > <domain type='kvm' id='17'> > <name>openeuler-test</name> > <memory unit='GiB'>32</memory> > <currentMemory unit='GiB'>32</currentMemory> > <memtune> > <hard_limit unit='GiB'>32</hard_limit> > </memtune> > <vcpu placement='static' cpuset='7-10'>60</vcpu> > <iothreads>5</iothreads> > <cputune> > <emulatorpin cpuset='7-10'/> > </cputune> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='aarch64' machine='virt-4.1'>hvm</type> > <loader readonly='yes' > type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> > <nvram>/var/lib/libvirt/qemu/nvram/fangying_openeuler_VARS.fd</nvram> > > <boot dev='hd'/> > <boot dev='cdrom'/> > </os> > <features> > <gic version='3'/> > <acpi/> > </features> > <cpu mode='host-passthrough' check='none'> > <topology sockets='2' cores='30' threads='1'/> > </cpu> > <clock offset='utc'> > </clock> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <devices> > <emulator>/usr/bin/qemu-kvm</emulator> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2' cache='none' io='native'/> > <source > file='/root/fangying/vm/openEuler-20.03-LTS.aarch64.qcow2'/> > <backingStore/> > <target dev='sda' bus='scsi'/> > <alias name='scsi0-0-0-0'/> > <address type='drive' controller='0' bus='0' target='0' > unit='0'/> > </disk> > <controller type='usb' index='0' model='ehci'> > <alias name='usb'/> > <address type='pci' domain='0x0000' bus='0x03' slot='0x01' > function='0x0'/> > </controller> > <controller type='scsi' index='0' model='virtio-scsi'> > <alias name='scsi0'/> > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' > function='0x0'/> > </controller> > <controller type='pci' index='0' model='pcie-root'> > <alias name='pcie.0'/> > </controller> > <controller type='pci' index='1' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='1' port='0x8'/> > <alias name='pci.1'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x0' multifunction='on'/> > </controller> > <controller type='pci' index='2' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='2' port='0x9'/> > <alias name='pci.2'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x1'/> > </controller> > <controller type='pci' index='3' model='pcie-to-pci-bridge'> > <model name='pcie-pci-bridge'/> > <alias name='pci.3'/> > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' > function='0x0'/> > </controller> > <controller type='pci' index='4' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='4' port='0xa'/> > <alias name='pci.4'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x2'/> > </controller> > <controller type='pci' index='5' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='5' port='0xb'/> > <alias name='pci.5'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x3'/> > </controller> > <controller type='pci' index='6' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='6' port='0xc'/> > <alias name='pci.6'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x4'/> > </controller> > <controller type='pci' index='7' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='7' port='0xd'/> > <alias name='pci.7'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x5'/> > </controller> > <controller type='pci' index='8' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='8' port='0xe'/> > <alias name='pci.8'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x6'/> > </controller> > <controller type='pci' index='9' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='9' port='0xf'/> > <alias name='pci.9'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x7'/> > </controller> > <controller type='pci' index='10' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='10' port='0x10'/> > <alias name='pci.10'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x0' multifunction='on'/> > </controller> > <controller type='pci' index='11' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='11' port='0x11'/> > <alias name='pci.11'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x1'/> > </controller> > <controller type='pci' index='12' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='12' port='0x12'/> > <alias name='pci.12'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x2'/> > </controller> > <controller type='pci' index='13' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='13' port='0x13'/> > <alias name='pci.13'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x3'/> > </controller> > <controller type='pci' index='14' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='14' port='0x14'/> > <alias name='pci.14'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x4'/> > </controller> > <controller type='pci' index='15' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='15' port='0x15'/> > <alias name='pci.15'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x5'/> > </controller> > <controller type='pci' index='16' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='16' port='0x16'/> > <alias name='pci.16'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x6'/> > </controller> > <controller type='pci' index='17' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='17' port='0x17'/> > <alias name='pci.17'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x7'/> > </controller> > <controller type='pci' index='18' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='18' port='0x18'/> > <alias name='pci.18'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0' multifunction='on'/> > </controller> > <controller type='pci' index='19' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='19' port='0x19'/> > <alias name='pci.19'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x1'/> > </controller> > <controller type='pci' index='20' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='20' port='0x1a'/> > <alias name='pci.20'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x2'/> > </controller> > <controller type='pci' index='21' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='21' port='0x1b'/> > <alias name='pci.21'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x3'/> > </controller> > <controller type='pci' index='22' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='22' port='0x1c'/> > <alias name='pci.22'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x4'/> > </controller> > <controller type='pci' index='23' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='23' port='0x1d'/> > <alias name='pci.23'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x5'/> > </controller> > <controller type='pci' index='24' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='24' port='0x1e'/> > <alias name='pci.24'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x6'/> > </controller> > <controller type='pci' index='25' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='25' port='0x1f'/> > <alias name='pci.25'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x7'/> > </controller> > <controller type='pci' index='26' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='26' port='0x20'/> > <alias name='pci.26'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x0' multifunction='on'/> > </controller> > <controller type='pci' index='27' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='27' port='0x21'/> > <alias name='pci.27'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x1'/> > </controller> > <controller type='pci' index='28' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='28' port='0x22'/> > <alias name='pci.28'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x2'/> > </controller> > <controller type='pci' index='29' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='29' port='0x23'/> > <alias name='pci.29'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x3'/> > </controller> > <controller type='pci' index='30' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='30' port='0x24'/> > <alias name='pci.30'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x4'/> > </controller> > <controller type='virtio-serial' index='0'> > <alias name='virtio-serial0'/> > <address type='pci' domain='0x0000' bus='0x03' slot='0x02' > function='0x0'/> > </controller> > <serial type='pty'> > <source path='/dev/pts/2'/> > <target port='0'/> > <alias name='serial0'/> > </serial> > <console type='pty' tty='/dev/pts/2'> > <source path='/dev/pts/2'/> > <target type='serial' port='0'/> > <alias name='serial0'/> > </console> > <channel type='unix'> > <source mode='bind' > path='/var/run/libvirt/qemu/fangying-test.extend'/> > <target type='virtio' name='org.qemu.guest_agent.1' > state='connected'/> > <alias name='channel0'/> > <address type='virtio-serial' controller='0' bus='0' port='1'/> > </channel> > <channel type='unix'> > <source mode='bind' > path='/var/run/libvirt/qemu/fangying-test.agent'/> > <target type='virtio' name='org.qemu.guest_agent.0' > state='connected'/> > <alias name='channel1'/> > <address type='virtio-serial' controller='0' bus='0' port='2'/> > </channel> > <channel type='unix'> > <source mode='bind' > path='/var/run/libvirt/qemu/fangying-test.hostd'/> > <target type='virtio' name='org.qemu.guest_agent.2' > state='connected'/> > <alias name='channel2'/> > <address type='virtio-serial' controller='0' bus='0' port='3'/> > </channel> > <channel type='unix'> > <source mode='bind' > path='/var/run/libvirt/qemu/fangying-test.upgraded'/> > <target type='virtio' name='org.qemu.guest_agent.3' > state='connected'/> > <alias name='channel3'/> > <address type='virtio-serial' controller='0' bus='0' port='4'/> > </channel> > <input type='tablet' bus='usb'> > <alias name='input0'/> > <address type='usb' bus='0' port='1'/> > </input> > <input type='keyboard' bus='usb'> > <alias name='input1'/> > <address type='usb' bus='0' port='2'/> > </input> > <graphics type='vnc' port='5903' autoport='yes' listen='0.0.0.0'> > <listen type='address' address='0.0.0.0'/> > </graphics> > <video> > <model type='virtio' heads='1' primary='yes'/> > <alias name='video0'/> > <address type='pci' domain='0x0000' bus='0x03' slot='0x04' > function='0x0'/> > </video> > <memballoon model='virtio'> > <alias name='balloon0'/> > <address type='pci' domain='0x0000' bus='0x03' slot='0x03' > function='0x0'/> > </memballoon> > </devices> > <seclabel type='dynamic' model='dac' relabel='yes'> > <label>+0:+0</label> > <imagelabel>+0:+0</imagelabel> > </seclabel> > </domain> > > I will test your cmdline way later to see if it can be reproduced. > > Thanks. > Ying. > .
Virt mailing list -- virt@openeuler.org To unsubscribe send an email to virt-leave@openeuler.org
-
Haibin Zhang -
Ying Fang