- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sparrow/READEME.md: fix format error
by Liu Yinsi 29 Dec '20

29 Dec '20

correct READEME format and content. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..c7a80e7 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下： - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，下面就可以开始进行测试了。 + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） ```bash @@ -81,7 +81,9 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… + 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) + 以查看iperf任务结果为例操如下： ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output ``` -- 2.23.0

1 0

[PATCH compass-ci 3/3] auto_submit.yaml: add l/linux/linux for pkg_build
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkg_build need specific submit args for l/linux/linux Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index fce1dc0..69b73ff 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -3,6 +3,9 @@ a/AvxToNeon/AvxToNeon: c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +l/linux/linux: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml - testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 2/3] auto_submit: add some log
by Li Yuanchao 29 Dec '20

29 Dec '20

Print git_repo to help find out if repo has been submitted job and the job id. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 4e26c87..f96be8c 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -42,6 +42,7 @@ class AutoSubmit def get_argvs(newrefs_info) git_repo = newrefs_info['git_repo'] + puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] -- 2.23.0

1 0

[PATCH compass-ci 1/3] auto_submit: pkgbuild_repo may have specific submit args
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkgbuild_repos all use the same submit args before. But some of them may need their own args, not the common one. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index d765094..4e26c87 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -50,7 +50,7 @@ class AutoSubmit return unless pkgbuild_repo submit_argv.push("pkgbuild_repo=#{pkgbuild_repo}") - repo2job_key = 'archlinux' + repo2job_key = @repo2job[git_repo] ? git_repo : 'archlinux' else return unless @repo2job[git_repo] -- 2.23.0

1 0

[PATCH compass-ci] optimize READE.md
by Liu Yinsi 29 Dec '20

29 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..4640974 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - �� - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci�� + �� + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci��8��rootfs�� 8. ��rootfs��openeuler/aarch64/20.03��/srv/os/��wget��cgz�� ```bash @@ -81,7 +81,8 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md),��compass-ci��srv/result��job id��output�� +14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/$job_id/output ``` + ��compass-ci��srv/result��yaml��/��/testbox/os-os-version-os-arch/job_id��output��(��tab��) -- 2.23.0

1 1

[PATCH v2 compass-ci] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

1.some comment and code is unnecessary, so remove. 2.put `echo **` together better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..fd921dc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,14 +18,9 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= @@ -47,8 +42,6 @@ do *) ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -63,6 +56,7 @@ done < $ipxe_script initrd=initrd cat $initrds > $initrd +echo $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append @@ -78,7 +72,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: get field state count
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get field state count for overview of report page. [input] suite/os/tbox_group/... eg: suite [output] {"build-pkg"=> {"failed"=>5543, "finished"=>906, "incomplete"=>197, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1782, "failed"=>675, "incomplete"=>279, "post_run"=>3, "OOM"=>1, "running"=>1}, "cci-makepkg"=> {"failed"=>1659, "finished"=>1231, "incomplete"=>384, "OOM"=>4, "post_run"=>1}, "iperf"=>{"finished"=>179, "failed"=>94}, "spinlock"=>{"finished"=>52, "failed"=>1}, "iperf-walk-os-test"=>{"finished"=>172, "failed"=>24, "OOM"=>1}, "stream"=>{"finished"=>242, "incomplete"=>157, "failed"=>7}, "deploy-cci"=>{"finished"=>27, "failed"=>19}, "borrow"=>{"finished"=>101, "incomplete"=>12, "failed"=>4}, "bisect"=>{"finished"=>169, "failed"=>151}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index ea67064..15e2ad0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -509,3 +509,37 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def query_field_state(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 }, + aggs: { + all_state: { + terms: { field: 'job_state' } + } + } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']["all_#{field}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_field_state(es_result) +end + +def parse_field_state(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_state = result['all_state']['buckets'] + state_count = {} + all_state.map { |state| state_count[state['key']] = state['doc_count'] } + result_hash[key] = state_count unless state_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..5becfd0 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,17 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

3 3

[PATCH compass-ci 2/3] providers/qemu: encapsulated into functions
by Liu Yinsi 29 Dec '20

29 Dec '20

test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272000 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 171 +++++++++++++++++++++++++----------------- 1 file changed, 104 insertions(+), 67 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index f84c5ae..48bacab 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,74 +7,111 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi - -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm - -echo $SCHED_PORT -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} - -append= -initrds= -while read a b c -do - case "$a" in - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - esac -done < $ipxe_script - -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi } -[ -n "$initrds" ] || { - exit +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm } -initrd=initrd -cat $initrds > $initrd - -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file - -sleep 5 - -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) - -"${kvm[@]}" --append "${append}" +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +get_rootfs() +{ + append= + initrds= + while read a b c + do + case "$a" in + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + esac + done < $ipxe_script +} + +check_rootfs() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} + +kvm_command() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) +} + +run_qemu() +{ + "${kvm[@]}" --append "${append}" +} + +check_logfile +write_logfile + +get_rootfs +check_rootfs + +qemu_command + +print_message + +kvm_command +run_qemu -- 2.23.0

2 2

[PATCH compass-ci 1/3] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

some comment and code is unnecessary, so remove. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.271979 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..f84c5ae 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -22,18 +22,12 @@ echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= while read a b c do case "$a" in - '#') - ;; initrd) file=$(basename "$b") wget --timestamping -a ${log_file} --progress=bar:force $b @@ -44,11 +38,7 @@ do wget --timestamping -a ${log_file} --progress=bar:force $b append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") ;; - *) - ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -78,7 +68,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

2 4