- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/2] assign-account: change get to post
by Luan Shengde 29 Dec '20

29 Dec '20

use post is more easier to use rest-client to send request with data for applying account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

1 0

[PATCH v5 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..162bf9f --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'nolab' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: add user lkp
by Li Yuanchao 29 Dec '20

29 Dec '20

[error message] auto-submit: whoami: unknown uid 1090 Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/auto-submit/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 5b44551..52cd996 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -3,6 +3,7 @@ FROM alpine:3.11 +RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/mail-post: add cc/bcc for send mail service
by Luan Shengde 29 Dec '20

29 Dec '20

enable user send Cc and Bcc email. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index ac455fa..9d3e601 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -20,6 +20,8 @@ post '/send_mail_yaml' do mail_info = { 'subject' => data['subject'], 'to' => data['to'], + 'cc' => data['cc'], + 'bcc' => data['bcc'], 'body' => data['body'] } @@ -32,6 +34,8 @@ post '/send_mail_text' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -45,6 +49,8 @@ post '/send_mail_encode' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -63,10 +69,16 @@ def send_mail(mail_info) from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] to mail_info['to'] + cc mail_info['cc'] + bcc mail_info['bcc'] body mail_info['body'] end mail.deliver! + check_to_store_email(mail) +end + +def check_to_store_email(mail) return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 10 +++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..5f88c36 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -40,9 +42,11 @@ class MailBisectResult commit: #{@commit_id} ("#{(a)git_commit.subject}") compiler: gcc (GCC) 7.3.0 - all errors/warnings: + all errors/warnings (new ones prefixed by >>): #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/READEME.md: fix format error
by Liu Yinsi 29 Dec '20

29 Dec '20

correct READEME format and content. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..c7a80e7 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下： - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，下面就可以开始进行测试了。 + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） ```bash @@ -81,7 +81,9 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… + 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) + 以查看iperf任务结果为例操如下： ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output ``` -- 2.23.0

1 0

[PATCH compass-ci 3/3] auto_submit.yaml: add l/linux/linux for pkg_build
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkg_build need specific submit args for l/linux/linux Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index fce1dc0..69b73ff 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -3,6 +3,9 @@ a/AvxToNeon/AvxToNeon: c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +l/linux/linux: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml - testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 2/3] auto_submit: add some log
by Li Yuanchao 29 Dec '20

29 Dec '20

Print git_repo to help find out if repo has been submitted job and the job id. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 4e26c87..f96be8c 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -42,6 +42,7 @@ class AutoSubmit def get_argvs(newrefs_info) git_repo = newrefs_info['git_repo'] + puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] -- 2.23.0

1 0

[PATCH compass-ci 1/3] auto_submit: pkgbuild_repo may have specific submit args
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkgbuild_repos all use the same submit args before. But some of them may need their own args, not the common one. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index d765094..4e26c87 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -50,7 +50,7 @@ class AutoSubmit return unless pkgbuild_repo submit_argv.push("pkgbuild_repo=#{pkgbuild_repo}") - repo2job_key = 'archlinux' + repo2job_key = @repo2job[git_repo] ? git_repo : 'archlinux' else return unless @repo2job[git_repo] -- 2.23.0

1 0

[PATCH compass-ci] optimize READE.md
by Liu Yinsi 29 Dec '20

29 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..4640974 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - �� - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci�� + �� + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci��8��rootfs�� 8. ��rootfs��openeuler/aarch64/20.03��/srv/os/��wget��cgz�� ```bash @@ -81,7 +81,8 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md),��compass-ci��srv/result��job id��output�� +14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/$job_id/output ``` + ��compass-ci��srv/result��yaml��/��/testbox/os-os-version-os-arch/job_id��output��(��tab��) -- 2.23.0

1 1

[PATCH v2 compass-ci] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

1.some comment and code is unnecessary, so remove. 2.put `echo **` together better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..fd921dc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,14 +18,9 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= @@ -47,8 +42,6 @@ do *) ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -63,6 +56,7 @@ done < $ipxe_script initrd=initrd cat $initrds > $initrd +echo $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append @@ -78,7 +72,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: get field state count
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get field state count for overview of report page. [input] suite/os/tbox_group/... eg: suite [output] {"build-pkg"=> {"failed"=>5543, "finished"=>906, "incomplete"=>197, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1782, "failed"=>675, "incomplete"=>279, "post_run"=>3, "OOM"=>1, "running"=>1}, "cci-makepkg"=> {"failed"=>1659, "finished"=>1231, "incomplete"=>384, "OOM"=>4, "post_run"=>1}, "iperf"=>{"finished"=>179, "failed"=>94}, "spinlock"=>{"finished"=>52, "failed"=>1}, "iperf-walk-os-test"=>{"finished"=>172, "failed"=>24, "OOM"=>1}, "stream"=>{"finished"=>242, "incomplete"=>157, "failed"=>7}, "deploy-cci"=>{"finished"=>27, "failed"=>19}, "borrow"=>{"finished"=>101, "incomplete"=>12, "failed"=>4}, "bisect"=>{"finished"=>169, "failed"=>151}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index ea67064..15e2ad0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -509,3 +509,37 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def query_field_state(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 }, + aggs: { + all_state: { + terms: { field: 'job_state' } + } + } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']["all_#{field}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_field_state(es_result) +end + +def parse_field_state(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_state = result['all_state']['buckets'] + state_count = {} + all_state.map { |state| state_count[state['key']] = state['doc_count'] } + result_hash[key] = state_count unless state_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..5becfd0 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,17 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

3 3

[PATCH compass-ci 2/3] providers/qemu: encapsulated into functions
by Liu Yinsi 29 Dec '20

29 Dec '20

test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272000 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 171 +++++++++++++++++++++++++----------------- 1 file changed, 104 insertions(+), 67 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index f84c5ae..48bacab 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,74 +7,111 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi - -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm - -echo $SCHED_PORT -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} - -append= -initrds= -while read a b c -do - case "$a" in - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - esac -done < $ipxe_script - -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi } -[ -n "$initrds" ] || { - exit +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm } -initrd=initrd -cat $initrds > $initrd - -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file - -sleep 5 - -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) - -"${kvm[@]}" --append "${append}" +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +get_rootfs() +{ + append= + initrds= + while read a b c + do + case "$a" in + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + esac + done < $ipxe_script +} + +check_rootfs() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} + +kvm_command() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) +} + +run_qemu() +{ + "${kvm[@]}" --append "${append}" +} + +check_logfile +write_logfile + +get_rootfs +check_rootfs + +qemu_command + +print_message + +kvm_command +run_qemu -- 2.23.0

2 2

[PATCH compass-ci 1/3] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

some comment and code is unnecessary, so remove. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.271979 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..f84c5ae 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -22,18 +22,12 @@ echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= while read a b c do case "$a" in - '#') - ;; initrd) file=$(basename "$b") wget --timestamping -a ${log_file} --progress=bar:force $b @@ -44,11 +38,7 @@ do wget --timestamping -a ${log_file} --progress=bar:force $b append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") ;; - *) - ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -78,7 +68,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

2 4

[PATCH v3 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..5572274 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,10 @@ dirs=( ) mkdir -p "${dirs[@]}" + +if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd +elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +fi -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs/common: add comment for set password file.
by Wang Chenglong 29 Dec '20

29 Dec '20

[Why] add password file is easy to ignore. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..2ee16c8 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -33,7 +33,8 @@ usage() dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: - ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists " } -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e95f55b --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/es-find: add -d/--desc-keyword option
by Lu Weitao 29 Dec '20

29 Dec '20

es-find support order result by keyword with desc [Example] es-find suite=iperf -d/--desc-keyword start_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/sbin/es-find b/sbin/es-find index 2397338..17eefe6 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -29,8 +29,8 @@ def parse_argv items end -options = {} -options[:nr_jobs] = 10 +options = { nr_jobs: 10, desc_keyword: nil } + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -45,6 +45,10 @@ opt_parser = OptionParser.new do |opts| options[:nr_jobs] = n.to_i end + opts.on('-d', '--desc-keyword KEYWORD', 'order by keyword: start_time|end_time|...') do |d| + options[:desc_keyword] = d + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -54,6 +58,6 @@ end opt_parser.parse!(ARGV) items = parse_argv es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs]) +query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/params_group.rb optimize filter_groups()
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] 1. We had for-each key-value in hash, needn't query value by key once more. 2. When remove empty value of Hash(String, Hash(String, Array(...)), remove the first level key-value at first. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..fa71653 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,13 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + if value.empty? + groups.delete(group_key) + next + end + value.each do |dim_key, job_list| + value.delete(dim_key) if job_list.empty? end - groups.delete(group_key) if groups[group_key].empty? end end -- 2.23.0

1 0

[PATCH v2 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..538f45d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,13 @@ dirs=( ) mkdir -p "${dirs[@]}" + +[ -n "$os_mount" ] && { + if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd + elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr + fi +} -- 2.23.0

4 3

[PATCH lkp-tests] INSTALL.md: add quick installation description
by Wei Jihui 29 Dec '20

29 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- INSTALL.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 1bf0acc7..d7ce3517 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,18 +1,30 @@ +# Quick installation +```shell + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source $HOME/.${SHELL##*/}rc +``` +all is ok! + +# Step-by-Step installation + system wide setup ================= - ## debian - +```shell sudo apt-get install ruby-git ruby-activesupport +``` ## openEuler - +```shell sudo dnf install ruby rubygems gem install git activesupport +``` per-user setup ============== - +```shell git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests echo "export LKP_SRC=$PWD" >> $HOME/.${SHELL##*/}rc @@ -24,3 +36,4 @@ per-user setup hdd_partitions: ssd_partitions: EOF +``` -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 29 Dec '20

29 Dec '20

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..21396c4 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,33 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? + matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +99,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 29 Dec '20

29 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0