December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] srv-http/start: add os images data download directory
by Cui Yili 29 Jan '21

29 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index b56fe4d..8c8ca0b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/os:/srv/os:ro -d srv-http ) -- 2.23.0

3 4

[PATCH compass-ci] sbin/auto_submit: take the first pkgbuild_repo if no '-git'
by Li Yuanchao 12 Jan '21

12 Jan '21

As new pkgbuild_repos don't have '-git' at the end, return the first one instead. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 8845b13..d765094 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -18,7 +18,7 @@ class AutoSubmit end def get_pkgbuild_repo(repo_array) - pkgbuild_repo = nil + pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| next unless repo =~ /-git$/ -- 2.23.0

2 1

[PATCH compass-ci] container: mount default config dir to delimiter container for submitting job
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/delimiter/start b/container/delimiter/start index 27e8d45..5d7037d 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -23,6 +23,7 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm 'delimiter' cmd = %w[ @@ -34,8 +35,9 @@ cmd = %w[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} - -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro -v /srv/git:/srv/git:ro -w #{DEFAULT_CCI}/src delimiter -- 2.23.0

2 1

[PATCH v2 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 12 Jan '21

12 Jan '21

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..4ff4285 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,32 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +98,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] assign-account: add 'lab' to default yaml
by Luan Shengde 08 Jan '21

08 Jan '21

the default yaml file will contain: my_email my_name lab lab is used to read the lab yaml according to its value. example: if the value of lab is 'z9', the sched will read token from file: z9.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 1

[PATCH v5 compass-ci] providers/qemu: refactor kvm script
by Liu Yinsi 07 Jan '21

07 Jan '21

encapsulating into functions. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 160 +++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..c926d3d 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,112 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + [ -f "$log_file" ] || { + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + } +} + +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +parse_ipxe_script() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || exit +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +get_initrd() +{ + initrd=initrd + cat $initrds > $initrd } -[ -n "$initrds" ] || { - exit +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 } -initrd=initrd -cat $initrds > $initrd +run_qemu() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) + + "${kvm[@]}" --append "${append}" +} -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +check_logfile +write_logfile -sleep 5 +qemu_command +parse_ipxe_script +check_option_value +get_initrd -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +print_message -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

2 2

[PATCH lkp-tests] jobs/build-linux.yaml: submit 100 randconfig job
by Lin Jiaxin 04 Jan '21

04 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/build-linux.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/jobs/build-linux.yaml b/jobs/build-linux.yaml index 080a84b3..1ced05ad 100644 --- a/jobs/build-linux.yaml +++ b/jobs/build-linux.yaml @@ -1,4 +1,7 @@ suite: build-pkg category: functional build-pkg: -config: <%= 'aarch64-randconfig-'+SecureRandom.random_number(1000).to_s %> +config: <%= + today = Time.new + Array.new(100) { |i| 'aarch64-randconfig-' + today.strftime('%Y-%m-%d') + '-' + format('%02d', i) } + %> -- 2.23.0

2 1

[PATCH v2 compass-ci] container/srv-http: import h5ai
by Lu Weitao 04 Jan '21

04 Jan '21

[Why] Optimize all pages style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 16 +++++++++++++--- container/srv-http/build | 7 +++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 8f9a36d..987e40e 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,15 +1,25 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM nginx:alpine +FROM alpine:3.9 + ADD root / -RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo +RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm \ + && mkdir /run/nginx/ \ + && mkdir /h5ai RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN chown -R nginx:nginx /srv/_h5ai +WORKDIR /h5ai +COPY h5ai /h5ai + +# build h5ai package +RUN npm install \ + && npm run build \ + && cp -r ./build/_h5ai /srv \ + && chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..7b0ea74 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $H5AI_SRC ]] || H5AI_SRC=/c/h5ai + +cp -r $H5AI_SRC ./ + docker build -t srv-http . + +# clean h5ai folder +rm -rf ./h5ai diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

2 2

[PATCH v2 compass-ci] docker-rootfs/common: add comment for set password file.
by Wang Chenglong 04 Jan '21

04 Jan '21

[Why] add password file is easy to ignore. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..2ee16c8 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -33,7 +33,8 @@ usage() dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: - ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists " } -- 2.23.0

3 4

[PATCH v3 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 04 Jan '21

04 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..1b6f42f --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +## Use the tool to create a new nfsroot for os + +## Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +## Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists + +## Some configuration items: + ./packages-to-install + # If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + # Set the password for the image into this file. -- 2.23.0

2 2

[PATCH lkp-tests] lib/bootstrap.sh: 'run-ipconfig' step should be before the check step
by Yu Chuan 03 Jan '21

03 Jan '21

[Before] test job: result/borrow/2020-12-31/vm-2p8g/debian-sid-aarch64/3600/z9.275666 key log: + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + cat /sys/class/net/enp0s1//operstate + [ down = up ] [After] test job: result/borrow/2020-12-31/vm-2p8g/debian-sid-aarch64/3600/z9.275673 key log: + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + cat /sys/class/net/enp0s1//operstate + [ up = up ] Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..770d550724f1 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -160,6 +160,9 @@ setup_network() return } + $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig + network_ok && return + local net_devices=$(get_net_devices) if [ -z "$net_devices" ]; then @@ -181,9 +184,6 @@ setup_network() fi fi - $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig - network_ok && return - local err_msg='IP-Config: Auto-configuration of network failed' dmesg | grep -q -F "$err_msg" || { # Include $err_msg in the error message so that it matches -- 2.23.0

2 2

git count
by Wu Fengguang 31 Dec '20

31 Dec '20

Cao Xueliang is our code star again, thanks xueliang! 22 Cao Xueliang 21 Luan Shengde 16 Wu Zhende 13 Wei Jihui 13 Li Yuanchao 11 Hu Xuejiao 11 Ren Wen 10 Lu Weitao 9 Liu Shaofei 7 Lin Jiaxin 7 Lu Kaiyi 6 Liu Yinsi 6 Cui Yili 6 Wang Yong 5 Li Ping 4 Xiao Shenwei 3 Wu Fengguang 3 Zhang Yu 3 Yu Chuan 2 Xu Xijian 2 Wang Chenglong 1 Zhang Yale 0 Bai Jing 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Dewan 0 Shi Zhichao

1 0

[PATCH v2 compass-ci 2/2] mail-robot: optimize assign-account-email
by Luan Shengde 31 Dec '20

31 Dec '20

set different greetings for mail-robot and bisect Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index ee3b574..b71b3fe 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -4,13 +4,19 @@ # frozen_string_literal: true def build_apply_account_email(my_info) + email_greeting = if my_info['bisect'] + 'We have automatically created the following information for you.' + else + 'Thank you for joining us.' + end + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready Dear #{my_info['my_name']}, - Thank you for joining us. + #{email_greeting} You can use the following info to submit jobs: -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/assign_account_client: optimize
by Luan Shengde 31 Dec '20

31 Dec '20

add new key for bisect way to apply account use rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index d7e0623..4831545 100755 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -7,6 +7,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' @@ -52,7 +53,8 @@ class AutoAssignAccount apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -69,6 +71,7 @@ class AutoAssignAccount end def send_mail + @my_info['bisect'] = true message = build_apply_account_email(@my_info) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: optimize assign-account-email
by Luan Shengde 31 Dec '20

31 Dec '20

set different greetings for mail-robot and bisect Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index ee3b574..b71b3fe 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -4,13 +4,19 @@ # frozen_string_literal: true def build_apply_account_email(my_info) + email_greeting = if my_info['bisect'] + 'We have automatically created the following information for you.' + else + 'Thank you for joining us.' + end + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready Dear #{my_info['my_name']}, - Thank you for joining us. + #{email_greeting} You can use the following info to submit jobs: -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/assign_account_client: optimize
by Luan Shengde 31 Dec '20

31 Dec '20

add new key for bisect way to apply account use rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index d7e0623..15d3f35 100755 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -7,6 +7,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' @@ -52,7 +53,8 @@ class AutoAssignAccount apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -69,6 +71,7 @@ class AutoAssignAccount end def send_mail + @my_info['bisect'] = true message = build_apply_account_email(@my_info) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end @@ -81,3 +84,12 @@ class AutoAssignAccount send_mail end end + +my_info = { + 'my_email' => 'shdluan(a)163.com', + 'my_name' => 'lsd', + 'my_commit_url' => '' +} + +test = AutoAssignAccount.new(my_info) +test.send_account -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: send bisect report to author email
by Cao Xueliang 31 Dec '20

31 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index f0f0194..209a28d 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -18,8 +18,9 @@ class MailBisectResult @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) + @to = @git_commit.author_email # now send mail to review - @to = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com' + @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' end def create_send_email @@ -37,6 +38,7 @@ class MailBisectResult data = <<~BODY To: #{@to} + Bcc: #{@bcc} Subject: #{subject} Hi #{(a)git_commit.author_name}, -- 2.23.0

1 0

[PATCH compass-ci] delimiter: use docker to run build-pkg job
by Cao Xueliang 31 Dec '20

31 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/delimiter.rb | 2 +- src/delimiter/utils.rb | 10 ++++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index bf36db3..1afffbc 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id'].inspect} bisect.yaml) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id'].inspect} bisect.yaml queue='dc-bisect') rescue StandardError => e puts e sleep(60) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 49d48ee..de5a302 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -135,10 +135,16 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'vm-2p32g' : 'vm-2p16g' + job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' + job['os'] = 'openeuler' + job['os_arch'] = 'aarch64' + job['os_mount'] = 'container' + job['os_version'] = '20.03-pre' + job['docker_image'] = 'openeuler:20.03-pre' - job.delete('error_ids') job.delete('start_time') + job.delete('tbox_group') + job.delete('error_ids') job.delete('job_state') job.delete('end_time') job.delete('loadavg') -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/tutorial.md: fix abnormal display in markdown
by Lu Kaiyi 31 Dec '20

31 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 4) 备注目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + PKGBUILD 存放路径： `/srv/git/archlinux/*/*/PKGBUILD`。 ### depends -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: add runtime field in borrow-*.yml and update borrow-machine.*.md
by Zhang Yale 31 Dec '20

31 Dec '20

User can specify the borrow time in "runtime" field Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 7 ++++--- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b598bec 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..5288377 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # 使用vim来修改你的sleep字段的值 + # 使用vim来修改你的runtime字段的值 hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # 修改完毕后重新submit即可 hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

2 1

[PATCH v3 compass-ci] sparrow/4-docker/buildall: build dist image when it doesn't exist
by Xu Xijian 31 Dec '20

31 Dec '20

If the dist docker image doesn't exist for some reason, run build script to create it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 32801d1..ca4406b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -32,7 +32,8 @@ do_one() ( cd $container [ "$container_name" == 'ssh-r' ] && exit - if [ "$action" != "reboot" ]; then + docker images | grep -wq "$container_name" + if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/4-docker/buildall: build dist image when it doesn't exist
by Xu Xijian 31 Dec '20

31 Dec '20

If the dist docker image doesn't exist for some reason, run build script to create it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 32801d1..9e66ac9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -32,7 +32,8 @@ do_one() ( cd $container [ "$container_name" == 'ssh-r' ] && exit - if [ "$action" != "reboot" ]; then + docker images | grep -w -q "$container_name" + if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 1

[PATCH lkp-tests] tests/build-pkg: fix to generate the directory of *.cgz file
by Liu Shaofei 31 Dec '20

31 Dec '20

[why] Since the file upload path saved by ES is inconsistent with the file path generated by the build-pkg script, the *.cgz file upload failed. [error] ==> /tmp/stdout <== renamed '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz.tmp' -> '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz' ==> Leaving fakeroot environment. ==> Finished making: kubernetes 1.18.5-1 (Wed Dec 30 16:50:17 CST 2020) create soft link: latest.cgz -> HEAD.cgz ==> /tmp/stderr <== curl: (22) The requested URL returned error: 403 Forbidden Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..e21397ee6 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -88,6 +88,9 @@ request_pkg() create_cgz_name() { + pkgname="${pkgname}" + [ -n "${pkgbase}" ] && pkgname="${pkgbase}" + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" pkg_args="-A --check --skippgpcheck" -- 2.23.0

3 5

[PATCH compass-ci] doc: add runtime field in borrow-*.yml and update borrow-machine.*.md
by Zhang Yale 31 Dec '20

31 Dec '20

User can specify the borrow time in "runtime" field Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 7 ++++--- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b598bec 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..5288377 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v3 lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 31 Dec '20

31 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 52 ++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..8e6acbac 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +export_env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c && cd /c + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +export_env +deploy -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..8891d6c7 --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='http://ftp.us2.freshrpms.net/linux/RPM/epel/7/aarch64/Packages/h/httpress-1…' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm from source maybe not be available + # httpress-1.1.0-13.fc31.aarch64.rpm has been validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 31 Dec '20

31 Dec '20

[why] make es result more easy access. [example] input: result_hash = {} es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 29d50e9..dcd735b 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -129,7 +129,9 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + result_hash = {} + parse_fields(result, result_hash) + result_hash end end @@ -217,3 +219,46 @@ def build_aggs_from_fields(fields, aggs_hash) build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) end + +# input: +# result_hash = {} +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result, result_hash) + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + result_hash[key] = {} + all_field = result[sub_field]['buckets'] + parse_fields(all_field, result_hash[key]) + else + result_hash[key] = result['doc_count'] + end + end +end -- 2.23.0

2 1

[PATCH compass-ci] matrix2.rb: filter groups to make compare faster
by Li Yuanchao 31 Dec '20

31 Dec '20

When do compare, after group jobs, there are many one_size group, which can not find another job to make a group. Delete such groups to save time. But, if there is no size > 1 group, 3 one_size groups will be kept to avoid 'No Data'. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/matrix2.rb | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..0e80858 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -123,18 +123,37 @@ end def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) + have_multi_member = multi_member?(groups) + one_size_count = 0 groups.each do |group_key, value| - suite_list = [] - value.each do |dimension_key, jobs| - groups[group_key][dimension_key], suites = create_matrix(jobs) - suite_list.concat(suites) + if value.size < 2 + one_size_count += 1 + if have_multi_member || one_size_count > 3 + groups.delete(group_key) + next + end end - suites_hash[group_key] = suite_list + get_groups_matrix(groups, group_key, value, suites_hash) end - return groups, suites_hash end +def get_groups_matrix(groups, group_key, value, suites_hash) + suite_list = [] + value.each do |dimension_key, jobs| + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) + end + suites_hash[group_key] = suite_list +end + +def multi_member?(groups) + groups.each do |_key, value| + return true if value.size > 1 + end + return false +end + # input: # 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() # 2. params from user's template include: -- 2.23.0

2 3

[PATCH v4 compass-ci] providers/qemu: refactor kvm script
by Liu Yinsi 31 Dec '20

31 Dec '20

encapsulating into functions. test: /srv/result/host-info/2020-12-30/vm-2p8g/openeuler-20.09-aarch64/z9.275381 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 160 +++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..0845099 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,112 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + [ ! -f "$log_file" ] || { + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + } +} + +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +parse_ipxe_script() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || exit +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +get_initrd() +{ + initrd=initrd + cat $initrds > $initrd } -[ -n "$initrds" ] || { - exit +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 } -initrd=initrd -cat $initrds > $initrd +run_qemu() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) + + "${kvm[@]}" --append "${append}" +} -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +check_logfile +write_logfile -sleep 5 +qemu_command +parse_ipxe_script +check_option_value +get_initrd -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +print_message -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix to generate the directory of *.cgz file
by Liu Shaofei 31 Dec '20

31 Dec '20

[why] Since the file upload path saved by ES is inconsistent with the file path generated by the build-pkg script, the *.cgz file upload failed. [error] ==> /tmp/stdout <== renamed '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz.tmp' -> '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz' ==> Leaving fakeroot environment. ==> Finished making: kubernetes 1.18.5-1 (Wed Dec 30 16:50:17 CST 2020) create soft link: latest.cgz -> HEAD.cgz ==> /tmp/stderr <== curl: (22) The requested URL returned error: 403 Forbidden Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..436d43840 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -88,6 +88,8 @@ request_pkg() create_cgz_name() { + [ -n "${pkgbase}" ] && pkgname="${pkgbase}" + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" pkg_args="-A --check --skippgpcheck" -- 2.23.0

1 0

[PATCH lkp-tests] api-avx2neon/PKGBUILD: add multi arch build
by Wang Yong 31 Dec '20

31 Dec '20

build packages through $(arch) Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- pkg/api-avx2neon/PKGBUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/api-avx2neon/PKGBUILD b/pkg/api-avx2neon/PKGBUILD index 3ccebca86..fac047add 100644 --- a/pkg/api-avx2neon/PKGBUILD +++ b/pkg/api-avx2neon/PKGBUILD @@ -9,7 +9,7 @@ md5sums=('SKIP') build() { cd "$srcdir/AvxToNeon/tests" - make + make ARCH=$(arch) } package() { -- 2.23.0

1 0

[PATCH lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..b906303d --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm which from source maybe unavailable + # httpress-1.1.0-13.fc31.aarch64.rpm is validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..44af00ea --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm from source maybe bot be available + # httpress-1.1.0-13.fc31.aarch64.rpm has been validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH v2 lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 31 Dec '20

31 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 52 ++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..8e6acbac 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +export_env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c && cd /c + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +export_env +deploy -- 2.23.0

2 1

[PATCH lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 30 Dec '20

30 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH compass-ci] 3-code/dev-env: add variables
by Hu Xuejiao 30 Dec '20

30 Dec '20

[why] $GIT_SERVER will be defined in job.yaml $LKP_SERVER is ip in test-machine for upload_files Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..ac38f56 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -23,6 +23,8 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port +LKP_SERVER: $server_ip +GIT_SERVER: $GIT_SERVER TASKQUEUE_HOST: $sched_host OS_HTTP_HOST: $os_http_host INITRD_HTTP_HOST: $initrd_http_host -- 2.23.0

1 0

[PATCH v4 compass-ci] sbin/run_job: Automatic task execution
by Hu Xuejiao 30 Dec '20

30 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 0000000..c84df0d --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,74 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + docker_images=${single_arg[4]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/mail_bisect_result: optimize bisect report
by Cao Xueliang 30 Dec '20

30 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- doc/bisect_email.en.md | 28 ++++++++++++++++++++++++++++ lib/constants.rb | 2 ++ lib/mail_bisect_result.rb | 28 ++++++++++++++++++---------- 3 files changed, 48 insertions(+), 10 deletions(-) create mode 100644 doc/bisect_email.en.md diff --git a/doc/bisect_email.en.md b/doc/bisect_email.en.md new file mode 100644 index 0000000..fee7ee1 --- /dev/null +++ b/doc/bisect_email.en.md @@ -0,0 +1,28 @@ +# Who are we? + +Compass-CI runs a number of tests for open source projects for free. +We aim to improve OSS quality in a number of hardware and OS environment. +We are part of the openEuler community. + +# Why do you get the email? + +Whenever you git push, our test robot will pull and test the new code. +When new errors are found, Compass-CI will auto run git bisect to find out the first bad commit, +and send email report to the commit author. + +# About the email + +The first bad commit job result directory: +you can read the below file to understand the content of each file. +- https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-res… + +Notice: +You may need to read the output file of the first bad commit job result directory, +it is the build log about the first bad commit. + +# how to confirm bug fix + +- git push your fix patch to your project. +- download the job.yaml from the first bad commit job directory. +- submit -m job.yaml upstream_commit=$your_fix_commit. +- check result from the new result directory which is in output above command and key is result_root. diff --git a/lib/constants.rb b/lib/constants.rb index f0938f2..c3c7543 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -13,3 +13,5 @@ MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 + +SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5f88c36..f0f0194 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -3,6 +3,8 @@ require 'json' require 'base64' require_relative 'git' +require_relative 'es_query' +require_relative 'constants' require_relative 'mail_client' require_relative 'assign_account_client' @@ -27,10 +29,11 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" - prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" - bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' - pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" + prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" + bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' + bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} @@ -38,17 +41,19 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, - url: #{(a)git_commit.url} - commit: #{@commit_id} ("#{(a)git_commit.subject}") - compiler: gcc (GCC) 7.3.0 + We found some error/warning(s) and the first bad commit in the below project: + git url: #{(a)git_commit.url} + git commit: #{@commit_id} ("#{(a)git_commit.subject}") - all errors/warnings (new ones prefixed by >>): + All error/warning(s) (new ones prefixed by >>): #{@all_errors} - reference information: + Reference information: + compiler: gcc (GCC) 7.3.0 #{pkgbuild_repo_url} - #{bisect_job_url} #{first_bad_commit_job_url} + #{bisect_job_url} + #{bisect_report_doc} Regards, Compass CI team BODY @@ -67,6 +72,9 @@ class MailBisectResult 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } + account_info = ESQuery.new(index: 'accounts').query_by_id(@to) + return if account_info + apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: optimize bisect report
by Cao Xueliang 30 Dec '20

30 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5f88c36..4883072 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -3,6 +3,7 @@ require 'json' require 'base64' require_relative 'git' +require_relative 'es_query' require_relative 'mail_client' require_relative 'assign_account_client' @@ -28,9 +29,9 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" - bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' - pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" + bisect_job_url = ENV['result_root'] ? "bisect job result directory: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD file directory: #{prefix_srv}/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} @@ -46,9 +47,9 @@ class MailBisectResult #{@all_errors} reference information: + #{first_bad_commit_job_url} #{pkgbuild_repo_url} #{bisect_job_url} - #{first_bad_commit_job_url} Regards, Compass CI team BODY @@ -67,6 +68,9 @@ class MailBisectResult 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } + account_info = ESQuery.new(index: 'accounts').query_by_id(@to) + return if account_info + apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end -- 2.23.0

1 1

[PATCH v1 compass-ci] lib/mail_bisect_result: optimize bisect report
by Cao Xueliang 30 Dec '20

30 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 2 ++ lib/mail_bisect_result.rb | 28 ++++++++++++++++++---------- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index f0938f2..c3c7543 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -13,3 +13,5 @@ MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 + +SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5f88c36..bc3c74a 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -3,6 +3,8 @@ require 'json' require 'base64' require_relative 'git' +require_relative 'es_query' +require_relative 'constants' require_relative 'mail_client' require_relative 'assign_account_client' @@ -27,10 +29,11 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" - prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" - bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' - pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" + prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" + bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' + bisect_report_doc = "bisect report:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_r…" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} @@ -38,17 +41,19 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, - url: #{(a)git_commit.url} - commit: #{@commit_id} ("#{(a)git_commit.subject}") - compiler: gcc (GCC) 7.3.0 + We found some error/warning(s) and the first bad commit in the below project: + git url: #{(a)git_commit.url} + git commit: #{@commit_id} ("#{(a)git_commit.subject}") - all errors/warnings (new ones prefixed by >>): + All error/warning(s) (new ones prefixed by >>): #{@all_errors} - reference information: + Reference information: + compiler: gcc (GCC) 7.3.0 #{pkgbuild_repo_url} - #{bisect_job_url} #{first_bad_commit_job_url} + #{bisect_job_url} + #{bisect_report_doc} Regards, Compass CI team BODY @@ -67,6 +72,9 @@ class MailBisectResult 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } + account_info = ESQuery.new(index: 'accounts').query_by_id(@to) + return if account_info + apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: add ${config} judge in common job
by Wang Yong 30 Dec '20

30 Dec '20

if not exist ${config} in general build-pkg job, ${cci_dir} should not be judeged and do not need upload file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..d356b2f14 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -49,7 +49,9 @@ get_config() upload_config() { local upload_dir="${cci_dir}/${pkgname}" - [ -d "${upload_dir}" ] || return + + [ -n "${config}" ] || return + [ -d "${upload_dir}" ] || return upload_to_target_dir "${upload_dir}" } -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: import h5ai into container
by Lu Weitao 30 Dec '20

30 Dec '20

[Why] Optimize result page style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 16 +++++++++++++--- container/srv-http/build | 7 +++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 8f9a36d..987e40e 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,15 +1,25 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM nginx:alpine +FROM alpine:3.9 + ADD root / -RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo +RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm \ + && mkdir /run/nginx/ \ + && mkdir /h5ai RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN chown -R nginx:nginx /srv/_h5ai +WORKDIR /h5ai +COPY h5ai /h5ai + +# build h5ai package +RUN npm install \ + && npm run build \ + && cp -r ./build/_h5ai /srv \ + && chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..bdda319 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $H5AI_SRC ]] || H5AI_SRC=/c/h5ai + +cp -r $H5AI_SRC ./ + docker build -t srv-http . + +# clean h5ai foler +rm -rf ./h5ai diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

1 1

Re: [PATCH v3 compass-ci] providers/qemu: refactor kvm script
by Ren Wen 30 Dec '20

30 Dec '20

>>+ if [ ! -f "$log_file" ]; then >>>+ touch $log_file >>>+ # fluentd refresh time is 1s >>>+ # let fluentd to monitor this file first >>>+ sleep 2 >>>+ fi >> >>how about: [ -f "$log_file" ] || { ... } ? > >there are 2 line codes, if just 1 line code use like you say is good. {} is a way to group a list of commands to be executed as a unit, you can also use like this: [ ] || { expression1 expression2 } Thanks, RenWen > >Thanks, >Yinsi > > >> >>-------- >>Thanks >>Yu Chuan >> >>>+} >>> >>>-qemu=qemu-system-aarch64 >>>-command -v $qemu >/dev/null || qemu=qemu-kvm >>>+qemu_command() >>>+{ >>>+ qemu=qemu-system-aarch64 >>>+ command -v $qemu >/dev/null || qemu=qemu-kvm >>>+} >>> >>>-ipxe_script=ipxe_script >>>-curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script >>>-cat $ipxe_script >> ${log_file} >>>+write_logfile() >>>+{ >>>+ ipxe_script=ipxe_script >>>+ curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script >>>+ cat $ipxe_script >> ${log_file} >>>+} >>> >>>-append= >>>-initrds= >>>-while read a b c >>>-do >>>- case "$a" in >>>- '#') >>>- ;; >>>- initrd) >>>- file=$(basename "$b") >>>- wget --timestamping -a ${log_file} --progress=bar:force $b >>>- initrds+="$file " >>>- ;; >>>- kernel) >>>- kernel=$(basename "$b") >>>- wget --timestamping -a ${log_file} --progress=bar:force $b >>>- append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") >>>- ;; >>>- *) >>>- ;; >>>- esac >>>-done < $ipxe_script >>>+option_value() >>>+{ >>>+ append= >>>+ initrds= >>>+ while read a b c >>>+ do >>>+ case "$a" in >>>+ '#') >>>+ ;; >>>+ initrd) >>>+ file=$(basename "$b") >>>+ wget --timestamping -a ${log_file} --progress=bar:force $b >>>+ initrds+="$file " >>>+ ;; >>>+ kernel) >>>+ kernel=$(basename "$b") >>>+ wget --timestamping -a ${log_file} --progress=bar:force $b >>>+ append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") >>>+ ;; >>>+ *) >>>+ ;; >>>+ esac >>>+ done < $ipxe_script >>>+} >>> >>>-[ -s "$kernel" ] || { >>>- echo "can't get kernel or kernel size is 0" >>>- exit >>>+check_option_value() >>>+{ >>>+ [ -s "$kernel" ] || { >>>+ echo "can't get kernel or kernel size is 0" >>>+ exit >>>+ } >>>+ >>>+ [ -n "$initrds" ] || exit >>>+ >>>+ initrd=initrd >>>+ cat $initrds > $initrd >>> } >>> >>>-[ -n "$initrds" ] || { >>>- exit >>>+print_message() >>>+{ >>>+ echo $SCHED_PORT >>>+ echo kernel: $kernel >>>+ echo initrds: $initrds >>>+ echo append: $append >>>+ echo less $log_file >>>+ >>>+ sleep 5 >>> } >>> >>>-initrd=initrd >>>-cat $initrds > $initrd >>>+run_qemu() >>>+{ >>>+ kvm=( >>>+ $qemu >>>+ -machine virt-4.0,accel=kvm,gic-version=3 >>>+ -kernel $kernel >>>+ -initrd $initrd >>>+ -smp $nr_cpu >>>+ -m $memory >>>+ -cpu Kunpeng-920 >>>+ -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd >>>+ -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} >>>+ -k en-us >>>+ -no-reboot >>>+ -nographic >>>+ -serial file:${log_file} >>>+ -monitor null >>>+ ) >>>+ >>>+ "${kvm[@]}" --append "${append}" >>>+} >>> >>>-echo $SCHED_PORT >>>-echo kernel: $kernel >>>-echo initrds: $initrds >>>-echo append: $append >>>-echo less $log_file >>>+check_logfile >>>+write_logfile >>> >>>-sleep 5 >>>+qemu_command >>>+option_value >>>+check_option_value >>> >>>-kvm=( >>>- $qemu >>>- -machine virt-4.0,accel=kvm,gic-version=3 >>>- -kernel $kernel >>>- -initrd $initrd >>>- -smp $nr_cpu >>>- -m $memory >>>- -cpu Kunpeng-920 >>>- -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd >>>- -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} >>>- -k en-us >>>- -no-reboot >>>- -nographic >>>- -serial file:${log_file} >>>- -monitor null >>>-) >>>+print_message >>> >>>-"${kvm[@]}" --append "${append}" >>>+run_qemu >>>-- >>>2.23.0 >>>

2 1

[PATCH v3 compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 30 Dec '20

30 Dec '20

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 71 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..29d50e9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,50 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = {} + build_aggs_from_fields(fields, aggs_hash) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +190,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# aggs_hash = {} +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields, aggs_hash) + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] = { + "all_#{field}" => { + 'terms' => { field: field, size: 1000 } + } + } + + build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) +end -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/qemu: refactor kvm script
by Liu Yinsi 30 Dec '20

30 Dec '20

encapsulating into functions. test: /srv/result/host-info/2020-12-30/vm-2p8g/openeuler-20.09-aarch64/z9.275242 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 156 +++++++++++++++++++++++++----------------- 1 file changed, 93 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..c48c0c7 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,108 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +option_value() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || exit + + initrd=initrd + cat $initrds > $initrd } -[ -n "$initrds" ] || { - exit +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 } -initrd=initrd -cat $initrds > $initrd +run_qemu() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) + + "${kvm[@]}" --append "${append}" +} -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +check_logfile +write_logfile -sleep 5 +qemu_command +option_value +check_option_value -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +print_message -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

3 4

[PATCH v2 compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 30 Dec '20

30 Dec '20

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example1] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... [example2] input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... [example3] input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 52 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..a9b651e 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,45 @@ class ESQuery result end + + # select field1, field2, count(*) from index where query_items group by field1, field2 + # example: + # fields = ['suite', 'job_state'] + # query_items = { + # 'os' => 'openeuler' + # } + # input: + # fields, query_items + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = {} + build_aggs_from_fields(fields, aggs_hash) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +185,16 @@ def assign_desc_body(keyword) }] } end + +def build_aggs_from_fields(fields, aggs_hash) + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] = { + "all_#{field}" => { + 'terms' => { field: field, size: 1000 } + } + } + + build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) +end -- 2.23.0

2 1

[PATCH lkp-tests] distro/common: fix the scenario when the directory not exist
by Liu Shaofei 30 Dec '20

30 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- distro/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/distro/common b/distro/common index 166ff8550..1fd7df4e8 100755 --- a/distro/common +++ b/distro/common @@ -87,8 +87,9 @@ upload_to_target_dir() . $LKP_SRC/lib/upload.sh local dir="$1" - local file="$(ls -F ${dir} | grep -v "@")" + [ -d "${dir}" ] || return 0 + local file="$(ls -F ${dir} | grep -v "@")" [ -z "${file}" ] && return 0 upload_one_curl "${dir}/${file}" "${dir}" >/dev/null -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor: print ssh info
by Wu Zhende 30 Dec '20

30 Dec '20

After the user logs out, can copy this printed information to log in again. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 283cd788d..12f5280f3 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -87,8 +87,11 @@ class Monitor def ssh_connect(ssh_host, ssh_port, web_socket) web_socket.close - exec "ssh-keygen -R #{ssh_host}; - ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" + cmd = "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" + puts cmd + + cmd = "ssh-keygen -R #{ssh_host};" + cmd + exec cmd end def stop(data, web_socket, code = 1000, reason = 'normal') -- 2.23.0

1 0

[PATCH compass-ci 3/3] monitoring/filter.cr: fix send message error
by Wu Zhende 30 Dec '20

30 Dec '20

filter message error: Error writing to socket: Broken pipe Exception: Error writing to socket: Broken pipe (Errno) from /usr/lib/crystal/core/socket.cr:82:13 in 'unbuffered_write' from /usr/lib/crystal/core/io/buffered.cr:192:5 in 'flush' from /usr/lib/crystal/core/io/buffered.cr:200:5 in 'close' from /usr/lib/crystal/core/http/server/handlers/websocket_handler.cr:44:9 in 'call' from /usr/share/crystal/app/lib/kemal/src/kemal/websocket.cr:11:7 in 'call' from /usr/share/crystal/app/lib/kemal/src/kemal/websocket_handler.cr:14:7 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/static_file_handler.cr:64:9 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/log_handler.cr:8:37 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/init_handler.cr:12:7 in 'call' from /usr/lib/crystal/core/http/server/request_processor.cr:48:11 in 'process' from /usr/lib/crystal/core/http/server/request_processor.cr:22:3 in 'process' from /usr/lib/crystal/core/http/server.cr:497:5 in 'handle_client' from /usr/lib/crystal/core/http/server.cr:463:13 in '->' from /usr/lib/crystal/core/fiber.cr:255:3 in 'run' from /usr/lib/crystal/core/fiber.cr:48:34 in '->' from ??? Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 5d3bc8b..79af3ab 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -41,6 +41,9 @@ class Filter @hash[query].each do |socket| socket.send msg.to_json + rescue e + puts "send msg failed: #{e}" + remove_filter_rule(query, socket) end end -- 2.23.0

1 0

[PATCH compass-ci 2/3] monitoring/monitoring.cr: change rabbitmq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Get logs from different rabbitmq queues. And use multi-process to improve service processing capabilities. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/monitoring.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/monitoring/monitoring.cr b/src/monitoring/monitoring.cr index d09c363..8d52d46 100644 --- a/src/monitoring/monitoring.cr +++ b/src/monitoring/monitoring.cr @@ -9,7 +9,8 @@ module Monitoring message_queue_client = MessageQueueClient.new - spawn message_queue_client.monitoring_message_queue(filter, "logging-test", "logging-test") + spawn message_queue_client.monitoring_message_queue(filter, "serial-logging", "serial-logging") + spawn message_queue_client.monitoring_message_queue(filter, "docker-logging", "docker-logging") ws "/filter" do |socket| query = JSON::Any.new("") -- 2.23.0

1 0

[PATCH compass-ci 1/3] master-fluentd/docker-fluentd.conf: send logs to different mq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Send serial logs to serial-logging queue. Send docker logs to docker-logging queue. The two do not affect each other. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 7bf2229..410f2eb 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -18,7 +18,7 @@ <match serial.**> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange serial-logging exchange_type fanout exchange_durable false heartbeat 10 @@ -57,7 +57,7 @@ <store> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange docker-logging exchange_type fanout exchange_durable false heartbeat 10 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: print ssh info
by Wu Zhende 30 Dec '20

30 Dec '20

After the user logs out, can copy this printed information to log in again. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index 283cd788d..522df713f 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -86,6 +86,7 @@ class Monitor def ssh_connect(ssh_host, ssh_port, web_socket) web_socket.close + puts "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" exec "ssh-keygen -R #{ssh_host}; ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" -- 2.23.0

2 1

[PATCH v2 compass-ci] providers/qemu: encapsulated into functions
by Liu Yinsi 30 Dec '20

30 Dec '20

use function better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/z9.273758 ssssss-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 163 ++++++++++++++++++++++++++---------------- 1 file changed, 100 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..6c4ad00 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,115 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +option_value() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +qemu_options() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) } -[ -n "$initrds" ] || { - exit +run_qemu() +{ + "${kvm[@]}" --append "${append}" } -initrd=initrd -cat $initrds > $initrd +check_logfile +write_logfile -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +qemu_command +option_value +check_option_value -sleep 5 +print_message -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +qemu_options -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

3 4

[PATCH v3 compass-ci] sbin/run_job: Automatic task execution
by Hu Xuejiao 30 Dec '20

30 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 0000000..c6cf9e8 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH compass-ci] container/assist-result: fix failed to start
by Liu Yinsi 30 Dec '20

30 Dec '20

[error] docker: invalid publish opts format (should be name=value but got ':'). [why] $ASSIST_RESULT_PORT not set default value, it's null when deploy compass-ci. [how] set default value ASSIST_RESULT_PORT=8102 in 3-code/dev-env. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..ffaa6e1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${ASSIST_RESULT_PORT:=8102} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -39,6 +40,7 @@ SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when received email_file's from like "Xing Ming", add has double quotes with it, the email message may end unexpectedly From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready Dear Luan ----email end [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH v2 compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] parsed email address for received email file may like: "Xing Ming <email_address(a)163.com>" or "\"Xing Ming\" <eamil_address(a)163.com>" if the email address contains double quotes as part of string, the email message may end unexpectedly, like: Date: Wed, 23 Dec 2020 18:32:06 +0800^M From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready^M Dear Luan ====email message ends [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH compass-ci 2/4] assign-account: use post request type
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when send requise use rest-client, request data is not supportted for get, so we need to change it to post. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

2 2

[PATCH v4 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 30 Dec '20

30 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

2 2

[PATCH lkp-tests] makepkg.conf: use the my_curl command to use the cache
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- etc/makepkg.conf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 1fffdc2d2..5d73399d9 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,10 +8,10 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u' - 'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' +DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' 'scp::/usr/bin/scp -C %u %o') -- 2.23.0

2 1

[PATCH compass-ci] delimiter/utils: set job testbox
by Cao Xueliang 29 Dec '20

29 Dec '20

If job has config field, set the testbox to "vm-2p32g", otherwise set the testbox to "vm-2p16g". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 17128ab..49d48ee 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -135,12 +135,14 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job['testbox'] = job['config'] ? 'vm-2p32g' : 'vm-2p16g' job.delete('error_ids') job.delete('start_time') job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('queue') job.delete('stats') job.delete('id') -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: try to install ntpdate if there is no such pkg
by Yu Chuan 29 Dec '20

29 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ae90220750db..7d15bb370fa9 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -6,6 +6,7 @@ . $LKP_SRC/lib/reboot.sh . $LKP_SRC/lib/ucode.sh . $LKP_SRC/lib/tbox.sh +. $LKP_SRC/lib/detect-system.sh # borrowed from linux/tools/testing/selftests/rcutorture/doc/initrd.txt # Author: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com> @@ -236,7 +237,13 @@ run_ntpdate() [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - [ -x '/usr/sbin/ntpdate' ] || return + has_cmd ntpdate || { + detect_system + local DISTRO=$_system_name_lowercase + . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) + } + + has_cmd ntpdate || return local hour="$(date +%H)" -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] my_curl: change the curl request so that the cache can be used
by Cui Yili 29 Dec '20

29 Dec '20

change curl request, in order to use local cache. e.g. https://download.redis.io/releases/redis-4.0.2.tar.gz ==> https://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k http://download.redis.io/releases/redis-4.0.2.tar.gz ==> http://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k Signed-off-by: cuiyili <2268260388(a)qq.com> --- rootfs/addon/usr/bin/my_curl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 rootfs/addon/usr/bin/my_curl diff --git a/rootfs/addon/usr/bin/my_curl b/rootfs/addon/usr/bin/my_curl new file mode 100755 index 000000000..97d5352a4 --- /dev/null +++ b/rootfs/addon/usr/bin/my_curl @@ -0,0 +1,22 @@ +#!/bin/bash + +declare -a array +i=0 + +if [ -n "$HTTP_PROXY_HOST" ]; then + for param in "$@"; + do + if [[ "$param" =~ http://(.*) ]] && [ -n "$HTTP_PROXY_PORT" ]; then + array[$i]="http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT/${BASH_REMATCH[1]}" + elif [[ "$param" =~ https://(.*) ]] && [ -n "$HTTPS_PROXY_PORT" ]; then + array[$i]="https://$HTTP_PROXY_HOST:$HTTPS_PROXY_PORT/${BASH_REMATCH[1]}" + else + array[$i]="$param" + fi + ((i++)) + done + + command curl -k ${array[*]} +else + command curl "$@" +fi -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] lkp-setup-rootfs: all use of env http_proxy and delete specific proxy settings
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 9 --------- 1 file changed, 9 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..a7f1a7ffb 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -105,7 +105,6 @@ install_proxy_yum() { [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo fi @@ -117,11 +116,6 @@ install_proxy_apt() { [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy -Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; -Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; -EOF - for cfg in /etc/apt/sources.list /etc/apt/sources.list.d/* do [ -f "$cfg" ] || continue @@ -176,9 +170,6 @@ install_proxy() install_goproxy - [ -z "$SQUID_HOST" ] && return - [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum } -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add query job stats multiple times
by Cao Xueliang 29 Dec '20

29 Dec '20

Sometimes, the job state is already extract_finished, but we can't query the job stats, so add a temporary solutions, query the job stats multiple times. The extract container will be improved in future. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 372e821..17128ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -83,10 +83,10 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? - es = ESQuery.new - new_job = es.query_by_id(new_job_id) + stats = query_stats(new_job_id, 10) + raise "es cant query #{new_job_id} stats field!" unless stats - status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + status = stats.key?(error_id) ? 'bad' : 'good' puts "new_job_id: #{new_job_id}" puts "upstream_commit: #{job['upstream_commit']}" record_jobs(new_job_id, job['upstream_commit']) @@ -94,6 +94,21 @@ module Utils return status end + # sometimes the job_state is extract_finished + # but we cant query the job stats field in es, so, add many times query + # this is a temporary solution, the extract container will be improved in future. + def query_stats(job_id, times) + (1..times).each do |i| + new_job = ESQuery.new.query_by_id(job_id) + puts "query stats times: #{i}" + return new_job['stats'] if new_job['stats'] + + sleep 60 + end + + return nil + end + def record_jobs(job_id, job_commit) FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') -- 2.23.0

1 0

[PATCH lkp-tests 2/2] my_curl: change the curl request so that the cache can be used.
by Cui Yili 29 Dec '20

29 Dec '20

change curl reuqest, in order to use local cache. E..g https://download.redis.io/releases/redis-4.0.2.tar.gz ==> https://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k http://download.redis.io/releases/redis-4.0.2.tar.gz ==> http://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k Signed-off-by: cuiyili <2268260388(a)qq.com> --- rootfs/addon/usr/bin/my_curl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 rootfs/addon/usr/bin/my_curl diff --git a/rootfs/addon/usr/bin/my_curl b/rootfs/addon/usr/bin/my_curl new file mode 100755 index 000000000..697f53d65 --- /dev/null +++ b/rootfs/addon/usr/bin/my_curl @@ -0,0 +1,22 @@ +#!/bin/bash + +declare -a array +j=0 + +if [ -n "$HTTP_PROXY_HOST" ]; then + for param in "$@"; + do + if [[ "$param" =~ http://(.*) ]] && [ -n "$HTTP_PROXY_PORT" ]; then + array[$j]="http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT/${BASH_REMATCH[1]}" + elif [[ "$param" =~ https://(.*) ]] && [ -n "$HTTPS_PROXY_PORT" ]; then + array[$j]="https://$HTTP_PROXY_HOST:$HTTPS_PROXY_PORT/${BASH_REMATCH[1]}" + else + array[$j]="$param" + fi + ((j++)) + done + + command curl -k ${array[*]} +else + command curl "$@" +fi -- 2.23.0

2 1

[PATCH lkp-tests 1/2] lkp-setup-rootfs: all use of env http_proxy and delete specific proxy settings
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 9 --------- 1 file changed, 9 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..a7f1a7ffb 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -105,7 +105,6 @@ install_proxy_yum() { [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo fi @@ -117,11 +116,6 @@ install_proxy_apt() { [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy -Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; -Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; -EOF - for cfg in /etc/apt/sources.list /etc/apt/sources.list.d/* do [ -f "$cfg" ] || continue @@ -176,9 +170,6 @@ install_proxy() install_goproxy - [ -z "$SQUID_HOST" ] && return - [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum } -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: don't return queues individually
by Ren Wen 29 Dec '20

29 Dec '20

- avoid to auto submit idle job if finding no job from sub-queues, because idle-queues are not found yet. - sub-queues should be shuffled. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..a1e3497 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -72,14 +72,14 @@ class Sched end end - return sub_queues unless sub_queues.empty? - idle_queues = [] of String default_queues.each do |queue| idle_queues << "#{queue}/idle" end - return idle_queues + all_queues = rand_queues(sub_queues) + idle_queues + + return all_queues end def get_job_from_queues(queues, testbox) -- 2.23.0

1 0

[PATCH compass-ci 2/2] scheduler: fix confused comments and add comments
by Ren Wen 29 Dec '20

29 Dec '20

fix: - use 'is joined with' instead of 'is link with'. - fix format. add: - add a TODO comment. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index f60c0e5..a7b4bd8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -25,10 +25,11 @@ class Sched @log.warn(e) end - # auto submit a job to collect the host information - # grub hostname is link with ":", like "00:01:02:03:04:05" - # remind: if like with "-", last "-05" is treated as host number - # then hostname will be "sut-00-01-02-03-04" !!! + # auto submit a job to collect the host information. + # + # grub hostname is joined with ":", like "00:01:02:03:04:05". + # remind: if joined with "-", last "-05" is treated as host number + # then hostname will be "sut-00-01-02-03-04" !!! def submit_host_info_job(mac) host = "sut-#{mac}" @redis.hash_set("sched/mac2host", normalize_mac(mac), host) @@ -63,6 +64,7 @@ class Sched sub_queues = [] of String default_queues.each do |queue| + # TODO: this could be high cost and should be improved in future : keys(pattern). matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") next if matched_queues.empty? -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler: find_job_boot: remove useless functions
by Ren Wen 29 Dec '20

29 Dec '20

- find_job : used to find job according to testbox. Now use sub-queues. - get_idle_job : used to find and submit idle jobs. Now use idle-queues. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 38 ---------------------------------- 1 file changed, 38 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..f60c0e5 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -183,32 +183,6 @@ class Sched end end - private def find_job(testbox : String, count = 1) - tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox_group.partition("--")[0] - - queue_list = query_consumable_keys(tbox) - - boxes = ["sched/" + testbox, - "sched/" + tbox_group, - "sched/" + tbox, - "sched/" + tbox_group + "/idle"] - boxes.each do |box| - next if queue_list.select(box).size == 0 - count.times do - job = prepare_job(box, testbox) - return job if job - - sleep(1) unless count == 1 - end - end - - # when find no job, auto submit idle job at background - spawn { auto_submit_idle_job(tbox_group) } - - return nil - end - private def prepare_job(queue_name, testbox) response = @task_queue.consume_task(queue_name) job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 @@ -232,18 +206,6 @@ class Sched return job end - private def get_idle_job(tbox_group, testbox) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - - # if there has no idle job, auto submit and get 1 - if job.nil? - auto_submit_idle_job(tbox_group) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - end - - return job - end - private def auto_submit_idle_job(testbox) full_path_patterns = "#{CCI_REPOS}/#{LAB_REPO}/allot/idle/#{testbox}/*.yaml" fields = ["testbox=#{testbox}", "subqueue=idle"] -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/params_group.rb optimize filter_groups()
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] 1. We had for-each key-value in hash, needn't query value by key once more. 2. When remove empty value of Hash(String, Hash(String, Array(job)), remove the first level key-value at first. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..9be2cbe 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,11 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + if value.empty? + groups.delete(group_key) + next end - groups.delete(group_key) if groups[group_key].empty? + value.delete_if { |_dim_key, job_list| job_list.empty? } end end -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: take the pkgbuild_repo if it is linux
by Lin Jiaxin 29 Dec '20

29 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index f96be8c..51ecc6a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -20,7 +20,7 @@ class AutoSubmit def get_pkgbuild_repo(repo_array) pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| - next unless repo =~ /-git$/ + next unless repo =~ /(-git|linux)$/ pkgbuild_repo = "archlinux/#{repo}" break -- 2.23.0

2 1

[PATCH lkp-tests] auto_submit: add build-linux.yaml for linux
by Lin Jiaxin 29 Dec '20

29 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/build-linux.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/build-linux.yaml diff --git a/jobs/build-linux.yaml b/jobs/build-linux.yaml new file mode 100644 index 00000000..080a84b3 --- /dev/null +++ b/jobs/build-linux.yaml @@ -0,0 +1,4 @@ +suite: build-pkg +category: functional +build-pkg: +config: <%= 'aarch64-randconfig-'+SecureRandom.random_number(1000).to_s %> -- 2.23.0

2 1

[PATCH compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get multi-field aggregation count for overview of report page. [example] es = ESQuery.new query_items = { 'os' => 'openeuler' } field1 = 'suite' field2 = 'job_state' es.query_fields(field1, field2, query_items) [input]: field1, field2, query_items (optional for query_items, default no scope limitation) [output]: {"build-pkg"=> {"failed"=>3804, "finished"=>800, "incomplete"=>196, "submit"=>136, "OOM"=>11}, "cci-depends"=>{"finished"=>580, "failed"=>218, "incomplete"=>1}, "iperf"=>{"finished"=>136, "failed"=>57}, "spinlock"=>{"finished"=>52, "failed"=>1}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..d881f6b 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,52 @@ class ESQuery result end + + # select field1, field2, count(*) from index where query_items group by field1, field2 + # example: + # es = ESQuery.new + # query_items = { + # 'os' => 'openeuler' + # } + # field1 = 'suite' + # field2 = 'job_state' + # es.query_fields(field1, field2, query_items) + # input: + # field1, field2, query_items + # (optional for query_items, default no scope limitation) + # output: + # {"build-pkg"=> + # {"failed"=>3804, "finished"=>800, "incomplete"=>196, "submit"=>136, "OOM"=>11}, + # "cci-depends"=>{"finished"=>580, "failed"=>218, "incomplete"=>1}, + # "iperf"=>{"finished"=>136, "failed"=>57}, + # "spinlock"=>{"finished"=>52, "failed"=>1}, + # ... + # } + def query_fields(field1, field2, query_items = {}) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: { + "all_#{field1}" => { + terms: { field: field1, size: 1000 }, + aggs: { + "all_#{field2}" => { + terms: { field: field2, size: 1000 } + } + } + } + }, + size: 0 + } + es_result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_fields(es_result, field2) + end end # Range Query Example: @@ -146,3 +192,16 @@ def assign_desc_body(keyword) }] } end + +def parse_fields(es_result, field) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_field = result["all_#{field}"]['buckets'] + field_count = {} + all_field.map { |item| field_count[item['key']] = item['doc_count'] } + result_hash[key] = field_count unless field_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] scheduler: fix: only use nodes job specified
by Ren Wen 29 Dec '20

29 Dec '20

[Before]: Every node in cluster config file will be distributed a job when submitting cluster jobs. [Now]: Job will be distributed according to job content. For example: If job has: if role server: if role client: then only nodes which roles contains 'server' and 'client' will use. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c09d145..bcb265f 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -38,6 +38,7 @@ class Sched job_messages = Array(Hash(String, String)).new lab = job.lab subqueue = job.subqueue + roles = get_roles(job) # collect all job ids job_ids = [] of String @@ -52,6 +53,9 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| + # continue if role in cluster config matches role in job + next if (config["roles"].as_a.map(&.to_s) & roles).empty? + queue = host.to_s job_id = add_task("#{queue}/#{subqueue}", lab) @@ -157,6 +161,11 @@ class Sched end end + def get_roles(job) + roles = job.hash.keys.map { |key| $1 if key =~ /^if role (.*)/ } + roles.compact.map(&.strip) + end + def get_commit_date(job) if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", -- 2.23.0

1 0

[PATCH lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 53 ++++++++++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..21308ecf 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,39 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c + cd /c || exit + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +env +deploy -- 2.23.0

2 1

[PATCH v1 compass-ci 2/2] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 8 ++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..a4a604b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -43,6 +45,8 @@ class MailBisectResult all errors/warnings: #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 1

[PATCH compass-ci] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 7 +++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..2d2e855 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -43,6 +45,7 @@ class MailBisectResult all errors/warnings: #{@all_errors} + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 1

[PATCH compass-ci] container: mount "/srv/git/archlinux" to srv-http
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 8c8ca0b..d9f461b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -15,6 +15,7 @@ cmd=( -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro + -v /srv/git/archlinux:/srv/git/archlinux:ro -d srv-http ) -- 2.23.0

1 1

[PATCH v3 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..8edd28e 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,11 @@ dirs=( ) mkdir -p "${dirs[@]}" + +if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd +elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t "$os_mount" //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr +fi -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] mail-robot: optimize parse email_file
by Luan Shengde 29 Dec '20

29 Dec '20

[why] avoid build too much messages in the code files [how] just raise error code in the code file, and build the error message in the file building email message Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 4 +-- .../lib/parse-apply-account-email.rb | 31 ++++++------------- 2 files changed, 11 insertions(+), 24 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 7cac0d5..af31e17 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -38,9 +38,7 @@ class GiteeCommitUrlCheck def gitee_commit_exist(email_index) return if email_index - error_message = "We can not confirm whether the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with your email.' + error_message = 'COMMIT_URL_NOT_AVAILABLE' raise error_message end diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 7024731..a41a89a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey @@ -27,7 +28,7 @@ require_relative 'gitee-commit-url-check' # gitee.com: # call GiteeCommitUrlCheck to check the commit # non-gitee.com: -# execute curl to check the commit +# get request date for the commit url to check the commit # parse_pub_key # check my_ssh_pubkey exists and return it class ParseApplyAccountEmail @@ -63,10 +64,8 @@ class ParseApplyAccountEmail # the commit url should be in a standart format, example: # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - error_message = "No matched commit url found.\n" - error_message += 'Ensure that you have add a right commit url, ' - error_message += "and with prefix 'my oss commit:'." - raise error_message + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) + raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) end mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] @@ -88,16 +87,13 @@ class ParseApplyAccountEmail return unless match_out.empty? - error_message = "The repo url for your commit is not in our upstream-repo list.\n" - error_message += 'Use a new one, or consulting the manager for available repo list.' - - raise error_message + raise 'NOT_REGISTERED' end def commit_url_availability(url, base_url) hub_name = url.split('/')[2] - # it requires authentication when execute curl to get the commit information + # it requires authentication when sending request to get the commit information # clone the repo and then validate the commit for the email address if hub_name.eql? 'gitee.com' gitee_commit(url, base_url) @@ -112,24 +108,17 @@ class ParseApplyAccountEmail end def non_gitee_commit(url) - url_fdback = %x(curl #{url}) + url_fdback = RestClient.get(url).body email_index = url_fdback.index @my_info['my_email'] return unless email_index.nil? - error_message = "We can not confirm the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with you email.' - - raise error_message + raise 'COMMIT_URL_NOT_AVAILABLE' end def parse_pub_key - error_message = "No pub_key found.\n" - error_message += 'Please add a pub_key as an attachment to your email.' - - raise error_message if @mail_content.attachments.empty? - raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + raise 'NO_PUBKEY' if @mail_content.attachments.empty? + raise 'PUBKEY_NAME_ERR' unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ pub_key = @mail_content.attachments[0].body.decoded -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] mail-robot: add error code/message
by Luan Shengde 29 Dec '20

29 Dec '20

add error message for each error code before building the email_message. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 87 +++++++++++++++---- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 593f345..242dfe1 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -3,32 +3,83 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_fail_email(my_info, message) - email_msg = <<~EMAIL_MESSAGE - To: #{my_info['my_email']} - Subject: [compass-ci] apply account failed +def email_err_message(message) + case message + when 'NO_COMMIT_URL' + err_message = <<~EMAIL_MESSAGE + No commit url found. + You should add a commit url for the 'apply account' email. - Dear user: + For example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'URL_PREFIX_ERR' + err_message = <<~EMAIL_MESSAGE + Please add a correct prefix for the commit url. + + my oss commit + + For example: + + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'NOT_REGISTERED' + err_message = <<~EMAIL_MESSAGE + Your repo has not been registered to our upstream-repos yet. + You should register your repo to the upstream first. + Try again after you have done it. + + Reference the following url to learn how to register the repo: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… + EMAIL_MESSAGE + when 'COMMIT_URL_NOT_AVAILABLE' + err_message = <<~EMAIL_MESSAGE + We cannot confirm the commit url matches your email. + + Make sure the commit exists. + Make sure it is truely submitted with your email. - Your application for account failed with following error: + Go to the commit history and find your commit. + Enter the commit and you will get your commit url. - #{message} + Try again after you have correctted the url + EMAIL_MESSAGE + when 'NO_PUBKEY' + err_message = <<~EMAIL_MESSAGE + No pub_key found. + Please add a pubkey to the email and then try again. + EMAIL_MESSAGE + when 'PUBKEY_NAME_ERR' + err_message = <<~EMAIL_MESSAGE + Pubkey file name error, keep its name as when it was generated. + The pubkey file name should like: - In order to successfully apply an account, please pay attention to the following points: + id_{{ xxx }}.pub + EMAIL_MESSAGE + end - 1. mail subject - The subject should exactly: apply account + err_message += <<~EMAIL_MESSAGE - 2. commit url - When you writing the url, add prefix: my oss commit - example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + Manual for how to apply account: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… + EMAIL_MESSAGE + + err_message +end + +def build_apply_account_fail_email(my_info, message) + err_message = email_err_message(message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: - attention: - Ensure you commit url exist and available to access. + Your application for account failed. - 3. ssh pubkey - You need to add a pubkey as an attachment to the email. + #{err_message} regards compass-ci -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] assign-account: change get to post
by Luan Shengde 29 Dec '20

29 Dec '20

use post is more easier to use rest-client to send request with data for applying account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

1 0

[PATCH v5 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..162bf9f --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'nolab' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: add user lkp
by Li Yuanchao 29 Dec '20

29 Dec '20

[error message] auto-submit: whoami: unknown uid 1090 Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/auto-submit/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 5b44551..52cd996 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -3,6 +3,7 @@ FROM alpine:3.11 +RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/mail-post: add cc/bcc for send mail service
by Luan Shengde 29 Dec '20

29 Dec '20

enable user send Cc and Bcc email. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index ac455fa..9d3e601 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -20,6 +20,8 @@ post '/send_mail_yaml' do mail_info = { 'subject' => data['subject'], 'to' => data['to'], + 'cc' => data['cc'], + 'bcc' => data['bcc'], 'body' => data['body'] } @@ -32,6 +34,8 @@ post '/send_mail_text' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -45,6 +49,8 @@ post '/send_mail_encode' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -63,10 +69,16 @@ def send_mail(mail_info) from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] to mail_info['to'] + cc mail_info['cc'] + bcc mail_info['bcc'] body mail_info['body'] end mail.deliver! + check_to_store_email(mail) +end + +def check_to_store_email(mail) return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 10 +++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..5f88c36 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -40,9 +42,11 @@ class MailBisectResult commit: #{@commit_id} ("#{(a)git_commit.subject}") compiler: gcc (GCC) 7.3.0 - all errors/warnings: + all errors/warnings (new ones prefixed by >>): #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/READEME.md: fix format error
by Liu Yinsi 29 Dec '20

29 Dec '20

correct READEME format and content. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..c7a80e7 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下： - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，下面就可以开始进行测试了。 + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） ```bash @@ -81,7 +81,9 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… + 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) + 以查看iperf任务结果为例操如下： ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output ``` -- 2.23.0

1 0

[PATCH compass-ci 3/3] auto_submit.yaml: add l/linux/linux for pkg_build
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkg_build need specific submit args for l/linux/linux Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index fce1dc0..69b73ff 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -3,6 +3,9 @@ a/AvxToNeon/AvxToNeon: c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +l/linux/linux: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml - testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 2/3] auto_submit: add some log
by Li Yuanchao 29 Dec '20

29 Dec '20

Print git_repo to help find out if repo has been submitted job and the job id. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 4e26c87..f96be8c 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -42,6 +42,7 @@ class AutoSubmit def get_argvs(newrefs_info) git_repo = newrefs_info['git_repo'] + puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] -- 2.23.0

1 0

[PATCH compass-ci 1/3] auto_submit: pkgbuild_repo may have specific submit args
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkgbuild_repos all use the same submit args before. But some of them may need their own args, not the common one. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index d765094..4e26c87 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -50,7 +50,7 @@ class AutoSubmit return unless pkgbuild_repo submit_argv.push("pkgbuild_repo=#{pkgbuild_repo}") - repo2job_key = 'archlinux' + repo2job_key = @repo2job[git_repo] ? git_repo : 'archlinux' else return unless @repo2job[git_repo] -- 2.23.0

1 0

[PATCH compass-ci] optimize READE.md
by Liu Yinsi 29 Dec '20

29 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..4640974 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - �� - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci�� + �� + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci��8��rootfs�� 8. ��rootfs��openeuler/aarch64/20.03��/srv/os/��wget��cgz�� ```bash @@ -81,7 +81,8 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md),��compass-ci��srv/result��job id��output�� +14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/$job_id/output ``` + ��compass-ci��srv/result��yaml��/��/testbox/os-os-version-os-arch/job_id��output��(��tab��) -- 2.23.0

1 1

[PATCH v2 compass-ci] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

1.some comment and code is unnecessary, so remove. 2.put `echo **` together better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..fd921dc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,14 +18,9 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= @@ -47,8 +42,6 @@ do *) ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -63,6 +56,7 @@ done < $ipxe_script initrd=initrd cat $initrds > $initrd +echo $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append @@ -78,7 +72,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: get field state count
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get field state count for overview of report page. [input] suite/os/tbox_group/... eg: suite [output] {"build-pkg"=> {"failed"=>5543, "finished"=>906, "incomplete"=>197, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1782, "failed"=>675, "incomplete"=>279, "post_run"=>3, "OOM"=>1, "running"=>1}, "cci-makepkg"=> {"failed"=>1659, "finished"=>1231, "incomplete"=>384, "OOM"=>4, "post_run"=>1}, "iperf"=>{"finished"=>179, "failed"=>94}, "spinlock"=>{"finished"=>52, "failed"=>1}, "iperf-walk-os-test"=>{"finished"=>172, "failed"=>24, "OOM"=>1}, "stream"=>{"finished"=>242, "incomplete"=>157, "failed"=>7}, "deploy-cci"=>{"finished"=>27, "failed"=>19}, "borrow"=>{"finished"=>101, "incomplete"=>12, "failed"=>4}, "bisect"=>{"finished"=>169, "failed"=>151}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index ea67064..15e2ad0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -509,3 +509,37 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def query_field_state(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 }, + aggs: { + all_state: { + terms: { field: 'job_state' } + } + } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']["all_#{field}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_field_state(es_result) +end + +def parse_field_state(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_state = result['all_state']['buckets'] + state_count = {} + all_state.map { |state| state_count[state['key']] = state['doc_count'] } + result_hash[key] = state_count unless state_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..5becfd0 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,17 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

3 3

[PATCH compass-ci 2/3] providers/qemu: encapsulated into functions
by Liu Yinsi 29 Dec '20

29 Dec '20

test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272000 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 171 +++++++++++++++++++++++++----------------- 1 file changed, 104 insertions(+), 67 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index f84c5ae..48bacab 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,74 +7,111 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi - -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm - -echo $SCHED_PORT -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} - -append= -initrds= -while read a b c -do - case "$a" in - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - esac -done < $ipxe_script - -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi } -[ -n "$initrds" ] || { - exit +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm } -initrd=initrd -cat $initrds > $initrd - -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file - -sleep 5 - -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) - -"${kvm[@]}" --append "${append}" +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +get_rootfs() +{ + append= + initrds= + while read a b c + do + case "$a" in + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + esac + done < $ipxe_script +} + +check_rootfs() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} + +kvm_command() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) +} + +run_qemu() +{ + "${kvm[@]}" --append "${append}" +} + +check_logfile +write_logfile + +get_rootfs +check_rootfs + +qemu_command + +print_message + +kvm_command +run_qemu -- 2.23.0

2 2

[PATCH compass-ci 1/3] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

some comment and code is unnecessary, so remove. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.271979 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..f84c5ae 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -22,18 +22,12 @@ echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= while read a b c do case "$a" in - '#') - ;; initrd) file=$(basename "$b") wget --timestamping -a ${log_file} --progress=bar:force $b @@ -44,11 +38,7 @@ do wget --timestamping -a ${log_file} --progress=bar:force $b append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") ;; - *) - ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -78,7 +68,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

2 4

[PATCH v3 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..5572274 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,10 @@ dirs=( ) mkdir -p "${dirs[@]}" + +if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd +elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +fi -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs/common: add comment for set password file.
by Wang Chenglong 29 Dec '20

29 Dec '20

[Why] add password file is easy to ignore. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..2ee16c8 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -33,7 +33,8 @@ usage() dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: - ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists " } -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e95f55b --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/es-find: add -d/--desc-keyword option
by Lu Weitao 29 Dec '20

29 Dec '20

es-find support order result by keyword with desc [Example] es-find suite=iperf -d/--desc-keyword start_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/sbin/es-find b/sbin/es-find index 2397338..17eefe6 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -29,8 +29,8 @@ def parse_argv items end -options = {} -options[:nr_jobs] = 10 +options = { nr_jobs: 10, desc_keyword: nil } + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -45,6 +45,10 @@ opt_parser = OptionParser.new do |opts| options[:nr_jobs] = n.to_i end + opts.on('-d', '--desc-keyword KEYWORD', 'order by keyword: start_time|end_time|...') do |d| + options[:desc_keyword] = d + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -54,6 +58,6 @@ end opt_parser.parse!(ARGV) items = parse_argv es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs]) +query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/params_group.rb optimize filter_groups()
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] 1. We had for-each key-value in hash, needn't query value by key once more. 2. When remove empty value of Hash(String, Hash(String, Array(...)), remove the first level key-value at first. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..fa71653 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,13 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + if value.empty? + groups.delete(group_key) + next + end + value.each do |dim_key, job_list| + value.delete(dim_key) if job_list.empty? end - groups.delete(group_key) if groups[group_key].empty? end end -- 2.23.0

1 0

[PATCH v2 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..538f45d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,13 @@ dirs=( ) mkdir -p "${dirs[@]}" + +[ -n "$os_mount" ] && { + if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd + elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr + fi +} -- 2.23.0

4 3

[PATCH lkp-tests] INSTALL.md: add quick installation description
by Wei Jihui 29 Dec '20

29 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- INSTALL.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 1bf0acc7..d7ce3517 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,18 +1,30 @@ +# Quick installation +```shell + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source $HOME/.${SHELL##*/}rc +``` +all is ok! + +# Step-by-Step installation + system wide setup ================= - ## debian - +```shell sudo apt-get install ruby-git ruby-activesupport +``` ## openEuler - +```shell sudo dnf install ruby rubygems gem install git activesupport +``` per-user setup ============== - +```shell git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests echo "export LKP_SRC=$PWD" >> $HOME/.${SHELL##*/}rc @@ -24,3 +36,4 @@ per-user setup hdd_partitions: ssd_partitions: EOF +``` -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 29 Dec '20

29 Dec '20

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..21396c4 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,33 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? + matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +99,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 29 Dec '20

29 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 29 Dec '20

29 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 6/6] mail-robot: optimize parse email_file
by Luan Shengde 29 Dec '20

29 Dec '20

[why] avoid build too much messages in the code files [how] just raise error code in the code file, and build the error message in the file building email message Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 4 +-- .../lib/parse-apply-account-email.rb | 30 +++++++------------ 2 files changed, 11 insertions(+), 23 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 7cac0d5..af31e17 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -38,9 +38,7 @@ class GiteeCommitUrlCheck def gitee_commit_exist(email_index) return if email_index - error_message = "We can not confirm whether the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with your email.' + error_message = 'COMMIT_URL_NOT_AVAILABLE' raise error_message end diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 7024731..94becdf 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey @@ -27,7 +28,7 @@ require_relative 'gitee-commit-url-check' # gitee.com: # call GiteeCommitUrlCheck to check the commit # non-gitee.com: -# execute curl to check the commit +# get request date for the commit url to check the commit # parse_pub_key # check my_ssh_pubkey exists and return it class ParseApplyAccountEmail @@ -63,10 +64,8 @@ class ParseApplyAccountEmail # the commit url should be in a standart format, example: # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - error_message = "No matched commit url found.\n" - error_message += 'Ensure that you have add a right commit url, ' - error_message += "and with prefix 'my oss commit:'." - raise error_message + raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) end mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] @@ -88,16 +87,13 @@ class ParseApplyAccountEmail return unless match_out.empty? - error_message = "The repo url for your commit is not in our upstream-repo list.\n" - error_message += 'Use a new one, or consulting the manager for available repo list.' - - raise error_message + raise 'NOT_REGISTERED' end def commit_url_availability(url, base_url) hub_name = url.split('/')[2] - # it requires authentication when execute curl to get the commit information + # it requires authentication when sending request to get the commit information # clone the repo and then validate the commit for the email address if hub_name.eql? 'gitee.com' gitee_commit(url, base_url) @@ -112,24 +108,18 @@ class ParseApplyAccountEmail end def non_gitee_commit(url) - url_fdback = %x(curl #{url}) + url_fdback = RestClient.get(url).body email_index = url_fdback.index @my_info['my_email'] return unless email_index.nil? - error_message = "We can not confirm the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with you email.' - - raise error_message + raise 'COMMIT_URL_NOT_AVAILABLE' end def parse_pub_key - error_message = "No pub_key found.\n" - error_message += 'Please add a pub_key as an attachment to your email.' - raise error_message if @mail_content.attachments.empty? - raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + raise 'NO_PUBKEY' if @mail_content.attachments.empty? + raise 'PUBKEY_NAME_ERR' unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ pub_key = @mail_content.attachments[0].body.decoded -- 2.23.0

2 2

[PATCH v4 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 29 Dec '20

29 Dec '20

[why] systemctl is not supported when use docker Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..c2404dbfd 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -5,20 +5,35 @@ # sshr_port_len # sshr_port_base -. $LKP_SRC/lib/http.sh +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci 4/4] assign-account: add 'lab' to default yaml
by Luan Shengde 29 Dec '20

29 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 2

[PATCH v4 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..97ad757 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 2

[PATCH compass-ci 2/2] sbin/es-find: add -d/--desc-keyword option
by Lu Weitao 29 Dec '20

29 Dec '20

es-find support order result by keyword with desc [Example] es-find suite=iperf -d/--desc-keyword start_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index 2397338..985fdea 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -31,6 +31,8 @@ end options = {} options[:nr_jobs] = 10 +options[:desc_keyword] = nil + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| options[:nr_jobs] = n.to_i end + opts.on('-d', '--desc-keyword KEYWORD', 'order by keyword: start_time|end_time|...') do |d| + options[:desc_keyword] = d + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -54,6 +60,6 @@ end opt_parser.parse!(ARGV) items = parse_argv es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs]) +query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/5-build: adapt for x86_64 machine
by Liu Yinsi 29 Dec '20

29 Dec '20

build ipxe according different arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/ipxe | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/sparrow/5-build/ipxe b/sparrow/5-build/ipxe index a66e298..50a2af9 100755 --- a/sparrow/5-build/ipxe +++ b/sparrow/5-build/ipxe @@ -15,11 +15,25 @@ cd ipxe/src || exit cpus=$(nproc) -make ARCH=arm64 bin-arm64-efi/ipxe.efi -make ARCH=arm64 bin-arm64-efi/snponly.efi -make ARCH=arm64 bin-arm64-efi/snp.efi -make CONFIG=rpi bin-arm64-efi/rpi.efi - -mkdir -p /tftpboot/ipxe/bin-arm64-efi/ -cp -a bin-arm64-efi/*.efi /tftpboot/ipxe/bin-arm64-efi/ -chmod 664 /tftpboot/ipxe/bin-arm64-efi/*.efi +build_ipxe() +{ + make ARCH=$1 bin-$1-efi/ipxe.efi + make ARCH=$1 bin-$1-efi/snponly.efi + make ARCH=$1 bin-$1-efi/snp.efi + make CONFIG=rpi bin-$1-efi/rpi.efi + + mkdir -p /tftpboot/ipxe/bin-$1-efi/ + cp -a bin-$1-efi/*.efi /tftpboot/ipxe/bin-$1-efi/ + chmod 664 /tftpboot/ipxe/bin-$1-efi/*.efi +} + +ARCH=$(arch) + +case "$ARCH" in + "aarch64") + build_ipxe arm64 + ;; + "x86_64") + build_ipxe x86_64 + ;; +esac -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] container: mount "/srv/git/archlinux" to srv-http
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 8c8ca0b..d9f461b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -15,6 +15,7 @@ cmd=( -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro + -v /srv/git/archlinux:/srv/git/archlinux:ro -d srv-http ) -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..47ef92f --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER caoxl78320(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..a1dbe77 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load_file(File.join('/srv', "#{result_root}", 'job.yaml')) +end -- 2.23.0

1 0

[PATCH compass-ci] container/qemu-efi: adapt for x86_64
by Liu Yinsi 28 Dec '20

28 Dec '20

in x86_64 debian machine, UEFI support for qemu is ovmf/OVMF.fd, different arch use different UEFI support. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/qemu-efi/Dockerfile | 4 +++- container/qemu-efi/build | 5 ++++- container/qemu-efi/install | 6 +++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/container/qemu-efi/Dockerfile b/container/qemu-efi/Dockerfile index e1087c5..46f442e 100644 --- a/container/qemu-efi/Dockerfile +++ b/container/qemu-efi/Dockerfile @@ -3,9 +3,11 @@ FROM debian +ARG UEFI + ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y qemu-efi-aarch64 + apt-get install -y $UEFI diff --git a/container/qemu-efi/build b/container/qemu-efi/build index 1238e67..cac9426 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t debian:qemu-efi . +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker build -t debian:qemu-efi --build-arg UEFI=$uefi . diff --git a/container/qemu-efi/install b/container/qemu-efi/install index 96be9aa..04bf581 100755 --- a/container/qemu-efi/install +++ b/container/qemu-efi/install @@ -11,5 +11,9 @@ cmd=( ) "${cmd[@]}" -docker cp qemu-efi:/usr/share/qemu-efi-aarch64 /usr/share/qemu-efi-aarch64 + +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker cp qemu-efi:/usr/share/$uefi /usr/share/$uefi docker_rm qemu-efi -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/nginx-server: spelling error
by Wei Jihui 28 Dec '20

28 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/depends/nginx-server b/distro/depends/nginx-server index da35d5f6..85e9cc2e 100644 --- a/distro/depends/nginx-server +++ b/distro/depends/nginx-server @@ -1,6 +1,6 @@ gcc glibc -ethool +ethtool libssl-dev zlib1g-dev libpcre3-dev -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/qemu: adapt kvm.sh for x86_64 and riscv64
by Liu Yinsi 28 Dec '20

28 Dec '20

put same options in public_options, different options in individual_options, merge options according different arch. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272042 /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-x86_64/z9.272046 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-riscv64/z9.272055 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-x86_64/z9.272057 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 78 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 70 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 48bacab..83b9606 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -20,8 +20,19 @@ check_logfile() qemu_command() { - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm + qemu=( + qemu-system-aarch64 + qemu-system-x86_64 + qemu-kvm + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [ -n "$(echo $kernel | grep "riscv64")" ] && qemu=qemu-system-riscv64 } write_logfile() @@ -31,6 +42,17 @@ write_logfile() cat $ipxe_script >> ${log_file} } +options_value() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + ls $bios >/dev/null 2>&1 || bios=/usr/share/ovmf/OVMF.fd + + helper=/usr/libexec/qemu-bridge-helper + ls $helper >/dev/null 2>&1 || helper=/usr/lib/qemu/qemu-bridge-helper + + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + get_rootfs() { append= @@ -78,18 +100,14 @@ print_message() sleep 5 } -kvm_command() +public_options() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -98,6 +116,46 @@ kvm_command() ) } +individual_options() +{ + aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + + x86_64=( + -bios $bios + -nic $nic + ) + + riscv64=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) +} + +merge_options() +{ + case $qemu in + qemu-system-aarch64) + kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + qemu-system-x86_64) + kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + [ "$(arch)" == "x86_64" ] && kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-riscv64) + kvm=("${kvm[@]}" "${riscv64[@]}") + ;; + esac +} + run_qemu() { "${kvm[@]}" --append "${append}" @@ -110,8 +168,12 @@ get_rootfs check_rootfs qemu_command +options_value print_message -kvm_command +public_options +individual_options +merge_options + run_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 28 Dec '20

28 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 28 Dec '20

28 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..48741ec 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -15,7 +15,7 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git cd /c/compass-ci || exit - [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" ln -s compass-ci /c/cci } @@ -24,9 +24,12 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/lkp-tests.git cd /c/lkp-tests || exit - [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" + [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" } [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" } -- 2.23.0

1 0

[PATCH compass-ci] os/debian: add apt-get update
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the url for downloading the package is displayed as 404 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/0-package/os/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 6a58a0c..c45e8c2 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -22,4 +22,5 @@ pkgs=( hostname ) +apt-get update apt-get install -y --fix-missing "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 28 Dec '20

28 Dec '20

[why] systemctl is not supported when use docker Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..c2404dbfd 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -5,20 +5,35 @@ # sshr_port_len # sshr_port_base -. $LKP_SRC/lib/http.sh +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 1

[PATCH compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 1

[PATCH compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..4544835 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load(File.open(File.join('/srv', "#{result_root}", 'job.yaml'))) +end -- 2.23.0

1 1

[PATCH v1 compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..5672652 --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v1 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..4544835 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load(File.open(File.join('/srv', "#{result_root}", 'job.yaml'))) +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..5672652 --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 1

[PATCH compass-ci] lib/params_group.rb optimize filter_groups
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] We had for-each in hash, needn't query value by key once more Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..bf229e6 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,10 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + value.each do |dim_key, job_list| + value.delete(dim_key) if job_list.empty? end - groups.delete(group_key) if groups[group_key].empty? + groups.delete(group_key) if value.empty? end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..29b423b69 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || mkdir -p "${upload_dir}" + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] lib/json_logger: delete caller
by Wu Zhende 28 Dec '20

28 Dec '20

caller does not provide error information. It's not helpful for fault locating. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/json_logger.rb | 1 - src/lib/json_logger.cr | 1 - 2 files changed, 2 deletions(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 6d1173e..78f95ca 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -25,7 +25,6 @@ class JSONLogger < Logger } logger_hash['progname'] = progname if progname - logger_hash['caller'] = caller if level_num >= 2 msg = { 'message' => msg } unless msg.is_a?(Hash) logger_hash.merge!(msg) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 6d61197..fcd65fb 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -25,7 +25,6 @@ class JSONLogger < Logger })).as_h logger_hash.any_merge!({"progname" => progname}) unless progname.empty? - logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 begin message = JSON.parse(msg).as_h -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[why] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..f6c27a754 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,6 +48,8 @@ get_config() upload_config() { + [ -d "${cci_dir}/${pkgname}" ] || mkdir -p "${cci_dir}/${pkgname}" + upload_to_target_dir "${cci_dir}/${pkgname}" } -- 2.23.0

2 2

[PATCH compass-ci] 3-code/git: iadd repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the code downloaded from upstream_repo occupies too much memory, the repository version needs to specify a version Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..e2b3cbf 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -29,4 +29,7 @@ cd /c || exit [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commitid" ] && git reset --hard "$upstream_commitid" } -- 2.23.0

2 1

[PATCH compass-ci 5/6] mail-robot: add error code/message
by Luan Shengde 28 Dec '20

28 Dec '20

add error message for each error code before building the email_message. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 87 +++++++++++++++---- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 593f345..5871a6a 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -3,32 +3,83 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_fail_email(my_info, message) - email_msg = <<~EMAIL_MESSAGE - To: #{my_info['my_email']} - Subject: [compass-ci] apply account failed +def email_err_message(message) + case message + when 'NO_COMMIT_URL' + err_message = <<~EMAIL_MESSAGE + No commit url found. + You should add a commit url for the 'apply account' email. - Dear user: + For example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'URL_PREFIX_ERR' + err_message = <<~EMAIL_MESSAGE + Please add a correct prefix for the commit url. + + my oss commit + + For example: + + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'NOT_REGISTERED' + err_message = <<~EMAIL_MESSAGE + Your repo has not been registered to our upstream-repos yet. + You should register your repo to the upstream first. + Try again after you have done it. + + Reference the following url to learn how to register the repo: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… + EMAIL_MESSAGE + when 'COMMIT_URL_NOT_AVAILABLE' + err_message = <<~EMAIL_MESSAGE + We cannot confirm the commit url matches your email. + + Make sure the commit exists. + Make sure it is truely submitted with you email. - Your application for account failed with following error: + Go to the commit history and find your commit. + Enter the commit and you will get your commit url. - #{message} + Try again after you have correctted the url + EMAIL_MESSAGE + when 'NO_PUBKEY' + err_message = <<~EMAIL_MESSAGE + No pub_key found. + Please add a pubkey to the email and then try again. + EMAIL_MESSAGE + when 'PUBKEY_NAME_ERR' + err_message = <<~EMAIL_MESSAGE + Pubkey file name error, keep it's name as when it was generated. + The pubkey file name should like: - In order to successfully apply an account, please pay attention to the following points: + id_{{ xxx }}.pub + EMAIL_MESSAGE + end - 1. mail subject - The subject should exactly: apply account + err_message += <<~EMAIL_MESSAGE - 2. commit url - When you writing the url, add prefix: my oss commit - example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + Manual for how to apply account: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… + EMAIL_MESSAGE + + err_message +end + +def build_apply_account_fail_email(my_info, message) + err_message = email_err_message(message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: - attention: - Ensure you commit url exist and available to access. + Your application for account failed. - 3. ssh pubkey - You need to add a pubkey as an attachment to the email. + #{err_message} regards compass-ci -- 2.23.0

1 0

[PATCH compass-ci 4/6] mail-robot: optimize message message
by Luan Shengde 28 Dec '20

28 Dec '20

optimize mail message for successfully applied account [why] its used linux command 'curl' to send request data when sending email. so the ${HOME} and ${SHELL##*/} is treated as a system variable and replaced with the variable value. as a result, we will send a unexpected email. to avoid this problem, need add '\\' before '$' in the old. [how] use rest-client instead of command 'curl' to send the request. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 466586d..ee3b574 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source \\${HOME}/.\\${SHELL##*/}rc + source ${HOME}/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

1 0

[PATCH compass-ci 3/6] mail-robot: optimize apply-jumper-account
by Luan Shengde 28 Dec '20

28 Dec '20

use rest-client instead of linux command 'curl' to send http request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-jumper-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb index 44f412b..73d2946 100755 --- a/container/mail-robot/lib/apply-jumper-account.rb +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -22,8 +22,8 @@ class ApplyJumperAccount end def apply_jumper_account - account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ - -d '#{(a)my_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, @my_info.to_json account_info = JSON.parse account_info_str account_info_exist(account_info) -- 2.23.0

1 0

[PATCH compass-ci 2/6] mail-robot: optimize apply-account
by Luan Shengde 28 Dec '20

28 Dec '20

use rest-client instead of linux command 'curl' to send request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index aea1aa5..e8288d7 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative '../../../lib/es_client' require_relative 'assign-account-email' require_relative 'assign-account-fail-eamil' @@ -131,6 +132,7 @@ class ApplyAccount build_apply_account_fail_email(@my_info, error_message) end - %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + send_mail_url = "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" + RestClient.post send_mail_url, email_message end end -- 2.23.0

1 0

[PATCH compass-ci 1/6] mail-robot: update Dockerfile
by Luan Shengde 28 Dec '20

28 Dec '20

[why] the old way is use linux command 'curl' to send http request. it's not human friendly to read for user [how] install rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index 76dc94d..f0aebbd 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -8,7 +8,8 @@ MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client + RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: fix spell error
by Lu Kaiyi 28 Dec '20

28 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 0ca71e1..ea67064 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -328,7 +328,7 @@ def get_banner(git_repo, branches) } end -def get_optimize_jobs_braches(jobs) +def get_optimize_jobs_branches(jobs) branch_set = Set.new jobs.size.times do |i| branch = jobs[i]['upstream_branch'].to_s @@ -347,7 +347,7 @@ def get_jobs_body(params) page_size = get_positive_number(params.delete(:page_size), 20) page_num = get_positive_number(params.delete(:page_num), 1) - 1 jobs, total = search_job(params, page_size, page_num) - jobs, branches = get_optimize_jobs_braches(jobs) + jobs, branches = get_optimize_jobs_branches(jobs) { total: total, filter: params, -- 2.23.0

1 0

[PATCH compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 26 Dec '20

26 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..eb7270d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,6 @@ dirs=( ) mkdir -p "${dirs[@]}" + +mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: optimize answerback-email
by Luan Shengde 26 Dec '20

26 Dec '20

[why] the old way to send request is use linux command 'curl', it's not good for reading and has low efficiency use ruby gem resource rest-client instead of linux command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 21285e7..a45fef2 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -11,6 +11,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' @@ -180,7 +181,8 @@ def apply_account(my_info, conf_info) apply_info.update conf_info apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -285,7 +287,8 @@ def send_mail(my_info, account_info, conf_info) build_message(my_info['my_email'], account_info) end - %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") + send_mail_url = "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" + RestClient.post send_mail_url, message end def store_account_info(my_info) -- 2.23.0

1 0

[PATCH compass-ci 1/4] assign-account: update Dockerfile
by Luan Shengde 26 Dec '20

26 Dec '20

[why] no need to install command 'curl' in the container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index 12faafa..1d9c36c 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -8,7 +8,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ssh ruby-sinatra ruby-json curl && \ + apt-get install -y ssh ruby-sinatra ruby-json && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone COPY assign-account.rb /usr/local/bin/ -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 26 Dec '20

26 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..c6cf9e85 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v3 compass-ci] lib: add build_my_info_client
by Luan Shengde 26 Dec '20

26 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..2a664b4 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = nil + @my_info['my_commit_url'] = nil + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + complete_my_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 5

[PATCH compass-ci] providers: qemu/kvm.sh check kernel file exists
by Xiao Shenwei 26 Dec '20

26 Dec '20

[why] if os_http not working, the kernel file will can't be download or the size is 0, then the qemu-kvm will boot by pxe, the vm will request the next job again. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..fe2345f 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -51,6 +51,11 @@ do #done < /tftpboot/boot.ipxe-centos done < $ipxe_script +[ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit +} + [ -n "$initrds" ] || { exit } -- 2.23.0

1 0

[PATCH compass-ci] [RFC] providers/qemu: adapt kvm.sh for x86_64/riscv64
by Liu Yinsi 26 Dec '20

26 Dec '20

run qemu in different arch have different options, put same options in $kvm, different options in options_$arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..56cf8c0 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -16,7 +16,7 @@ if [ ! -f "$log_file" ]; then fi qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +command -v $qemu >/dev/null || qemu=qemu-kvm || qemu=qemu-system-riscv64 echo $SCHED_PORT ipxe_script=ipxe_script @@ -67,15 +67,10 @@ sleep 5 kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -device virtio-gpu-pci - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -83,4 +78,38 @@ kvm=( -monitor null ) +options_aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -device virtio-gpu-pci + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_x86_64=( + -device virtio-gpu-pci + -bios /usr/share/ovmf/OVMF.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_riscv64=( + -machine virt + -device virtio-net-device,netdev=net1,mac=${mac} + -netdev bridge,br=br0,id=net1,helper=/usr/lib/qemu/qemu-bridge-helper + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +case $qemu in + qemu-system-aarch64) + kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-kvm) + [ $(arch) == "x86_64" ] && kvm=(${kvm[@]} ${args_x86_64[@]}) + [ $(arch) == "aarch64" ] && kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-system-riscv) + kvm=(${kvm[@]} ${args_riscv64[@]}) + ;; +esac + "${kvm[@]}" --append "${append}" -- 2.23.0

1 1

[PATCH v2 compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..cc21926 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,7 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path - io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + io.popen("mkdir -p "..root_path.." && cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end redis_host = os.getenv("REDIS_HOST") -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..00abc30 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,6 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path + io.popen("mkdir -p "..root_path) io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: fix incorrect average and stddev
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We just skip useless job when make job_list --> matrix, had not did: job_list.size - 1, the average: sum($job['stats']['metric'])/job_list.size will be incorrect. standard deviation is the same reason [How] Remove useless job before job_list --> matrix, and do minor refactoring Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 11 ++--------- lib/params_group.rb | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..3c4035e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats && assign_suites(suites, job, stats) + suites << job['suite'] if job['suite'] stats.each do |key, value| next if key.include?('timestamp') @@ -97,17 +97,10 @@ def create_matrix(job_list) return matrix, suites end -def assign_suites(suites, job, stats) - return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] -end - # input: query results from es_query # return: matrix def combine_query_data(query_data) - job_list = query_data['hits']['hits'] - job_list.map! { |job| job['_source'] } + job_list = extract_jobs_list(query_data['hits']['hits']) create_matrix(job_list) end diff --git a/lib/params_group.rb b/lib/params_group.rb index 57cd5c1..7bb7f5b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -38,9 +38,21 @@ def auto_group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) - jobs_list.map do |job| - job['_source'] + jobs_list.map! do |job| + job['_source'] if job_is_useful?(job) end + + jobs_list.compact +end + +def job_is_useful?(job) + stats = job['_source']['stats'] + return unless stats + + suite = job['_source']['suite'] + return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + + true end def group(jobs_list, dimensions) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] lib/matrix2.rb: remove duplicate code
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] Had handle empty group in lib/params_group.rb def filter_groups(groups) groups.each do |group_key, value| value.each_key do |dim_key| value.delete(dim_key) if value[dim_key].empty? end groups.delete(group_key) if groups[group_key].empty? end end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index d9d86e8..14e9558 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -124,10 +124,6 @@ def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.empty? - groups.delete(group_key) - next - end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 25 Dec '20

25 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 4) 备注目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + PKGBUILD 存放路径： `/srv/git/archlinux/*/*/PKGBUILD`。 ### depends -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query: add select keyword from index
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We need only get specific keyword instead of entire document. it's like SQL: SELECT DISTINCT column FROM table Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..0e34bfc 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -61,6 +61,31 @@ class ESQuery return result[0]['_source'] end + + # select doc_field from index + # input: + # eg: suite (@index: jobs) + # output: + # [ + # {"key"=>"build-pkg", "doc_count"=>90841}, + # {"key"=>"cci-depends", "doc_count"=>4636}, + # {"key"=>"cci-makepkg", "doc_count"=>3647}, + # ... + # ] + def query_specific_fields(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 } + } + }, + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs: use array save suffix
by Wang Chenglong 25 Dec '20

25 Dec '20

[Why] the docker image usually use '.tar' and '.tar.xz' as the suffix. so use array save the suffix. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..89fad87 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -29,7 +29,7 @@ usage() Usage: ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> - src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ .tar, tar.xz ]. dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: @@ -43,14 +43,14 @@ check_docker_img_file() local docker_img_name local docker_img_suffix - allow_docker_img_suffix='tar.xz' + allow_docker_img_suffix=('tar' 'tar.xz') docker_img_name=$(basename "$1") docker_img_suffix=${docker_img_name##*.} [ "$docker_img_suffix" == 'tar.xz' ] || { - docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') - echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { - echo "[ERROR] Only support: .tar.xz file!" + docker_img_suffix=$(echo "$docker_img_name" | awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix[@]}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar .tar.xz file!" exit 2 } } -- 2.23.0

2 1

[PATCH compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 25 Dec '20

25 Dec '20

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..8483b25 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,31 +53,41 @@ class Sched end def get_queues(host) - queues = [] of String + default_queues = [] of String + sub_queues = [] of String queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return default_queues, sub_queues unless queues_str queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end - sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return default_queues, sub_queues end def get_job_from_queues(queues, testbox) + default_queues, sub_queues = queues + rand_sub_queues = rand_queues(sub_queues) + idle_queues = [] of String + + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + all_queues = rand_sub_queues + idle_queues + job = nil - queues.each do |queue| + all_queues.each do |queue| job = prepare_job("sched/#{queue}", testbox) return job if job end @@ -91,6 +101,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines now + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 25 Dec '20

25 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 25 Dec '20

25 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use email instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: judge whether has tail cmd before use it
by Wei Jihui 25 Dec '20

25 Dec '20

if is_docker is true, it will use tail cmd, so judge it before use it Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/bootstrap.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index c9a9f89f..ae902207 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -314,6 +314,8 @@ announce_bootup() redirect_stdout_stderr() { + has_cmd tail || return + if is_docker; then [ -e /tmp/stdout ] || touch /tmp/stdout [ -e /tmp/stderr ] || touch /tmp/stderr @@ -328,7 +330,6 @@ redirect_stdout_stderr() fi [ -c /dev/kmsg ] || return - has_cmd tail || return exec > /tmp/stdout exec 2> /tmp/stderr -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/matrix2.rb: fix incorrect average and stddev
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We just skip useless job when make job_list --> matrix, had not did: job_list.size - 1, the average: sum($job['stats']['metric'])/job_list.size will be incorrect. standard deviation is the same reason [How] Remove useless job before job_list --> matrix, and do minor refactoring Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 11 ++--------- lib/params_group.rb | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..ce64da1 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats && assign_suites(suites, job, stats) + suites << job['suite'] stats.each do |key, value| next if key.include?('timestamp') @@ -97,17 +97,10 @@ def create_matrix(job_list) return matrix, suites end -def assign_suites(suites, job, stats) - return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] -end - # input: query results from es_query # return: matrix def combine_query_data(query_data) - job_list = query_data['hits']['hits'] - job_list.map! { |job| job['_source'] } + job_list = extract_jobs_list(query_data['hits']['hits']) create_matrix(job_list) end diff --git a/lib/params_group.rb b/lib/params_group.rb index 57cd5c1..7bb7f5b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -38,9 +38,21 @@ def auto_group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) - jobs_list.map do |job| - job['_source'] + jobs_list.map! do |job| + job['_source'] if job_is_useful?(job) end + + jobs_list.compact +end + +def job_is_useful?(job) + stats = job['_source']['stats'] + return unless stats + + suite = job['_source']['suite'] + return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + + true end def group(jobs_list, dimensions) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/matrix2.rb: remove duplicate code
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] Had handle empty group in lib/params_group.rb def filter_groups(groups) groups.each do |group_key, value| value.each_key do |dim_key| value.delete(dim_key) if value[dim_key].empty? end groups.delete(group_key) if groups[group_key].empty? end end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index d9d86e8..14e9558 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -124,10 +124,6 @@ def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.empty? - groups.delete(group_key) - next - end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) -- 2.23.0

1 0

[PATCH lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 25 Dec '20

25 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..14e02b2f --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH v7 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 25 Dec '20

25 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 71 ++++++++++++------------------------------------- 1 file changed, 17 insertions(+), 54 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..2409c86 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,61 +150,24 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ + 1. �� Unable to mount root fs on unknown-block�� - �� ```bash -- 2.23.0

1 0

[PATCH v5 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 25 Dec '20

25 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..44bd1c10d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,24 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + [ -n "$config" ] || return + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +66,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +145,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 25 Dec '20

25 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..06a58d498 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

2 2

[PATCH v2 compass-ci] doc: update README.md
by Liu Yinsi 24 Dec '20

24 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 113 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 87 insertions(+), 26 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index ccb1d15..a9a4bb0 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -1,26 +1,87 @@ -Deploy a mini instance in one single machine. - -Prepare: - 1. hardware - Server: at least prepare a server - ProductType: ThaiShan200-2280 - Arch: aarch64 - Memory: 8G - CPU: 64 nuclear - DiskSpace: 500G - - 2. software - OS: openEuler-aarch64-20.03 LTS - git: suggest 2.23.0 - - 3. network - Internet is available - - 4. /os for store rootfs - >= 300G - -Steps: - umask 002 - git clone https://gitee.com/wu_fengguang/compass-ci.git - cd compass-ci/sparrow - ./install-tiny +# 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +## 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + +### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash + git clone https://gitee.com/wu_fengguang/compass-ci.git + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` +5. 生成lkp-aarch64.cgz压缩包 + ```bash + cd /c/compass-ci/container/lkp-initrd && ./run + ``` +6. 验证账号 + ```bash + cd /c/compass-ci/sbin && ./build-my-info.rb + 按照提示输入账号名和邮箱即可 + +7. 测试环境是否可以提交job测试 + ```bash + submit iperf.yaml testbox=vm-2p8g + ``` + + 执行上述命令正常情况下会提示信息如下： + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，下面就可以开始进行测试了。 + +8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） + ```bash + mkdir -p /srv/os/openeuler/aarch64/ + cd /srv/os/openeuler/aarch64 + wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + ``` + +9. 解压cgz包 + ```bash + gzip -dc 20.03.cgz | cpio -idv + ``` + +10. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +11. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +12. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + 注意：os_mount 必须指定为cifs + +13. 运行测试任务 + ```bash + cd /c/compass-ci/providers/ && ./my-qemu.sh + ``` + +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 + ```bash + cd /srv/result/iperf/$job_id/output + ``` -- 2.23.0

2 1

[PATCH lkp-tests] bootstrap.sh: need a blank line after single return
by Xu Xijian 24 Dec '20

24 Dec '20

A blank line is needed after single return, more than 1 return cmd together doesn't need. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ad1c4a2f2..bd168a62a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -235,7 +235,6 @@ run_ntpdate() [ -z "$NO_NETWORK" ] || return [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - [ -x '/usr/sbin/ntpdate' ] || return local hour="$(date +%H)" @@ -371,6 +370,7 @@ install_deb() # solved the dependent relationship, so here we ignore the dependency errors echo "install debs round one: dpkg -i --force-confdef --force-depends $files" dpkg -i --force-confdef --force-depends $files 2>/tmp/dpkg_error && return + grep -v "$filter_info" /tmp/dpkg_error # round two, install all debs one by one accroding to keep-deb which is in sequence @@ -397,6 +397,7 @@ install_deb() install_rpms() { [ -d /opt/rpms ] || return + echo "install $(ls /opt/rpms/*.rpm)" rpm -ivh --force --ignoresize /opt/rpms/*.rpm } @@ -458,6 +459,7 @@ fixup_packages() mount_debugfs() { is_docker && return + check_mount debug /sys/kernel/debug -t debugfs } @@ -467,6 +469,7 @@ cleanup_pkg_cache() local pkg_cache=$1 local cleanup_stamp=$pkg_cache/cleanup_stamp/$(date +%U) [ -d "$cleanup_stamp" ] && return + mkdir $cleanup_stamp -p find "$pkg_cache" $ -type f -mtime +7 -delete $ -or $ -type d -ctime +7 -empty -delete $ @@ -497,6 +500,7 @@ mount_rootfs() set_job_state 'load_disk_fail' return 1 } + ROOTFS_DIR=/opt/rootfs mkdir -p $ROOTFS_DIR mount $rootfs_partition $ROOTFS_DIR || { @@ -536,7 +540,6 @@ netconsole_init() { # don't init netconsole at local run [ "$LKP_LOCAL_RUN" = "1" ] && return - [ -z "$netconsole_port" ] && return [ -n "$NO_NETWORK" ] && return @@ -545,6 +548,7 @@ netconsole_init() # Select the interface that can access netconsole server netconsole_interface=$(show_default_interface) [ -n "$netconsole_server" ] || return + # eth0 is default interface if netconsole_interface is null. modprobe netconsole netconsole=@/$netconsole_interface,$netconsole_port@$netconsole_server/ } @@ -755,6 +759,7 @@ boot_init() install_pkgs() { [ -d /opt/pkgs ] || return + echo "install $(ls /opt/pkgs/*.pkg.tar*)" pacman -U --needed --noconfirm /opt/pkgs/*.pkg.tar* } -- 2.23.0

1 0

[PATCH compass-ci 3/3] sbin/auto_submit.yaml: deploy-cci is not completely adapted in centos
by Hu Xuejiao 24 Dec '20

24 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/auto_submit.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index e1eb39b..fce1dc0 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -2,7 +2,6 @@ a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- testbox=taishan200-2280-2s64p-256g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

2 1

[PATCH v2 lkp-tests] bin/run_job: Automatic task execution
by Hu Xuejiao 24 Dec '20

24 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..14e02b2f --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v2 lkp-tests] bin/run_job: Automatic task execution
by Hu Xuejiao 24 Dec '20

24 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..d45571b3 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,74 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + echo ${cmd[@]} + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH compass-ci 2/3] 3-code/git: add repository version pointing
by Hu Xuejiao 24 Dec '20

24 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 494107e..2159ddc 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -13,11 +13,18 @@ cd /c || exit # modify and manual run for now: [ -d "compass-ci/.git" ] || { git clone https://gitee.com/wu_fengguang/compass-ci.git - ln -s compass-ci cci + + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + + ln -s compass-ci /c/cci } [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git + + cd /c/lkp-tests || exit + [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" } [ -d "upstream-repos/.git" ] || { -- 2.23.0

1 0

[PATCH compass-ci 1/3] doc/job: add SCHED_HOST, SCHED_PORT description
by Hu Xuejiao 24 Dec '20

24 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- doc/job/SCHED_HOST-SCHED_PORT.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/job/SCHED_HOST-SCHED_PORT.md diff --git a/doc/job/SCHED_HOST-SCHED_PORT.md b/doc/job/SCHED_HOST-SCHED_PORT.md new file mode 100644 index 0000000..74c628a --- /dev/null +++ b/doc/job/SCHED_HOST-SCHED_PORT.md @@ -0,0 +1,15 @@ +# SCHED_HOST, SCHED_PORT +Meaning: +- The job task depends on scheduler service, which will distribute tasks to test + machine. All tasks will run in test machine. +- `Scheduler` is one of our service, `SCHED` is the abbreviation of `Scheduler` + these variables mean: + SCHED_HOST: the scheduler service ip. + SCHED_PORT: the scheduler service port. + +Usage example: + +```$LKP_SRC/include/lab/z9 +SCHED_HOST: api.compass-ci.openeuler.org +SCHED_PORT: 11306 +``` -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: fix redirect_stdout_stderr for docker
by Xiao Shenwei 24 Dec '20

24 Dec '20

[why] exec command will redirect stdout stderr to /tmp/stdout, /tmp/stderr but docker log based on /dev/stdout and /dev/stderr, it cause the task status cannot be viewed from the console, which is difficult to debug. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/bootstrap.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ad1c4a2f2..c9a9f89f9 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -315,6 +315,13 @@ announce_bootup() redirect_stdout_stderr() { if is_docker; then + [ -e /tmp/stdout ] || touch /tmp/stdout + [ -e /tmp/stderr ] || touch /tmp/stderr + tail -f /tmp/stdout > /dev/stdout & + echo $! >> /tmp/pid-tail-global + tail -f /tmp/stderr > /dev/stderr & + echo $! >> /tmp/pid-tail-global + exec > /tmp/stdout exec 2> /tmp/stderr return -- 2.23.0

1 0

[PATCH v2 lkp-tests] adaptation/archlinux: add archlinux mapping
by Wang Yong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 89163cf41..c52212e08 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -72,3 +72,11 @@ libtirpc: libtirpc1 initramfs-tools: openssh-server: openssh openssh-client: openssh +autopoint: gettext +g++: gcc +libarchive13: libarchive +libbz2-1.0: bzip2 +libssl: openssl +libtool-bin: libtool +linux-perf: perf +ruby-dev: ruby -- 2.23.0

1 0

[PATCH compass-ci] lib/env.cr: rename to web_env.cr
by Wu Zhende 24 Dec '20

24 Dec '20

[Why] env.cr is the same as the file name provided by the crystal. [Error] /c/cci/scheduler $ crystal a.cr Showing last frame. Use --error-trace for full trace. In /usr/lib/crystal/core/json/from_json.cr:140:5 140 | def Set.new(pull : JSON::PullParser) Error: undefined constant Set Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 2 +- src/lib/{env.cr => web_env.cr} | 0 src/scheduler/scheduler.cr | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) rename src/lib/{env.cr => web_env.cr} (100%) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6caa5b9..2d9dba6 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -5,7 +5,7 @@ require "kemal" require "yaml" require "./job" -require "./env" +require "./web_env" require "./block_helper" require "./taskqueue_api" require "./remote_git_client" diff --git a/src/lib/env.cr b/src/lib/web_env.cr similarity index 100% rename from src/lib/env.cr rename to src/lib/web_env.cr diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 2103f7c..6b032ee 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -3,7 +3,7 @@ require "kemal" -require "../lib/env" +require "../lib/web_env" require "../lib/sched" require "../lib/json_logger" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/4-docker/buildall: build dist image when it doesn't exist
by Xu Xijian 24 Dec '20

24 Dec '20

If the dist docker image doesn't exist for some reason, run build script to create it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 32801d1..9e66ac9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -32,7 +32,8 @@ do_one() ( cd $container [ "$container_name" == 'ssh-r' ] && exit - if [ "$action" != "reboot" ]; then + docker images | grep -w -q "$container_name" + if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] code-spec.md: adjust markdown document format
by Xu Xijian 24 Dec '20

24 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```SHELL content SHELL Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/code-spec.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index d44c6fe..a717f64 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -21,10 +21,12 @@ code注意事项 - puts/printf字符串应当容易搜索(grep)。特别注意包含足够长的固定内容substring. 例如： +```SHELL die "test yaml$${test_yaml}$ not exist in ${LKP_SRC}/jobs !!!" ==> die "cannot find test yaml ($test_yaml) in $LKP_SRC/jobs" ^^^^^^^^^^^^^^^^^^^^^ grep-friendly fixed substring +``` 重构友好编码 ============ -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_client: add send_mail_text function
by Luan Shengde 24 Dec '20

24 Dec '20

for user use mail_client to send mail with 'mail content' data type Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail_client.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/mail_client.rb b/lib/mail_client.rb index b13492f..f63e65d 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -24,4 +24,9 @@ class MailClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_encode") resource.post(Base64.encode64(mail_data)) end + + def send_mail_text(mail_text) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_text") + resource.post(mail_text) + end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-depends: update depends for cci-depends
by Wang Yong 24 Dec '20

24 Dec '20

[Why] cci-depends need base packages to generate cgz file, and mount operation is deprecated Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-depends | 3 ++- tests/cci-depends | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/distro/depends/cci-depends b/distro/depends/cci-depends index 1f7f73ac3..0dc9d1501 100644 --- a/distro/depends/cci-depends +++ b/distro/depends/cci-depends @@ -1,2 +1,3 @@ -cifs-utils cpio +curl +gzip diff --git a/tests/cci-depends b/tests/cci-depends index 42d5ef676..8baf023f3 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -44,6 +44,8 @@ check_shared_package "$benchmark" && { share_debian_package +distro_install_depends cci-depends + fixup_distro_mirror update -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: add runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 24 Dec '20

24 Dec '20

User can specify the borrow time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 9 +++++---- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b14ea0a 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..80c15d1 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -207,6 +207,7 @@ suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci] lib: add build_my_info_client
by Luan Shengde 24 Dec '20

24 Dec '20

purpose: for user build their infos when installing compassci - store my_info to ES - write my_info to default/lab yaml file if not, job will fail for verification problems for user private environment Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..e4a4335 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = '' + @my_info['my_commit_url'] = '' + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + complete_my_info + store_account_info + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: add runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 24 Dec '20

24 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 9 +++++---- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..cfb67aa 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..66f6876 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -207,6 +207,7 @@ suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

2 2

[PATCH v7 compass-ci] docker-rootfs: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..62e1c05 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -2,10 +2,19 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $LKP_SRC/lib/env.sh + install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ has_cmd yum ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ has_cmd apt-get ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ has_cmd pacman ]; then + pacman -Syu --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 1

[PATCH v2 compass-ci] update document
by Luan Shengde 24 Dec '20

24 Dec '20

update documents with new email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- doc/mail-robot.md | 2 +- doc/manual/apply-account.md | 2 +- doc/manual/borrow-machine.en.md | 2 +- doc/manual/borrow-machine.zh.md | 2 +- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.en.md b/README.en.md index c082167..632a480 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index c2140b0..1339ef0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **登录测试环境** -1. 向 compass-ci(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +1. 向 compass-ci-reply(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… 3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… diff --git a/doc/mail-robot.md b/doc/mail-robot.md index d3778e7..e97912c 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -11,7 +11,7 @@ apply account/uuid for user ## usage build email as the following section: allowed email format -send mail to: compass-ci(a)qq.com +send mail to: compass-ci-reply(a)qq.com ## allowed email format: diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 7e1a6a6..08e7618 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci(a)qq.com + To: compass-ci-reply(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..bed9fcc 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -3,7 +3,7 @@ Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - Send an email to apply for an account. -- Receive an email from compass-ci(a)qq.com. +- Receive an email from compass-ci-reply(a)qq.com. - Configure the local environment. # Applying for a Test Machine (VM) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..3f500e2 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -2,7 +2,7 @@ 请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - send apply account email. -- receive email from compass-ci(a)qq.com. +- receive email from compass-ci-reply(a)qq.com. - local environment configuration. diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index dd115f7..6f0736b 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -6,7 +6,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 :exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email -- receive email from compass-ci(a)qq.com +- receive email from compass-ci-reply(a)qq.com - get reply email and follow its instructions to - setup default config -- 2.23.0

1 0

[PATCH compass-ci] update document
by Luan Shengde 24 Dec '20

24 Dec '20

update documents with new email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- doc/mail-robot.md | 2 +- doc/manual/apply-account.md | 2 +- doc/manual/borrow-machine.en.md | 2 +- doc/manual/borrow-machine.zh.md | 2 +- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.en.md b/README.en.md index c082167..632a480 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index c2140b0..1339ef0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� diff --git a/doc/mail-robot.md b/doc/mail-robot.md index d3778e7..e97912c 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -11,7 +11,7 @@ apply account/uuid for user ## usage build email as the following section: allowed email format -send mail to: compass-ci(a)qq.com +send mail to: compass-ci-reply(a)qq.com ## allowed email format: diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 7e1a6a6..08e7618 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci(a)qq.com + To: compass-ci-reply(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..bed9fcc 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -3,7 +3,7 @@ Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - Send an email to apply for an account. -- Receive an email from compass-ci(a)qq.com. +- Receive an email from compass-ci-reply(a)qq.com. - Configure the local environment. # Applying for a Test Machine (VM) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..3f500e2 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -2,7 +2,7 @@ �� [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - send apply account email. -- receive email from compass-ci(a)qq.com. +- receive email from compass-ci-reply(a)qq.com. - local environment configuration. diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index dd115f7..6f0736b 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -6,7 +6,7 @@ Compass-CI �� [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) �� :exclamation: �� [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… �� - send apply account email -- receive email from compass-ci(a)qq.com +- receive email from compass-ci-reply(a)qq.com - get reply email and follow its instructions to - setup default config -- 2.23.0

1 0

[PATCH compass-ci] doc: and runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 24 Dec '20

24 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 9 +++++---- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..cfb67aa 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..66f6876 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -207,6 +207,7 @@ suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

2 2

[PATCH v6 compass-ci] docker-rootfs: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..ffe4937 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Syu --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH v4 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 24 Dec '20

24 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinux instead of vmlinuz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

[PATCH v2 compass-ci] modify assign-account-email
by Li Ping 24 Dec '20

24 Dec '20

[why]: source /root/.bashrc && /root/.bash_profile follow this step can not success. Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..466586d 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source \\${HOME}/.\\${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

4 10

[PATCH compass-ci] doc/manual: update apply-account.md
by Luan Shengde 24 Dec '20

24 Dec '20

used new email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/apply-account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 7e1a6a6..08e7618 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci(a)qq.com + To: compass-ci-reply(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 24 Dec '20

24 Dec '20

[why] 1. load job from result's job yaml has key which is start with "#! ", like '#! hosts/vm-2p16g': this key should be deleted. 2. load job from a yaml maybe has symbol key, like # key ---> :'# key': nil but symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..35e0ceee 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -290,6 +289,12 @@ class Job jobs_array = [] hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ + next unless key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

2 1