December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 26 Dec '20

26 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..eb7270d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,6 @@ dirs=( ) mkdir -p "${dirs[@]}" + +mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: optimize answerback-email
by Luan Shengde 26 Dec '20

26 Dec '20

[why] the old way to send request is use linux command 'curl', it's not good for reading and has low efficiency use ruby gem resource rest-client instead of linux command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 21285e7..a45fef2 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -11,6 +11,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' @@ -180,7 +181,8 @@ def apply_account(my_info, conf_info) apply_info.update conf_info apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -285,7 +287,8 @@ def send_mail(my_info, account_info, conf_info) build_message(my_info['my_email'], account_info) end - %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") + send_mail_url = "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" + RestClient.post send_mail_url, message end def store_account_info(my_info) -- 2.23.0

1 0

[PATCH compass-ci 1/4] assign-account: update Dockerfile
by Luan Shengde 26 Dec '20

26 Dec '20

[why] no need to install command 'curl' in the container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index 12faafa..1d9c36c 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -8,7 +8,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ssh ruby-sinatra ruby-json curl && \ + apt-get install -y ssh ruby-sinatra ruby-json && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone COPY assign-account.rb /usr/local/bin/ -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 26 Dec '20

26 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..c6cf9e85 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v3 compass-ci] lib: add build_my_info_client
by Luan Shengde 26 Dec '20

26 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..2a664b4 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = nil + @my_info['my_commit_url'] = nil + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + complete_my_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 5

[PATCH compass-ci] providers: qemu/kvm.sh check kernel file exists
by Xiao Shenwei 26 Dec '20

26 Dec '20

[why] if os_http not working, the kernel file will can't be download or the size is 0, then the qemu-kvm will boot by pxe, the vm will request the next job again. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..fe2345f 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -51,6 +51,11 @@ do #done < /tftpboot/boot.ipxe-centos done < $ipxe_script +[ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit +} + [ -n "$initrds" ] || { exit } -- 2.23.0

1 0

[PATCH compass-ci] [RFC] providers/qemu: adapt kvm.sh for x86_64/riscv64
by Liu Yinsi 26 Dec '20

26 Dec '20

run qemu in different arch have different options, put same options in $kvm, different options in options_$arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..56cf8c0 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -16,7 +16,7 @@ if [ ! -f "$log_file" ]; then fi qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +command -v $qemu >/dev/null || qemu=qemu-kvm || qemu=qemu-system-riscv64 echo $SCHED_PORT ipxe_script=ipxe_script @@ -67,15 +67,10 @@ sleep 5 kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -device virtio-gpu-pci - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -83,4 +78,38 @@ kvm=( -monitor null ) +options_aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -device virtio-gpu-pci + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_x86_64=( + -device virtio-gpu-pci + -bios /usr/share/ovmf/OVMF.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_riscv64=( + -machine virt + -device virtio-net-device,netdev=net1,mac=${mac} + -netdev bridge,br=br0,id=net1,helper=/usr/lib/qemu/qemu-bridge-helper + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +case $qemu in + qemu-system-aarch64) + kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-kvm) + [ $(arch) == "x86_64" ] && kvm=(${kvm[@]} ${args_x86_64[@]}) + [ $(arch) == "aarch64" ] && kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-system-riscv) + kvm=(${kvm[@]} ${args_riscv64[@]}) + ;; +esac + "${kvm[@]}" --append "${append}" -- 2.23.0

1 1

[PATCH v2 compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..cc21926 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,7 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path - io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + io.popen("mkdir -p "..root_path.." && cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end redis_host = os.getenv("REDIS_HOST") -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..00abc30 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,6 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path + io.popen("mkdir -p "..root_path) io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: fix incorrect average and stddev
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We just skip useless job when make job_list --> matrix, had not did: job_list.size - 1, the average: sum($job['stats']['metric'])/job_list.size will be incorrect. standard deviation is the same reason [How] Remove useless job before job_list --> matrix, and do minor refactoring Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 11 ++--------- lib/params_group.rb | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..3c4035e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats && assign_suites(suites, job, stats) + suites << job['suite'] if job['suite'] stats.each do |key, value| next if key.include?('timestamp') @@ -97,17 +97,10 @@ def create_matrix(job_list) return matrix, suites end -def assign_suites(suites, job, stats) - return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] -end - # input: query results from es_query # return: matrix def combine_query_data(query_data) - job_list = query_data['hits']['hits'] - job_list.map! { |job| job['_source'] } + job_list = extract_jobs_list(query_data['hits']['hits']) create_matrix(job_list) end diff --git a/lib/params_group.rb b/lib/params_group.rb index 57cd5c1..7bb7f5b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -38,9 +38,21 @@ def auto_group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) - jobs_list.map do |job| - job['_source'] + jobs_list.map! do |job| + job['_source'] if job_is_useful?(job) end + + jobs_list.compact +end + +def job_is_useful?(job) + stats = job['_source']['stats'] + return unless stats + + suite = job['_source']['suite'] + return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + + true end def group(jobs_list, dimensions) -- 2.23.0

1 0